The Wisdom in Log Fies
by Vikas Kamat
First Online: April 15, 2001
Page Last Updated: August 20, 2024
Every web hosting plan (even the inexpensive plans) provides what are known as server logs that contain information on the user activity on your website. They can be a single file or separate files depending on the server configuration , but primarily contain three types of information: The Error Log, The Access Log, and the The Referer Log.
The
Nosy Surfer See how people come to Kamat's Potpourri. and find out what they are searching for. You can implement a similar system to study how surfers come to your site. |
The Error Log
This error_log contains all the errors that were generated while processing users' request. Typical examples are missing files, CGI or script errors. As a webmaster, you should always watch this file, for signs of trouble in the website. Familiarize yourself with the HTTP error messages to diagnose the problems.What is the favicon.ico?
Many times, in your error log, you find an entry of the
type:
[error] File does not exist: /usr/local/etc/httpd/htdocs/favicon.ico
favicon.ico is a 16x16
Windows® icon, the browser looks for so it
can display it in the address toolbar of the browser. (If you are
currently using Microsoft Internet Explorer (IE), it is the small graphic
displayed to the left of URL)
If you have numerous entries for a file favicon.ico in the error
log, it is a good thing! It shows how many people using IE 4.0 and
above have book marked your site. To remove reporting of this error,
place an icon with that name in your root folder.
Most of the problems we encounter at Kamat's Potpourri are people linking to invalid URLs. Sometimes they make typographical mistakes, but by watching this log, you know of those mistakes and could provide aliases to the correct URLs.
The Access Log
This large file keeps track of all the resources (images, scripts, pages) that were accessed on the website. It also contains the time, and the location (IP Address) of the user.What Pages are Popular?
By sorting the access log by number of hits you can find out which
contents are popular among your readers. You will be surprised that
they will also be your best sections; so highlight them!
Who are Your Patrons?
Sophisticated software systems available today (some are listed
below), can mine the access log and tell you demographics
information about your patrons. For example, if you had a website on
medicine, the way a medical student browses your site is different
than a practicing doctor. Of course, the access log does not tell
you the names of the patrons, so you can assure the patrons on
protection of their privacy.
What Days or Times are Busy?
This is important so you can perform maintenance tasks. For
example, we have determined that our usage was highest in the beginning
part of the week, and hence we publish most contents for that
audience. We perform updates and indexing when it is late at night in
the USA, when our usage is the lowest.
The Referer Log
Along with every http request is sent a small piece of information called the HTTP_REFERER, which tells the name and location of the page that has a link to your website. This important information can be effectively used to find out the effectiveness of Search engine listings, advertisement campaigns, or simply to find our how users come to your site (see our example, Adhik Prasangi).How Do Users Find Your Site?
The Referer Log answers this important question, so you can optimize
the site for
Build a Smarter Site
By carefully studying the search patterns, you can find out
interesting information on general web usage. Here are some
discoveries from our own logs:
- More and more people are searching for multiple words now, rather than just one word which was prevalent in earlier days of the web.
- Yahoo! and Google are leading search engines and by a huge margin. So, you can spend energy on getting a better rank on those.
- People make typing mistakes. By addressing the mistakes the users often make (for example, many Europeans spell Gandhi as Ghandi), you can get more traffic.
So do find out from your hosting company where the server log files are. They contain wisdom about your site that no one else can tell you about!
Internet Links
-
Webtrends Log Analyzer -- an out of the box commercial solution
-
WebLog -- an inexpensive comprehensive access log analysis tool