Search Engine Optimization: Analyzing referrer logs of your traffic is necessary to measure your success

by Robin Nobles

Congratulations! Your hard work and persistence in optimizing your pages have resulted in a dramatic increase in traffic to your site, which is just what you wanted.

But, are you analyzing that traffic and using it to strengthen your site? Or, are you quickly glancing at your log files every week or so to see how much traffic you're getting, and letting it go at that?

Before we go on, let's define exactly what log files are.

When someone visits your website, server software counts and tracks, i.e. "logs", that visit. It also keeps a record of it for a certain period of time. Part of the saved information is called a referrer log.

Referrer logs can help you analyze the traffic to your site. Though each referrer log program provides slightly different data, some of the more common information includes:

• Which engines have sent you traffic;
• What keywords were used to find your site;
• Which pages were accessed the most or the least;
• Who are the visiting spiders;
• User profile by region;
• Average length of time someone remains on your site;
• Average number of user sessions or page views per day;
• Top entry and exit pages;
• Top referring sites;
• Summary of activity by day;
• Server errors;
• Bandwidth, which is the measure (in kilobytes of data transferred) of the traffic on the site; and,
• Type of technology used by your visitors.

But why is it so important to study your traffic? Isn't it enough to know that your traffic is increasing, without having to spend valuable time analyzing it?

Think about it this way. If you know which engines are sending you the most traffic, you can boost your optimization strategies for those engines by creating additional pages for other relevant keyword phrases. This could increase your traffic even more.

Or, if you know that you're not getting any traffic at all from a particular engine, you'll be able to consider strategies for findability on that engine.

Through your referrer logs, you'll probably discover that you're getting found through keyword phrases that you haven't even considered before. In that case, you certainly don't want to change those pages and lose the traffic. By the same token, if you're getting found under a keyword phrase in one engine, wouldn't it be worth creating pages for the other engines for that same keyword phrase to see if you can bring in some additional traffic?

You can also find out through which pages you are losing visitors. This begs the questions...why are you losing visitors? ...and what changes can you make to keep them from leaving?

Simply put, a referrer log can give you an enormous amount of information and can serve as a road map for future changes to your site.

So, how can you view your referrer logs?

Ask your Internet Service Provider (ISP) to provide referrer logs in extended log format. If they don't provide referrer logs, you're missing out on some extremely valuable information about your web site. You may even want to consider changing ISP's.

However, even if your provider captures referrer information, you may want to get a program to read it, since the raw data can be a little cumbersome to analyze.

Here's an example of such an entry:

216.219.177.29 - - [15/May/2000:23:03:36 -0800] "GET /index.htm HTTP/1.0" 200 3956 "http://www.altavista.digital.com/cgi- bin/query? pg=aq&text=yes&d0=1%2fnov %2f99&q=email+marketing%2a +AND+email marketing%2a&stq=30" "Mozilla/2.0 (compatible; MSIE 4.0; SK; Windows 98)"

Not exactly easy reading, is it?

However, through the above entry, you can learn:

• The IP address of your visitor -- 216.219.177.29
• The date and time of the visit -- [15/May/2000:23:03:36 -0800]
• The first file requested -- "GET /index.htm HTTP/1.0"
• The fact that the request was completed -- 200
• The number of bytes that were transferred -- 3956
• Where your visitor came from and the keywords used to find your site -- "http://www.altavista.digital.com/cgi-bin/query? pg=aq&text=yes&d0=1%2fnov %2f99&q=email+marketing%2a +AND+email marketing%2a&stq=30"
• Browser and operating system of the visitor -- "Mozilla/2.0 (compatible; MSIE 4.0; SK; Windows 98)"

Though the information in the raw data provides invaluable information to you, you can easily see that it requires a little effort to analyze it yourself.

If you're serious about analyzing your traffic, consider obtaining a quality log analysis software program. Examples of programs are listed at the bottom of this article.

When analyzing your referrer log information, what should you look for?

• Number of page views per day

  Page views (or page impressions) refer to how many "hits" your site has had to HTML pages only. A hit is an action on a web page, such as when a user views a web page.

• Number of user sessions per day

  User session refers to the activity of one visitor to a web site.

• How long users are staying on your pages

  How long did the users stay at your website? Are they finding what they're looking for, or are they getting frustrated and leaving?

  If visitors are immediately clicking out of your site, maybe it's time to set up an onsite search engine. After all, once you get visitors to your site, you want them to be able to find what they're looking for. <http://www.searchbutton.com>SearchButton.com is an excellent service. It even provides statistics on who is searching your site, the most popular searches, and will also alert you to searches that produced no results.

  Ginette Degner, Professional Optimize, adds,

  "The time users spend at a site tells me if they are actually reading the site or just clicking in only to leave immediately. Perhaps I am not conveying the purpose of the site well enough to make them stay. Or, I am listed under the wrong phrases. If I am buying traffic from GoTo or another engine, I like to know if the words I chose are valuable to me or just a waste."

• Most requested and least requested pages

  Degner further explains,

  "Which page is attracting the most visits and how long are they there? This helps me decide what areas of a site need to be expanded upon and what areas can be dropped. For an example with a sports picks site, we found that the least visited page was the record the handicapper used to show everyone his win/lose ratio for picks and the chat room. So we dropped the page and spent the programming money on live scores and a sports news page."

• Top entry pages

  How are people first coming in to your site? Which pages are bringing you the most traffic? What about some of your other pages? What can you do to make them "top entry pages" too?

• Top exit pages

  Exit pages are another very important area of a log file, according to Degner. She goes on to ask and then elaborates on the answers...

  "Where are they leaving? What off site links are they clicking on the most? If this is an intro page or another sub page that is a doorway, I may need to get rid of it or use a redirect. It tells me where I am losing visitors."

  "As an example, a client insisted upon having a second intro page that played their radio commercial, so you clicked on the index page to enter and were stuck in a second media-enhanced page. We could see half of our traffic leaving right there and going no further into the site, which was a really big clue that it was a turn off to the surfing public."

  "A lesson to corporate sites - a website is the wrong place to be vain. Serve up your product or service immediately or pay the price with an impatient dotCom'r."

  "On other sites, we have discovered that a screen shots page made a huge difference in converting sales, and that was where the most orders came from (exit link was the order link)."

• Single access pages

  Which pages are being viewed by themselves, where visitors aren't even clicking to go to another page? Again, look at these pages carefully to see what you can provide on the page to keep up the interest of your visitors. You're losing them, and you need to figure out why.

• Errors, such as 404 pages

  If your visitors encounter too many error messages when visiting your website, they'll assume that you don't do your "house cleaning," and the professional image of your site will plummet several notches.

• Most active countries

  If you want a corner of the international market, study this data carefully. How many user sessions are being generated for each country that's important to your business? How you can beef up efforts to improve those numbers? Are you creating highly targeted information pages for your international keywords?

• Top referring sites and URLs
  
  
• Top referring search engines

  Do you have some top ranking pages in certain search engines, but you're not seeing coinciding traffic through those engines? If so, you may need to rethink your keyword strategy, because a truckload of #1's won't do you any good if traffic doesn't follow.

• Keywords that searchers are using to find your site

  If you're being found under a particular keyword in one engine but not another, boost efforts in the other search engine and try to bring in more traffic. Also, study this section closely for any holes in your keyword-thinking strategies.

  Remember that search engine positioning strategies begin with a simple keyword or keyword phrase. If you're having problems finding a keyword phrase that will bring you more traffic. Also most search engines have "related search" results that can give you some clues, don't over look that information.

• Browsers used by your visitors

  Check this section periodically to make sure that the technology offered at your website can be used by the majority of your visitors. In other words, if many of your visitors are accessing the web using older browsers, you will want to be careful about using technology that will prevent them from fully utilizing your website.

• Visiting spiders

  Which search engine spiders have visited your site recently? After submitting your pages to the engines, be sure to monitor this section closely for spider activity.

What do the experts feel are the most valuable parts of a referrer log?

Charlie Morris, Managing Editor of Web Developer's Journal, looks at the list of most popular pages first.

"This is something that's easy to act on - whatever content is most popular, simply produce more of the same sort of thing. The list of 404s should also be one of the first sections to look at, as sometimes (though not always) it's easy to find and fix these errors, greatly improving short- term traffic and your long-term reputation."

"The most important thing of all, however, is simply to compare a site's traffic from month to month, to measure how well your promotional efforts are working. It's surprising how many sites don't do this carefully."

To Rocky Rawstern, Professional Optimizer with SeventhWave.com, the most important areas of the referrer log are

"specific search strings (what the browser was looking for), search engine traffic counts, and finding keywords that are hit but aren't on our list to work on. We will work harder on a search engine that we aren't getting much traffic from."

The spider/bot sections are the most valuable areas of a log file to Ginette Degner. "When did they spider our site and how many pages? Which engines are sending me the most traffic? What words or phrases are being used consistently? What exact phrases were used to get to my site? All of this information assists me with positioning. I can find the strengths and weaknesses of a website and exploit the strengths and work to fix the areas that are lacking."

Excellent closing advice for positioners

Ginette Degner says,

"Keep track of when a spider hits your site and how deep and compare it to the dates you submit and the dates the pages actually appear in the index. You will start to see a pattern emerge with each engine. Yes, there are hiccups, but it will help you time your submissions, and when a client asks you when they can expect to see results, you can answer intelligently."

"When you are stumped as to why none of the words you optimized for are hitting, look at the logs. Use the words you see actually hitting with the search engines in your reporting files (Top Dog or WebPosition Gold). It can open your eyes and help you find more avenues of traffic for your client."

"At the very least, you can show some positions which will make a client more comfortable because you are showing they are getting traffic from the engines. This helps immensely when you hear 'I do not have any sales' from a client. You can steer the client to rethink their approach or sales copy."

So, take the time to analyze your traffic, and then put that valuable information to work on your website and reap the benefits of even more traffic!

Log Analysis Programs

WebTrends
http://www.webtrends.com

FlashStats
http://maximized.com/products/flashstats

Northern Web's Keyword Sniffer (Free Perl Script)

This article was written by Robin Nobles. Over the past few years, she has trained over 900 people in her online and onsite courses in search engine positioning strategies.

This article first published by Planet Ocean in their Search Engine News Update Newsletter.