“It is a capital mistake to theorize before one has data. Insensibly one begins to twist facts to suit theories, instead of theories to suit facts.”–Sherlock Holmes, A Scandal in Bohemia
Well said, Mr. Holmes. Well said.
Just like Sherlock Holmes, as SEO consultants, we stand behind our work by showing performance changes with our data. We’ve all come in on a morning ready to face the day ahead of us. We sit down to view our weekly report with our morning coffee in hand. Bam! We see something we’ve never seen before – traffic either completely fell off the planet or it’s spiked over 1,000%.
Needless to say, such a spike in any metric comes with questions. Questions which you, as an SEO expert, need to have answers for (most likely before 9:30am, when your boss looks at the report). Time to put on your detective cap and follow our recommended checklist to troubleshoot your data fluctuations.
Seasonality
The first thing I look at when analyzing changes in traffic is seasonality. Was this trend seen on a holiday or a day that might have been particularly big or small for your client? For example, a major flower seller sees multiple spikes in the search volume on their brand name from December through May because of the Christmas, Valentine’s Day and Mother’s Day holidays.
These holidays most likely correlate with their analytics’ spikes in orders and visits as well. To truly understand your data you must understand it within the context of time.
Figure 1 – Google Trends for a flower seller shows seasonality driven spikes.
If your reporting isn’t already monitoring things Year over Year (YoY), then it’s time to set that up. Many client’s business cycles follow the same YoY trends. Last year’s trends are a great benchmark for seasonal expectations. I have found that comparing the growth rates Year Week over Year Week (YWoYW) has been the most helpful for determining seasonality.
Figure 2 – Year Week over Year Week (YWoYW) Growth comparison example.
Channel Analysis
Once you’ve determined that the traffic is abnormal, and once you’ve gathered some historical information for proof, move on to a cross-channel analysis. The main thing you want to figure out here is “Is organic the only channel seeing this minor oscillation or is everyone’s data going berserk?”
Pay specific attention to paid search marketing tactics, because engines are known to change the aesthetics of their results to make it difficult to identify the differences between the organic and paid links. One time, we watched traffic from organic search taper off while paid traffic increased, only to find that our paid team had implemented expanded site links on high value brand keywords.
Cross-channel knowledge can help you to pinpoint whether or not you have a problem. However, if only one channel is seeing a change in performance, then it’s time to keep digging.
The data should be viewed as both a total count and as a percentage against other channels. The total numbers for the month long pull below give a good view of weekly trends, and one can clearly see the regular dips that occur during weekends.
Figure 3 – Trended channel visits by numbers.
Percentages are naturally normalized, and can give a better picture of cross channel fluctuations. In Figure 4, you can see clearly see one channel spike, taking incremental traffic from another.
Figure 4 – Trended channel visits by percent. When looking at percentages, the spike in visits is much clearer.
News: Client Industry & SEO
The next thing I like to do is check the news: specifically news relating to my client’s industry and SEO news. Here you look for whether something in the news might affect your client’s performance. Maybe your client released a new product without telling you. Maybe they got a lot of press for an ad or bad customer service. SEO is pull marketing, where customers actively seek information about products, and sometimes there are clear external reasons why traffic is flowing in.
Another consideration is whether there was a special event that would affect performance. For example, a pay-per-view event might increase traffic to our clients pay-per-view page and site. A great way to diagnose these events later is to perform a Week over Week (WoW) keyword analysis with Google Webmaster Tools (GWT) Search Query data. Once GWT updates its data you can compare last week’s keyword clicks to the current week’s keyword clicks.
Finally, if there was no news in your client’s industry, was there something in the SEO space that might have affected your performance? Did Google come out with another update that affected performance? Are other SEO client’s seeing a similar trend? How is MozCast, a report showing the “turbulence in the Google algorithm”, looking? Perhaps something occurred within the SERP, causing a fluctuation in traffic.
Figure 5 – Traffic for a new customer offering. The first spike represented the announcement and the second (highest spike) represents the launch.
Figure 6 – Moz Cast shows you the “turbulence in the Google algorithm”.
Visit Your Client’s Site
At this point in your analysis it’s time to check your client’s website to confirm that everything is functioning properly. I like to check core pages that drive conversions to make sure that they are functioning properly, focusing on meta robots tags, tracking codes, and the robots.txt file.
There is nothing worse (than maybe lemon juice poured in a paper cut, while you’re performing this research) than having a noindex tag added to a highly converting page, but it has happened. If nothing out of the ordinary is coming up, I like to check the Way Back Machine to see if it caught any shots of the site during the period in which the data fluctuated.
Segment Your Data
Three more sips of coffee later, it’s time to break out the magnifying glass. The next thing I do is try to segment the issue to a particular dimension. Dimensions are ways to group data, and metrics are measured across dimensions. Segmenting the data will help you to isolate factors related to this particular data fluctuation.
A tip from Mr. Holmes,
“I consider that a man’s brain originally is like a little empty attic, and you have to stock it with such furniture as you choose…The skillful workman is very careful indeed as to what he takes into his brain-attic… It is of the highest importance, therefore, not to have useless facts elbowing out the useful ones.” -Sherlock Holmes, A Study in Scarlet
Like Sherlock suggests, you will see a lot of non-related data. Diagnosing data requires a little bit of intuition, so don’t be afraid to skip steps. Your goal here is to find the most relevant information and explore it.
Device Type:
I usually start with segmenting device type, due to the tremendous impact that mobile has on overall traffic. If you see the trend on mobile, but not on desktop, you might be able to pinpoint the issue to a particular operating system or cell phone carrier. For example, there was a period when the iOS 6 operating system was not passing referrer data.
Another (more horrifying) example – what if your mobile site was down? Through segmenting this data, you can either prove or eliminate device type as the cause.
Search Engines:
If this is not the cause, simply move through the checklist to search engines. In terms of search engines, we’ve seen referrer data not passing from Yahoo affect performance. We’ve also seen Bing go down for a few hours at the beginning of this year. We’ve seen Mozilla choose Yahoo as its primary search engine. Maybe your ranking has dropped in one engine on your highest converting terms.
Segmented by search engine, your data can help you to understand these kind of issues.
Figure 7 – The Mozilla switch to Yahoo has given incremental volume to Yahoo.
Browsers:
Up next, segment by browsers. This will show you if any particular browser is affecting metrics. One example of when segmenting the data by browser proved helpful was when IE8 dropped referrer data. When looking at the total traffic coming from IE8, we see a relatively consistent number of page views; however, organic traffic from IE8 falls significantly.
Figure 8 – IE8 All Channels – You can see there is consistent amount of page views.
Figure 9 – IE8 Organic Channel – You can clearly see when the referrer data was dropped. Despite consistent page views seen in all channels, organic traffic plummets.
If you haven’t pinpointed the trend yet, it’s time to go a level deeper. At this point you’ve proven that this data is only an organic problem, that there is nothing (to the SEO World’s knowledge) in Google’s algorithm which could have effected your numbers, that there is nothing new which could have caused a spike, your site is still functioning, and the issue affects all devices, all search engines, and all browsers. It’s time to dig deeper within your site to detect the cause of this data fluctuation.
Entry Page:
The cause of this spike/dip is probably more specific than the overarching concepts we’ve searched. Looking at entry page data is a great way to start digging. If you can find a specific page which contains the upward trend, you’ll be able to confidently say it’s the cause.
For example, I have a client with a World Cup page. During the World Cup this page saw a huge traffic spike. Segmenting the data by this entry page showed us how many incremental visits we received from this event.
Another example is an outages page. Whenever there is an outage we see a huge spike to this particular page.
Location-Based:
If I can’t find any specific page, I turn to locally based dimensions like region or division. I have seen clients with service outages in particular regions, which inspired more visits to the site from those regions. This can also help identify a server failure in a particular region.
The idea is to continue to segment data until you can diagnose the fluctuation. Once you can segment the trend to a particular dimension, you’ve struck the goldmine. Jinkies!
Internal Site Search:
When nothing appears and I’m desperate, I usually turn to internal site search. Occasionally, finding what people are searching on the site is telling. I’ve seen users look to internal site search for specific error messages appearing on my client’s products.
Ask your Client
If you’ve gone through the entire checklist and nothing appears, ask your client. Something might have happened on the backend. The style of attribution or event (eVar) may have changed. Maybe something broke. Although it can be hard to tell your client that you don’t know everything, they will definitely appreciate your proactive approach to figuring out their analytics.
Closing & Takeaways
Hopefully, this article helps you to determine the cause of fluctuations in your data (or at least make you look like an all-star to your client). I would love to hear your insights in the comments.
I’d like to leave you with a check listing and a modified quote by Sherlock Homes from The Sign of the Four. For our modern analytical age, “When you have eliminated the impossible through data analysis, whatever remains, however improbable, must be the truth. Now prove it with your data.”
