It’s not about the tools you have, it’s about how you interpret the data.
Most importantly, it’s how you evaluate what your data is really telling you.
Look at this chart:
Now, ask yourself what you see. Say it out loud, or in your head, whatever.
I bet the first thing you said was:
“The rankings on this site dropped”.
That’s not correct.
If you said,
“The keyword rankings for this site have dropped according to Searchmetrics.”
You’d be perfectly right.
The problem with keyword intelligence
The very name implies someone else is doing the thinking for you. I think that’s the problem. Our industry loves a new tool – and I’m no exception. SEOgadget is a happy customer of Searchmetrics (and SEMrush) and we have been for a long time. I use the tools as useful starting points for keyword research and competitive analysis and sales calls!
But, a few weeks back, one of our clients got in touch, extremely concerned, because their Searchmetrics visibility had dropped.
As it turns out, nothing was actually wrong, in fact, they’d had a 30% increase in their organic traffic, rankings were up all over the place and therefore there’d been a positive net revenue uplift on their SEO channel. Everyone had become so dependent on referring to visibility metrics via this data, and no one had actually checked what the data meant, or how it correlated with the real world. In this case, there were enough keywords outside of the SM database referring traffic to the site to make the aggregated visibility score irrelevant.
We had to point out to our team that everyone should be careful how we interpret single points of data. It happens.
What’s keyword intelligence good for?
Well, it’s definitely good for getting a flavor of how websites generate traffic from search. You can see where a site ranks for a keyword, provided that keyword is in the intelligence database, you can see what paid ads your competitors might be running (provided that keyword’s in the database). You can see how the rankings for those keywords have changed changed over time.
It’s fair to hope that the bigger the keyword database is, the more likely it is we’ll see data resembling actual analytics data. The reality is that the data is gappy, and therefore you need to be very careful how you interpret it.
Never forget the classic advice: don’t rely on a single data point to draw a conclusion. With that most basic of principles in mind, you can’t go wrong.
What I found
Well, first a caveat – we know this will be specific to individual domains. So you have to try this yourself.
Now I’ve said that, here’s what we did for that particular client:
We looked at Searchmetrics and SEMrush. We found that these tools have a spread of coverage – defined in our case by the number of keywords found in either tool that match actual traffic driving keywords coming from Google.
I think the right way to describe both tools for this domain is “generally gappy, more accurate for higher volume keyword coverage than long tail”.
To calculate spread, I exported 5,000 keywords from both SEMrush and Searchmetrics databases and compared those to the top 5,000 keywords we retrieved from the client’s analytics tool.
What we looked for is the count of keywords found in the site’s top 5,000 referring keywords from both tools.
Search metrics contained 1891 – 37.82% coverage, SEMrush 613 keywords – 12%. Essentially, there were lots more higher volume terms appearing in the analytics data than the intelligence tools databases.
You could be more wrong than you are right
Think about that for a second. If you make a judgement on your “SEO visibility” from an incomplete list of keywords, what’s going to happen? If you’ve got less than a 40% spread, won’t you have more chance of being wrong about the outcome as you could have to be correct? That’s precisely what happened – the keywords that were included in the intelligence database *did* drop, while may volume terms that were not in the database shot up.
I asked Pete to take a look at the data, and he gave me this response (we looked at SEMrush at the same time)
Interestingly, he pointed out, the higher the search volume, the more likely the keyword is to be in the data. When we ringfenced “head” terms that had “very high traffic” levels, SearchMetric’s coverage was much better: 78%
Here’s what we saw:
- SM knows about 80% of VHT keywords (very high traffic) - SM knows about 64% of HT keywords (high traffic) - SM knows about 33% of LT keywords (lower traffic)
- SEMr knows about 69% of VHT keywords - SEMr knows about 21% of HT keywords - SEMr knows about 8% of LT keywords
The odds that reports are accurate for this particular site are a reflection of the distribution of their high traffic phrases (15% high traffic / 85% low traffic), and how often Searchmetrics has the data on the site (38% of the time SearchMetrics had the data, 12% of the time SEMrush did).
Following on from this, we examined the distributions for specific volumes of phrases as well, and compared the estimated volume accuracy against the actual traffic. The estimated average volume for SearchMetrics for very high traffic phrases was 3802, out by 16%, with a standard deviation of 962. Across the bottom 15% of phrases, these numbers became 10.1, 51% and 32.
The takeaway from this is clear: whilst SM and SEMr are undoubtedly useful tools for alerting you to things that are going on with the rankings they track, what they track isn’t necessarily what you actually rank for and what drives your most important traffic, nor may the scoring values actually equate to traffic. That’s what we found, for this domain, and now we’re taking a look at more, to see what the differences are. Go do the same and let me know in the comments. I wish I had more time, and I certainly wish I had the chance to take a look at Sistrix.
Pete and I discussed this for some time – the only way you’d get a really accurate impression of a domain’s referring keyword portfolio is if you had many, many more keywords available. There’s probably a better package on all the tools for that, but that is not the point.
A more complete view (in the UK, at least) is most likely Hitwise territory – and if you’re arriving at that point, you’ve got to ask yourself what the business case should be for increasing spend, massively on this type of tool.
For me, I see SM and SEMr as good tools for sites that target high traffic, generic “head” keywords. For an enterprise class site, where often the most keywords that form the organic search traffic portfolio are “long tail”, or highly seasonal, emerging terms, Searchmetrics aggregate rank tracking (the visibility charts) could be missing out.
If you really want to track your search visibility, and your competitors, best to devise a large, potentially seasonal keyword tracking list and use a tool like AWR to monitor daily.
Indeed – SEOgadget.com is quite a good example. Searchmetrics reports on about 8% of the keywords that actually drive traffic to the site. You could easily derive a partial head term keyword strategy from the data, but you’d struggle to gain any clear idea of our real rankings.
Anyway, that’s what we found, for this particular domain, and now we’re taking a look at more sites, to see what the differences are. Go do the same and let me know in the comments.
Image: Tambako The Jaguar