ucchau13
12-23-2012, 11:17 PM
On the morning of December 14th, MozCast registered the largest 24-hour Google ranking flux on record since we started tracking data in early April. The temperature for Thursday, December 13th was 102.2° F (for reference, the original Penguin update was 93.1°):
This was especially striking since I had just rolled out a small fix in our computations for a problem that was slightly overestimating temperatures on some days since the rollout of 7-result SERPs. SERPmetrics confirmed substantial levels of 24-hour flux, and webmaster chatter suggested that people were seeing major ranking and organic traffic changes.
Unfortunately, Google was unable to confirm an algorithm update. So, where does that leave us? It turns out that it’s not an easy question.
The Big Signals
A while back we launched a set of five top-view metrics to help provide an at-a-glance view of patterns across the entire set of rankings MozCast tracks. Only one of those metrics moved noticeably between December 13th and 14th – PMD Influence suffered a sizeable one-day drop. PMD Influence is the percentage of Top 10 results occupied my partial-match domains (PMDs). This includes hyphenated and non-hyphenated domains that contain the keyword phrase but are not an exact match. Here’s the 30-day view:
PMD Influence dropped from 3.73% on 12/13 to 3.54% on 12/14 (about a 5.1% drop in 24 hours). While my gut says that drop wasn’t the full picture, it’s a good place to start. So, which sites lost out in this change?
Across the 1,000 SERPs tracked, this PMD drop represents a change of only 18 partial-match domains that fell out of the top ten. It’s a bit more complicated than that, though. There were actually 36 PMDs that fell out of the top ten, and 18 new PMDs that entered the top ten, for a net difference of 18. Analyzing these domains one-by-one can turn into a wild goose chase pretty quickly, so let’s look at a couple of situations where a keyword lost multiple PMDs.
One query that lost two PMDs was “barbeque”. On 12/13 the following PMDs ranked in the top ten:
www.springcreekbarbeque.com
www.qbarbeque.com
www.barbequeman.com
scbarbeque.com
www.waltsbarbeque.com
The next day, domains (4) and (5) fell out of the top ten. Domain (5) had been floating near the #10 spot, so that may be a fluke. Interestingly, for just one day, Wikipedia’s barbecue page fell completely out of the top ten, after ranking in the #1 position consistently. We’ll explore that in the next section.
Here’s another example with multiple PMD losses – the keyword “joannes" had three PMDs ranking on 12/13:
www.joannesbedandback.com
www.joannesbb.com
www.joannesgourmetpizza.com
The next day, only (1) remained. Again, (2) and (3) were taking up the tail end of the top ten, and in this case were bumped out by Yelp and Urban Spoon, so this change may be smaller than it initially looks.
One PMD that lost ranking caught my eye – a query for “gmaps”. On 12/13, the domain [www.mgmaps.com] fell out of the top ten. This turns out to be a shift from a 10-result SERP to a 7-result SERP, and the PMD was sitting at #8 prior to the shift. Interestingly, though, Google Maps, which had been sitting at #2, took the #1 spot and got site-links and a 7-result SERP. We’ll come back to this one.
Sorry - we’re not exactly making the situation clearer, are we? I want to illustrate just how complex the situation really is. I’ve come to believe that not even Google fully understands the dynamic system they’ve created. Ultimately, there were no clear patterns across the PMD changes, so let’s dive into a couple of specific situations.
A Wiki Situation
Wikipedia suffered a rare (albeit temporary) loss of their coveted #1 position for the query “barbeque”. Since Wikipedia holds the largest share of top-ten real estate in our data set, a major change to the site (such as a technical problem that caused temporary de-indexation) could cause very large-scale flux in the rankings. Luckily, we can run these numbers.
On 12/13, Wikipedia had a 4.56% top-ten share in our data set, which dropped to 4.41%, for a net loss of 14 rankings. This may not sound like much, phauthuathanquoc.org (http://thammyhanquoc.org/) until you recall that that change is on par with the 18 ranking PMD shift (and Wikipedia is just one site). In some ways, this seems to be an anomaly of 12/13 more than 12/14, as Wikipedia held a 4.46% share on 12/12. Historically, the 12/14 numbers aren’t unheard of – Wikipedia had a 4.82% share back in June, for example.
I should also note that the Wikipedia page in question for the query “barbeque” was actually the “/Barbecue” (alternate spelling) page. It’s possible that a spell-check adjustment or other very minor code tweak could have had unexpected repercussions.
This does go to show, though, how a site as powerful as Wikipedia can definitely have an impact on the overall SERP landscape. Like the PMDs, I don’t think it’s the entire picture, but it is a piece of the puzzle.
The Curious Case
Let’s go back to another oddity in the PMD analysis – the query for “gmaps”. On the morning of 12/14, the official Google Maps site not only jumped from #2 to #1, but it got site-links and a 7-result SERP, pushing out three domains. It’s easy to jump to conclusions and assume Google is favoring their own products, except that two pieces of data make that unlikely here.
The first clue is that Google Maps returned to the #2 position on 12/15 (and a 10-result SERP). The second is that we know that something big happened on 12/13 – Google Maps finally re-launched on Apple’s iOS6. Here’s a headline and time-stamp from Forbes:
Obviously, this story had a ripple effect across 12/13, and probably had a huge impact on metrics (CTR, dwell time, etc.) related to Google Maps and the official site. While this doesn’t help our quest to find the source of the update, it is interesting to note that a major news item could not only change a ranking, but cause a 7/10 shift in results. My ongoing investigations indicate that 7-result SERPs are highly dynamic and automatically change based on factors that may include user metrics and QDF (“freshness”).
The Big Movers
Everything to this point came out of just one data point – the PMD shift. Let’s go back to the beginning and ask the other obvious question – which queries changed the most from 12/13 to 12/14? This turns out to be a tricky question, because some queries are just naturally higher-flux than others. Typically, I compare the 24-hour “temperature” for any given query to the 7-day average for that query, to get a ratio. This helps indicate which queries are unusually high-flux. For 12/14, here are ten unusually high-flux queries (with temperatures):
“knockout roses” (181°)
“condo rentals” (168°)
“rosatis pizza” (161°)
“aerosoles store locator” (158°)
“bj wholesale hours” (151°)
“party stores” (143°)
“kitchen sinks” (137°)
“millionaire matchmaker” (125°)
“celiac disease diet” (119°)
“garnishment” (115°)
Any one query is an anecdote – the web changes. What we’re looking for in the data is a calling card of sorts – a story that ties these queries together. Unfortunately, the patterns are all over the place. Our top mover (1) was just a case of an eHow page jumping up the rankings. Two of these queries (6 and 10) have no clear explanation other than multi-spot shifts. Query (8) seems to be a case of QDF and has high volatility outside of the 7-day window.
Four queries (2, 4, 5, and 9) showed shifts in domain diversity. For three of them, one domain went from a single spot in the top ten to multiple spots. For query (5), though, one domain lost spots (diversity increased). Our top-view metrics aren’t showing any big overall shifts in domain diversity, but there are always winners and losers day-to-day.
Query (3) was another case where Wikipedia dropped out of the top ten, and (7) saw an Amazon product page fall from #1 to #10. In the case of (3), Yelp moved up and went from one ranking in the top ten to two. In both cases, the big sites regained their positions on 12/15, which is certainly interesting. If we look at the MozCast “Big 10” data, though, Wikipedia was still #1 and Amazon #2 on 12/14, and the overall SERP share of the Big 10 didn’t move much.
The Bigger Mystery
So, where does all of this leave us? A handful of people were kind enough to send me evidence of search traffic losses on 12/14, but it’s very difficult to reconcile these specific cases against MozCast’s sampling of top ten SERPs. I can’t pinpoint any single factor here, but it seems clear that the amount of change was unusual, and it can’t be simply explained by any single event (this data was all recorded prior to the tragic events in Connecticut, for example).
It’s possible that Google made a small change – so small that they didn’t even consider it an “update” – that had unexpected repercussions. It’s possible that something non-algorithmic but still under Google’s control happened, such as processing a large chunk of disavow requests (we have no evidence of this – just covering the bases). It could be that a small set of highly influential sites, like Wikipedia, made large-scale changes. Or it could just be a massive coincidence (although my gut still says no on this one).
I’d welcome further data and discussion. We’re actively working to expand the MozCast data set, and the next version of it will include some enhancements, including a keyword set that’s cleanly divided across some major categories/verticals. We’ll also be working in the new year to automate some of the analysis tools, so that we can process large numbers of SERPs more quickly. We’re learning as we go, and I hope the exploration is useful.
Power:Seomoz.org
This was especially striking since I had just rolled out a small fix in our computations for a problem that was slightly overestimating temperatures on some days since the rollout of 7-result SERPs. SERPmetrics confirmed substantial levels of 24-hour flux, and webmaster chatter suggested that people were seeing major ranking and organic traffic changes.
Unfortunately, Google was unable to confirm an algorithm update. So, where does that leave us? It turns out that it’s not an easy question.
The Big Signals
A while back we launched a set of five top-view metrics to help provide an at-a-glance view of patterns across the entire set of rankings MozCast tracks. Only one of those metrics moved noticeably between December 13th and 14th – PMD Influence suffered a sizeable one-day drop. PMD Influence is the percentage of Top 10 results occupied my partial-match domains (PMDs). This includes hyphenated and non-hyphenated domains that contain the keyword phrase but are not an exact match. Here’s the 30-day view:
PMD Influence dropped from 3.73% on 12/13 to 3.54% on 12/14 (about a 5.1% drop in 24 hours). While my gut says that drop wasn’t the full picture, it’s a good place to start. So, which sites lost out in this change?
Across the 1,000 SERPs tracked, this PMD drop represents a change of only 18 partial-match domains that fell out of the top ten. It’s a bit more complicated than that, though. There were actually 36 PMDs that fell out of the top ten, and 18 new PMDs that entered the top ten, for a net difference of 18. Analyzing these domains one-by-one can turn into a wild goose chase pretty quickly, so let’s look at a couple of situations where a keyword lost multiple PMDs.
One query that lost two PMDs was “barbeque”. On 12/13 the following PMDs ranked in the top ten:
www.springcreekbarbeque.com
www.qbarbeque.com
www.barbequeman.com
scbarbeque.com
www.waltsbarbeque.com
The next day, domains (4) and (5) fell out of the top ten. Domain (5) had been floating near the #10 spot, so that may be a fluke. Interestingly, for just one day, Wikipedia’s barbecue page fell completely out of the top ten, after ranking in the #1 position consistently. We’ll explore that in the next section.
Here’s another example with multiple PMD losses – the keyword “joannes" had three PMDs ranking on 12/13:
www.joannesbedandback.com
www.joannesbb.com
www.joannesgourmetpizza.com
The next day, only (1) remained. Again, (2) and (3) were taking up the tail end of the top ten, and in this case were bumped out by Yelp and Urban Spoon, so this change may be smaller than it initially looks.
One PMD that lost ranking caught my eye – a query for “gmaps”. On 12/13, the domain [www.mgmaps.com] fell out of the top ten. This turns out to be a shift from a 10-result SERP to a 7-result SERP, and the PMD was sitting at #8 prior to the shift. Interestingly, though, Google Maps, which had been sitting at #2, took the #1 spot and got site-links and a 7-result SERP. We’ll come back to this one.
Sorry - we’re not exactly making the situation clearer, are we? I want to illustrate just how complex the situation really is. I’ve come to believe that not even Google fully understands the dynamic system they’ve created. Ultimately, there were no clear patterns across the PMD changes, so let’s dive into a couple of specific situations.
A Wiki Situation
Wikipedia suffered a rare (albeit temporary) loss of their coveted #1 position for the query “barbeque”. Since Wikipedia holds the largest share of top-ten real estate in our data set, a major change to the site (such as a technical problem that caused temporary de-indexation) could cause very large-scale flux in the rankings. Luckily, we can run these numbers.
On 12/13, Wikipedia had a 4.56% top-ten share in our data set, which dropped to 4.41%, for a net loss of 14 rankings. This may not sound like much, phauthuathanquoc.org (http://thammyhanquoc.org/) until you recall that that change is on par with the 18 ranking PMD shift (and Wikipedia is just one site). In some ways, this seems to be an anomaly of 12/13 more than 12/14, as Wikipedia held a 4.46% share on 12/12. Historically, the 12/14 numbers aren’t unheard of – Wikipedia had a 4.82% share back in June, for example.
I should also note that the Wikipedia page in question for the query “barbeque” was actually the “/Barbecue” (alternate spelling) page. It’s possible that a spell-check adjustment or other very minor code tweak could have had unexpected repercussions.
This does go to show, though, how a site as powerful as Wikipedia can definitely have an impact on the overall SERP landscape. Like the PMDs, I don’t think it’s the entire picture, but it is a piece of the puzzle.
The Curious Case
Let’s go back to another oddity in the PMD analysis – the query for “gmaps”. On the morning of 12/14, the official Google Maps site not only jumped from #2 to #1, but it got site-links and a 7-result SERP, pushing out three domains. It’s easy to jump to conclusions and assume Google is favoring their own products, except that two pieces of data make that unlikely here.
The first clue is that Google Maps returned to the #2 position on 12/15 (and a 10-result SERP). The second is that we know that something big happened on 12/13 – Google Maps finally re-launched on Apple’s iOS6. Here’s a headline and time-stamp from Forbes:
Obviously, this story had a ripple effect across 12/13, and probably had a huge impact on metrics (CTR, dwell time, etc.) related to Google Maps and the official site. While this doesn’t help our quest to find the source of the update, it is interesting to note that a major news item could not only change a ranking, but cause a 7/10 shift in results. My ongoing investigations indicate that 7-result SERPs are highly dynamic and automatically change based on factors that may include user metrics and QDF (“freshness”).
The Big Movers
Everything to this point came out of just one data point – the PMD shift. Let’s go back to the beginning and ask the other obvious question – which queries changed the most from 12/13 to 12/14? This turns out to be a tricky question, because some queries are just naturally higher-flux than others. Typically, I compare the 24-hour “temperature” for any given query to the 7-day average for that query, to get a ratio. This helps indicate which queries are unusually high-flux. For 12/14, here are ten unusually high-flux queries (with temperatures):
“knockout roses” (181°)
“condo rentals” (168°)
“rosatis pizza” (161°)
“aerosoles store locator” (158°)
“bj wholesale hours” (151°)
“party stores” (143°)
“kitchen sinks” (137°)
“millionaire matchmaker” (125°)
“celiac disease diet” (119°)
“garnishment” (115°)
Any one query is an anecdote – the web changes. What we’re looking for in the data is a calling card of sorts – a story that ties these queries together. Unfortunately, the patterns are all over the place. Our top mover (1) was just a case of an eHow page jumping up the rankings. Two of these queries (6 and 10) have no clear explanation other than multi-spot shifts. Query (8) seems to be a case of QDF and has high volatility outside of the 7-day window.
Four queries (2, 4, 5, and 9) showed shifts in domain diversity. For three of them, one domain went from a single spot in the top ten to multiple spots. For query (5), though, one domain lost spots (diversity increased). Our top-view metrics aren’t showing any big overall shifts in domain diversity, but there are always winners and losers day-to-day.
Query (3) was another case where Wikipedia dropped out of the top ten, and (7) saw an Amazon product page fall from #1 to #10. In the case of (3), Yelp moved up and went from one ranking in the top ten to two. In both cases, the big sites regained their positions on 12/15, which is certainly interesting. If we look at the MozCast “Big 10” data, though, Wikipedia was still #1 and Amazon #2 on 12/14, and the overall SERP share of the Big 10 didn’t move much.
The Bigger Mystery
So, where does all of this leave us? A handful of people were kind enough to send me evidence of search traffic losses on 12/14, but it’s very difficult to reconcile these specific cases against MozCast’s sampling of top ten SERPs. I can’t pinpoint any single factor here, but it seems clear that the amount of change was unusual, and it can’t be simply explained by any single event (this data was all recorded prior to the tragic events in Connecticut, for example).
It’s possible that Google made a small change – so small that they didn’t even consider it an “update” – that had unexpected repercussions. It’s possible that something non-algorithmic but still under Google’s control happened, such as processing a large chunk of disavow requests (we have no evidence of this – just covering the bases). It could be that a small set of highly influential sites, like Wikipedia, made large-scale changes. Or it could just be a massive coincidence (although my gut still says no on this one).
I’d welcome further data and discussion. We’re actively working to expand the MozCast data set, and the next version of it will include some enhancements, including a keyword set that’s cleanly divided across some major categories/verticals. We’ll also be working in the new year to automate some of the analysis tools, so that we can process large numbers of SERPs more quickly. We’re learning as we go, and I hope the exploration is useful.
Power:Seomoz.org