What we learned when Google deindexed our site for a day

Last Friday, Google mistakenly delisted Search Engine Land from its index and search results. That meant none of our pages showed up in Google search — none.

Fortunately, we were able to get a response from Google on Friday morning. We were told that its “system misidentified the site as being hacked” and thus was mistakenly removed from the index. Google said it had caught and fixed the issue on its end early that morning, removing the hack classification. However, our site didn’t return to the index at around 5:30 p.m. that evening.

The irony of a website dedicated to covering search engine marketing getting accidentally delisted by Google was not lost on us (or the community). Following are more details on what happened, how Google responded to our follow-up questions, and most importantly what SEOs, webmasters can learn from our dramatic day.

What we’ve learned so far

In total, Google did not include Search Engine Land in the results for more than 12 hours on Nov. 30.

What happened? We’re told the site was completely removed (rather than given a warning) because Google’s hacking classifiers had some sort of bug and classified our site as being hacked when it was not. “This was a mistake in our systems — a false positive that indicated that the site had been hacked,” a Google spokesperson said.

Why no warning before removal? One of the biggest questions we received from the SEO community was why didn’t Google just label the site as being hacked in the search results? Why did Google take the severe step of removing the site completely from the index?

“If we determine a site is hacked, there are rare instances in which we may deindex the site,” Google told us. “For example, if spammers deface organic pages, including the homepage, and replace the original content entirely with spam. These types of hacks are uncommon.”

This makes sense if a site is so badly hacked that Google does not want searchers to risk having their computers infected after visiting a compromised site. So Google delists these types of hacked sites from the index entirely. But obviously, when Google makes a mistake, it can have significant ramifications for the site affected.

Do false-positives happen often? We asked Google if this is a common occurrence. That it will deindex sites due to detected bugs in the hacking classifiers. “Mislabeling a site is something that should never happen,” Google responded. “Unfortunately we don’t always get it right,” Google added. Then Google explained a bit more about how it works, “There are complex systems at play to label sites, as you can imagine automatically determining the hacked status of a website is no easy task.”

“In this specific case we weren’t able to spot the mistake in the backend before the site was delisted as hacked from our Search results,” Google told us.

We do know it’s relatively rare, but Google did not indicate how often these mistakes happen.

Why did it take several hours for an email? Google Search Console is programmed to send automated email notifications about site hacks and other website issues to site owners. For some reason, in this case, it took several hours for that email to go out.

The site was likely deindexed by Google around 3 a.m. Friday morning. We learned of the problem from readers on social media. We never received a warning or a notice that there was a hack that we needed to fix within a certain timeframe before the site was removed.

“We strive to be as fast as possible to ensure website owners are in the know as things evolve on Search,” Google said. “Ideally the machinery should work in unison and in real-time. As you saw in this case, the reality is a bit different: there’s definitely room for improvement here.”

Of course, we would love more answers. We still are unclear what specifically caused Search Engine Land to be classified as a hacked site. Was it truly just a random error or was there something that triggered the false positive? Google had yet to provide further detail on this question by the time we published this story.

We would also like further clarification about the average time it takes Google Search Console system to send out email notifications, particularly notifications about serious problems like the one supposedly detected on Search Engine Land.

What happened to our traffic that day?

Every site is different, of course, but we’re sure you can imagine that essentially losing organic search traffic for a day stung in terms of traffic. But the good news was other sources picked up a lot as users and community members sought alternate avenues to access our content. Compared to an average Friday, Bing organic traffic increased by more than 15 percent, for example. What really minimized the losses, though were direct, referral and social hits. Direct traffic was up more than 25 percent from the average. Referral increased 90 percent, and, social media traffic exploded, jumping more than 300 percent, mostly directed to our day’s coverage of our unfortunate deindexing.

Most of that gain, honestly, ties right back to our community, which helped boost our traffic from other sources.

What can you do if this happens to your site?

We’ll be honest, it was not fun, but in the end, there’s not likely to be lasting damage for us. But we can’t help wondering how tragic it would have been if a similar thing happened to an e-commerce site, especially during Cyber Week. As digital marketers, we all know the precarious nature of relying on Google — or any one company — to drive a significant share of traffic to our websites. But it’s instances like this that  underscore the inherent vulnerability of being subject to the whims or errors of a primary source of traffic. Particularly when there aren’t humans you can call to get a problem addressed.

If this kind of error can happen to us, it can happen to anyone, so here’s some advice we can pass on from this experience:

1. If you think you’ve been penalized in error, make noise in multiple places and hope you get heard quickly. Google suggested Google Webmaster support forums where, it said, the are hundreds of “Product Experts” and Googlers who are reachable every day. “These people can escalate issues on their behalf.” But as someone who knows these forums pretty well, often it is hard to get things escalated via those forums. Use Twitter to get heard, too. There’s the main Google account, and as SEOs, you’re likely already following the accounts of John Mueller (@JohnMu) and Danny Sullivan (@SearchLiason).
2. If your organization is distributed, try to give people in multiple time zones access to Google Search Console.
3. Have a plan to leverage other channels to try to make up for traffic losses. Tweets, for example, do show in Google Search results. We didn’t have a plan, and again, were lucky that news of our plight that day got a lot of attention from the community. That ended up driving significant direct, referral and social traffic. Make a contingency plan now — that should go for any channel that drives a significant share of your traffic or revenue. This will require coordination with content, social, email, advertising and other teams beyond digital. There are likely promotions, pieces of content or ways to tap influencers that you can consider deploying in case of emergency.

The takeaways

This episode just reinforced the saying “You don’t want to keep all of your eggs in one basket.” Thankfully, with our active community, social media presence and our general reputation we came out relatively unscathed. But Google is still a black box, by its own admission, and its machinery doesn’t always function as promised. Try to diversify as much as possible and have contingency plans.

If we learn more, you’ll be the first to know.

---