When Penguin, an update of Google’s search algorithm, swam online five years ago, it just about sank StairSupplies.
“In the course of a week we went from 1,000-1,200 organic clicks a day [from Google search results] to 100-200,” says Caleb Morris, marketing manager at the Goshen, Ind.-based vendor of stairway components for house remodelers. “The only traffic we had left was from paid ads.”
StairSupplies was a victim of an ongoing, behind-the-scenes battle between search engines and the experts who try to decipher their algorithms — or game them, depending on whom you ask or the color of the expert’s hat. Yes, hat, as in black hat versus white hat. Search engines to some extent operate in the Wild West.
Google, like other search engine proprietors, is constantly shifting, enhancing and tweaking its search algorithm, which determines who is listed in what order on a search results page, and some of those recalibrations are noteworthy enough to earn names, as in the case of Penguin (see the “Evolution of the Google search algorithm” sidebar). Those changes often try to nullify deceptive tactics that websites use to improve their Google rankings — and can leave companies such as StairSupplies scrambling to regain lost ranking stature. Trying to manipulate search algorithms — and coping with the tide of changes — has been the goal of experts in search engine optimization (SEO) for almost as long as we’ve had search engines.
The stakes are high. “Number one on the search results list will get double the traffic of number two, and number two gets triple the traffic of number five, so SEO really matters,” says Bruce Clay, who heads up digital marketing optimization firm Bruce Clay Inc.
Except that no one can actually define search engine algorithms. Google, for example, employs a kaleidoscope of hundreds of constantly shifting “signals” to rank pages. (Google declined to comment on its search algorithm for this story, which is in keeping with a thickening veil of secrecy that shrouds the process. The company stopped updating the public version of its database of page rankings in 2013 and cut off all access to it last year.)
“There are those who claim to be able to reverse engineer Google’s search algorithm,” says former Google search engineer Vanessa Fox, now head of Keylime Toolbox, a search analytics software firm. “But even Google doesn’t know what it is at a given moment — algorithms are being used to change other algorithms. Teams work on different areas of it. There is no way to see it all at once. And it will change every few hours.”
Make no mistake: All SEO experts are gunning for Google. There are other search engines, of course, most prominently Bing and Yahoo. However, “We look at Bing, etc., but 99% of our effort is for Google,” says Kyle Sanders, head of Complete Web Resources in Austin. “If you are ranking well there, you will rank well with the others.”
But if Google’s algorithm cannot be precisely dissected, sources agree that it has two major concentrations: It looks for incoming links, which give a page the presumption of authoritativeness, and it looks for content that matches the user’s query.
There are likewise two major types among SEO experts and their clients: Those who labor to build sites with compelling content that other sites would find it worthwhile to link to (popularly known as “white hats”) and those who aren’t concerned with content but seek only to manipulate the algorithm to suck in traffic (sometimes called “black hats”).
Early on, the black-hat approach led to the creation of content farms, with superficial or copied text crammed with selected keywords; and link farms, sites that existed solely to link to other sites. That led to attempts by Google to suppress value-free content and link farms, using algorithm updates with names such as Penguin and Panda.
Penguins and penalties
At StairSupplies, previous management had used some overseas firms for SEO, recalls Morris. “They realized they were getting a huge boost in traffic, so they did not look too closely at how it was done, but it was largely black-hat,” he says. Because Penguin set out to lower rankings for companies that used link farms, StairSupplies got hit with an algorithmic penalty.
There are two kinds of Google penalties for trying to game the system, explains Fox: manual and algorithmic. Manual penalties are levied against sites that have been flagged by a human employee of Google. They can range from reduced search rankings to complete deletion from the Google database, and are accompanied by notices placed in the site owner’s Google Search Console. The owner can apply for reconsideration after fixing the problem. Figures released by Google in 2014 showed 400,000 manual penalties per month, of which fewer than 5% triggered a reconsideration application. The rest were presumably black hats who abandoned the penalized URLs.
Algorithmic penalties are the cumulative negative page rankings that sites experience when a refresh of Google’s search algorithm flags their SEO practices as unsavory, Fox says. There is no notification or reconsideration. The owners must notice the problem, diagnose it and fix it on their own.
After being burned because it hadn’t been paying attention to how its SEO was being doctored, StairSupplies switched to another SEO firm that migrated the company from a dot-com to a dot-net address, Morris recalls. But the junk links followed them and the penalty remained in force.
StairSupplies then switched to yet another SEO firm that said it would have to go through Google channels and disavow the bad links. “We emailed the linking sites up to three times asking them to drop the links,” says Morris. “If there was no response, we uploaded a list of the links to Google to disavow them. Over three or four months, we saw our ranking slowly go back up. With the addition of better content rather than just keyword-stuffing, our traffic is now again at 1,200 organic clicks per day.”
After the battle
While Google’s initiatives may have taken the wind out of classic black-hat practices, not everyone has gotten the memo.
“A large majority of [so-called SEO experts] are sales guys who outsource to India, and the result is really bad content by non-native English speakers,” complains Josh Rubin, co-founder of Post Modern Marketing in Sacramento. “I get a dozen requests a day from India wanting me to outsource my SEO work to them for pennies on the dollar — I can have them do the work for 20% of what I get. But they would do black-hat SEO and get the customers penalized.”
The Wild West ambience of the SEO field has certainly struck Rubin, who notes that it has neither a governing authority nor professional credentialing. He recalls that a stranger from the Philippines started complaining to him that Rubin had not paid him for six months. “It turns out that a firm pretending to be mine had hired people in the Philippines to build fake sites to do back-linking and other spammy tactics, with Skype interviews and a 30-page contract. They were getting free work,” Rubin explains.
True black-hat these days has moved toward negative SEO and malware, says Clay. Negative SEO, he explains, involves pointing black-hat links at the sites of competitors to poison their rankings, while malware involves hacking into unsecured sites to add links from them to one’s own site.
“But you still see sites using old black-hat tactics that talk to robots rather than to people, where the language is poor, keywords are stuffed everywhere, and the site has 10,000 incoming links,” says Rubin.
The white-hat way
Penguin aside, credible, engaging, mobile-friendly, fast-loading, secured sites with fresh and original content (i.e., things Google is known to like) still need legitimate incoming links if they are going to have decent rankings, sources agree. “You need links in order to rank in a competitive market — just making [SEO-related] site changes won’t produce that big a lift,” says Sanders.
As for getting links the white-hat way, “It’s very labor-intensive,” says Julie Joyce, director of operations at link-building firm Link Fish Media. “You search the web like any human and find a site that looks good and makes sense. If you saw this link in an article on the site you would click it. If that is the case you would say that’s a quality link.”
Then you have to get that site to link back to your site. “We do almost all of that with email,” says Joyce. “After deciding it is a good site, we write a personalized email and send it. But if we can’t find legitimate contact information for the site, we don’t bother with them — they probably don’t want to be contacted for a reason.”
Not every site responds. “I am usually happy with a 5% response rate, but it can sometimes be 30%, and sometimes 2%,” says Joyce. “It depends on the industry and reputation of the client. DIY bloggers want to connect to home furnishing sites. But we once had a loan site that no one wanted to connect to.”
“If they say they will do it for money and the client will buy links, we will negotiate a no-follow link that does not violate Google guidelines. The link is still good for traffic. Some webmasters say they want $25 to cover the effort to go into their code. Others just say, ‘Yes, we did it and here’s the link,'” explains Joyce. “But from my experience, linking does not involve big money.”
In any event, “Google wants people to think that they can tell the difference between paid and unpaid links, but if a link looks natural, they can’t tell if you paid for it,” says Joyce. “If you are a small site and have three links, a couple of paid links a month would not be noticeable.”
Future voice
Meanwhile, the search environment is changing. Users are increasingly speaking their search queries into mobile devices, using services such as Apple’s Siri and Google Assistant, and getting spoken answers. SEO will have to change to keep up.
“Mobile search now amounts to more than half of all Google searches, and of those, 20% to 25% are voice,” says Barry Schwartz, head of web marketing firm RustyBrick. “Those are usually longer queries, such as ‘How do I do X-Y-Z?’ Your site has to have content that answers those kinds of questions.”
“You need to identify the top 10 questions that your customers ask and have a page that answers each of them,” Rubin explains. “You must answer each of them clearly and concisely.”
“Voice search and mobile search are the future of the web, and therefore any SEO firm who knows what they are doing is working in this area,” says Clay. “Find the terms that are important and define them, using ‘what is.’ Then use schema to define the words further — if it said ‘mercury,’ put schema around it saying it is the chemical, not the planet.”
For the schema itself, Google recommends marking up selected text with JavaScript Object Notation for Linked Data (JSON-LD).
Admittedly, being the source of a straight voice answer is, at this juncture, of little benefit to the site owner, since the process ends with the utterance of the answer and there is no link back to the answer’s source. But Google has been adding other formats to its results pages as well, and schema also affects who gets included in those new formats.
These include the “Knowledge Graph” summation box, a sort of mini-encyclopedia entry on an established topic; a “snippet box,” offering an outtake from the search result with a link to the source; news excerpts; answers to questions stated or inferred by the search; thumbnails of video or image results; and a list of related searches.
“There is only one snippet result, and people are winning out big-time who worked hard to be in that snippet box,” notes Britney Muller, head of SEO at Moz, a Seattle-based web consulting firm. “If you are already on the first page, you need to evaluate how the current box’s page owner is marked up and simulate what is being rewarded with that snippet.”
Obviously the Google algorithm is an increasingly stern taskmaster. “You cannot get traffic accidentally anymore,” notes Clay. “Quite frankly there is only room for 10 in the top 10, but there are usually a million or more search results. When I started 21 years ago, I could get a site ranked in an hour. Now it takes six months. A lot of the things you have to do to satisfy the Google algorithm require that you be first among equals.”
Alternately, “You should not think about the algorithm too much,” says Chris Koller, head of Idealgrowth, a digital marketing agency in Dallas. “As long as you are giving the user the best possible experience, you are going to do well, now and in the future.”