Why should you block your internal search result pages for Google? Well, how would you feel if you are in dire need for the answer to your search query and end up on the internal search pages of a certain website? That’s one crappy experience. Google thinks so too. And prefers you not to have these internal search pages indexed.
Google considers these search results pages to be of lower quality than your actual informational pages. That doesn’t mean these internal search pages are useless, but it makes sense to block these internal search pages.
Back in 2007
Many years ago, Google, or more specifically Matt Cutts, told us that we should block these pages in our robots.txt. The reason for that:
Typically, web search results don’t add value to users, and since our core goal is to provide the best search results possible, we generally exclude search results from our web search index. (Not all URLs that contains things like “/results” or “/search” are search results, of course.)
– Matt Cutts (2007)
Nothing changed, really. Even after more than 10 years of SEO changes, this remains the same. The Google Webmaster Guidelines still state that you should “Use the robots.txt file on your web server to manage your crawling budget by preventing crawling of infinite spaces such as search result pages.” Furthermore, the guidelines state that webmasters should avoid techniques like automatically generated content, in this case, “Stitching or combining content from different web pages without adding sufficient value”.
However, blocking internal search pages in your robots.txt doesn’t seem the right solution. In 2007, it even made more sense to simply redirect the user to the first result of these internal search pages. These days, I’d rather use a slightly different solution.
Blocking internal search pages in 2018
I believe nowadays, using a noindex, follow meta robots tag is the way to go instead. It seems Google ‘listens’ to that meta robots tag and sometimes ignores the robots.txt. That happens, for instance, when a surplus of backlinks to a blocked page tells Google it is of interest to the public anyway. We’ve already mentioned this in our Ultimate guide to robots.txt.
The 2007 reason is still the same in 2018, by the way: linking to search pages from search pages delivers a poor experience for a visitor. For Google, on a mission to deliver the best result for your query, it makes a lot more sense to link directly to an article or another informative page.
Yoast SEO will block internal search pages for you
If you’re on WordPress and using our plugin, you’re fine. We’ve got you covered. Your internal search results pages are noindexed by default in Yoast SEO. In addition, you can set a template for the page title of those pages, so the correct title will display in your browser tab or if your title is used in your theme.
You can set the template in the Search Appearance section of Yoast SEO. Just click on SEO › Search Appearance › Archives › Special pages in the left hand menu, and you’ll get this. This is the default template, but you can amend it, if you consider it necessary:
Most other content management systems allow for templates for your site’s search results as well, so adding a simple line of code to that template will suffice:<meta name="robots" content="noindex,follow"/>
Meta robots AND robots.txt?
If you try to block internal search pages by adding that meta robots tag and disallowing these in your robots.txt, please think again. Just the meta robots will do. Otherwise, you’ll risk losing the link value of these pages (hence the follow in the meta tag). If Google listens to your robots.txt, they will ignore the meta robots tag, right? And that’s not what you want. So just use the meta robots tag!
Back to you
Did you block your internal search results? And how did you do that? Go check for yourself! Any further insights or experiences are appreciated; just drop us a line in the comments.
Read more: Robots.txt: the ultimate guide »