Crawl budget is a vital SEO concept that often gets overlooked.
There are so many tasks and issues an SEO expert has to keep in mind that it’s often put on the back burner.
In short, crawl budget can, and should, be optimized.
In this article, you will learn:
- How to improve your crawl budget along the way.
- Go over the changes to crawl budget as a concept in the last couple of years.
What Is Crawl Budget
So for those of us who’ve had so much to think/worry/sweat about that we forgot what crawl budget even means, here’s a quick recap.
Crawl budget is simply the frequency with which search engine’s crawlers (i.e., spiders and bots) go over the pages of your domain.
That frequency is conceptualized as a tentative balance between Googlebot’s attempts to not overcrowd your server and Google’s overall desire to crawl your domain.
Crawl budget optimization is just a series of steps that you can take specifically to up the rate at which search engines’ bots visit your pages.
The more often they visit, the quicker it gets into the index that the pages have been updated.
Consequently, your optimization efforts will take less time to take hold and start affecting your rankings.
With that wording, it certainly sounds like the most important thing we all should be doing every second, right?
Well, not entirely.
Why Is Crawl Budget Optimization Neglected?
To answer that question, you only need to take a look at this official blog post by Google.
As Google explains plainly, crawling by itself is not a ranking factor.
So that alone is enough to stop certain SEO professionals from even thinking about crawl budget.
To many of us, “not a ranking factor” is equated to “not my problem.”
I disagree with that wholeheartedly.
But even forgetting that, there are Google’s Gary Illyes’ comments. He has stated outright that, sure, for a huge website of millions and millions of pages, crawl budget management makes sense.
But if you’re a modestly-sized domain, then you don’t have to actually concern yourself too much with crawl budget. (And in fact added that if you really have millions and millions of pages, you should consider cutting some content, which would be beneficial for your domain in general.)
But, as we all know, SEO is not at all a game of changing one big factor and getting the results.
SEO is very much a process of making small, incremental changes, taking care of dozens of metrics.
Our job, in a big way, is about making sure that thousands of tiny little things are as optimized as possible.
In addition, although it’s not a big crawling factor by itself, as Google’s John Mueller points out, it’s good for conversions and for the overall website health.
With all that said, I feel it’s important to make sure that nothing on your website is actively hurting your crawl budget.
How to Optimize Your Crawl Budget Today
There are still things that are super heavy-duty and others’ importance has changed dramatically to a point of not being relevant at all.
You still need to pay attention to what I call the “usual suspects” of website health.
1. Allow Crawling of Your Important Pages in Robots.Txt
This is a no-brainer, and a natural first and most important step.
Managing robots.txt can be done by hand, or using a website auditor tool.
I prefer to use a tool whenever possible. This is one of the instances where a tool is simply more convenient and effective.
Simply add your robots.txt to the tool of your choice will allow you to allow/block crawling of any page of your domain in seconds. Then you’ll simply upload an edited document and voila!
Obviously, anybody can pretty much do it by hand. But from my personal experience I know that with a really large website, where frequent calibrations might be needed, it’s just so much easier to let a tool help you out.
2. Watch Out for Redirect Chains
This is a common-sense approach to website health.
Ideally, you would be able to avoid having even a single redirect chain on your entire domain.
Honestly, it’s an impossible task for a really large website – 301 and 302 redirects are bound to appear.
But a bunch of those, chained together, definitely hurt your crawl limit, to a point where search engine’s crawler might simply stop crawling without getting to the page you need indexed.
One or two redirects here and there might not damage you much, but it’s something that everybody needs to take good care of nevertheless.
3. Use HTML Whenever Possible
Now, if we’re talking Google, then it has to be said that its crawler got quite a bit better at crawling JavaScript in particular, but also improved in crawling and indexing Flash and XML.
On the other hand, other search engines aren’t quite there yet.
Because of that, my personal standpoint is, whenever possible, you should stick to HTML.
That way, you’re not hurting your chances with any crawler for sure.
4. Don’t Let HTTP Errors Eat Your Crawl Budget
Technically, 404 and 410 pages eat into your crawl budget.
And if that wasn’t bad enough, they also hurt your user experience!
This is exactly why fixing all 4xx and 5xx status codes is really a win-win situation.
In this case, again, I’m in favor of using a tool for website audit.
SE Ranking and Screaming Frog are a couple of great tools SEO professionals use to do a website audit.
5. Take Care of Your URL Parameters
Always keep in mind that separate URLs are counted by crawlers as separate pages, wasting invaluable crawl budget.
Again, letting Google know about these URL parameters will be a win-win situation, save your crawl budget, as well as avoid raising concerns about duplicate content.
So be sure to add them to your Google Search Console account.
6. Update Your Sitemap
Once again, it’s a real win-win to take care of your XML sitemap.
The bots will have a much better and easier time understanding where the internal links lead.
Use only the URLs that are canonical for your sitemap.
Also, make sure that it corresponds to the newest uploaded version of robots.txt.
7. Hreflang Tags Are Vital
In order to analyze your localized pages, crawlers employ hreflang tags. And you should be telling Google about localized versions of your pages as clearly as possible.
First off, use the <link rel="alternate" hreflang="lang_code" href="url_of_page" /> in your page’s header. Where “lang_code” is a code for a supported language.
And you should use the <loc> element for any given URL. That way, you can point to the localized versions of a page.
Summary
So if you were wondering whether crawl budget optimization is still important for your website, the answer is clearly yes.
Crawl budget is, was, and probably will be an important thing to keep in mind for every SEO professional.
Hopefully these tips will help you optimize your crawl budget and improve your SEO performance.
Good luck!
More Resources:
Image Credits
Featured Image: Created by author, August 2019
Screenshots taken by author, August 2019