Crawl budget is a key concept for SEO professionals because mismanaging your website crawl budget can lead to Google not indexing pages of your site, and ultimately losing valuable search traffic on your site.
While most of the sites don’t need to worry about crawl budget, if you run a website with more than 500k pages, you have to focus on optimizing your site’s crawl budget.
A few things which can affect your site crawl budget are:
- On-site duplicate content
- Soft-error pages
- Low quality and spam content
- Faceted navigations and URL parameters
- And hacked pages.
The best approach to optimize the crawl budget is to start with improving overall site speed and simplifying site structure as both of these will help both users and the Googlebot. Then work on the internal links, fix duplicate content issues and remove redirect chains.
Improve site speed. Google states that “making a site faster improves the users’ experience while also increasing the crawl rate.” So, enable compression, remove render-blocking JS, leverage browser caching and optimize images to give Googlebot time to visit and index all of your pages.
Simplify website architecture. Structure the website layer by layer, starting with the homepage, then categories/tags, and finally, the content pages. Review your site structure, organize pages around topics, and use internal links to guide crawlers.
Avoid orphan pages. As orphan pages have no external or internal links, and no direct connection with the web, Google has a really tough time finding these pages.
Limit duplicate content. Everyone, including Google, only wants quality and unique content. So, implement the canonical tags properly, noindex category/tag pages, and limit URLs with parameters.
Remove redirect chains. Redirects, especially in massive sites, are very common. But redirects having more than three hops can create problems for Googlebot. Use a log analyzer to find redirects and fix them by pointing the first URL to the last one in the chain.
Use internal links. Google prioritizes pages with many external and internal links, but it’s not possible to get backlinks to every single page of the site. With proper internal linking, Googlebot can reach to every page of the website.
If you are dealing with a massive site (a huge e-commerce brand, for example), crawl budget will be an important thing to keep in mind.
Opinions expressed in this article are those of the guest author and not necessarily Search Engine Land. Staff authors are listed here.
About The Author
Suganthan Mohanadasan is an experienced international digital marketing consultant from Norway. He is presently working as a freelancer and helps medium to large companies around the world. His specialties include SEO, paid search, social and native advertising.