The Complete Guide for Beginners

Looking to learn what duplicate content is, and how it might be hurting your SEO?

Duplicate content is a source of constant anxiety for many site owners.

Read almost anything about it, and you’ll come away believing that your site is a ticking time bomb of duplicate content issues. A Google penalty is merely days away.

Thankfully, this isn’t true—but duplicate content can still cause SEO issues. And with 25–30% of the web being duplicate content, it’s useful to know how to avoid and fix such issues.

In this guide, you’ll learn:

What is duplicate content?

Duplicate content is exact or near-duplicate content that appears on the web in more than one place. It can occur on a single website or cross-domain.  

For example, if I were to republish this post at ahrefs.com/blog/duplicate-content-copy/, then that would be duplicate content. That would also be true if I were to republish it on another website.

Google states that most duplicate content is not deceptive in origin.

Why is duplicate content bad for SEO?

Duplicate content may harm your SEO performance for a few reasons.

1. Undesirable or unfriendly URLs in search results;
2. Backlink dilution;
3. Burns crawl budget;
4. Scraped or syndicated content outranking you.

Let’s explore these in more depth.

1. Undesirable or unfriendly URLs in search results

Imagine that the same page is available at three different URLs:

1. domain.com/page/
2. domain.com/page/?utm_content=buffer&utm_medium=social
3. domain.com/category/page/

The first should show up in search results, but Google can get this wrong. If that happens, an undesirable URL may take its place.

Because people may be less inclined to click on an unfriendly URL, you may get less organic traffic.

2. Backlink dilution

If the same content is available at many URLs, then each of those URLs may attract backlinks. That results in the splitting of “link equity” between URLs.

To show an example of this in the wild, take a look at these two pages on buffer.com:

https://buffer.com/library/social-media-manager-checklist
https://buffer.com/resources/social-media-manager-checklist

These pages are almost exact duplicates. And they have 106 and 144 referring domains (links from unique websites), respectively.

Before you panic, know that this isn’t always a problem because of how Google handles duplicate content.

In simple terms, when they detect duplicate content, they group the URLs into one cluster. They then “select what [they] think is the ‘best’ URL to represent the cluster in search results” and “consolidate properties of the URLs in the cluster, such as link popularity, to the representative URL.”

So, in the case above, Google should show only one of the URLs in organic search and attribute all referring domains in the cluster (106+144) to that URL.

But that’s not what happens, as we see both URLs ranking in Google for similar keywords.

In this instance, Google likely isn’t consolidating “link equity” at one URL.

3. Burns crawl budget

Google finds new content on your website via crawling, which means they follow links from existing pages to new pages. They also recrawl pages they know about from time to time to see if anything has changed.

Having duplicate content serves only to create more work for them. That can affect the speed and frequency at which they crawl your new or updated pages.

That’s bad because it may lead to delays in indexing new pages and reindexing updated pages.

4. Scraped content outranking you

Occasionally, you may permit another website to republish your content. That’s known as syndication. Other times, sites may scrape your content and republish it without permission.

Both of these scenarios lead to duplicate content across multiple domains, but they usually don’t cause problems. It’s only when the scraped or republished content starts outranking the original on your site that issues arise.

The good news is this is a rare occurrence, but it can happen.

Does Google have a duplicate content penalty?

Google has stated on multiple occasions that they don’t have a duplicate content penalty.

We don’t have a duplicate content penalty. It’s not that we would demote a site for having a lot of duplicate content. 

Let’s put this to bed once and for all, folks: There’s no such thing as a duplicate content penalty. 

DYK Google doesn’t have a duplicate content penalty.

But, this isn’t entirely true. If your duplicate content is accidental and not the result of intentional manipulation of search results or spammy practices, then you won’t get penalized. If it is, then you might.

Google confirms that here:

In the rare cases in which Google perceives that duplicate content may be shown with intent to manipulate our rankings and deceive our users, we’ll also make appropriate adjustments in the indexing and ranking of the sites involved. As a result, the ranking of the site may suffer, or the site might be removed entirely from the Google index, in which case it will no longer appear in search results.

The question is, what counts as “intent to manipulate our rankings and deceive our users”?

Google has a lot of information on that here. But basically, it’s things like:

• Intentionally creating multiple pages, subdomains, or domains with lots of duplicate content.
• Publishing lots of scraped content
• Publishing affiliate content scraped from Amazon or other sites (and adding no additional value)

However, as discussed above, duplicate content can still hurt SEO—even without a penalty.

Common causes of duplicate content

There’s no single cause of duplicate content. There are many.

Faceted/filtered navigation

Faceted navigation is where users can filter and sort items on the page. Ecommerce websites use it a lot.

This kind of navigation appends parameters to the end of the URL.

Because there are usually many combinations of these filters, faceted navigation often results in lots of duplicate-or-near-duplicate content.

Take a look at these two pages, for example:

bbclothing.co.uk/en-gb/clothing/shirts.html?new_style=Checked
bbclothing.co.uk/en-gb/clothing/shirts.html?Size=S&new_style=Checked

The URLs are unique, but the content is almost identical.

Plus, the order of the parameters often doesn’t matter. For example, the same page is accessible at both of these URLs:

bbclothing.co.uk/en-gb/clothing/shirts.html?new_style=Checked&Size=XL
bbclothing.co.uk/en-gb/clothing/shirts.html?Size=XL&new_style=Checked

Tracking parameters

Parameterized URLs are also used for tracking purposes. For example, you may use UTM parameters to track visits from a newsletter campaign in Google Analytics:

Example: example.com/page?utm_source=newsletter

Session IDs

Session IDs store information about your visitors. They usually append a long string to the URL like so:

Example: example.com?sessionId=jow8082345hnfn9234

HTTPS vs. HTTP, and non-www vs. www

Most websites are accessible at one of these four variations:

• https://www.example.com (HTTPS, www)
• https://example.com (HTTPS, non-www)
• http://www.example.com (HTTP, www)
• http://example.com (HTTP, non-www)

If you’re using HTTPS, it’ll be one of the first two. Whether it’s the www or non-www version is your choice.

However, if you don’t correctly configure your server, your site will be accessible at two or more of these variations. That isn’t good and can lead to duplicate content issues.

Case-sensitive URLs

Google sees URLs as case-sensitive.

URLs are case-sensitive, but pick whatever case you want.— 🍌 John 🍌 (@JohnMu) June 22, 2017

That means these three URLs are all different:

• example.com/page
• example.com/PAGE
• example.com/pAgE

Trailing slashes vs. non-trailing-slashes

Google treats URLs with and without trailing slashes as unique. That means these two URLs are unique in Google’s eyes:

• example.com/page/
• example.com/page

If your content is accessible at both URLs, then that can lead to duplicate content issues.

To check if this is an issue, try to load a page with and without the trailing slash. Ideally, only one version will load. The other will redirect.

For example, if you try to load this post without the trailing slash, it will redirect to the URL with the trailing slash.

Google states that this behavior is ideal.

If only one version can be returned (i.e., the other redirects to it), that’s great! This behavior is beneficial because it reduces duplicate content.

Print-friendly URLs

Print-friendly versions have the same content as the original. It’s only the URL that differs.

• example.com/page
• example.com/print/page

Mobile-friendly URLs

Mobile-friendly URLs, like print-friendly URLs, are duplicates.

• example.com/page
• m.example.com/page

AMP URLs

Accelerated Mobile Pages (AMP) are duplicates.

• example.com/page
• example.com/amp/page

Tag and category pages

Most CMS’ create dedicated tags pages when you use tags.

For example, if you have an article about organic whey protein, and you use both “protein powder” and “whey” as tags, then you’ll end up with two tag pages like these:

https://www.caltonnutrition.com/tag/whey/
https://www.caltonnutrition.com/tag/protein-powder/

That doesn’t always cause duplicate content in itself, but it can.

That’s the case here because there’s only one page on the site with those two tags—so each tag page is identical.

Showing caltonnutrition.com/tag/whey/.

Showing caltonnutrition.com/tag/protein-powder/.

Attachment image URLs

Many CMS’ create dedicated pages for image attachments. These pages usually show nothing but the image and some boilerplate copy.

Because this copy is the same across all auto-generated pages, it leads to duplicate content.

Paginated comments

WordPress and other CMS’ allow for paginated comments. This causes duplicate content as it effectively creates multiple versions of the same URLs.

• example.com/post/
• example.com/post/comment-page‑2
• example.com/post/comment-page‑3

Localization

If you’re serving similar content to people in different locales who speak the same language, then that can cause duplicate content.

For example, you might have different versions of your site for people in the US, UK, and Australia. Because there are likely only minor differences between the content served to each locale (e.g., prices in dollars versus pounds sterling), the versions will be near duplicates.

Search results pages

Lots of websites have search boxes. Using these typically takes you to a parameterized search URL.

Example: example.com?q=search-term

Google’s former Head of Webspam, Matt Cutts, stated that:

Typically, web search results don’t add value to users, and since our core goal is to provide the best search results possible, we generally exclude search results from our web search index. (Not all URLs that contain things like “/results” or “/search” are search results, of course.) 

Staging environment

A staging environment is a duplicate or near-duplicate version of your site used for testing purposes.

For example, imagine that you want to install a new plugin or change some code on your website. You might not want to push that straight to a live site with hundreds of thousands of daily visitors. The risk of catastrophe is too high. The solution is to test the changes in a staging environment first.

Staging environments become an SEO issue when Google indexes them because it results in duplicate content.

How to check for duplicate content on your site

Head over to Ahrefs’ Site Audit and start a crawl.

Once done, head to the Content quality report.

Look for clusters of duplicates and near-duplicates without a canonical. These are highlighted in orange.

Click any of these clusters to see the affected pages.

Investigate the reason for the duplicate content, then take the appropriate action.

Note that these won’t always be issues that need rectifying, especially in the case of near duplicates.

You can also check for duplicate title tags, meta descriptions, and H1s in the HTML tags report.

Bad duplicates are what you’re looking for. These are pages with duplicate meta tags but different canonicals.

Select these by clicking the “Bad duplicates” toggle under HTML tags & content.

Click on any of the yellow bars to see the affected pages.

Pages with duplicate titles, meta descriptions, or H1’s are often very similar.

For example, these two have the same title tag, and the content is almost identical because the product is the same. The only difference is that one of the pages is for a 3‑pack of instant lighting firelogs, whereas the other is for just one.

https://www.xs-stock.co.uk/big-k-instant-light-the-wrapper-firelog-3-pack-camp-fire-fuel/
https://www.xs-stock.co.uk/big-k-instant-light-the-wrapper-firelog-camp-fire-chiminea/

Google states that you should minimise similar content like this:

If you have many pages that are similar, consider expanding each page or consolidating the pages into one.  

However, a small number of similar pages is unlikely to be much of an issue.

How to check for duplicate content issues across the web

Content scraping and syndication can also lead to duplicate content issues. But it’s only usually an issue if you see scraped versions of your content outranking you.

Does that happen? Yes, but it’s often more of an issue for new or weak websites. Why? Because the sites scraping your content are often more authoritative. That sometimes “tricks” Google into thinking that theirs is the original.

If you have a small website, then you can often find scraped content by searching Google for a snippet of text from your page in quotes.

For larger sites, you’ll need to use an automated tool like Copyscape. This searches the web for other occurrences of the content on your page(s).

Whichever method you use, most results will be from spammy and low-quality sites.

Generally speaking, these are nothing to worry about. However, if you see that a legitimate website scraped your content, and are concerned that it may be stealing your traffic, throw the URL into Ahrefs’ Site Explorer to see an organic traffic estimate.

If it’s getting more traffic than your page, then there may be an issue.

In this case, you have three options:

1. Reach out and request that they remove the content.
2. Reach out and request they add a canonical link to the original on your site.
3. Submit a DMCA takedown request via Google.

If you intentionally syndicate content to other websites, then it’s worth asking them to add a canonical link to the original. That will eliminate the risk of duplicate content issues.

Final thoughts

Don’t stress over duplicate content too much. It’s usually much less of an issue than it’s thought to be.

If you have a handful of duplicate or near-duplicate pages, there’s unlikely to be much of a problem. The same is true when quoting content from another website or other pages on your site. Small amounts of duplicate or boilerplate content should be okay. Google has systems in place to deal with such things.

What you need to be on the lookout for are technical SEO mishaps that lead to the generation of hundreds or thousands of pages of duplicate content, such as the improper implementation of faceted navigation on ecommerce sites.

These can wreak havoc on your crawl budget, amongst other things.

Let me know in the comments or on Twitter if you’re struggling with duplicate content.