Tricks For Avoiding Duplicate Content Penalties

Having quite a bit of experience handling duplicate content problems in A Little Google DeIndexing Puts Things In Perspective, An Accident Discovers The Cause Of My Google Deindexing, Site Pages Reindexed, and from running WM Media, here’s a few tips for bloggers (and other website owners) to avoid Google from assigning a duplicate content penalties to their blogs.

www vs non www – If your website is accessible via both www.domain.com and domain.com, then Google will end up indexing both at some point. To fix it, you should redirect everything (using a 301 redirect) to www.domain.com. Just modify the .htaccess file of your public_html folder and use the following code:

RewriteEngine On
RewriteBase /
RewriteCond %{HTTP_HOST} !^www\.yourdomain\.com [NC]
RewriteRule ^(.*) http://www.yourdomain.com/$1 [L,R=301]

slash vs no slash – If any of your website addresses is accessible with a trailing slash or with no trailing slash (for example, www.whatithinkabout.com/do-you-get-money and www.whatithinkabout.com/do-you-get-money/), Google also treats those as two separate webpages. Again, you’ll have to redirect no slashes to slashes. For wordpress, simply use the redirect plugin.

index.php vs no index.php – Another way the same page can be indexed in Google is if your webpage is available via the index file also. For example, www.whatithinkabout.com and www.whatithinkabout.com/index.php. In this case, you should redirect index.php to www.whatithinkabout.com via the following snippet in your .htaccess file:

RewriteEngine On
RewriteBase /
RewriteCond %{REQUEST_URI} index.html
RewriteRule .* http://www.yourdomain.com/? [R=301,L]

Categories & Archives – Make sure you add a noindex tag to these. Since these pages are all snippets of your other pages, it’s all just duplicate content organized differently. Therefore, you don’t want Google to index these pages and possibly devalue your article page(s)! Here’s another handy plugin for this: No index plugin

Search Box – The search box is another dangerous place where duplicate content may be indexed. For example, www.whatithinakbout.com/?s=development would be indexed if someone referenced this search result from another blog. Therefore, it is recommended that you disable search all together (by redirecting anything with parameters) and just putting in a Google search box (you make more money from that anyway!)

Oh, so to disable it, you can probably just redirect everything with parameters (since you should have permalinks for your posts anyway):

RewriteEngine On
RewriteBase /
RewriteCond %{QUERY_STRING} .
RewriteCond %{REQUEST_URI} !^/wp
RewriteRule .* http://www.yourdomain.com/? [R=301,L]

Title And Description Tags – If all your pages have the same title and description, then you’re making Google work that much harder to decipher what your pages are about. It would be wise to add a different description tag to every one of your pages, just in case the content seems similar. Besides, search engine visitors need to find your description compelling to click through to your blog! If you don’t have one, Google may not be so great at picking one for you! To do this, just install the description plugin!

Why Avoid Duplicate Content?

There are three good reasons to avoid duplicate content.

1. The ranking power of your page is decreased if the pages are spread apart. For example, incoming links to www.whatithinkabout.com and www.whatithinkabout.com/index.php could have been pointing to the same page. That means neither page ranks as highly on Google as just one would.

2. Possible duplicate content penalties. Since Google’s algorithm isn’t completely open and they need to prevent spam, it’s logical to assume that they may assign some sort of penalty to your website if there’s too much duplicate content. Imagine if tons of your blog’s searche pages got indexed (such as www.whatithinkabout.com/?s=blog), in which case you may have 1000s of pages of duplicate content!

3. The number of indexed pages is capped. Google has some limits on how many pages of a website can appear in their index. For example, site:www.google.com only shows 30 million pages or so indexed, whereas it probably has trillions of pages. Therefore, if you have 10 duplicate pages indexed, then that might just crowd out that all those important page you spent hours writing!

The Wrath Of Duplicate Content

That’s about it for duplicate content. It might not seem like a big deal, but it actually makes the difference between your blog being virtually unnoticed in search engines vs. your blog being one of the best ranked ones out there!

Just think about all the points above! Let’s say you don’t do any of them. Then google may index all of these URLs:

http://www.whatithinkabout.com

http://www.whatithinakbout.com/

http://www.whatithinakbout.com/index.php

http://www.whatithinakbout.com/index.php/

http://whatithinkabout.com

http://whatithinakbout.com/

http://whatithinakbout.com/index.php

http://whatithinakbout.com/index.php/

Additionally, let’s say you have one category and 5 pages of content. You’ll also have these pages indexed:

http://www.whatithinkabout.com/category

http://www.whatithinkabout.com/category/

http://www.whatithinkabout.com/post-1

http://www.whatithinkabout.com/post-1/

http://www.whatithinkabout.com/post-2

http://www.whatithinkabout.com/post-2/

http://www.whatithinkabout.com/post-3

http://www.whatithinkabout.com/post-3/

http://www.whatithinkabout.com/post-4

http://www.whatithinkabout.com/post-4/

http://www.whatithinkabout.com/post-5

http://www.whatithinkabout.com/post-5/

http://www.whatithinkabout.com/2008/02/post-1

Plus other archive links
…
Plus the non www versions of these
…

Not to mention if someone links to www.whatithinkabout.com/?s=post-1 in a search, that’s another 10 or 20 links.

Yet, how much content do you actually have here? Just 5 posts! What if Google indexes say only 90% of your website? With this setup, it’s possible that Google indexes all the duplicate content pages and none of your real content pages get indexed! Compare that to five pages ranking really well!

It seems like such a small thing, but it’s such a huge difference!

1) How much value these visitors receive and contribute – Since the primary goal of a website is to create value through the dissemination of information, it makes sense to see how each type of traffic helps to accomplish that goal.
2) How much these type of traffic is worth – After all, you deserve to be compensated for bringing this value to people!

Okay! On to the different types of web traffic:

Search Engine Traffic – People from search engines are looking for specific information. They type what they are looking for into a little box and up comes your website describing what they want. Depending on how accurate the description is, the value of the information that comes up ranges widely. For example, a search for mystery shopping jobs will bring up massreps.com, whose description is “Merchandisers and Mystery Shoppers: post or find merchandising jobs and mystery shopping jobs”. When the visitor clicks on it, he is taken to a job board, giving him exactly what he was looking for. On the complete opposite end of the spectrum, a search for say “Viagra” will probably bring up a bunch of spam sites, offering the visitor no value when they click through.

Because the visitors are only searching for a specific bit of information, they generally won’t stay for a very long time and will leave once they’ve found what they’re looking for. For example, 78% of search engine visitors for this blog click the back button immediately after reading one article. Unless they feel very strongly about something in the article, they are unlikely to participate beyond that.

Of course, the advantage to this type of visitor is that because they don’t really stick around, your marketing campaign can reach a lot of “unique” people for the number of page they view. You get a large number of different people looking at your ad, increasing the chance that someone would be interested in the product being advertised. As a bonus, because they leave immediately after finding what they’re looking for, they don’t use up a lot of server resources. As a result, this is probably one of the most desirable forms of traffic in terms of profit per amount of resource used.

Direct Traffic – These are people who have heard about your website from a friend or already have been to your website and came back. Instead of going to Google to find what they are looking for, they go to you! Therefore, the value you bring to this type of visitor is enormous. You give them exactly what they want to see! Of course, there is still a range of people, from the ones who just want to “check it out” to a loyal subscriber. In general though, this type of traffic benefits the most from your website.

This is also the type of traffic that exists in community type websites like forums. It’s a place where people of a like mind gather to discuss common things of interest to them. It’s not surprising then that great projects can come out of this group of people. For example, this final fantasy website has the largest forum out of all its competitors and as a result, contains the most comprehensive information on final fantasy games. Not only do these people enjoy your material, they enjoy it so much that they actively contribute to its creation!

All this reading and contributing takes a huge toll on your server resources though. A visitor that spends so much time on your website gets used to the positioning of the ads and may just scroll past them. In addition, he is not likely to purchase the same products over and over again (plus, he only has so much money!). You are more likely to make a sale to this person than a search engine visitor of course, but it’s much lower in terms of $$/resource usage – short term.

One key benefit of this type of visitor is that they create a lot of content and can spread the word via their close friends. So indirectly, they could certainly refer you a lot of prospective customers!

Referring Traffic – Somebody has told them about your website, be it a friend or a link from another website. As such, this type of visitor is more targeted than say, search engine traffic. They were referred by a reliable and trusted source that is already familiar with your website, so they already know there is value to be found here. After taking a look, they are more likely to convert into a loyal subscriber of your website and actively participate in it than other forms of non-direct traffic.

Of course, this does depend on the quality of the referring source. If you spam 100000 people and get 100 visitors via email referrals, that probably brings close to no value to those visitors. This source could also act like social bookmarking visitors if posted on a “check it out” type website like Digg.

Monetization wise, referring traffic is similar to search engine traffic in that there are a high number of unique visitors per resource expenditure. They are more likely to dig deeper into your website though, making the resource expenditure higher. However, that is balanced by the increased exposure to advertisements, so it’s generally a good thing!

Social Bookmarking Traffic – This is probably the least desirable form of traffic in terms of both value and monetization. People see an interesting title that has been voted high up on social bookmarking websites like Digg, and click through to “check it out”. They aren’t actively searching for the information you provided, so are less interested in your material than any other type of traffic. It’s the difference between someone sitting in front of their TV flipping channels vs. someone eagerly turning on their TV for their favorite 7pm show.

The upside of this type of traffic is that there is a lot of it. When an article of yours ranks high on social bookmarking websites, it’s not unusual to receive a few thousand visitors a day! Therefore, they can cause a general buzz about your website that bring about the other forms of more desirable traffic.

Of course, a small % of the visitors will actually be very interested in your material and may convert to be active participants. Since the amount of visits is just simply so large, this small % may be on par with other forms of traffic.

And there you have it, the four major forms of traffic you will experience throughout the course of your webmaster career. Depending on your goals, you may want to target specific forms of traffic. For example, if your main goal is to make money via search engine rankings, then you can devote more of your efforts to SEO. On the other hand, if your main goal is to form a community for developing future projects, then you would probably want to keep the site adless and ask for donations instead.

Either way, keep your goals in mind, and you’re sure to succeed!

I just read an article from Webomatica that commented on how hard blogging is. Amongst other things, the author mentioned how he has the perception of being greeted by a blank page day after day, needing to come up with something by a certain deadline. When I read it, it seemed like such a grim view of blogging!

From my experience, rarely have I gotten such a feeling! In fact, I only started a blog because there were just so many ideas floating around in my head. I would think, “Hmmm, what a shame to think of these unique ideas and then forget them right afterwards without anyone hearing them?”

From time to time, I would walk around and suddenly think of a unique perspective on something and jot it down. Blogging is just my means of not losing these ideas, and also allowing other people to experience of benefits of an interesting perspective. As a result, I am never short on material nor do I ever set a deadline for myself. The ideas come up when they come up and happen when they happen! I didn’t start a blog to find content – I already have unique ideas and just need a place to write them down!

And that is why exactly some bloggers feel “dry”. You only experience the feeling of “blank page after blank page” if you have nothing to talk about! And if you have nothing to talk about, how can you succeed at a venture that is essentially talking to people?! It seems to me that people fail at blogging because they shouldn’t have started in the first place!

Let’s look at the difference between a person who blogs because he has something he needs to put down on paper vs. someone who’s blogging for say, profit, and just picks some random high CPM (the amount advertisers pay for 1000 impressions of an ad on the site) niche to write about.

Someone Who Needs To
- Naturally occurring ideas and topics; never runs “dry”
- Has an in depth (and often unique) understanding of the topic
- Takes only minutes to write an article because the idea is already there! For example, this article only took me about 40 minutes
- Will keep writing in the face of obstacles. For example, I chose to keep writing even when my pages got deindexed from Google.
- It’s fun and satisfying!

Someone Who Picks A High CPM Niche
- Needs to keep “researching for ideas”
- Just re-summarizing ideas that are out there for the most part
- Takes a really long time to write articles because he needs to research the topic first, understand it in his head, and then write it
- He is results driven because the writing itself gives him no satisfaction. Will easily give up when the money doesn’t come in a few months. Is it a wonder then why so many blogs go dormant?

This is yet another example of why knowing what you want in life is so important! If you cultivate long term interests, then those subjects will always be something that you can immerse yourself in. Most things in life take sustained interest to accomplish – your efforts compound and grow exponentially. By not having long term interest, you actually waste your initial efforts in setting up the foundation.

For example, one business venture that I’ve dabbed in is selling virtual currency in online rpgs. For a while, I established a decent reputation on ebay, sparter, etc. and gained some loyal customers. However, having discovered that this endeavor doesn’t create a lot of value (and as a result, have a lot of opposition), I’d decided to exit this business. Yet, even now, people still contact me asking if I have anything for sale! Had this been a good value generating business, these leads would have been the result of my previous efforts. Instead, because I stopped, these leads are now more of a source of annoyance!

Applied to blogging, if you feel that you have something to say that is important, then it’s worth starting a blog. If you feel that you would blog anyway even if there was no one reading it, then you should start a blog. If you feel that other people would really benefit by reading your blog and you would be proud to show it to everyone, then you should start a blog. Otherwise, your blog is just doomed to failure!