Getting Everything Indexed Is NOT Good
Quick scenario for you…
You have a search on your site that say… searches for your stock of hamster wheels (why I chose that example, I don’t know!). You are a good (nay great) pet store and you have hundreds. You notice the search could be a bit more helpful to your more discerning visitor if it broke things down a little more at the second level. You say to your web developer “could you tweak the search results to reorder by things like colour, size, etc.”. He says “sure, no problem” (he comes from America).
You now have a great search that shows all your hamster wheels and now you can click on the category and it will group them for you too. Great! Before you pat yourself on the back too hard, bear in mind a few things.
- Have you now created a lot more pages that the search engines will index?
- Will your new sorting part of the URL (e.g. ?sort=blue) be added to the new selection and create a big issues with the ranking for your www.petpetpetstore.com/product/search?=id245 page which ranks really well for “blue hamster wheel” (because you now have about 10 different URLs for the same page)?
- Are you leaking “link juice” out to these new pages?
- Will the search engine robots now devote enough time to crawl the important pages when they visit?
- Have you just created a load of duplicate content?
- Should you get out more?
Apart from number 6 (you might like hamster wheels!), you have probably done the top five. What to do? Well…
- Get your database guy to dynamically add “noindex” to the header of all the duplicate content product page URLs.
- Get your database guy to dynamically add “noindex, nofollow” to the header of all the duplicate content “advanced search” URLs.
- Try to create a URL structure that lets all of the main pages still get indexed, but the advanced search ones don’t (put a # in there, that usually does the trick).
- Exclude the duplicate content URLs from crawling in your robots.txt file.
Every scenario is different here, so do some analysis first as to what the best course of action is. I have listed above a typical approach, but each case needs to be look at separately.
Next time I will pick an example less abstract than “hamster wheels”, promise…
