This is an add-on to my post on SEO Link Juice Leakage and deals with making your 404 error page a better experience for your user.

As mentioned in my previous article you should always ensure that you protect your pages from leaking juice (I really must try to think of a better term…) by…

1. Not changing the URLs at all
2. If you have to change URLs then make sure you 301 redirect all of them to the new pages (or if they don’t have a defined new page then the closest match or home page)

However, there will always be some pages that slip through the net and there will always be some people who key in the wrong page into the URL bar. This is where a nice and friendly 404 page will be a great help to your visitors.

Before we get started here, please remember that this will not help your SEO efforts, but it won’t hinder them either. You need to make the effort to deal individually with all the pages that you have changed URLs on and do a 301 redirect. This is the only way that, some or all of, the power (juice!) will not be lost and instead passed on to the new page.

The dangerous part of customising your 404 page is that it will create confusion for search engines. This is because many people who create a custom 404 error page allow it to return a ’200 OK’ header. This means that the page is actually alive and can be found and indexed by search engines.

So think of this scenario; you have a site that you have changed from a static site to a CMS based one. Your programmer either didn’t know or could be bothered to rewrite the URLs (in any case URL rewriting on this scale is a very delicate thing to do to get right), so you have a site that has completely changed its URL structure.

You think I know what I will do I’ll (or get my web guy to) create a 404 page that says something like:

• “sorry, we have changed our site and the page you want does not exist anymore, please click here to visit our home page”
• “we think you may want one of these pages” – and then list a few relevant pages that might be the best match

Even worse you may just set up a (302) redirect to the home page (from your 404 page!). There is an argument that directing the bulk of your pages with individual 301 redirects and then letting a 404 page handle all remaining redirects with a 301 redirect to the home page is an OK SEO way to do this. Well, as long as you really put the effort into redirecting your pages with this (the 301 redirect on your 404 page) element being the final bit taking up the slack, then it can be OK. However, from a user experience perspective it is better, in my view, to consentrate on doing a good job on your 301 redirects and then creating a (SEO friendly) 404 error page that will act as a guide for now and in the future.

So, what is the problem with letting your 404 page give a ’200 OK’ header? Well it means that any page that does not really exist (whether it existed before or is just a ‘keying error’ now has a life of its own and can be indexed by search engines. Basically, the search engine spider will visit the page and index what it finds, which means that you will have lots of pages that seem to be alive and well, but really don’t exist anymore. This will mean that your power (juice) will never be passed over to your shiny new pages and you will have a load of meaningless pages that will continue to take up the majority of the search bots time. Also, your pages will cease to be relevant for the things there were before (because they will all say something like “sorry, try this pages”, etc.) and will drop like a stone down the rankings. Not a good situation.

So, after all that… here is how I have found the best way to create your customised 404 page. Please bear in mind that this is an Apache sever solution (I will look at a IIS solution at a later date, but please feel free to suggest one if you have it).

Firstly, add a line to your .htaccess file like this:

ErrorDocument 404 /404error.html

Substitute your own 404 page’s URL for the ‘/404error.html’ bit. However, please remember not to use the full CNAME e.g. http://www.yoursite.co.uk/404error.html here or this will send the request to the web and create a ’200 OK’ header response instead of the “404 NOT FOUND” response we want.

You can then create your ’404error.html’ (or whatever you want to call it) to be anything you want. My preferred choice at the moment is to use the new 404 page creator feature on Google’s Webmaster Tools, that will create some Javascript code for you to drop into your normal page template that uses the details Google has of your site to help the user be redirected in a nice way.

So you get a customised 404 error page that returns a ’404 NOT FOUND’ code, helps your visitor get to where they want to go on your site AND does not confuse the life out of search engines by giving them an incorrect header code.

Before you ask I haven’t got around to doing it on this site yet… However, try putting in a few random characters at the end of the URL on this site here, Large Format Printers Samples. You will see what you can do with this approach. Also, if you check the header code for your mistyped URL here you will see that your 404 URL actually returns a ’404 NOT FOUND’ header code.

Do we really know what we are looking for? And if so can we be bothered to tell a search engine exactly what we want?

Semantic search has become something that we now know about and also something that search companies are genuinely trying to do. With Hakia, Cuil and alike (by the way can anyone imagin saying I am going to “Hakia them” or “Cuil them” in the same way we say “Google them”? Thought not) we now have a new wave of semantic based search engines that are trying to go a step further than what we now know as our way of searching the web.

Back in the day when Ask were Ask Jeeves and they started doing TV commercials the concept was “ask me a question, and I will answer it”. So we all turned in Bertie Wooster and started to (try) to treat our web queries in the same way we would treat a servant. Then we realised that search engines weren’t a form of artificial intelligence, but instead a database of the web. So we learnt to direct our searches in a more database driven sort of way; “hats london”, “boil on leg”, sort of thing.

Everything at this stage started to work fine. Google then came along and cleaned everything up, applied a relevance to the things we were searching for and pushed back much of the SPAM. All was (and is) well.

I am interested in semantic search and think that pushing this forward is a good thing. However, have we already learnt how we search the web? Are we happy with it? I think broadly speaking the answer is yes, which leads us to think about where the semantic engines go with their service and concept. They really don’t have enough money to push what they do into our consciousness. How many of us have used Cuil apart from when it was launched? Also, if any of these guys get really good at it won’t Google (or Microsoft) just buy them? Maybe this or a small slice of a big pie is what the game plan is anyway?

Let’s see how it pans out…

As a footnote this is not to be confused with the ‘semantic web’. There are (and will be further) cross overs here, but they still sit quiet separately at the moment.