Good post from Matt Cutts regarding the updating of many elements of the Google definitions and documentation on subjects like:

• Doorway pages
• SPAM reporting
• Robots.txt
• And some more!

It is worth taking a look at the full post at http://www.mattcutts.com/blog/improved-seo-documentation-galore/ and also checking out the new reference documents.

P.S. does anyone really do doorway pages anymore? I guess so, maybe, from a long tail keyword perspective? Seems like a lot of work to create somthing that will (probably) just go into (what was) the Supplemental Results…

IP delivery is a technique where a website/server looks at who a visitor is or where they are located from an IP address perspective. Once this is known (not always completely accurate, but god enough), the site/server can serve a page/site that will be catered for that visitor.

Sounds simple? Well, give or take a bit of server side programming it is. The main part of the process is taking the time and effort to create the individual content for the groups of users you are targeting. This work mainly fits into two categories, namely, specific regional information for that visitor (contact details, in country taxes, etc.) or translating words into a specific language. Google, for example, does this all the time when you type in one of its URLs (try typing in www.google.com from an UK IP address for instance and see where you end up, for an extra part of the test try setting you language in Google to French… see what I mean?).

So, what is the connection between this and SEO? Well, if you treat Google as a specific user, then you can also serve specific information to them by recognising the IP addresses of their robots. This (in a Google ‘rules” sort of way) is called cloaking. The ‘black hat’ version of this technique is where you deliver different content to fool search engines into indexing you for something ‘they’ see, but when a visitor gets to the actual page you get something completely different. The level of ‘completely different’ will define what side you stay of the ‘rules’. Think carefully before you do any of this kind of stuff for this purpose, even if you are doing it with the best of intentions.

The real area where you can learn here is if you have a multiple country and multiple language website. From a usability perspective it is really desirable to serve relevant content based upon who your visitor is. Many (many, many, many) large companies have attempted this (and still do) and get this completely wrong from a search engine perspective. Bad IP delivery will only confuse search engines and their robots. Conversely good regional IP delivery will enhance your search engine presence in the regions you target and operate in. Also, if you still bring everyone in to the same domain you will not be diluting your power with many different regional sites, but still serving relevant content to each visitor. However, there are pluses and minuses for each approach, you need to find one that is right for you and you objectives.

If you are someone responsible for a multiple country/language website or still don’t know which side of the ‘rules’ you come down on from an IP delivery perspective. Here is a good video from Google Webmaster Central. It is worth a watch.

In essence, look at what Google want and are doing with IP delivery and you won’t go far wrong. There are other benefits that you can incorporate from a SEO perspective, but more of those in a future post.

OK, on my “post per day” quest, sometimes we need to cover in depth topics and sometimes the more basic. Today is one of the latter, but get the robots.txt file wrong and you will omit your pages from all (decent) search engines.

Well, robots.txt is a file that (should, in most cases) sit in the root directory of your web server and it is a file that all the major search engine robots access before they start doing their collection work on your site. The best way of thinking about this file is that it is like leaving a note for someone with directions, the main difference is that the note (mainly) contains instructions for the things that you do NOT want that person to do.

A basic robots.txt file (which you can create in any text editor or there are also some automated programmes like this. Also, Google has a basic editor in its Webmaster Tools programme) looks like this:

User-agent: *
Disallow: /bank/details.html
Disallow: /personal/stuff.html
Disallow: /testscripts/

The user agent part is where you can specify which robots you want to give instructions to; the ‘*’ indicated “all spiders”. Alternatively, you can input ‘google’ here or whatever if you only want that robot to have that particular instruction. The disallow part is (as you would expect) the parts of your webserver that you would rather the robots not visit. This could be for privacy reasons (suck eggs time – although, you need to pretty much not upload things in unprotected areas of your web server that you don’t want people to access) or because you want the spider (who is short on time and patience) to not bother with the unimportant parts of your site and get quickly into the important stuff.

The main thing to remember about disallowing is NOT to do this:

Disallow: /

This will tell the robot not to index anything on your site (unless this is what you want to happen).

Will the pages/sections I disallow stop them getting indexed by Google, etc? Err, no, actually. If someone on the web links to pages that you would rather not be indexed then the spiders will still follow those links and index the pages. If there are pages you truly don’t want indexed then you need to use “noindex” in your meta tag for that page, something like this:

meta name=”ROBOTS” content=”NOINDEX,FOLLOW”

This means that you don’t really want the search engine to index the page, but you are quite happy for it to follow links on the page (more of this in another post).

A good element is that you can tell the robot where your XML sitemap (again, more in another post) is, by using this:

User-agent: *
Sitemap: http://www.searchkingdom.co.uk/sitemap.xml
Disallow:

Another handy thing is that because you know that all good robots access the file first before they visit your site, if you look at the log files and see how many times the robots.txt file has been accessed you can get an idea, over time, of how frequently those little robots are accessing your stuff.

If you are still worried or unsure about all of this, Google have a nice little checker in their Webmaster Tools programme. You can see if the file is in the right place and also see if you have made some (big and small) errors.

Robot.txt won’t make or break your SEM efforts, but get it right and it will help. Get it wrong and prepare to have to sit around until the search engines reindex your site… not nice.

Thought I had better make the first proper post something deep and meaningful… not saying this is, but here goes…

Black hat, white hat… what does it all mean? Also, who makes the rules? Well, that one is easy, Google does. Even though I am quite sure that the good people of Google never meant to be portrayed as the police, that it what they have become. Why? Well, power (usually) brings responsibility and Google are the most powerful force in the most well used information retrieval system ever created. So, they make the rules and they get to say what is black (hat) and white (hat). This is altogether fine in the main (someone has to make the rules I guess).

So what are the rules? Well, you all know them really. You just decide if you want to play by them. Just in case you are unsure of the rules, here is my cut:

1. Produce fantastic websites that people will like, use, love and share.

2. Make the website friendly to search engines remembering that (sorry guys) they are glorified databases (I don’t mean that to sound harsh… I like search engines).

3. Market your site like no tomorrow, but don’t use tactics that your mother would not be proud of you using.

4. Make the biggest effort possible to continually improve your site and make it better, more usable, more commercial, etc. and use analytics (and user feedback) to do this.

Give or take the things I have omitted (forgotten), that is it really. The hard bit? Well, in the bad (good?) old days when you could SPAM your way to the top of the search engines, all it took was a little knowledge and a little effort. The main part about the rules of today is that Google makes you work hard, be cleaver, etc. and the good bit about that is that most people won’t bother. The sensible people who don’t bother just get someone great to do it for them. However, great people cost a lot of money and they are in short supply.

Can you still bend the rules? Yes, sure you can. Will it last? No. Search engine marketing is inherently a long term strategy. Think short term and be prepared to get short term results (if that it what you are aiming for then this may still be for you).

In essence, Google (sorry Yahoo, MSN, Ask, etc. for not mentioning you too, you will be mentioned in the future, you just don’t set the rules, sorry) protects two main things, namely, the quality of its product so it can continue to make money and our ability to use search engines to enrich our lives, save us time. etc. (the two are completely linked).

Where does it leave all of us who want to have fantastic exposure on search engines? Quite simply (and boringly), THINK!, work hard, work smart and set your objective realistically. Most importantly, remember that Google (and Yahoo, and MSN, and Ask, et al) are (just) intelligent databases; people are people are people. The rules are really made by all of us, Google just set them.

Firstly… bold statement…

We aim to be the foremost website in the UK for Search Engine Marketing. There… said it…

How are we going to do this? Well, by providing everything you need in order to do your own SEM (the first of, no doubt, many abbreviations) or get some one else to do it, work in SEM, find out about all the things you need for SEM, SEM news and much, much more. Oh, and why just the UK you ask? Well, I think that a site of this type has been a long time coming. Of course, anyone in the world can use the information here, but the slant is towards what is happening in the good old UK.

Who am I? Well I am Rob Andrews and I have been working in the SEM industry for many years and (think!) I have a wealth of knowledge to share. Over time I aim to get a number of quality contributors to add to the site and in the meantime all of your comments are very welcome (well, not the SPAM ones!).

This site has been put together with the fantastic assistance of WordPress and the base template is from Chris Pearson (thank you). I have hacked it around so much that you might not recognise it, but it was from your original base.

Anyway, I suppose I now have live up to my claim and make this the best darn SEM site in the UK. Wish me luck.