
To someone in our line of work, it is the most chilling phrase of all: "Why can't I see I my site in Google when I type shoes/books/welsh nose flutes/etc?" Alas that there is never one straightforward answer. But there are some very good reasons why your site mightn't be at the top. Here is a selection of factors that might be contributing.

Every day sees hundreds if not thousands of new websites being launched. The huge majority of these will be gone within a few months as their owners run out of money, lose heart or get involved in some other project. So to stop these sites - however good they may be - getting straight to the top of the rankings, all of the major search engines have pretty stiff ageing filters. Google in particular, might not even index your site for up to 6 months (although you might fare better on MSN).
In addition to the ageing filter for new sites, older sites have big in-built advantages. Firstly, if a site has been around for some years, it tells Google that the business or person behind it is committed (and solvent!) It is also likely that an old site will naturally have built up lots of links and plenty of content over time. These are potentially massive competitive advantages, and your brand new, four page website simply isn't going to have the muscle to shove these sites aside to get to the top.
Most sites are, in terms of content, pretty thin. The typical company website consists of a few pages about the company and its services or products. The sites that are sitting at the top of the rankings usually (but not always) will have lots of content.
The search engines love content. If you don't have much of it - invest in some straight away!
Aside from sheer volume of content, the search engines are looking for quality of content.
Most websites have:
Avoid those pitfalls and you're starting to get the mix right.
A link is seen as a vote of confidence. If your site has plenty of incoming links then it suggests that people like your site for whatever reason and are happy to recommend it. This makes it easier for the search engines to recommend your site in turn to people doing the searching.
Not all links are created equal though. If you've signed up to a free directory or joined a linkfarm then you are saying to the search engines "the only links I can get are crappy, cheap links". Frankly, you'd be better off having no links than go down this route.
If you can get a link from a big organisation like a news site or a major supplier or reseller, then that will certainly help. Unfortunately, most websites can't or don't through lack of contacts or lack of knowledge.
Because the search engines like links, it is in their interest to reward sites that link out to good sources of information. If you sell shoes, link to the relevant Wikipedia entry. This says: "I want to give my visitors access to the best information." Search engines like that kind of stuff.
If you're trying to hoard links like some lunatic pirate zealously guarding his plunder it just looks bad. Share the love and find some sites that you think your visitors will appreciate.
Most web design companies don't do optimisation as standard. A good agency is likely to give you a site that is built so that it can be indexed efficiently by the search engines, and with a reassuring level of quality. To optimise the site for particular keywords however takes the input of a specialist agency who can offer copywriting and keyword research skills.
If you've paid £50 for someone to 'do SEO' then chances are it isn't being done very well.
Unless you've discovered a unique niche (like "welsh nose flutes") then 6 million websites have already beaten you to the punch. You've got to convince the search engines that your site is better than all those other 6 million competitors. Just doing a bit of optimisation and buying a link from your supplier ain't going to cut the mustard.
If your site isn't at the top of the rankings, it probably doesn't deserve to be - no matter how nice it looks and how much you've spent on it. It might not have been there long enough. It might have 4 pages of boring copy you've nicked from a competitor. There might be thousands of other websites just like yours already.
The good news is: you can overcome all of this. Invest in quality content, a good linkbuilding strategy and be prepared to wait things out. And if you don't have time to wait, you can always buy your way to the top.
Seems obvious enough, but thanks for pointing it out, and laying it out so easily so i can follow it step by step!