Just couldn’t see it. There were pages Google didn’t have any idea we had, and were never going to even get a shot of getting indexed because we didn’t give Google any way of finding them.
Yeah, ok, big woops.
As you can see from the graphic to the right, what Google can see and what Google can’t see is something we take very seriously, as even with our site a mess and most emphatically not optimized for search engines they still sent us an awful lot of traffic and slapped us with a PR5 (which ain’t too shabby for a little web hosting company).
Our answer was sitemaps.
So, what are sitemaps?
You don’t need to built them by hand, though – a Google search for “sitemap generator” will find both online and downloadable sitemap generators. We decided to use WonderWebWare Sitemap Generator.
When we first used it, we had around 750 pages on the site – which was astounding considering I didn’t remember writing 750 pages worth of stuff. What was happening was that the WordPress Blog has the same post indexed several times – by category, by keyword, and on and on running the risk of the search engines seeing the posts as duplicate content and nuking them altogether.
To address this, we split the site into two sitemaps, one that we used the SiteMap Generator for (the main site), and one sitemap just for the blog in the form of a WordPress Sitemap plugin (XML Sitemap Generator for WordPress 3.2.2). We placed both of those sitemap locations in our robots.txt file to let the search engines know we had them, like so:
Sitemap: http://www.sitename.tld/sitemap.xml Sitemap: http://www.sitename.tld/blog/sitemap.xml
We whittled the site down to 37 links, and the Blog sitemap lists each post only once, with the added benefit that it will regenerate each time a post is published to add its content to the list so we don’t have to manually mess with it.
Once you get a sitemap created, you should go the extra mile and submit your sitemap to the search engines – Google Webmaster Tools allows you to claim your site with Google, submit the locations of your sitemaps, and will give you an overview of any errors it finds when crawling your site. For Yahoo! there’s Site Explorer, and for Bing there’s Webmaster Center.
Make a site with a clear hierarchy and text links. Every page should be reachable from at least one static text link.
so for good measure, we used WonderWebWare Sitemap Generator to create an HTML site map as well which, depending on your navigation, you may wish to think about doing.