Sitemaps, and why you should use them. - Blogging, Small Business, Web Design & Hosting Tips - A Small Orange

Sitemaps, and why you should use them.

When doing a site redesign, I made a discovery – while our spiffy Javascript fly out menu was awesome in that it could be used across multiple servers, it was cross-compatible, and the code could load once no matter where you traveled making it pretty fast once you cached it, crawlers couldn’t see the darn thing.

Just couldn’t see it. There were pages Google didn’t have any idea we had, and were never going to even get a shot of getting indexed because we didn’t give Google any way of finding them.

Yeah, ok, big woops.

As you can see from the graphic to the right, what Google can see and what Google can’t see is something we take very seriously, as even with our site a mess and most emphatically not optimized for search engines they still sent us an awful lot of traffic and slapped us with a PR5 (which ain’t too shabby for a little web hosting company).

Our answer was sitemaps.

Creating a Site Map

So, what are sitemaps?

Sitemaps are an easy way for webmasters to inform search engines about pages on their sites that are available for crawling. In its simplest form, a Sitemap is an XML file that lists URLs for a site along with additional metadata about each URL (when it was last updated, how often it usually changes, and how important it is, relative to other URLs in the site) so that search engines can more intelligently crawl the site. (

You don’t need to built them by hand, though – a Google search for “sitemap generator” will find both online and downloadable sitemap generators. We decided to use WonderWebWare Sitemap Generator.

When we first used it, we had around 750 pages on the site – which was astounding considering I didn’t remember writing 750 pages worth of stuff. What was happening was that the WordPress Blog has the same post indexed several times – by category, by keyword, and on and on running the risk of the search engines seeing the posts as duplicate content and nuking them altogether.

To address this, we split the site into two sitemaps, one that we used the SiteMap Generator for (the main site), and one sitemap just for the blog in the form of a WordPress Sitemap plugin (XML Sitemap Generator for WordPress 3.2.2). We placed both of those sitemap locations in our robots.txt file to let the search engines know we had them, like so:

Sitemap: http://www.sitename.tld/sitemap.xml
Sitemap: http://www.sitename.tld/blog/sitemap.xml

We whittled the site down to 37 links, and the Blog sitemap lists each post only once, with the added benefit that it will regenerate each time a post is published to add its content to the list so we don’t have to manually mess with it.

Once you get a sitemap created, you should go the extra mile and submit your sitemap to the search engines – Google Webmaster Tools allows you to claim your site with Google, submit the locations of your sitemaps, and will give you an overview of any errors it finds when crawling your site. For Yahoo! there’s Site Explorer, and for Bing there’s Webmaster Center.

Google has design advice located here, and we kind of flunked the first set from their perspective due to the Javascript navigation, which was:

Make a site with a clear hierarchy and text links. Every page should be reachable from at least one static text link.

so for good measure, we used WonderWebWare Sitemap Generator to create an HTML site map as well which, depending on your navigation, you may wish to think about doing.

Comments are closed.