If you ask me, there’s one big reason why “primarily” sneaks itself into that statement: Faceted Navigation. Oh, how nice it would be to be able to build faceted navigation into a site without concern for the search engines. Unfortunately, this isn’t the case, and in this post I’ll attempt to break down what it takes to do it right for both users and search engines.
But hold on here. Let’s first define faceted navigation. Faceted navigation is most often found on eCommerce sites, and is a means to allow the user to apply filters (ok, or “facets”) as they browse through thousands of products. You’ll find faceted navigation on most any eCommerce site with a ton of products, like Best Buy.
In the example above, I’ve filtered my way down to find the perfect product:
I want a laptopI hear good things about Apple. I want an Apple laptop.I need at least 300 GB of hard drive space.I’m not interested in the cheap stuff! Just show me laptops over $900.
As a user, that was enjoyable. I can add and subtract filters all day until I’ve found exactly what I want.
For the search engine, there can be a number of unfavorable consequences.
The search engine might begin to crawl through these facets, wrecklessly adding and subtracting filters and indexing whatever the hell it wants. Eventually crawl fatigue catches up to it, and it leaves your site.
This is the naive way to build faceted navigation because it allows all pages to be crawled and indexed, and the hope is that the engines figure out what’s important on their own.
SPOILER ALERT: They won’t. More than likely, plenty of actually important pages will not get the crawl love they deserve.
There are plenty of eCommerce packages out there that utilize the robots = noindex and nofollow as a solution to this problem, but it’s really not helping too much. Each has their own shortcomings.
Noindex – Well, you’ve managed to keep the really ugly, over-faceted pages out of the index, but the noindex does nothing to stop the search engines from wasting crawl bandwidth.
Nofollow – Remember, robots=nofollow doesn’t mean the engines aren’t going to crawl through the link, it just means no link equity will flow through the link. Again, this isn’t a good method to preserving crawl bandwidth.
Now, the extreme opposite way to handle faceted navigation…
This is the method that Best Buy appears to be employing. The top-level categories are crawled and indexed, but once any sort of facet is applied the pages are excluded via robots.txt. It’s the reason why their main “Laptops” page is in the index, but “Apple Laptops” is not.
And therein lie the fault with the hatchet approach. You might very well be missing out on some really strong organic landing pages. Don’t you think Best Buy would like to rank for “Apple Laptops”?* They don’t.
* I’m merely using Best Buy as an example of the hatchet approach. There could very well be some ulterior reasoning for not having an indexed “Apple Laptops” page.
The principles of great faceted navigation should be becoming clear…
Be simple and easy to use for the userNot allow the search engines to go buck wild crawling in and out of facets (whether those facets are being indexed or not)Allow for indexation of particular facet combinations with high-volume search traffic
Easier said then done, but let’s talk about potential ways to make it happen.
Over the last 2 years or so, the search engines have made strides in the indexation of AJAX content. By following some standards set by Google, webmasters are now able to have their AJAX content indexed (to some degree). With the user experience created by AJAX so favorable, websites are beginning to take advantage.
All that aside, in my example I want use AJAX in a way that keeps the engines from crawling through and indexing pages that are only navigable through AJAX. There are a few eCommerce sites that are doing this right now with their navigation.
AJAX can make the user experience of applying and unapplying filters to your navigation fast and enjoyable. In this example, we want an uncrawlable AJAX faceted navigation. For this segment of our user base, our needs are met. Take the ‘narrow your search’ feature on FramesDirect for a live example.
Here we are on the Oakley glasses static page. We can add and subtract facets. The page won’t reload, the URL won’t change, and the items will filter right there on-page.
Now, we’ve got our ‘top categories’ (AKA organic landing pages) indexed, and the engines are only spending time crawling pages that matter.
Unfortunately, I’ve never actually seen this solution implemented anywhere, so I can’t link to a live example. In theory, this is a great way to tackle this problem.
Mockups courtesy of Mockingbird. I wish there embed feature was better, but it’s a great mockup tool!
This solution is the happy medium between the Naive and the Hatchet approach. With the selective robots.txt solution, we’ll define a URL parameter that will serve as road block to the search engines. Let’s call that “crawl=no”. Our robots.txt file might look like this:
Now, we’ll need to create some server-side rules that define when that “crawl=no” is appended to a URL. One simple idea might be to append “crawl=no” from the third facet application forward. To describe:
User is on “laptops” page (indexed)User applies “Apple” filter. Page changes to /laptops?manufacturer=apple. (indexed)User applies “Macbook” filter. Page changes to /laptops?manufacturer=apple&model=macbook. (indexed)Any filter applied from here on is blocked. So, user applies “below $1,000” filter, and page changes to /laptops?manufacturer=apple&model=macbook&price=1000&crawl=no
Take a look at the faceted navigation on TrendToGo. They’ve implemented almost exactly this method of dealing with facets.
Honestly, I like the AJAX solution better because of the AJAX user experience, but both should get the job done.
I expect that a lot of the folks who are able to innovate with their navigation are those smaller, more agile businesses. These are the sites that might be under most peoples’ radars, including my own. If you know of any other excellently built faceted navigation implementations on the web, I’d love to hear about it in the comments or let me know on Twitter.
Hello, we are Ice Clear Media, a Creative Social Media and SEO Agency based in London. We help clients to dramatically accelerate online visibility, website traffic and revenue growth through Digital Marketing Strategies. We specialise in SEO, Social Media, Link Building, Mobile Marketing, Traffic Generation, Creative Design and SEO Consultancy and Social Media Consultancy. We are a passionate and extremely creative team; dedicated to inspiration and innovation in Digital Marketing and online.
brinkka2011 says: Im speechless. This is a superb blog and very enticing too. Great work! Thats not in point of fact so much coming from an amateur publisher like me, however its all I may say after diving into your posts. Nice grammar and vocabulary. No longer like other blogs. You really recognize what you?re speaking approximately too. So much that you just made me want to explore more. Your weblog has grow to be a stepping stone for me, my friend.
buy seroquel online
September 5, 2011
Thanks for tris interesting information! I found it very useful =)
August 28, 2011
Really interesting blog, keep up the good work!
August 25, 2011
Really great article with very interesting information. You might want to follow up to this topic!?! 2012
fat burning furnace ebook
August 20, 2011
Hello. Neat article. There is a problem with your web site in chrome, and you might want to check this… The browser is the market chief and a good section of folks will omit your great writing because of this problem.
August 19, 2011
I’m really impressed along with your writing talents neatly as with the layout for your blog. Is this a paid topic or did you customize it yourself? Anyway stay up the excellent quality writing, it is uncommon to peer a nice blog like this one nowadays.