Module # 7 User Retrieval Evaluation

In this module, we were asked to locate and review three articles that measure and discuss user retrieval evaluation. The article topics could discuss the subject of the user in search engines, library catalogs or even social media. We were also asked to discuss if the author addresses the subject of relevance, and to what extent.

In looking for articles for this module, I discovered three articles on usability tests of search, retrieval and display in prominent eCommerce websites from the Baymard Institute.  Together they address user search objectives, eCommerce query characteristics, search engine performance for top eCommerce sites, and UI strengths/weaknesses for these sites as found through their user observations.

Deconstructing eCommerce Search:  The Twelve Query Types:

In this article, the Baymard Institute assessed 19 major eCommerce sites including,, and others for support for query types that they consider to be essential for transactional inquiries. To do this, a usability test was conducted with subjects who articulated their search goals, search process, and subjective assessments of the results they received.   Through this test, the Institute identified twelve query types and rated sites’ performance in delivering results that met (or didn’t meet) user expectations.  These query types were divided into four groups and the Institute noted that users’ searches frequently included combinations across groups.  While I will not repeat all of their findings on all of the query types ,I will summarize some of their more interesting (to me) key findings, which came from the first functional group they called “Query Spectrum” searches that defined the domain of products/information of interest:

  • Support for phonetic misspellings and alternate names/titles is deficient: In eCommerce, users are often searching for an exact manufacturer or product number.   In fact, they often copy/paste from other sites into the one they are searching.  Not only were localized variations ignored (AT&T Wireless vs Cingular Wireless) but even exact match searching was an issue in 18% of the test searches.    These search engine deficiencies greatly impacted results relevance.  The Institute suggested partnering with industry databases to improve Exact Keyword search results.
  • Product category searches require a robust database of synonym surrogates,e.g. “hair dryer” vs “blow dryer,” to retrieve results that are perceived as relevant. Additionally, from a UI perspective, once the broad category results are retrieved (recall), filters are necessary for the user to effectively explore.
  • Symptom based searches: These are often a last resort for the user and are a way for the user to find a product-based solution to a problem.  The examples given were “stained rug” or “dry cough.”  These are difficult to support, and the Institute suggests that Help references and links be offered to aid the user in identifying what type of product would be of assistance.  Assumedly these help references would redirect the user to refine his/her search to an Exact Keyword or Product Category search.
  • Non-Product searches on eCommerce sites are generally responsive: The Institute placed searches for company information, return policies, shipping information and other ancillary queries related to a Product search into this category and rated the performance of the sites that were assessed, well.  They found that when searching on these types of terms, 86% of the sites returned the results users expected either as the top page, or one of the top pages.

eCommerce Sites Should Include Contextual Search Snippets (96% Get It Wrong)

In this next article, which was derived from the same study, the Institute discussed  the value that they called “search snippets” and what we have called “key words in context” to internet search engine results, and how eCommerce sites have failed to follow their lead.  The Institute emphasized that these “snippets” perform the valuable task of connecting the user’s query to the retrieved results, explaining why they were delivered as responsive/relevant.

Specifically, the Institute stated that users want to know the following information about any search results in order to feel confident about their queries and the search engine that delivers the response  to it:

  • Why was this search result included?
  • How does it relate to my search query? How is it relevant to me?
  • How does it differ from the next result?

Surprisingly, it was not eCommerce giant Amazon that performed the best, but rather Walmart.   In a demonstration of Amazon’s weaknesses, the article recounted a user’s search for “Steven Spielberg” on, which delivered the movies Lincoln, Schindler’s List, and Empire of the Sun, among others.  The user did not know why they were retrieved and wondered aloud, “Did he direct these movies?”

In contrast, a screen shot was shown for results from WalMart’s site for the search “high quality tea kettle.”  It was noted that the context sensitive snippets in the results affirmed the user’s understanding of why the results retrieved were selected, and created confidence that the results delivered were based on the user’s search, not Walmart’s preconceptions of what the user would think is a high quality tea kettle:


This article closed with the recommendation that all eCommerce sites should explore  offering user search terms in their search results to minimize site abandonment.

 eCommerce Search Field Design and Its Implications

 This final article is more focused on the aspects of user experience akin to those discussed in the Chu chapter that was assigned this week on Modes of User-System Interaction.  Specifically, this article discussed best practices for designing menu selection, graphical interaction, forms, hyperlinks, and other display dimensions that aid users in creating queries that will deliver more relevant results.

The institute recommends thinking about the approaches users take to locating products on a specific eCommerce site, and adjusting two important design features accordingly.  Specifically, it is recommended that search fields be more prominently featured when users are more likely to know the specifics regarding what they want – a particular model, brand, etc.  Further, the Institute recommends leveraging placement, size, color constrast, and other design tools to “nudge” the user towards the search field if keyword searching is expected.

In contrast, if users are more likely to browse categories for what they want, the Institute advocates that a hierarchy navigation (like a left nav consisting of words that denote categories and subcategories) be used.  In analyzing eCommerce sites, the Institute suggested that this technique works best on apparel sites and home furnishing sites where the visual appeal of the products among those similarly categorized is beneficial.  The Urban Outfitters site is highlighted, as the hierarchy navigation is so prominent that it is difficult to find the “search” function.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s