When choosing a research method for your usability project, you might ask what the best method is. You have probably heard or read that remote testing will yield a higher number of findings or that in-lab testing yields higher quality findings.

Well there are different reasons as to why you would use one method over the other. Examples below:

Remote Testing

• Cost and time saving
• Global distribution
• Natural/native environments

Lab Testing

• Highly secure client/server
• Handheld products (requires physical interaction)
• Observation of the user
• Long test sessions

In this post I wanted to discuss how both research methods were used in conjunction, and how they both supported the overall goal of leveraging valuable information about search engine users. Google recently released a usability study that demonstrated how a lab study and an online study can complement one another. The usability lab test provided the hypothesis and the online study quantitatively supported the hypothesis using a more generalizable data set.

We are a huge believers in integrated methods, such as utilizing analytics with remote testing. One method isn’t necessarily better than the other. Designing a research project has to be thought out strategically. Below we will analyze Google’s Study “How Does Search Behavior Change as Search becomes More Difficult?” and showcase how both methods supported the overall results.

Overall Summary & Results

Search engines make it easy to check facts online, but finding specific kinds of information sometimes proves to be difficult. Google studied the behavioral signals that suggest that a user is having trouble in a search task, first by running a lab study to gain a preliminary understanding on how users’ behavior changes when they struggle finding the information they’re looking for.

Experiment #1: Usability lab study
23 participants
Two or three difficult search tasks (closed informational tasks)

Task: “I was recently watching footage of one of Prada’s fashion shows from a few months ago, where two models fell (and several others stumbled) due to the footwear. Find the names of the models who fell.”

Example of how users changed their strategy from keywords to direct questions. User A:

1. Prada fashion show models falling
2. Prada fashion show models name falling
3. Prada fashion model name
4. Prada fashion model name fall
5. Prada fashion model name falling
6. Prada fashion models names that fell
7. Prada models fall
8. what were the names of the models that fell at Prada

The analysis of the laboratory study generated qualitative findings that were used as hypotheses to be tested with the larger-scale online study. The data suggested that as users’ keyword queries failed, they also tended to:

• enter direct questions as queries (question queries may start with who, what, when, how, where, why or end with a question mark)
• enter additional keywords, increasing the total length of their search string
• enter advanced operators (such as “+”)

Experiment #2: Online study
179 participants
22 avg search tasks (closed informational tasks)

To test if these hypotheses hold within the general population for a diverse set of tasks, Google analyzed the number of question queries for the successful and failed search tasks within a larger data set online.

Harder tasks (those in which fewer participants reported success) tended to have longer queries. In the tasks where more than 80% of the participants are successful, the queries tend to be between 2-5 terms long.

Google found five behaviors that were prevalent among users who were struggling in their search task. As we can see, the data suggests that all three behaviors hypothesized as tending to occur were accurate.

Those signals were:

• use of question queries
• use of advanced operators
• spending more time on the search results page
• formulating the longest query in the middle of the session
• spending a larger proportion of the time on the search results page

Based on these experiments, Google was able to conclude that the signals related to successful and less successful search strategies could be used to build a model that would predict the user satisfaction in a search session. These insights could be used to gain a better understanding of how often users leave search engines unhappy - or how often they are frustrated and in need of help, and perhaps an intervention, at some point during the search session.

Final Thoughts

This study exemplifies how a qualitative lab test can be used to uncover possible causes for certain behaviors and how a quantitative method such as remote testing can be used to test the accuracy of such cause and effect relationships. Google might just know the next time your frustrated based on these signals but the real value creating a better user experience.

What are you thoughts? Do you like one method over the other, or has integrating both methods been more of a pain than joy of awesome data. I would love to know!