Jun 15th, 2009
Know your research!
Are you studying usability problems or looking for user preference?
When it comes to user research it’s important for everyone involved in the design process to have an understanding about the insights we can gain with a particular research approach. As interaction designers we look to user experience (UX) research to discover usability problems or barriers to the experiences we create. As stakeholders get involved in UX research, questions are sometimes introduced that are intended to uncover user preference instead of usability problems. This can profoundly affect research results.
Consider this scenario: You’re observing a UX research session for a media player interface. Your goals include determining if a user can create a playlist and manage a list of videos. The session is also set up to question which media player design the user likes most and to ask the user to rate the visual appeal of the design. Any cause for concern?
Yes!
The scenario represents an attempt to combine two very different types of user research: user experience research and market research.
User experience research studies user behavior. The goal is to understand users’ cognitive expectations and mental models. UX research is informed by what people do and tends to change little over time.
Market research studies user preference. The goal is to understand user’s opinions and likes/dislikes. Market research is informed by what people say and tends to change significantly over time.
Most interaction designers understand that we don’t usually engage in market research. We’re most concerned with uncovering demonstrable user experience problems and less concerned with user opinions on a particular design. Besides having differing goals, UX research and market research have drastically different requirements.
User experience research is highly effective with relatively few users. Since humans tend to categorize and organize in similar ways, behavior doesn’t differ much across demographics. Studies show sharply diminishing returns after only 5 users.
Market research requires a large sample of users. Since user opinions vary widely across different regions, incomes and lifestyles, large number of users are necessary for any meaningful data. It’s not fair to assume opinions of a small number of users accurately represent a larger group.
Problems can arise when we mix market research with UX research. Results focused around user experience research goals can be positioned alongside relatively meaningless market research results. These mixed results can mislead decision makers and designers and distract from the actual experience problems we’re trying to address. Decisions based on these results are not grounded in solid research.
If you’re a designer hopefully you’ve had the opportunity to validate your designs through some form of user research. When approaching user experience research it’s critical that we remind ourselves of the difference between UX research and market research and the pitfalls of mixing the two. It’s also our responsibility to articulate these differences and educate others involved in the design process — project managers, marketing professionals, product managers — so we can stay focused on the goals that we’re best suited to meet.
(Thanks to Frances James at Bolt | Peters for helping me articulate my points on user experience research!)
2 Responses to “Know your research!”
Leave a Reply
You must be logged in to post a comment.
That’s well put! I’ve seen this too often. Surprisingly, also experienced usability professionals fall into this. Reports of usability tests have often quotes - they look convincing, don’t they - about user preferences, “I really like this. — User A” or similar. That should NOT be in a usability test report among the slightly-more-objective usability test data.
What is worrying also is that recent graduates from the local university couldn’t identify this problem. So we’ll be seeing the same.
[...] Know your research!When it comes to user research it’s important for everyone involved in the design process to have an understanding about the insights we can gain with a particular research approach. As interaction designers we look to user experience (UX) research to discover usability problems or barriers to the experiences we create. As stakeholders get involved [...]