Testing web sites with users in the lab is the gold standard. But there's a problem. Observational research in the lab means small sample sizes. And small sample sizes means it's hard to generalize about some important things, like users' opinions. Opinions need big samples.
Enter Kantuit Online™. Why are opinions about web sites important?
Some usability purists need reminding that knowing users' opinions is critical. You want to know what content users might give priority to for example. You want to know what features users value. You want to know what users might want to do versus what they actually do.
Why Not Just Do an Online Survey?
Most online survey tools make it easy to create survey questions and get responses online. This is what makes them "online surveys". But just because you can create them online and get responses online, doesn't mean they are good for testing things that are online (i.e. web sites).
On the contrary, most online survey tools are designed to make it easy to collect data on just about everything, from opinions about the president, to opinions about a new potato chip bag design.
The way we do online research (that is research about web sites) is very different. Most online survey tools lack at least three key ingredients that Kantuit Online has.
- The ability to show users the proper stimulus (that is the ability to show users a real web page)
- The ability to let users answer questions before, during, and after, they interact with a real web site
- The ability to track the click-stream of the user while he interacts with the web site (or sites) we're testing
All of this requires some fancy technology that has been developed precisely to test web sites. The point of it all is to connect user opinions with behavior in meaningful ways.
Getting the Stimulus Right
We think it is bad to do things like ask users to prioritize features or content without showing them pages or letting them interact with pages.
It all has to do with human memory. Someone calls to ask you about real estate web sites. Last week you visited sites looking at houses. The interviewer ticks off a list of features that real estate web sites have and asks you to prioritize them on a scale of 1 to 5.
Such questions assume that you know and remember what each item on the list actually is—something that is unlikely for a significant number of people.
Although it is possible to produce cool charts and graphs from surveys that collect data this way, and even though you might have a decent margin of error and so on, it is often basically just garbage in/garbage out.
Showing Images of Web Pages
Showing an image of a web page and asking people to respond to it is better than nothing. At least people can look at what you're asking them to respond to. And sometimes we resort to this technique when there is no site to interact with, because it's an early-stage prototype, or because it is a screen-shot from a web site that we can't let people interact with for other reasons.
But the problem with most online survey tools is that they don't show screen-shots of web pages very well. Many have a limit on the size of the image that can be shown, or present other challenges for getting feedback on full page screen-shots, all because they are designed for testing shampoo bottle designs (and everything else) and not web sites.
Connecting Opinion with Behavior
It's a well known research phenomenon. People often answer survey questions in the way they think they should or the way they think the interviewer wants them to. In real life, people often say one thing and actually do another. Ensuring respondents mean what they say is hard.
It is all about asking the right questions in the right way at the right time. With health care web sites, the goal is to get people to do stuff after using a web site: talk to their doctor, choose a hospital, live a healthier life, and so on.
We wanted to measure the influence of hospital quality information on decision making. Would people be more likely to choose a hospital for lower back surgery after seeing information about the quality of the hospitals in their area?
Using Kantuit Online, we showed people three web sites (a non-profit, a government site, and a health insurance site) that allow people to search for hospitals in their area, providing quality scores on each across a range of criteria.
What we found was that people didn't understand hospital quality search results. Only about half the time were people able to say which hospital had the best outpatient experience for lower back surgery (even though all the sites provided an answer). People reported they would use the results of a hospital quality search in roughly the same ratio (about half the time).
The government and health insurance sites did slightly worse than the non-profit. But when only half your users understand something, you've got a big problem on your hands.
In another part of our health care research we found that, when asked, people said they felt they were effective finding cause and treatment information for lower back pain about 80% of the time. In reality, since we tracked their click-streams, people only found the information about half the time.
Survey Research Deliverables
In most of our survey projects have us delivering a combination of the following items, depending on the needs of the project.
- Survey design and administration
- Full data analysis, charting, cross-tabs and report
- Actionable findings and recommendations
- Perceived success rates compared with actual success rates
- Click-stream analysis
- Management presentation
The Uses of Kantuit Online
Online surveys of user experience can be useful in many contexts.
- Getting strategic direction
- Determining best practices across competitors
- Refining and prioritize content
- Refining nomenclature and navigation
- Mapping demographics to behavior
- A/B or version testing
- Early-stage concept testing
- Prototype testing
- Card sorting
- Persona and scenario development
- Quarterly assessments
Typical Project Plan
3-6 weeks from data design to final report and presentation.
Costs
The costs for Kantuit Online surveys tend to be higher than traditional online surveys, with a typical project coming in at $15-40k.