User Experience Surveys with Kantuit Online™

Most online survey tools make it easy to create survey questions and get responses online. This is what makes them "online surveys". But just because you can create them online and get responses online, does't mean they are good for testing things that are online (i.e. web sites).

On the contrary, most online survey tools are designed to make it easy to collect data on just about everything, from opinions about the president, to opinions about a new potato chip bag design.

The way we do online research (that is research about web sites) is very different. Most online surveys lack at least three key ingredients we have.

• The ability to show users the proper stimulus (that is the ability to show users a real web page)
• The ability to let users answer questions before, during, and after, they interact with a real web site
• The ability to track the clickstream of the user while he interacts with the web site (or sites) we're testing

All of this requires some fancy technology that has been developed precisely to test web sites. The point of it all is to connect user opinions with behavior in meaningful ways.

What's so bad about asking users to prioritze features or content without showing them things?

It all has to do with memory. Not RAM. Human memory. Someone calls to ask you about real estate web sites. Last week you visited sites looking at houses. The interviewer ticks off a list of features that real estate web sites have and asks you to prioritize them on a scale of 1 to 5.

Such questions assume that you know and remember what each item on the list actually is—something that is unlikely for a signficant number of people.

Although you can still produce cool charts and graphs from surveys that collect data this way, even though you might have a decent margin of error and so on, it's basically garbage in garbage out.

Showing an image of a web page and asking people to respond to it is better than nothing. At least people can look at what you're asking them to respond to. And sometimes we resort to this technique when there isn't a site to interact with, because it's an early-stage prototype, or because it is a screenshot from a web site that we can't let people interact with for other reasons.

But the problem with most online survey tools is that they don't show screenshots of web pages very well. Most have a limit on the size of the image that can be shown, or present other challenges for getting feedback on full page screenshots, all because they are designed for testing shampoo bottle designs (and everything else) and not web sites.

It's a well known research phenomenon. People often answer survey questions in the way they think they should or the way they think the interviewer wants them to. In real life, people often say one thing and actually do another. How do you ensure your users really mean what they say?

It's all about asking the right questions in the right way at the right time. By way of example, here's what we recently did in a survey for the healthcare industry.

In healthcare, everyone is trying to get people to do stuff after using a web site: talk to their doctor, choose a hospital, live a healthier life, and so on.

We wanted to measure the effectiveness of hospital quality information on decision making. Would people be more likely to choose a hospital for lower back surgery after seeing information about the quality of the hospitals in their area?

Using Kantuit Online, we showed people three web sites (a non-profit, a government site, and a health insurance site) that allow you to search for hospitals in your area, providing quality scores on each across a range of criteria.

What we found was that people didn't understand hospital quality search results. Only about half the time were people able to say which hospital had the best outpatient experience for lower back surgery (even though all the sites provided an answer). People reported they would use the results of a hospital quality search in roughly the same ratio (about half the time).

The government and health insurance sites did slightly worse than the non-profit. But when only half your users understand something, you've got a big problem on your hands.

Looking for evidence that people say one thing and do another? In another part of our healthcare research we found that, when asked, people said they felt they were effective finding cause and treatment information for lower back pain about 80% of the time. In reality (since we tracked their clickstreams) people only found the information about half the time.

• Survey design and administration
• Full data analysis, charting, cross-tabs and report
• Actionable findings and recommendations
• Perceived success rates compared with actual success rates
• Clickstream analysis
• Management presentation

• Getting strategic direction
• Determining best practices across competitors
• Refining and prioritize content
• Refining nomenclature and navigation
• Mapping demographics to behavior
• A/B or version testing
• Early-stage concept testing
• Prototype testing
• Card sorting
• Persona and scenario development
• Quarterly assessments

3-6 weeks from data design to final report and presentation.

The costs for Kantuit Online surveys tend to be higher than traditional online surveys, with a typical project coming in at around $25-40k. You knew there was a gotcha! But for many enterprises these types of surveys have become an indespensible tool for site development, monitoring and enhancement.

But the gotcha really isn't one when you think about it, since as we've said, in most cases doing a traditional survey about a web site is at best money down a hole, and at worst, going to give you bad data which could result in bad decisions. Don't let it happen to you!

Because we live and breathe this stuff. Meaning: we've spent the last 10 years designing, and administring online research for some of the biggest and best companies out there.