Update
In 2012 we will be introducing two new methodologies for user experience benchmarking: UX Live and UX Expert. These will supplement Kantuit UX Compare, which is discussed here. Get in touch for more details.
Usability is important. But even the most advanced user experience research focuses (usually) on one site. Doing otherwise is cost prohibitive. Few companies opt to do competitive lab-based testing.
Kantuit UX Compare™ is a process and a technology that makes it possible to compare how people experience web sites to identify web design best practices. It provides data that resembles a usability test for a much larger competitive set than would be possible otherwise.
In the world of Kantuit, good web sites are more informative and persuasive while being easy to use. You can describe informativeness, persuasiveness and ease of use with numbers. And once you have the numbers you can compare them to identify which site is best.
Two things make it possible: interactive experience models and advanced software.
Why Comparing Web Sites is Hard
Comparing web sites is hard for a few reasons. First, not everyone agrees on what to compare. Hits? Look and feel? Ease of use? Content? Features?
Tracking hits or click-streams isn't useless, but sites with less traffic are taking business from sites with more traffic all the time. Just emulating the design of high traffic sites won’t get you very far.
A good visual design doesn’t matter if people have trouble using it. Good content is important, but also pointless if people can’t find it.
Having the right features is also important, but not if people can't figure out how to use the features.
We think the only meaningful way to compare web sites is to compare the experience people have of them. We do this by building experience models.
Why Experience Models
Building models is critical to understanding complex phenomena in many fields. We think understanding online experience is no exception.
Building models allows us to get at things we simply couldn’t understand otherwise. For one, our fellow humans are not able to tell us about their experiences with much clarity. Models help us sharpen the focus.
Consider this: Can anyone say exactly how much clicking, scrolling, reading, mousing, looking, and thinking was required to buy their last book on Amazon?
Hard to say, yet all of these things affect how a person feels about their experience. If we are going to make sense of what people experience online, we need to measure these kinds of things consistently.
The Model Building Process
Each benchmarking project has the following steps.
- We visit sites with our special browser.
- We measure the experience people have as they complete common tasks.
- We compare the experience people have on one site with other competing sites and the best sites on the web.
- We show precisely how sites differ from the best sites, pointing to what to change in a series of detailed reports that everyone can understand.
Alternatives to Kantuit UX Compare
User surveys, tracking click-streams, or usability testing are some alternatives.
There are a handful of companies now providing data on how many users click through a given site at a given rate. It's useful stuff. But there's one problem. They can't tell why they're clicking. Also, clicks are just one dimension of user experience. When you're using your computer, you're doing more than just clicking. You're scrolling, reading, thinking and so on.
While usability testing is great, competitive usability testing (where you're looking at more than one site) is very expensive.
Other forms of web site benchmarking that go by the name of user experience benchmarking only look at things like page download time and server uptime. While page download time is certainly important, it's just one data point.
UX Benchmarking Deliverables
There two basic UX benchmarking deliverables.
Our benchmarks and best practices reports shows which sites are doing what well and why.
Our site review roadmap reports provide a detailed, page-by-page gap analysis of a site, showing precisely what works and what doesn't on each and every relevant page.
Models, Events and Metrics
Models contain both qualitative and quantitative metrics. When we’re comparing experiences we are comparing these metrics. Metrics are made up of experience events. Experience events can be either quantitative (clicks, keystrokes, mouse movements, eye movements, etc.) or qualitative (a usability problem, content problem, look and feel problem, etc.).
To get more depth on how the methodology works, download the whitepaper.
Standing on the Shoulders of Giants
We didn't invent the Kantuit UX Benchmarking methodology. It's based on several decades of academic research that looks at human-computer interaction. What we have done is create software that lets us build models efficiently.
A Typical Schedule
If you're fortunate enough to be in one of the industries we currently cover in our syndicated benchmarking reports you can get started right now. Otherwise a custom benchmarking project usually takes a few weeks.
Costs
Our off-the-shelf benchmarking reports range in price from a few hundred to a few thousand dollars. Custom benchmarking projects usually start at around $10k.
Why Change Sciences
Because nobody else does it! Although we haven't invented the methodology, our technology, process and metrics are all proprietary.