User Experience Benchmarking with Kantuit UX Compare™

Comparing web sites is hard for a few reasons. First, not everyone agrees on what to compare. Hits? Look and feel? Ease of use? Content? Features?

Tracking hits or clickstreams isn't useless, but sites with less traffic are taking business from sites with more traffic all the time. Just emulating the design of high traffic sites won’t get you very far.

What about look and feel or content? A good visual design doesn’t matter if people have trouble using it. Good content is important, but also pointless if people can’t find it.

Having the right features is also important, but what if people can't figure out how to use the features?

We think the only meaningful way to compare web sites is to compare the experience people have of them. We do this by building experience models.

Building models is critical to understanding complex phenomena in any field. We think understanding online experience is no exception.

Building models allows us to get at things we simply couldn’t understand otherwise. For one, our fellow humans are not able to tell us about their experiences with much clarity. We need models to sharpen the focus.

Consider this: Can you say exactly how much clicking, scrolling, reading, mousing, looking, and thinking was required to buy your last book on Amazon?

Hard to say, yet all of these things affect how you feel about your experience. If we’re going to make sense of what people experience online, we need to measure these kinds of things consistently.

1. We visit sites with our special browser.
2. We measure the experience people have as they complete common tasks.
3. We compare the experience people have on one site with other competing sites and the best sites on the web.
4. We show precisely how sites differ from the best sites, pointing to what to change in a series of detailed reports that everyone can understand.

Why not do a user survey, or track clickstreams, or do usability testing? And what about performance benchmarking?

Why not do a competitive web user experience survey? We do them when we're looking to compare user response to features. But surveys don't really work for comparing the whole user experience because so much of online experience is unconscious. People have trouble telling you about unconscious stuff (unless your name is Dr. Freud).

There are a handful of companies now who can tell you how many users are clicking through a given site at a given rate. It's useful stuff. But there's one problem. This doesn't tell you why they're clicking. Also, clicks are just one dimension of user experience. (When you're using your computer, you're doing more than just clicking. You're scrolling, reading, thinking and so on.)

While usability testing is great, competitive usability testing (where you're looking at more than one site) is very expensive.

Other forms of web site benchmarking that go by the name of user experience benchmarking only look at things like page download time and server uptime. While page download time is certainly important, it's just one data point.

There three basic UX benchmarking deliverables.

The benchmarks and best practices report shows which sites are doing what well and why.

In a site review roadmap we provide a detailed, page-by-page gap analysis of your site, showing precisely what's working and what isn't on each and every relevant page.

Models contain both qualitative and quantitative metrics. When we’re comparing experiences we are comparing these metrics. Metrics are made up of experience events. Experience events can be either quantitative (clicks, keystrokes, mouse movements, eye movements, etc.) or qualitative (usability problem, content problem, look and feel problem, etc). To wrap your head around qualitative events, think of smoke coming out of the user's ears.

To get more depth on how the methodology works, download the whitepaper.

We didn't invent the Kantuit UX Benchmarking methodology. It's based on several decades of academic research that looks at human-computer interaction. What we've done is create software that lets us build models efficiently. (It doesn't hurt that the software also lets us compare and present the data from the models in meaningful ways too.)

If you're fortunate enough to be in one of the industries we currently cover in our syndicated benchmarking reports you can get started right now. Otherwise a custom benchmarking project usually takes at least three weeks.

Our off-the-shelf benchmarking reports range in price from a few hundred to a few thousand dollars. Custom benchmarking projects usually start at around $10k.

Because nobody else does it! Although we haven't invented the methodology (we're standing on the shoulders of giants!) our technology, process and metrics are all proprietary.