What to Look for?

This is the hard one. Once you've had five people play with your product and you've asked them what they think about it, what does it all mean?

Well, first of all, this isn't statistics. It's tempting to say, "Two people liked the picture of the happy clown, and three people liked the angry clown, so that means that 60 percent of the audience likes angry clowns." As useful as it would be if you could tell that kind of information from this procedure, you can't.

However, you don't need statistics to tell if something is confusing to people. If all five people can't find the Start button, then a large chunk of your actual population is likely to have the same problem. Is everyone? Probably not. Will most people? It's impossible to say. But you can be sure that it'll be a significant enough chunk of your audience that making the Start button easier to find will improve people's experiences.

So what about features? Well, for one, remember that everyone wants everything. If you're given a choice between a jelly donut and a cruller, your first impulse will be to want both. That's how the rest of the world works, too, so don't take people's choices literally. Look for trends.

In a recent evaluation, we asked people what they wanted in an intranet. The specific results ranged from floor maps to org charts to project lists to titles. Implementing all of those exactly as people envisioned would have made for a lot more work than focusing on the general category of what they were asking for: information about people. Who does what where. Specifically, the people we asked did not care as much about the parking lot policy or the form for ordering a new stapler as they did about what their co-workers were doing and how they could best work with them.

Focusing the intranet on the larger task both helped to eliminate a lot of extra work for the development team and concentrated the work on what people did most often. As a byproduct, it gave the intranet team a list of things on which to base their feature set, though they would still have to use their judgement and insight to determine exactly which features to implement and in what way.