Evaluating Interfaces with Users: Qualitative Methods
Evaluating interfaces with users'
involvement form the basis for usability studies. A good evaluation process means
that designers will catch major problems (and successes!) early on, with lesser
problems being ironed out as the interface is being refined. Fortunately,
there are many low cost evaluation methods for discovering usability problems.
Most of these are based on observing people as they use your system to perform
An assignment on usability studies can
be used to complement this section by providing students with hands-on practice in various
usability study methods.
- Evaluating interfaces with Users
- Why bother?
- Natural vs laboratory approaches
- Qualitative evaluation methods
- Direct observation: think aloud and co-discovery learning
- Query techniques: interviews and questionnaires
- Continuous evaluation: user feedback and field studies
Chapter 6: Usability testing. Nielsen, J. (1993) In Usability
Engineering, p165-205, Academic Press. [1
page / side or
2 pages / side]
Discussion of guidelines for user
observation. Kathleen Gomoll
& Anne Nicol (1990) User Observation: Guidelines for Apple Developers, January.
In-Class Teaching tips
I have found that performing usability studies in class is an excellent use of time. I
do several of them, each emphasizing slightly different methods of performing a usability
- Initial conceptual model formation plus think aloud of a paper prototype.
a stylized picture of a Cannon fax machine on the overhead (included in the qualitative
evaluation overheads) and ask them to explain the meaning of labels and controls. I then
asked them to pretend they are sending a one page fax, and have them use think aloud to
say what controls they are selecting and why.
- Think aloud on a physical artifact. I have a student volunteer do a think-aloud
exercise as they try to display a slide on an overhead projector rigged with a burnt
bulb---it usually takes the student 20 minutes to discover the problem and realize that
the projector has a spare bulb that they can switch to. I also have them try to change the
bulb, and most have difficulties figuring out how to open the projector to reach the bulb.
The class, who are are taking notes, then critically analyze the design of the overhead
projector (relating back to Norman's design principles of everyday things) and suggest
improvements. As most recommendations are simple changes to the plastic overhead case, a
"better" projector could probably be built for the exact same price. The class
often wonders why the manufacturer (3M) had never bothered doing this simple exercise!
- Constructive interaction/co-discovery learning on an existing computer system.
pair of students are given a PC (the display is linked to a large screen so the class can
see what is going on). One student is usually familiar with conventional GUI file systems,
while another is a novice to it. The novice is the driver, while the 'expert' is the
coach. Students are asked to do a few tasks on well known systems; one year
we had people try to figure out the image placement system in Microsoft
Word. While the scenario is somewhat rigged to bring out the
interfaces worst features, it is surprising how much difficulty students have doing even the
most basic things. I also have done this for the IBM RealPhone Prototype.
- A Practical Guide to Usability Testing. Revised Edition. Dumas, J.S. and Redish, J.C. (1999)
- This how to text completely describes everything you need to know about
doing observational evaluations.
- Usability Engineering, Nielsen, J. (1993) Academic Press.
- while it covers much more than usability evaluation, it gives a great
well-rounded picture of this area
- Jakob Nielsen's Alertbox, March 1, 2004:
Risks of Quantitative Studies
- Ghostbusters (commercial video, available from any video store)
Ghostbusters has a very funny
clip early on that shows an example of poor ethics. In it, two subjects are in a
parapsychology experiment, with shock treatments being administered for a subject who
guesses wrong on a card. The experimenter, however, is totally oblivious to how the
subjects are performing, and always administers shocks to the "nerdy guy"
subject (who happens to start getting it right), while encouraging the beautiful woman
subject (who gets it wrong)! Ok, not a huge educational value here, but the students like
it as a break from the normal routine!
Major sources used to prepare lecture material
- How to design usable systems.
Gould, J. (1995) In Baecker Grudin
Buxton and Greenberg, Readings in Human Computer Interaction: Towards the Year
2000 (2nd Edition), Morgan Kaufman. Originally written in 1988, this great article shows how user
involvement and prototyping work together effectively
- Usability Engineering, by Jakob Nielsen, helped structure the idea of evaluating
systems with a user's involvement.
- For qualitative evaluation, the readings in Baecker, Grudin, Buxton
and Greenberg: How to design usable systems and
and guidelines for user observation are particularly valuable.
- Dumas, J. and Redish, J. (1993) A Practical Guide to Usability Testing.
Ablex.This book provides complete coverage of usability testing: