Evaluating Interfaces with Users: Qualitative Methods

Evaluating interfaces with users' involvement form the basis for usability studies. A good evaluation process means that designers will catch major problems (and successes!) early on, with lesser problems being ironed out as the interface is being refined. Fortunately, there are many low cost evaluation methods for discovering usability problems. Most of these are based on observing people as they use your system to perform particular tasks.

An [[X|assignment on usability studies] can be used to complement this section by providing students with hands-on practice in various usability study methods.

Overheads (Powerpoint)

Topics Covered

  • Evaluating interfaces with Users
    • Why bother?
    • Natural vs laboratory approaches
  • Qualitative evaluation methods
    • Introspection
    • Direct observation: think aloud and co-discovery learning
    • Query techniques: interviews and questionnaires
    • Continuous evaluation: user feedback and field studies
  • Ethics

Required Readings

  1. Nielsen, J. (1993)
    Usability Engineering, p273-4, Academic Press. Chapter 6: Usability testing. p165-205
  2. Gomoll, Kathleen & Nicol, Anne (1990)
    User Observation: Guidelines for Apple Developers, Apple Inc., January

Additional Readings

  1. Dumas, J.S. and Redish, J.C.
    A Practical Guide to Usability Testing. Revised Edition. (1999)
    • This how to text completely describes everything you need to know about doing observational evaluations.
  2. Nielsen, J. (1993)
    Usability Engineering, p273-4, Academic Press.
    • while it covers much more than usability evaluation, it gives a great well-rounded picture of this area
  3. Nielsen, J.
    Jacob Nielsen's Alertbox: Risks of Quantitative Studies, March 1, 2004


  1. Ghostbusters (commercial video, available from any video store)
    • Ghostbusters has a very funny clip early on that shows an example of poor ethics. In it, two subjects are in a parapsychology experiment, with shock treatments being administered for a subject who guesses wrong on a card. The experimenter, however, is totally oblivious to how the subjects are performing, and always administers shocks to the "nerdy guy" subject (who happens to start getting it right), while encouraging the beautiful woman subject (who gets it wrong)! Ok, not a huge educational value here, but the students like it as a break from the normal routine!

In-Class Teaching tips

I have found that performing usability studies in class is an excellent use of time. I do several of them, each emphasizing slightly different methods of performing a usability study.

  1. Initial conceptual model formation plus think aloud of a paper prototype. I place a stylized picture of a Cannon fax machine on the overhead (included in the qualitative evaluation overheads) and ask them to explain the meaning of labels and controls. I then asked them to pretend they are sending a one page fax, and have them use think aloud to say what controls they are selecting and why.
  2. Think aloud on a physical artifact. I have a student volunteer do a think-aloud exercise as they try to display a slide on an overhead projector rigged with a burnt bulb---it usually takes the student 20 minutes to discover the problem and realize that the projector has a spare bulb that they can switch to. I also have them try to change the bulb, and most have difficulties figuring out how to open the projector to reach the bulb. The class, who are are taking notes, then critically analyze the design of the overhead projector (relating back to Norman's design principles of everyday things) and suggest improvements. As most recommendations are simple changes to the plastic overhead case, a "better" projector could probably be built for the exact same price. The class often wonders why the manufacturer (3M) had never bothered doing this simple exercise!
  3. Constructive interaction/co-discovery learning on an existing computer system. A pair of students are given a PC (the display is linked to a large screen so the class can see what is going on). One student is usually familiar with conventional GUI file systems, while another is a novice to it. The novice is the driver, while the 'expert' is the coach. Students are asked to do a few tasks on well known systems; one year we had people try to figure out the image placement system in Microsoft Word. While the scenario is somewhat rigged to bring out the interfaces worst features, it is surprising how much difficulty students have doing even the most basic things. I also have done this for the IBM RealPhone Prototype

Major sources used to prepare lecture material

  1. Above readings are particularly valuable.
  2. Gould, J. (1988)
    How to design usable systems. In Readings in Human Computer Interaction: Towards the Year 2000 (2nd Edition). Baecker, R., Grudin, J., Buxton, W., and Greenberg, S. (1995). Morgan-Kaufmann.
    • Originally written in 1988, this great article shows how user involvement and prototyping work together effectively
  3. Nielsen, J. (1993)
    Usability Engineering, p273-4, Academic Press.
    • helped structure the idea of evaluating systems with a user's involvement.
  4. Dumas, J.S. and Redish, J.C.
    A Practical Guide to Usability Testing. Revised Edition. (1999)
    • This book provides complete coverage of usability testing