Evaluating Interfaces with Users: Qualitative Methods
This section introduces qualitative methods for evaluating
interfaces with users' involvement. These form the basis for usability
studies. I (and many other HCI people) strongly believe that evaluation
should occur continually through the design and implementation process.
Students should remember that these and other evaluation methods are applied
as the interface is being developed, rather than at the end. A good evaluation
process means that designers will catch major problems (and successes!)
early on, with lesser problems being ironed out as the interface is being
on usability studies can be used to complement this section by providing
students with hands-on practice in various usability study methods.
Introduction to evaluation: [PDF]
Evaluating interfaces with Users
Natural vs experimental approaches
Reliability and validity concerns
Qualitative evaluation methods
Direct observation: think aloud and co-discovery learning
Query techniques: interviews and questionnaires
Continuous evaluation: user feedback and field studies
Nielsen, J. (1993) Chapter 6: Usability testing.
Engineering, p165-205, Academic Press.
Kathleen Gomoll & Anne Nicol (1990) Discussion
of guidelines for user observation. From User Observation: Guidelines
for Apple Developers, January [Available
Optional Readings from Baecker Grudin Buxton and Greenberg
Chapter 2: Design and Evaluation, p.73-91, gives an
overview of the life cycle and methodologies of design and evaluation.
This chapter is really an introduction to the rest of the course, as it
touches upon themes that are covered later. The sections directly relevant
to qualitative evaluation are: a) Design and evaluation considered together;
b) Usability testing.
How to Design Usable Systems, p.93-121, is another
overview to the process of system design, and shows how evaluation fits
within many parts of the design cycle.
Methodology Matters: Doing Research in the Behavioral
and Social Sciences, p.152-169, covers many fundamental issues in empirical
Using Video in the BNR Usability Lab, p.182-185, discusses
the value of co-discovery learning and video as a means to record observations
within a product environment setting.
Optional reading: Conducting and Analyzing a Contextual
Interview, p.241-253, introduces and describes the process of contextual
interviews, an interviewing approach that acquires information on users'
work and tool use, and that can actively involve them in co-design of the
Ghostbusters (commercial video, available
from any video store) has a very funny clip early on that shows an example
of poor ethics. In it, two subjects are in a parapsychology experiment,
with shock treatments being administered for a subject who guesses wrong
on a card. The experimenter, however, is totally oblivious to how the subjects
are performing, and always administers shocks to the "nerdy guy" subject
(who happens to start getting it right), while encouraging the beautiful
woman subject (who gets it wrong)! Ok, not a huge educational value here,
but the students like it as a break from the normal routine!
Ghostbusters (commercial video, available from any
In-Class Teaching tips
I have found that performing usability studies in class is
an excellent use of time. I do several of them, each emphasizing slightly
different methods of performing a usability study.
Initial conceptual model formation plus think aloud of
a paper prototype.
I place a stylized picture of a Cannon fax machine
on the overhead (included in the qualitative evaluation overheads) and
ask them to explain the meaning of labels and controls. I then asked them
to pretend they are sending a one page fax, and have them use think aloud
to say what controls they are selecting and why.
Think aloud on a physical artifact. I have a student
volunteer do a think-aloud exercise as they try to display a slide on an
overhead projector rigged with a burnt bulb---it usually takes the student
20 minutes to discover the problem and realize that the projector has a
spare bulb that they can switch to. I also have them try to change the
bulb, and most have difficulties figuring out how to open the projector
to reach the bulb. The class, who are are taking notes, then critically
analyze the design of the overhead projector (relating back to Norman's
design principles of everyday things) and suggest improvements. As most
recommendations are simple changes to the plastic overhead case, a "better"
projector could probably be built for the exact same price. The class often
wonders why the manufacturer (3M) had never bothered doing this simple
Constructive interaction/co-discovery learning on an existing
computer system. A pair of students are given a PC (the display is
linked to a large screen so the class can see what is going on). One student
is usually familiar with conventional GUI file systems, while another is
a novice to it. The novice is the driver, while the 'expert' is the coach.
Students are asked to do a few basic tasks i.e., one year we used Windows
95 and had them copy files around, finding files, and so on. While the
scenario is somewhat rigged to bring out the interfaces worst features,
its surprising how much difficulty students have doing even the most basic
things. I also have done this for the IBM RealPhone System.
Major sources used to prepare lecture material
Gould's reading in Chapter 2 lays much of the foundation
for evaluating interfaces
Usability Engineering, by Jakob Nielsen, helped structure
the idea of evaluating systems with a user's involvement.
For qualitative evaluation, the readings How to design
usable systems and Discussions and guidelines for user observation
are particularly valuable.
This book provides complete coverage: Dumas, J. and Redish,
J. (1993) A Practical Guide to Usability Testing. Ablex. I