Instructor: Sheelagh Carpendale
Credits: from Lecture Topics in HCI, by Saul Greenberg
Contents for: CPSC 481 

Evaluating Interfaces with Users: Qualitative Methods

This section introduces qualitative methods for evaluating interfaces with users' involvement. These form the basis for usability studies. I (and many other HCI people) strongly believe that evaluation should occur continually through the design and implementation process. Students should remember that these and other evaluation methods are applied as the interface is being developed, rather than at the end. A good evaluation process means that designers will catch major problems (and successes!) early on, with lesser problems being ironed out as the interface is being refined.

An assignment on usability studies can be used to complement this section by providing students with hands-on practice in various usability study methods.


Overheads

Topics Covered

Required Readings

Optional Readings from Baecker Grudin Buxton and Greenberg

Videos

Ghostbusters (commercial video, available from any video store) has a very funny clip early on that shows an example of poor ethics. In it, two subjects are in a parapsychology experiment, with shock treatments being administered for a subject who guesses wrong on a card. The experimenter, however, is totally oblivious to how the subjects are performing, and always administers shocks to the "nerdy guy" subject (who happens to start getting it right), while encouraging the beautiful woman subject (who gets it wrong)! Ok, not a huge educational value here, but the students like it as a break from the normal routine!

In-Class Teaching tips

I have found that performing usability studies in class is an excellent use of time. I do several of them, each emphasizing slightly different methods of performing a usability study.
  1. Initial conceptual model formation plus think aloud of a paper prototype. I place a stylized picture of a Cannon fax machine on the overhead (included in the qualitative evaluation overheads) and ask them to explain the meaning of labels and controls. I then asked them to pretend they are sending a one page fax, and have them use think aloud to say what controls they are selecting and why.
  2. Think aloud on a physical artifact. I have a student volunteer do a think-aloud exercise as they try to display a slide on an overhead projector rigged with a burnt bulb---it usually takes the student 20 minutes to discover the problem and realize that the projector has a spare bulb that they can switch to. I also have them try to change the bulb, and most have difficulties figuring out how to open the projector to reach the bulb. The class, who are are taking notes, then critically analyze the design of the overhead projector (relating back to Norman's design principles of everyday things) and suggest improvements. As most recommendations are simple changes to the plastic overhead case, a "better" projector could probably be built for the exact same price. The class often wonders why the manufacturer (3M) had never bothered doing this simple exercise!
  3. Constructive interaction/co-discovery learning on an existing computer system. A pair of students are given a PC (the display is linked to a large screen so the class can see what is going on). One student is usually familiar with conventional GUI file systems, while another is a novice to it. The novice is the driver, while the 'expert' is the coach. Students are asked to do a few basic tasks i.e., one year we used Windows 95 and had them copy files around, finding files, and so on. While the scenario is somewhat rigged to bring out the interfaces worst features, its surprising how much difficulty students have doing even the most basic things. I also have done this for the IBM RealPhone System.

Major sources used to prepare lecture material