Evaluating Interfaces with Users: Controlled Experiments

This section introduces quantitative methods for evaluating interfaces with users, concentrating on experimental design, controlled experimentation, and simple statistics. It also includes a more detailed introduction to ANOVA

An [[X|assignment on quantitative evaluation] gives students practice in a controlled experiment.

Overheads (Powerpoint)

Topics Covered

  • Quantitative evaluation methods
    • User performance data collection
    • Controlled experiments
    • The experimental method
    • Planning an experiment
    • The value of statistics
    • Example: T-test
    • Significance levels, types of error
    • Other statistical tests
  • Analysis of variance
    • terminology
    • factorial design
    • case study

Additional Readings

  1. Eberts, R.E.
    User Interface Design, Prentice Hall 1994
  2. Baecker, R., Grudin, J., Buxton, W., and Greenberg, S.
    Readings in Human Computer Interaction: Towards the Year 2000 (2nd Edition). Morgan-!Kaufmann, San Francisco, CA, 1995


  1. Touch Typing with a Stylus, by Xerox (1993, SGVR 88) shows a pen-based character input system. This is presented as a radically different alternative to the screen-based keyboards evaluated in the example assignment.

In-Class Teaching tips

  1. In-class quantitative controlled experiment. Sometimes, there was no time for students to do the related assignment. When this is the case, I run a subject in class through a controlled experiment to give people an idea of what they have to do (their task was mouse-typing on two soft keyboards with different layouts, specified in the assignment 1). Students then run each other as subjects out of class (since the software is set up for them, it takes them only 15 minutes). They hand in the data, and we analyze and interpret it in a later class.

Major sources used to prepare lecture material

  1. The readings above provide general background.
  2. Other sources I used include general psychology statistics books, and chapters in HCI textbooks dealing with controlled experiments (e.g., Chapter 11.5.1 in Human Computer Interaction by Dix, Finlay, Abowd and Beale).
  3. The research planning chart included in the slides came from an old CHI tutorial, but I've since lost track of who actually created it.