Evaluating Interfaces with Users:
This section introduces quantitative methods for evaluating interfaces with users,
concentrating on experimental design, controlled experimentation, and simple statistics.
An assignment on quantitative
evaluation gives students practice in a controlled experiment.
- Quantitative evaluation methods
- User performance data collection
- Controlled experiments
- The experimental method
- Planning an experiment
- The value of statistics
- Example: T-test
- Significance levels, types of error
- Other statistical tests
- Analysis of variance
- factorial design
- case study
In-Class Teaching tips
In-class quantitative controlled experiment. Sometimes, there was no time for students to do the related assignment.
When this is the case, I run a
subject in class through a controlled experiment to give people an idea of what they have
to do (their task was mouse-typing on two soft keyboards with different layouts, specified
in the assignment 1). Students then run each other as subjects out of class (since the
software is set up for them, it takes them only 15 minutes). They hand in the data, and we
analyze and interpret it in a later class.
- Chapter 2: Design and Evaluation, p.73-91. The sections directly relevant to
quantitive evaluation are c) Controlled experiments.
- Methodology Matters: Doing Research in the Behavioural and Social
p.152-169, covers many fundamental issues in empirical research methodologies.
- Touch Typing with a Stylus, by Xerox (1993, SGVR 88) shows a
pen-based character input system. This is presented as a radically different
alternative to the screen-based keyboards evaluated in the example assignment.
Major sources used to prepare lecture material
- For quantitative evaluation, the reading in Baecker, Grudin, Buxton
and Greenberg: Methodology matters provides good
- Other sources I used include general psychology statistics books, and
chapters in HCI textbooks dealing with controlled experiments (e.g., Chapter 11.5.1 in
Computer Interaction by Dix, Finlay, Abowd and Beale).
- The research planning chart included in the slides came from an old CHI tutorial, but I've since lost track of
who actually created it.