Evaluating Interfaces with Users: Controlled Experiments

HCILectures.EvaluationControlled History

Hide minor edits - Show changes to output

Changed lines 9-11 from:
* [[HCILectures:evaluation_controlled-part1.ppt|Controlled experiments - Part 2]]. Basic statistics
* [[HCILectures:evaluation_anova.ppt|Analysis of Variance]]
to:
* [[HCILectures:evaluation_controlled-part2.ppt|Controlled experiments - Part 2]]. Basic statistics
* [[HCILectures:evaluation_controlled-part3.ppt|Controlled experiments - Part 3]]. Analysis of Variance
Changed lines 8-9 from:
* [[HCILectures:evaluation_controlled.ppt|Controlled experiments]]
to:
* [[HCILectures:evaluation_controlled-part1.ppt|Controlled experiments - Part 1]]. What it is
* [[HCILectures:evaluation_controlled-part1.ppt|Controlled experiments - Part 2]]. Basic statistics
Changed lines 8-10 from:
* [[HCILectures:evaluation-controlled.ppt|Controlled experiments]]
* [[HCILectures:evaluation-anova.ppt|Analysis of Variance]]
to:
* [[HCILectures:evaluation_controlled.ppt|Controlled experiments]]
* [[HCILectures:evaluation_anova.ppt|Analysis of Variance]]
Changed lines 35-36 from:
** [[HCIPapers:mcgrath-methodologymatters-bgbg95.pdf |'''Methodology matters: Doing research in the behavioural and social sciences.''']], by McGrath, J. p152-169 - covers many fundamental issues in empirical research methodologies.
to:
** [[HCIPapers:mcgrath-methodologymatters-bgbg95.pdf |Methodology matters: Doing research in the behavioural and social sciences.]], by McGrath, J. p152-169 - covers many fundamental issues in empirical research methodologies.
Changed lines 34-39 from:
** [[HCIPapers:baecker-chapter2-bgbg95.pdf| Chapter 2: Design and Evaluation]]
** The sections directly relevant to quantitive evaluation are c) Controlled experiments.
# McGrath, J. \\
[[HCIPapers:mcgrath-methodologymatters-bgbg95.pdf |'''Methodology matters: Doing research in the behavioural and social
sciences.''']] In R. Baecker, J. Grudin, W. Buxton and S. Greenberg (eds) Readings in Human Computer Interaction: Towards the Year 2000, p152-169, Morgan-Kaufmann. (1996).
**
covers many fundamental issues in empirical research methodologies.
to:
** [[HCIPapers:baecker-chapter2-bgbg95.pdf| Chapter 2: Design and Evaluation]] - the section on Controlled experiments.
** [[HCIPapers:mcgrath-methodologymatters-bgbg95.pdf |'''Methodology matters: Doing research in the behavioural and social
sciences.''']], by McGrath, J. p152-169 - covers many fundamental issues in empirical research methodologies.
Deleted lines 25-30:
!!Required Readings
# Nielsen, J. (1993) \\
'''Usability Engineering,''' p273-4, Academic Press. [[HCIPapers:nielson93-usabilityengch6-usabilitytesting-2page.pdf|Chapter 6: Usability testing]]. p165-205
# Gomoll, Kathleen & Nicol, Anne (1990) \\
[[HCIPapers:gomol-userobservations-apple90.pdf|'''User Observation: Guidelines for Apple Developers''',]] Apple Inc., January
Changed lines 27-30 from:
# Chapter 2: Design and Evaluation, p.73-91. The sections directly relevant to quantitive evaluation are c) Controlled experiments.
# Methodology Matters: Doing Research in the Behavioural and Social Sciences, p.152-169, covers many fundamental issues in empirical research methodologies
.
to:
# Eberts, R.E. \\
'''User Interface Design
,''' Prentice Hall 1994
** [[HCIPapers:eberts-uidesign-ch4.pdf|Chapter 4: Experimental methodology; ]]
** [[HCIPapers:eberts-uidesign-ch5
.pdf|Chapter 5: Experimental designs and analysis; ]]
** [[HCIPapers:eberts-uidesign-ch6.pdf|Chapter 6: Hazards to conducting and interpreting HCI experiments]]
# Baecker, R
., Grudin, J., Buxton, W., and Greenberg, S. \\
'''Readings in Human Computer Interaction: Towards the Year 2000 (2nd Edition).''' Morgan-!Kaufmann, San Francisco, CA, 1995
** [[HCIPapers:baecker-chapter2-bgbg95.pdf| Chapter 2: Design and Evaluation]]
** The sections directly relevant to quantitive evaluation are c) Controlled experiments.
# McGrath, J. \\
[[HCIPapers:mcgrath-methodologymatters-bgbg95.pdf |'''Methodology matters: Doing research in the behavioural and social sciences.''']] In R. Baecker, J. Grudin, W. Buxton and S. Greenberg (eds) Readings in Human Computer Interaction: Towards the Year 2000, p152-169, Morgan-Kaufmann. (1996).
** covers many fundamental issues in empirical research methodologies.
Added lines 1-46:
(:title Evaluating Interfaces with Users: Controlled Experiments :)

This section introduces quantitative methods for evaluating interfaces with users, concentrating on experimental design, controlled experimentation, and simple statistics. It also includes a more detailed introduction to ANOVA

An [[X|'''assignment on quantitative evaluation'''] gives students practice in a controlled experiment.

!! Overheads (Powerpoint)
* [[HCILectures:evaluation-controlled.ppt|Controlled experiments]]
* [[HCILectures:evaluation-anova.ppt|Analysis of Variance]]

!!Topics Covered
* Quantitative evaluation methods
** User performance data collection
** Controlled experiments
** The experimental method
** Planning an experiment
** The value of statistics
** Example: T-test
** Significance levels, types of error
** Other statistical tests
* Analysis of variance
** terminology
** factorial design
** case study

!!Required Readings
# Nielsen, J. (1993) \\
'''Usability Engineering,''' p273-4, Academic Press. [[HCIPapers:nielson93-usabilityengch6-usabilitytesting-2page.pdf|Chapter 6: Usability testing]]. p165-205
# Gomoll, Kathleen & Nicol, Anne (1990) \\
[[HCIPapers:gomol-userobservations-apple90.pdf|'''User Observation: Guidelines for Apple Developers''',]] Apple Inc., January

!!Additional Readings
# Chapter 2: Design and Evaluation, p.73-91. The sections directly relevant to quantitive evaluation are c) Controlled experiments.
# Methodology Matters: Doing Research in the Behavioural and Social Sciences, p.152-169, covers many fundamental issues in empirical research methodologies.


!!Videos
# '''Touch Typing with a Stylus''', by Xerox (1993, SGVR 88) shows a pen-based character input system. This is presented as a radically different alternative to the screen-based keyboards evaluated in the example assignment.

!!In-Class Teaching tips
# '''In-class quantitative controlled experiment.''' Sometimes, there was no time for students to do the related assignment. When this is the case, I run a subject in class through a controlled experiment to give people an idea of what they have to do (their task was mouse-typing on two soft keyboards with different layouts, specified in the assignment 1). Students then run each other as subjects out of class (since the software is set up for them, it takes them only 15 minutes). They hand in the data, and we analyze and interpret it in a later class.

!! Major sources used to prepare lecture material
# The readings above provide general background.
# Other sources I used include general psychology statistics books, and chapters in HCI textbooks dealing with controlled experiments (e.g., Chapter 11.5.1 in Human Computer Interaction by Dix, Finlay, Abowd and Beale).
# The research planning chart included in the slides came from an old CHI tutorial, but I've since lost track of who actually created it.