back to CPSC 681

Course texts: You should buy these texts. Some of the readings below will be from them.

• Required
  1. Papers will be made available to you at cost.
• Some recommended classics
  1. A Practical Guide to Usability Testing. Revised Edition. Dumas, J.S. and Redish, J.C. (1999)
  2. Usability Engineering, Nielsen, J. (1993) Academic Press.
  3. Usability Inspection Methods (1994) J. Nielsen and R. Mack (eds), Wiley and Sons.

Overview of methodologies

The following papers introduces (but do not detail) many methodologies. They also discuss and compare methodologies from various perspectives: to influence design, to evaluate products, to encourage end-user feedback, and to compare when and where each is useful. Because you do not yet know the details of these methodologies, the discussions in the papers may be hard to follow at times. Because these papers are just intended to give you a feel for the area, I suggest you give them a quick read now. As the course progresses, you will find that you will be able to go back to them for a more careful read.

1. Gould, J.D. (1996)
   How to design usable systems. In R. Baecker, J. Grudin, W. Buxton and S. Greenberg (eds) Readings in Human Computer Interaction: Towards the Year 2000, p93-121, Morgan-Kaufmann.
   • This delightful paper, originally written in 1988, presents an overview of many informal methods and how they are applied to the usability design process.
2. Dix, A., Finlay, J., Abowd, G., and Beale, R.
   Human Computer Interaction, 2nd Edition. Prentice Hall. 1998
   • Chapter 11: Evaluation Techniques. 363-400
   • This useful overview chapter briefly surveys a variety of methodologies.
3. McGrath, J.
   Methodology matters: Doing research in the behavioural and social sciences. In R. Baecker, J. Grudin, W. Buxton and S. Greenberg (eds) Readings in Human Computer Interaction: Towards the Year 2000, p152-169, Morgan-Kaufmann. (1996).
   • See also a student-created summary by Stephanie Smale
   • This excellent paper discusses and compares fundamental concepts in evaluation methods. While it may be a 'heavy' read. its worth re-reading as the course progresses.

Optional Readings

1. Norman, D. Applying the Behavioral, Cognitive, and Social Sciences to Products. Mirrored from http://www.jnd.org/dn.mss/BCCSandProducts.html 2001.
   • This essay contrasts the academic vs business approach to product development.

User Observation

Experimental methodologies

1. Dix, A., Finlay, J., Abowd, G., and Beale, R.
   Human Computer Interaction, 2nd Edition. Prentice Hall. 1998
   • Selections from Chapter 11: Evaluation Techniques. 363-400
     • Section 11.3.1 Laboratory Studies (p365)
     • Section 11.5.1: Empirical methods (p375-385)
2. McGrath, J.
   Methodology matters: Doing research in the behavioural and social sciences. In R. Baecker, J. Grudin, W. Buxton and S. Greenberg (eds) Readings in Human Computer Interaction: Towards the Year 2000, p152-169, Morgan-Kaufmann. (1996).
   • Reread sections of this paper:
     • Experimental Strategies (p157-8)
     • Study Design, Comparison Techniques, and Validity (p159 - 164)
3. Eberts, R.E.
   User Interface Design, Prentice Hall 1994
   • Chapter 4: Experimental methodology;
   • Chapter 5: Experimental designs and analysis;
   • Chapter 6: Hazards to conducting and interpreting HCI experiments
   • All chapters give a general and high level introduction to experimental design.
4. Greenberg, S. and Witten, I. (1984)
   Comparison of menu displays for ordered lists. In Proc Canadian Information Processing Society National Conference, Calgary, Alberta, May
   • This paper describes the first experiment I ever did, and is an example of how statistical methods can be interpreted and applied to comparing particular system features. I use this example in the lectures.

Ethics

1. Nielsen, J. (1993)
   Usability Engineering, p273-4, Academic Press.
   • Chapter 6: Usability testing. p165-205 2 pages / side. In particular, read Chapter 6.4: Ethical aspects of tests with human subjects, p181-185.
   • A gentle and pragmatic introduction to ethics in HCI. See also my excerpt from CPSC 481 on Ethics, which summarizes some of Nielson's points.
2. University of Calgary Research Services: Ethics
   • University of Calgary’s website for Research Ethics
   • Do I Need to Apply For Ethics Approval?
   • Information to help applicants to fill out the application form
   • Ethics Application Form that you should fill out
   • Example template for a consent form
3. (optional) Rogers, Tim
   Research Participants' Bill of Rights. (Originally mirrored from http://www.psych.ucalgary.ca/research/ethics/participant_rights.html.
   • A concise statement of what participant's should expect as fair treatment

User Observation

Usability Testing

1. Nielsen, J. (1993)
   Usability Engineering, p273-4, Academic Press.
   • Chapter 6: Usability testing. 2 pages / side. p165-205 .
   • An excellent overview to various usability testing methodologies.
2. Gomoll, Kathleen & Nicol, Anne (1990)
   User Observation: Guidelines for Apple Developers, Apple Inc., January
   • This reading is an excellent recipe for performing usability studies.
3. Dumas, J.S. and Redish, J.C.
   A Practical Guide to Usability Testing. Revised Edition. (1999)
   • The chapters below provide an in-depth treatment of usability testing.
   • 2: Introducing usability testing
   • 6: Establishing a usability program in your organization
   • 7: Planning a usability test
   • 9: Deciding who should be participants
   • 11: Selecting and organizing tasks to test
   • 12: Creating task scenarios
   • 15: Preparing the testing environment
   • 18: Caring for the test participants
   • 19: Conducting the test
   • Appendix A: Setting up a usability laboratory
   • All of the above as a single PDF
4. (optional) Ehrlich, K., Butler, M. and Pernce, K.
   Getting the whole team into usability testing. IEEE Software, p89-90. (1994)
   • Tips and reasons for including developers as part of the evaluation team.
5. (optional) O'Malley, C., Draper, S. and Riley, M.
   Constructive interaction: A method for studying user-computer-user interaction. (1984) From Proceedings of Interact '84, p1-5.
   • Describes a method for observing people working together.
6. (optional) Kennedy, S.
   Using video in the BNR usability lab. SIGCHI Bulletin, 21(2), p92-95, ACM Press.(1989)
   • As in constructive interaction, they describe why observing multiple people working together is beneficial. Also describes benefits of videotaping.
7. (optional) Neal, L.
   The use of video in empirical research. (1989) SIGCHI Bulletin, 21(2), p100-101, ACM Press.
   • Further reasons why one should videotape usability sessions.
8. optional Krug, S.
   Don't Make Me Think: A Common Sense Approach to Web Usability. 1st Edition. Pearson Education, Inc and New riders. (2000) Extract: Chapters 9-11.
   • A simple guide to doing cut-rate usability tests. Written for the lay practitioner.
   • Chapter 9: Usability testing on 10 cents a day
   • Chapter 10: What to do if you're the facilitator
   • Chapter 11: Can this marriage be saved?

Excercise (to be handed out)

1. Get real with IBM RealPhone
   • We will perform a usability study on IBM's RealPhone system
   • Download the full handout, which contains:

     • the presentation
     • evaluator's instructions and tasks
     • consent form
     • Pre-test, During-test and Post-test questions
     • methods for running a basic observational study
     • discussion of guidelines for user observation
     • IBM Real Phone Instruction Sheet and their test-drive instructions
     • Isys' Hall of Shame critique of RealPhone

   • The system is available here, as an installable .exe
   • The presentation, in powerpoint

YOU SHOULD HAVE READ THE ABOVE PAPERS.

THE PAPERS BELOW ARE STILL TO BE ASSIGNED

Experimental methodologies

1. Quantitative methods Powerpoint or Adobe PDF
   Analysis of Variance Powerpoint or Adobe PDF
   * Lecture notes for the talk I give to the class.

Excercise

1. Mouse-Typing on Phone Pads and Keyboards
   • We will perform a 'pilot' controlled study on how people perform on different types of keyboards

Interface Inspection

Overview

1. Nielsen, J. and R. Mack (eds) ]]
   Usability Inspection Methods, p25-62, Wiley and Sons. (1994)
   • Chapter 1: Executive summary. p1-23
   • An exectuive summary and discussion of inspection methods.

Usability Heuristics and Heuristic Evaluation

1. Overview of Heuristic Evaluation Powerpoint or Adobe PDF
   • My presentation that I give in CPSC 481
2. Molich, R. and Nielsen, J.
   Improving a human-computer dialog. Comm ACM, 33(3), 1990
   • This article lists the heuristics and presents a working example, solutions, and alternatives.
3. Nielsen, J. (1993)
   Usability Engineering, p273-4, Academic Press.
   • Chapter 5: Usability heuristics. p115-163
   • Describes the heuristics in detail plus how and why it can be used to evaluate interfaces.
4. Nielsen, J. (1994)
   Enhancing the explanatory power of usability heuristics. In Proceedings of the CHI'94 Conference on Human Factors in Computing Systems, p152-158.
   • This article takes usability guidelines developed by different sources and sees which ones contribute most the the explanation of actual usability problems drawn from a database.
5. Nielsen, J. and R. Mack (eds) ]]
   Usability Inspection Methods, p25-62, Wiley and Sons. (1994)
   • Chapter 2: Heuristic evaluation. as 2 pages per side
   • A more in depth discussion of how heuristic evaluation works and its reliability
6. Nielsen, J. (1993)
   Usability Engineering, p273-4, Academic Press.
   • Exercise 8: Heuristic Evaluation of a Paper Mock-Up.
   • Another example, with answers, of a heuristic evaluation.

Task-Centered Walkthroughs

1. Overview of Task Centered Walkthrough Powerpoint or Adobe PDF
   • My presentation that I give in CPSC 481
2. Greenberg, S.
   Working through task-centered system design. in Diaper, D. and Stanton, N. (Eds) The Handbook of Task Analysis for Human-Computer Interaction. Lawrence Erlbaum Associates.
   • A how-to approach to task-centered system design.
3. Nielsen, J. and R. Mack (eds) ]]
   Usability Inspection Methods, p25-62, Wiley and Sons. (1994)
   • Chapter 3: The pluralistic usability walkthrough: Coordinated Empathies.
   • Describe the steps in the pluralistic walkthough process.

Supplemental Readings

1. Lewis, C. and Rieman, J. (1993)
   Task-Centered User Interface Design: A Practical Introduction. Available as Shareware.
   • The book that started it off.
   • Read the chapter Getting to Know Users and their Tasks., which was also published In R. Baecker, J. Grudin, W. Buxton and S. Greenberg (eds) Readings in Human Computer Interaction: Towards the Year 2000, p122-127, Morgan-Kaufmann.

Cognitive Walkthroughs

1. Nielsen, J. and R. Mack (eds) ]]
   Usability Inspection Methods, p25-62, Wiley and Sons. (1994)
   • Chapter 5: The Cognitive Walkthrough Method: A Practitioner's Guide. Wharton, C., Rieman, J. Lewis, C. and Polson, P. p25-62
   • This more complex but more complete cognitive walkthrough method inspired the discount task-centered walkthrough method.
2. Preece, J. et. al., (1994)
   Human Computer Interaction, Addison-Wesley
   • Walkthroughs (Excerpt), page 679-684

Prototyping

1. Nielsen, J. (1993)
   Usability Engineering, p273-4, Academic Press.
   • Chapter 4.8 Prototyping. (Extract) p93-101. (also as 2 pages / side )
   • Describes different styles of prototypes and how they can be used within scenarios.
2. Rettig, M. (1994)
   Prototyping for tiny fingers. Communications of the ACM, 37(4), ACM Press
   • A simple way to build low fidelity prototypes, similar to Pictive
3. Rudd, J., Stern, K. and Isensee, S. (1996)
   Low vs. high fidelity prototyping debate. Interactions 3(1), p76-85, ACM Press.
   • When are where you should use low vs. high fidelity prototypes
4. Muller, M. (1991)
   Pictive: An exploration in participatory design. In Proceedings of the ACM Conference on Human Factors in Computing Systems, p225-231, ACM Press.
   • A method for low fidelity prototype development that builds it on the fly with postit notes and other office supplies
5. Gould, J., Conti, J., and Hovanyecz, T. (1981)
   Composing letters with a simulated listening typewriter. Communications of the ACM, 26(4), 295-308. ACM Press.
   • The first description of the Wizard of Oz prototyping/testing method.

Supplemental Readings

• Maulsby, D., Greenberg, S. and Mander, R. (1993).
  Prototyping an intelligent agent through Wizard of Oz. Proceedings of the ACM CHI'93 Conference on Human Factors in Computing Systems, Amsterdam, The Netherlands, p277-284, ACM Press.
  • Describes a sophisticated Wizard of Oz experiment, plus gives advice on how to apply WOZ.
• Muller, Michael J.
  Retrospective on a year of participatory design using the PICTIVE technique. Proceedings of the SIGCHI conference on Human factors in computing systems, ACM Press June 1992
  • Experiences with Pictive and recommendations.
• Landay, James A., and Brad A. Myers
  Interactive sketching for the early stages of user interface design. Proceedings of the SIGCHI conference on Human factors in computing systems, ACM Press. May 1995
  • Describes a system that lets people sketch and storyboard through a live computer interface.

Field Study Methodologies

Introduction

1. Neustaedter, Carman Overview of Qualitative Data Analysis. (Powerpoint) and (Video) of his presenation
2. Nielsen, J.
   Jacob Nielsen's Alertbox
   • First rule of usability? Don't listen to users AND Field Studies done right: Fast and observational, August 5, 2001 and January 20, 2002.
   • http://www.useit.com/alertbox/20010805.html
   • 20020120.html

Contextual Inquiry

1. Holtzblatt, K. and Jones, S. (1996)
   Contextual Inquiry: A Participatory Approach. In D. Schuler and A. Namioka (eds) Participatory Design: Principles and Practices, p177-210.
2. Holtzblatt, K. and Beyer, H. (1993)
   Contextual Design: Principles and Practice. In D. Wixon and J. Ramey (eds) Field Methods Casebook for Software Design, p303-333.

Open Coding

• Strauss, A. and Corbin, J. (1998)
  Basics of Qualitative Research, 2nd Edition. Sage Publications.
  • Chapter 8. Open Coding.

Ethnography

1. Dourish, P. (2001)
   Where the Action Is. The Foundations of Embodied Interaction. MIT Press, Cambridge MMA.
   • Chapter 3: Social Computing p55-97.
   • Dourish introduces anthropology, ethnography and ethnomethodology and how it relates to Social Computing. While the 2nd half of this chapter is more related to the theme of his book, its still worth reading to the end. The book itself is great and is a must read for HCI researchers.
2. Bentley, R., Hughes, J.A., Randall, D., Rodden, T., Sawyer, P., Shapiro, D. and Summerville, I. (1992)
   Ethnographically-informed systems design for air traffic control, in Proc CSCW'92, p123-129, November.
3. Hughes, J., King, V., Rodden, T. and Andersen, H.
   Moving out of the control room: Ethnography in System Design. Proc ACM CSCW 1996.
4. Randall, D. (1996)
   Ethnography and Systems Development: Bounding the Intersection. Tutorial notes presented at CSCW'96.
   • Excerpts: Sections 3, 4,5,7

Comparing Methods

1. Greenberg, S. and Buxton, B. (2008)
   Usability Evaluation Considered Harmful (Some of the Time). In Proceedings of the ACM Conference on Human Factors in Computing Systems - ACM CHI'08. (Florence, Italy), ACM Press, pages 111-120, April 5-10.
2. Nielsen, J. (1993)
   Usability Engineering, p273-4, Academic Press.
   • Chapter 7: Usability Assessment Methods Beyond Testing
   • Introduces observations, questionnaires and interviews, focus groups, logging, user feedback, and how to choose between them.
3. Olson, J. and Moran, T. (1996)
   Mapping the method muddle: Guidance in using methods for user interface design. In M. Rudisill, C. Lewis, P. Polson and T. McKay (eds) Human-Computer Interface Design: Success Stories, Emerging Methods, and Real-World Context, p269-300, Morgan-Kaufmann.
   • The authors associate a variety of methodological approaches to specific interface design activities.
4. Preece, J. et. al., (1994)
   Human Computer Interaction, Addison-Wesley
   • Chapter 34: Comparing methods. p691-707
   • Provides an overview of previous comparative studies of evaluation methods.
5. Jeffries, R., Miller, J., Wharton, C. and Uyeda, K. (1991)
   User interface evaluation in the real world: A comparison of four techniques. In ACM Conference on Human Factors in Computing Systems. p119-124, ACM Press.
6. Karat, C., Campbell, R. and Fiegel, T. (1992)
   Comparison of empirical testing and walkthrough methods in user interface evaluation. In ACM Conference on Human Factors in Computing Systems. 397-4044, ACM Press.
7. Landauer, T. (1991)
   Lets Get Real: A Position Paper on the Role of Cognitive Psychology in the Design of Humanly Useful and Usable Systems, In J. Carroll (ed) Designing Interaction, Cambridge University Press.
8. Steve M. Easterbrook and J. Singer and M.-A. Storey and D. Damian, 2007
   Selecting Empirical Methods for Software Engineering Research, In Forrest Shull, Janice Singer and Dag I. K. Sjoberg (Eds) Guide to Advanced Empirical Software Engineering, Springer, 285-311.
9. Ioannidis, J. P. A. (2005)
   Why Most Published Research Findings Are False. PLoS Medicine, August; 2(8): e124. link
10. Norman, D. (various)
    • Technology First, Needs Last
    • Filling Much Needed Holes

Last updated Fall, 2006