Wizard of Oz in Human Computer Interaction

0 Suggested Reading

Composing letters with a simulated listening typewriter

An online video giving a quick introduction to the method

I Overview

"Oh no, my dear...I'm a very good man. I‘m just a very bad wizard." - The Wizard of Oz


Figure 1 up: the "powerful" wizard down: the man behind the curtain who operated the wizard

The Wizard of Oz is an American musical fantasy adventure film produced by a media company known as Metro-Goldwyn-Mayer in 1939. It is based on the 1900 novel The Wonderful Wizard of Oz by L. Frank Baum. In short, it is a story of the adventures of a little girl, Dorothy through a colorful magical Land of Oz. One interesting character of the story is an ordinary man who hides behind a curtain and acts as a powerful Wizard. This deception forms the basis of the evaluation technique in human-computer interaction (HCI) also known as Wizard of Oz (WOz).

The term in the HCI context is coined by John F. Kelly when he was working on his Ph.D. dissertation in Johns Hopkins University. His original work introduced human intervention in the work flow of a natural language processing application [1].

Nowadays WOz has been widely accepted as an evaluation and prototyping methodology in the HCI domain. It is mainly used to analyze a partially implemented computer application for design improvements.

In a WOz experiment, subjects interact with a seemingly autonomous application whose unimplemented functions are actually simulated by a human operator, or known as the Wizard. The Wizard intercepts the communication between the participants and the computer application. Then following an algorithm or written instructions, he/she mimics the response of the application to the participants. Participants‘ interactions are observed or measured to serve the evaluation purpose as if they are working with a real system. Usually the participants are not informed of this involvement of human intelligence to encourage more natural reaction but it‘s not always the case. If deception is used, it should be debriefed after the study.

II When to Use Wizard of Oz in a Design Process

As the Wizard of Oz (WOz) method is usually employed to simulate and test unimplemented technology, it is a common notion that it mainly fits in the early stage of design. Indeed, the early phase of a design process is a suitable time to apply WOz method since it could solicit valuable feedback on future design direction and rule out wrong assumptions before they lead to any serious consequences. But the WOz method can actually go beyond these preliminary stages and be applied along the course of designing.


Figure 2 Using Wizard of Oz in the design process [2]

As the figure above illustrated, WOz is commonly used in the initial prototyping stage (red dots) but in fact, it can serve to fill the gap between the current implementation and envisioned system and to provide user experience insights throughout the entire design process. On the other hand though WOz may be the quickest and simplest way to actualize a design, engineering a WOz interface and integrating it into the current implementation still requires considerable efforts and operating it places extra burden on experimenters. Therefore designers must consider the value of the interface to be evaluated, the efforts to simulate it and whether there are alternative means before starting a WOz study.

III Typical Contexts where Wizard of Oz is Used

Traditionally, WOz is mostly adopted in evaluating natural language interfaces (NLI). Poineering work using WOz on NLI include [1, 3] and a summary of WOz on NUI before 1993 is [4] As its value got gradually recognized, its application has been extended to various systems that rely on sophisticated artificial intelligence functions, such as motion tracking [5], programming by demostration [6] and trace recognition [7]. The common characteristic of these functions simulated is that they are relatively hard to code, but easy for humans. A contrary example is finding patterns in large dataset, which may be hard to code and impossible for humans. Broadly speaking, the Wizard can simulate all system functions or interface designs which bear this characteristic, and serve for evaluation but whether the results will outweigh the efforts is left to designers to decide.

IV Adavantages and Disadvantages of Wizard of Oz

Advantages

1. Faster, cheaper and more
-This is probably the most significant and obvious advantage of WOz. Using simulation works around difficult, time-consuming and expensive development and with the saved resources more ideas and variations of design can be studied to open up new interaction space.
2. Available in early stages of design
-Even at the very beginning of the design process, given a rough idea WOz can work. Thus it is able to help detect wrong design assumptions, such as mistaken target groups, before they lead to any expensive mistakes later on.
3. More "real" than other early prototyping methods
-As compared to paper prototyping and other early prototyping methods, WOz apparently offers more "real" experience. Though realism is not always needed in early evaluation, presenting users a prototype that "feels" closer to completion is likely to elicit richer and more comprehensive feedback.
4. Useful for envisioning and proactively evaluating hard-to-build interfaces
-While those in high performance computing or artificial intelligence are working on the fundamentals of the next-generation interfaces, researchers in HCI can seek the answer to this question: how can we maximize the benefits these technologies bring to people? WOz is one of the possible methods to envision their appropriate form and to direct their advancement.
5. A designer learns when playing as the wizard
-Reacting for her system lets a designer know its otherwise concealed unreasonable logic.

Disadvantages

1. Simulations may conceal possible errors or problems in technology implementation
-Simulations are essentially illusions. The perfect illusions may let the team ignore errors and problems in actual implementation but, of course, not existent in simulations. For example, one team working on natural language interfaces manages to use WOz to find the optimal response time. But if they never consider the chance of misrecognition, their finds may be invalid.
2. Simulated technologies may not be possible to implement now, or ever.
-It is possible that the yet-to-exist technology simulated by WOz could not be implemented or is not feasible at all.
3. Challenges of a Wizard
-Playing the Wizard is sometimes a demanding job posing challenges such as
  • It causes both physical and cognitive fatigue. The simulator must be careful about inconsistent responses across sessions due to fatigue.
  • A Wizard who is also one of the design team should not be biased towards their design and perform beyond the programmed behavior thus influencing user feedback.
  • Wizard needs to properly match the responses of a computer. For example, the wizard should not make typing mistakes.
4. Cost-intensive
-The wizard needs to be trained to conduct the experiment effectively. So it takes more time to setup the study environment than other prototyping method such as paper prototype.
5. Some features cannot be easily simulated by human
-As we discussed above, the use of human simulation is usually limited to functions hard to computers, but easy to human. Analyzing large dataset or rendering complex graphics are not suitable for being simulated in WOz.

V Designing and Running a Wizard of Oz Study

Designing

Here are several steps to follow when designing a WOz study

  1. Plan scenarios and application flow
  2. Deploy the interface "skeleton"
  3. Decide the form of input from the Wizard. Then implement it
  4. Rehearsal

Running

Several points to note when running a WOz study

References

  1. Kelly, J. F., An Empirical Methodology for Writing User-Friendly Natural Language Computer Applications. In Proceedings of CHI ‘83, ACM Press (1983), 193-196.
  2. This is the J. F. Kelly‘s paper on his work in which the methodology was proposed. It was not focusing on evaluation but took evaluation as one step in the iterative design process.

  3. Dow, S., Maclntyre. B, Lee, J., Oezbek, C., Bolter, J.D., Gandy, M. Wizard of Oz Support throughout an Iterative Design Process. In Pervasive Computing, IEEE, 4(4), 18-26.
  4. In this relatively recent paper, the authors re-emphasized the value of using Wizard of Oz in an iterative design process. Along with this point, a system supporting WOz in iterative design was discussed.

  5. Gould, J. Conti, J. and Hovanyecz T. Composing Letters with a Simulated Listening Typewriter. In Communications of the ACM, 26(4), 295-308.
  6. This is a poineering paper giving a thorough description and discussion on an early work using WOz in natural language interfaces. It is also in the suggested reading list.

  7. Dahlback, N., Jonsson, A. and Arhenberg, L. Wizard of Oz Studies -- why and how. In Knowledge-Based Systems, 6(4), 258-266.
  8. This paper gave a summary of WOz on natural language interfaces before the point it was written.

  9. Andersson, G., Hook, K., Mourao, M., Paiva, A. and Costa, M. Using a Wizard of Oz Study to Inform the Design of SenToy. In Proceedings of DIS ‘02, ACM Press (2002), 349-355
  10. In the paper, the authors reported their use of WOz on the design of a bodily motion interaction game and its implications.

  11. Maulsby, D., Greenberg, S. and Mander, R. Prototyping an Intelligent Agent through Wizard of Oz. In Proceedings of CHI ‘93, ACM Press (1993), 277-284.
  12. The paper gave a deep discussion on designing intelligent agent (programming by demonstration system) with a WOz methodology.

  13. Davis, R. C., Saponas, T. S., Shilman, M. and Landy, J. A. SketchWizard: Wizard of Oz Prototyping of Pen-based User Interfaces. In Proceedings of UIST ‘07, ACM Press (2007), 277-284.
  14. The paper talked about WOz on pen-based interfaces. It is included mainly to show the breadth to which WOz was used.