Multi User Speech and Gesture Interaction

Personal.ExploringMultiUserSpeechAndGestureInteraction History

Hide minor edits - Show changes to output

May 09, 2007, at 05:17 PM by 136.159.7.55 -
Changed lines 25-26 from:
%width=512px% Attach:LargeWallInteraction.jpg Sometimes on a large digital wall it is exceedingly difficult to interact with items at a distance. We have some software that allows graphics to be drawn on a large digital wall display. You could develop applications to perform tasks or games over the large display
to:
%width=512px% Attach:LargeWallInteraction.jpg
Sometimes on a large digital wall it is exceedingly difficult to interact with items at a distance. We have some software that allows graphics to be drawn on a large digital wall display. You could develop applications to perform tasks or games over the large display
May 09, 2007, at 05:16 PM by 136.159.7.55 -
Changed lines 25-27 from:
%width=512px% Attach:LargeWallInteraction.jpg
Sometimes on a large digital wall it is exceedingly difficult to interact with items at a distance. We have some software that allows graphics to be drawn on a large digital wall display. You could develop applications to perform tasks or games over the large display
to:
%width=512px% Attach:LargeWallInteraction.jpg Sometimes on a large digital wall it is exceedingly difficult to interact with items at a distance. We have some software that allows graphics to be drawn on a large digital wall display. You could develop applications to perform tasks or games over the large display
May 09, 2007, at 05:16 PM by 136.159.7.55 -
Changed lines 5-6 from:
You'll have an opportunity to explore multi user speech and gesture interaction over digital walls and tables.
to:
When people work face to face they often use a combination of speech and gestures to interact with the computer. We now have technologies that can detect the speech and gesture actions of multiple people over a digital table. In this project you'll have an opportunity to explore multi user speech and gesture interaction over digital walls and tables.
Changed lines 8-11 from:
When multiple people are interacting over a shared digital tabletop, they often communicate with others using many speech and gestural actions. Example applications show multi user interaction with [[http://edwardhtse.googlepages.com/edwardtse-videos.html|Google Earth, Warcraft III, and The Sims]].

Recently, toolkits have been developed to dramatically simplify the development of speech
and gesture enabled applications in two ways: first, wrappers can be built to allow speech and gesture actions to be converted to mouse and keyboard events on an existing single user application. Second, custom speech and gesture applications can be built from the ground up (e.g., Blended Lenses, an Air Traffic Control game, be creative!)
to:
When multiple people are interacting over a shared digital tabletop, they often communicate with others using many speech and gestural actions. Example applications show that multi user speech and gestures can be used to interacti with existing single user applications such as [[http://edwardhtse.googlepages.com/edwardtse-videos.html|Google Earth, Warcraft III, and The Sims]]. These actions serve dual purpose as commands to the computer and as awareness for other collaborators (e.g., "Fly to Calgary").

Recently, tools have been developed to dramatically simplify the development of
speech and gesture enabled applications in two ways: first, wrappers can be built to allow speech and gesture actions to be converted to mouse and keyboard events on an existing single user application. Second, custom speech and gesture applications can be built from the ground up (e.g., [[http://grouplab.cpsc.ucalgary.ca/cookbook/index.php?n=Toolkits.BlendedLenses|GSI Colour Blender]]) However, there are a limited number of applications in this area, you're responsibility will be to try to develop applications that explore multi user speech and gesture interaction over digital tables.
May 09, 2007, at 05:14 PM by 24.64.76.194 -
Changed lines 2-3 from:
'-Return to [[http://pages.cpsc.ucalgary.ca/~saul/wiki/pmwiki.php/Personal/IdeaSketches | Idea Sketches]]-'
to:
'-Return to [[http://pages.cpsc.ucalgary.ca/~saul/wiki/pmwiki.php/Personal/IdeaSketches | Idea Sketches]]-'
May 09, 2007, at 05:12 PM by 24.64.76.194 -
Deleted lines 0-1:
[[IdeaSketches|Back to Idea Sketches]]
Changed lines 2-4 from:
to:
'-Return to [[http://pages.cpsc.ucalgary.ca/~saul/wiki/pmwiki.php/Personal/IdeaSketches | Idea Sketches]]-'
May 09, 2007, at 05:09 PM by 136.159.7.55 -
Changed lines 36-37 from:
GSI Demo is a toolkit that supports multi user speech and gesture interaction over large digital displays [[http://grouplab.cpsc.ucalgary.ca/cookbook/index.php?n=Toolkits.GSIDemo|Link]]. Also check out the [[http://grouplab.cpsc.ucalgary.ca/cookbook/index.php?n=Toolkits.SDGToolkit|SDG Toolkit]] for multiple mice support. Contact [[http://www.edwardtse.com|Ed Tse]] if you'd like to learn more about using GSI Demo or SDG Toolkit.
to:
[[http://grouplab.cpsc.ucalgary.ca/cookbook/index.php?n=Toolkits.GSIDemo|GSI Demo]] is a toolkit that supports multi user speech and gesture interaction over large digital displays . Also check out the [[http://grouplab.cpsc.ucalgary.ca/cookbook/index.php?n=Toolkits.SDGToolkit|SDG Toolkit]] for multiple mice support. Contact [[http://www.edwardtse.com|Ed Tse]] if you'd like to learn more about using GSI Demo or SDG Toolkit.
May 09, 2007, at 05:03 PM by 136.159.7.55 -
Changed lines 36-37 from:
GSI Demo is a toolkit that supports multi user speech and gesture interaction over large digital displays. [[http://grouplab.cpsc.ucalgary.ca/cookbook/index.php?n=Toolkits.GSIDemo|Link]] also check out the [[http://grouplab.cpsc.ucalgary.ca/cookbook/index.php?n=Toolkits.SDGToolkit|SDG Toolkit]] for multiple mice support. Contact [[http://www.edwardtse.com|Ed Tse]] if you'd like to learn more about using GSI Demo or SDG Toolkit.
to:
GSI Demo is a toolkit that supports multi user speech and gesture interaction over large digital displays [[http://grouplab.cpsc.ucalgary.ca/cookbook/index.php?n=Toolkits.GSIDemo|Link]]. Also check out the [[http://grouplab.cpsc.ucalgary.ca/cookbook/index.php?n=Toolkits.SDGToolkit|SDG Toolkit]] for multiple mice support. Contact [[http://www.edwardtse.com|Ed Tse]] if you'd like to learn more about using GSI Demo or SDG Toolkit.
May 09, 2007, at 05:03 PM by 136.159.7.55 -
Changed lines 36-37 from:
GSI Demo is a toolkit that supports multi user speech and gesture interaction over large digital displays. Contact Ed Tse if you'd like to learn more about using GSI Demo in practice.
to:
GSI Demo is a toolkit that supports multi user speech and gesture interaction over large digital displays. [[http://grouplab.cpsc.ucalgary.ca/cookbook/index.php?n=Toolkits.GSIDemo|Link]] also check out the [[http://grouplab.cpsc.ucalgary.ca/cookbook/index.php?n=Toolkits.SDGToolkit|SDG Toolkit]] for multiple mice support. Contact [[http://www.edwardtse.com|Ed Tse]] if you'd like to learn more about using GSI Demo or SDG Toolkit.
May 09, 2007, at 05:01 PM by 136.159.7.55 -
Deleted lines 14-16:
In this project, you will have the opportunity to explore speech and gesture wrappers over existing single user applications and building your very own multi user applications.
A location-dependan object controller (yes, its a horrible name)is an untethered mobile device with a reasonable size screen (e.g., tablet, or a PDA). If a person approaches a 'controllable' object (it could be digital or real world), the controller senses that object (e.g., through RFID or bar codes) and links to a view of that object. The person can then see information related to that object, and perhaps even control its properties.
May 09, 2007, at 04:58 PM by 136.159.7.55 -
Changed line 31 from:
%width=600px% Attach:LargeWallInteraction.jpg
to:
%width=512px% Attach:LargeWallInteraction.jpg
May 09, 2007, at 04:57 PM by 136.159.7.55 -
Changed line 31 from:
%width=800px% Attach:LargeWallInteraction.jpg
to:
%width=600px% Attach:LargeWallInteraction.jpg
May 09, 2007, at 04:54 PM by 136.159.7.55 -
Changed line 31 from:
Attach:LargeWallInteraction.jpg
to:
%width=800px% Attach:LargeWallInteraction.jpg
May 09, 2007, at 04:54 PM by 136.159.7.55 -
Changed lines 31-32 from:
Attach:LargeWallInteraction.jpg Sometimes on a large digital wall it is exceedingly difficult to interact with items at a distance. We have some software that allows graphics to be drawn on a large digital wall display. You could develop applications to perform tasks or games over the large display
to:
Attach:LargeWallInteraction.jpg
Sometimes on a large digital wall it is exceedingly difficult to interact with items at a distance. We have some software that allows graphics to be drawn on a large digital wall display. You could develop applications to perform tasks or games over the large display
Changed line 35 from:
Attach:gsistudytranscripter.jpg
to:
%width=400px% Attach:gsistudytranscripter.jpg
May 09, 2007, at 04:53 PM by 136.159.7.55 -
Changed lines 31-32 from:
Sometimes on a large digital wall it is exceedingly difficult to interact with items at a distance. We have some software that allows graphics to be drawn on a large digital wall display. You could develop applications to perform tasks or games over the large display
to:
Attach:LargeWallInteraction.jpg Sometimes on a large digital wall it is exceedingly difficult to interact with items at a distance. We have some software that allows graphics to be drawn on a large digital wall display. You could develop applications to perform tasks or games over the large display
Changed line 34 from:
gsistudytranscripter.jpg
to:
Attach:gsistudytranscripter.jpg
May 09, 2007, at 04:51 PM by 136.159.7.55 -
Changed line 34 from:
link:GsiStudyTranscripter.jpg
to:
gsistudytranscripter.jpg
May 09, 2007, at 04:07 PM by 136.159.7.55 -
Changed line 20 from:
%width=300px% http://www.news.navy.mil/management/photodb/webphoto/web_050316-N-7405P-111.jpg"Air Traffic Control"
to:
http://www.news.navy.mil/management/photodb/webphoto/web_050316-N-7405P-111.jpg"Air Traffic Control"
May 09, 2007, at 04:05 PM by 136.159.7.55 -
Changed line 34 from:
[[GsiStudyTranscripter.jpg]]
to:
link:GsiStudyTranscripter.jpg
May 09, 2007, at 04:04 PM by 136.159.7.55 -
Changed line 34 from:
GsiStudyTranscripter.jpg
to:
[[GsiStudyTranscripter.jpg]]
May 09, 2007, at 04:04 PM by 136.159.7.55 -
Changed line 20 from:
http://www.news.navy.mil/management/photodb/webphoto/web_050316-N-7405P-111.jpg"Air Traffic Control"
to:
%width=300px% http://www.news.navy.mil/management/photodb/webphoto/web_050316-N-7405P-111.jpg"Air Traffic Control"
Added line 24:
http://diamondspace.merl.com/images/papers/2006_forlines_adapting_single_user_geospatial_application.png
Added line 34:
GsiStudyTranscripter.jpg
May 09, 2007, at 03:59 PM by 136.159.7.55 -
Changed lines 23-41 from:
!!!Location-dependand displays
Similarly, Katherine Elliot's location dependant devices
can be viewed in further detail. Currently, one needs to swipe an RFID card to assign a function to these devices. But our controller could do this in a richer way, and perhaps give people further options as to how information (and what information) is assigned to these devices.

!!!Universal controller
There are already many devices that we control by remotes: televisions, cd players, ipods, dvd players, etc. The display could fuse these into a single control by being aware of what appliances are in a room.
Brad Myers did some work on Universal controllers
.

!!!Purchasing
If you approach a vending machine, you can buy things through your controller. This is already done with cell phones (often very badly!); the interface is often terrible due to the cell phone interface (many menus / buttons).

!!!Home inspection
We have many warning lights that tell us when things go right or wrong
. These are often presented as crypted LEDs (a green flashing may mean ok, but time for a checkup) or uninformative messages in cars (in my Suburu, there is a 'check engine' message; however, I don't know if its a serious problem or not). The controller can present this information in a much more meaningful way, and perhaps give me some options of what to do about it (e.g., connect to google and find local service people)

!!!Consumer information
Given a product, find out more information about it .e.g, when shopping (Marc Smith did this; get reference).

and on
and on.
to:
!!!Multi Display Interaction
Clifton Forlines explored how multiple displays
can be used for interaction with Digital Tabletop displays. Imagine exploring multiple display applications using speech and gestures. Some modern games also provide multiple display support (e.g., [[http://www.supremecommander.com/|Supreme Commander]])

!!!Are you talking to me or the computer?
Always on speech recognition often causes errors during regular conversation. It is hard to know when someone is talking to the computer vs. talking to another person. One solution would be to track where people are looking. [[http://www.naturalpoint.com/trackir/|Natural Point]] sells a cheap head tracking device that can be used to find the x,y,z position of a person's head
. This can be used for gaze tracking and might be useful for trying to find out when someone is looking at another person vs. looking at the screen. Might be interesting to run a study to see if this is actually effective or not.

!!!Interaction on the Digital Wall
Sometimes on a large digital wall it is exceedingly difficult to interact with items at a distance. We have some software that allows graphics to be drawn on a large digital wall display. You could develop applications to perform tasks or games over the large display

!!!Visualization of Multi User Input
While existing systems such as camtasia can synchronize mouse and keyboard actions with a live video there are few tools that allow this to be viewed for multiple users simultaneously. You could develop novel visualizations to show the speech
and gesture actions of multiple concurrent users on a digital table.
Changed lines 36-39 from:
Brad Myers describes an XML-based method to show and communicate relevant information between devices, where the device tries to generate an interface from it. An early version of VNC (Get Reference) actually had devices use the VNC protocol to generate a richer interface. Perhaps a better option is to have each device be associated with a web handle, where it publishes / subscribes to information in that handle (e.g., shared dictionary). Associated with that handle is also a program - perhaps a java applet - that can be downloaded automatically to the tablet.

We would implement location awareness through RFID tags, with the reader taped onto the tablet (Nic did this with his Shared Phidgets project)
.
to:
GSI Demo is a toolkit that supports multi user speech and gesture interaction over large digital displays. Contact Ed Tse if you'd like to learn more about using GSI Demo in practice.
Changed lines 39-42 from:
* flavours of this have been done before, but I am not sure if its been done as comprehensively as suggested here. Need to gather the background research. Rob Diaz started this once...
* Some devices we would want to control do not have network capability, e.g., my home thermostat, my stove, my car. We would have to simulate this (or make our own appliances using phidgets )
to:
* Speech recognition is not perfect, there are lots of recognition errors that occur during everyday use.
May 09, 2007, at 03:41 PM by 136.159.7.55 -
Changed line 20 from:
Image:http://www.news.navy.mil/management/photodb/webphoto/web_050316-N-7405P-111.jpg"Air Traffic Control"
to:
http://www.news.navy.mil/management/photodb/webphoto/web_050316-N-7405P-111.jpg"Air Traffic Control"
May 09, 2007, at 03:41 PM by 136.159.7.55 -
Changed lines 5-6 from:
[[Image:http://www.news.navy.mil/management/photodb/webphoto/web_050316-N-7405P-111.jpg]]
to:
Changed lines 20-21 from:
Often Air Traffic Controllers need to work together to direct air traffic. You're task would be to create a game that
to:
Image:http://www.news.navy.mil/management/photodb/webphoto/web_050316-N-7405P-111.jpg"Air Traffic Control"
Here
's an image of air traffic control in the real world. Perhaps you could think of how this might work in a digital world using speech and gesture commands.
May 09, 2007, at 03:38 PM by 136.159.7.55 -
Changed lines 1-3 from:
(:title Location-dependant Object controller :)
This is an idea I had several years ago, but had trouble finding a student to take it on
.
to:
[[IdeaSketches|Back to Idea Sketches]]

(:title Multi User Speech and Gesture Interaction :)

[[Image:http://www
.news.navy.mil/management/photodb/webphoto/web_050316-N-7405P-111.jpg]]
Changed lines 8-9 from:
A location-dependan object controller (yes, its a horrible name)is an untethered mobile device with a reasonable size screen (e.g., tablet, or a PDA). If a person approaches a 'controllable' object (it could be digital or real world), the controller senses that object (e.g., through RFID or bar codes) and links to a view of that object. The person can then see information related to that object, and perhaps even control its properties.
to:
You'll have an opportunity to explore multi user speech and gesture interaction over digital walls and tables.
Changed lines 11-12 from:
Many devices in our real world are small - too small to provide a reasonable view into the information it may contain. As well, interactions with that device are often compromised due to cost, size etc (think of, for example, your digital watch, your home thermostat, an ambient display, etc.). Why not bring larger screens with better viewing and interaction facilities to the device?
to:
When multiple people are interacting over a shared digital tabletop, they often communicate with others using many speech and gestural actions. Example applications show multi user interaction with [[http://edwardhtse.googlepages.com/edwardtse-videos.html|Google Earth, Warcraft III, and The Sims]].

Recently, toolkits have been developed to dramatically simplify the development of speech and gesture enabled applications in two ways: first, wrappers can be built
to allow speech and gesture actions to be converted to mouse and keyboard events on an existing single user application. Second, custom speech and gesture applications can be built from the ground up (e.g., Blended Lenses, an Air Traffic Control game, be creative!)

In this project, you will have the opportunity to explore speech and gesture wrappers over existing single user applications and building your very own multi user applications.
A location-dependan object controller (yes, its a horrible name)is an untethered mobile
device with a reasonable size screen (e.g., tablet, or a PDA). If a person approaches a 'controllable' object (it could be digital or real world), the controller senses that object (e.g., through RFID or bar codes) and links to a view of that object. The person can then see information related to that object, and perhaps even control its properties.
Changed lines 19-21 from:
!!!Ambient displays
The purpose behind most ambient displays is
to provide awareness information about something. These devices are usually located in some context amenable to that dispaly e.g., so it is easily seen as people walk by. The catch is that it is sometimes difficult to move from awareness to exploration and even interaction with that information. For example, imagine we had a figurine (connected to an Instant Messenging system) that lights up to different degrees when a particular person is online. While we know that that person is there, actually moving into conversation requires a much more complex interface. Instead, we would approach the figuring with our controller, and the controller would immediately display more information about that person, the ability to chat with them, and also the ability to reassign who that figurine represents. A version of this is presented in a video (see Harrison's work; look up reference).
to:
!!!Air Traffic Control Game
Often Air Traffic Controllers need
to work together to direct air traffic. You're task would be to create a game that
May 09, 2007, at 03:28 PM by 136.159.7.55 -
Added lines 1-42:
(:title Location-dependant Object controller :)
This is an idea I had several years ago, but had trouble finding a student to take it on.

!!Basic premise.
A location-dependan object controller (yes, its a horrible name)is an untethered mobile device with a reasonable size screen (e.g., tablet, or a PDA). If a person approaches a 'controllable' object (it could be digital or real world), the controller senses that object (e.g., through RFID or bar codes) and links to a view of that object. The person can then see information related to that object, and perhaps even control its properties.

!!Motivation
Many devices in our real world are small - too small to provide a reasonable view into the information it may contain. As well, interactions with that device are often compromised due to cost, size etc (think of, for example, your digital watch, your home thermostat, an ambient display, etc.). Why not bring larger screens with better viewing and interaction facilities to the device?

!!Examples
!!!Ambient displays
The purpose behind most ambient displays is to provide awareness information about something. These devices are usually located in some context amenable to that dispaly e.g., so it is easily seen as people walk by. The catch is that it is sometimes difficult to move from awareness to exploration and even interaction with that information. For example, imagine we had a figurine (connected to an Instant Messenging system) that lights up to different degrees when a particular person is online. While we know that that person is there, actually moving into conversation requires a much more complex interface. Instead, we would approach the figuring with our controller, and the controller would immediately display more information about that person, the ability to chat with them, and also the ability to reassign who that figurine represents. A version of this is presented in a video (see Harrison's work; look up reference).

!!!Location-dependand displays
Similarly, Katherine Elliot's location dependant devices can be viewed in further detail. Currently, one needs to swipe an RFID card to assign a function to these devices. But our controller could do this in a richer way, and perhaps give people further options as to how information (and what information) is assigned to these devices.

!!!Universal controller
There are already many devices that we control by remotes: televisions, cd players, ipods, dvd players, etc. The display could fuse these into a single control by being aware of what appliances are in a room.
Brad Myers did some work on Universal controllers.

!!!Purchasing
If you approach a vending machine, you can buy things through your controller. This is already done with cell phones (often very badly!); the interface is often terrible due to the cell phone interface (many menus / buttons).

!!!Home inspection
We have many warning lights that tell us when things go right or wrong. These are often presented as crypted LEDs (a green flashing may mean ok, but time for a checkup) or uninformative messages in cars (in my Suburu, there is a 'check engine' message; however, I don't know if its a serious problem or not). The controller can present this information in a much more meaningful way, and perhaps give me some options of what to do about it (e.g., connect to google and find local service people)

!!!Consumer information
Given a product, find out more information about it .e.g, when shopping (Marc Smith did this; get reference).

and on and on.


!!Implementation
Brad Myers describes an XML-based method to show and communicate relevant information between devices, where the device tries to generate an interface from it. An early version of VNC (Get Reference) actually had devices use the VNC protocol to generate a richer interface. Perhaps a better option is to have each device be associated with a web handle, where it publishes / subscribes to information in that handle (e.g., shared dictionary). Associated with that handle is also a program - perhaps a java applet - that can be downloaded automatically to the tablet.

We would implement location awareness through RFID tags, with the reader taped onto the tablet (Nic did this with his Shared Phidgets project).

!!Risk
* flavours of this have been done before, but I am not sure if its been done as comprehensively as suggested here. Need to gather the background research. Rob Diaz started this once...
* Some devices we would want to control do not have network capability, e.g., my home thermostat, my stove, my car. We would have to simulate this (or make our own appliances using phidgets )