(:quicktime http://pages.cpsc.ucalgary.ca/~boyd/gait/g53.mpg width=320 height=180 playeveryframe=true autoplay=false :)

  (:quicktime http://pages.cpsc.ucalgary.ca/~boyd/gait/g53u.mpg width=320 height=180 playeveryframe=true autoplay=false :)\\

  (:quicktime http://pages.cpsc.ucalgary.ca/~boyd/gait/g53v.mpg width=320 height=180 playeveryframe=true  autoplay=false :)\\

  (:quicktime http://pages.cpsc.ucalgary.ca/~boyd/gait/g53mag.mpg width=320 height=180 playeveryframe=true autoplay=false :)\\

  (:quicktime http://pages.cpsc.ucalgary.ca/~boyd/gait/g53blob.mpg width=320 height=180 playeveryframe=true autoplay=false :)\\

  (:quicktime http://pages.cpsc.ucalgary.ca/~boyd/gait/g53scl.mpg width=320 height=180 playeveryframe=true autoplay=false :) \\

• (:quicktime http://pages.cpsc.ucalgary.ca/~boyd/gait/g13.mpg width=320 height=180 playeveryframe=true autoplay=false :)\\

• (:quicktime http://pages.cpsc.ucalgary.ca/~boyd/gait/g23.mpg width=320 height=180 playeveryframe=true autoplay=false :)\\

• (:quicktime http://pages.cpsc.ucalgary.ca/~boyd/gait/g33.mpg width=320 height=180 playeveryframe=true autoplay=false :)\\

• (:quicktime http://pages.cpsc.ucalgary.ca/~boyd/gait/g43.mpg width=320 height=180 playeveryframe=true autoplay=false :)\\

• (:quicktime http://pages.cpsc.ucalgary.ca/~boyd/gait/g53.mpg width=320 height=180 playeveryframe=true autoplay=false :)\\

• (:quicktime http://pages.cpsc.ucalgary.ca/~boyd/gait/g63.mpg width=320 height=180 playeveryframe=true autoplay=false :)\\

• (:quicktime http://pages.cpsc.ucalgary.ca/~boyd/gait/g13.mpg width=320 height=180 playeveryframe=true :)
  Person #1
• (:quicktime http://pages.cpsc.ucalgary.ca/~boyd/gait/g23.mpg width=320 height=180 playeveryframe=true :)
  Person #2
• (:quicktime http://pages.cpsc.ucalgary.ca/~boyd/gait/g33.mpg width=320 height=180 playeveryframe=true :)
  Person #3
• (:quicktime http://pages.cpsc.ucalgary.ca/~boyd/gait/g43.mpg width=320 height=180 playeveryframe=true :)
  Person #4
• (:quicktime http://pages.cpsc.ucalgary.ca/~boyd/gait/g53.mpg width=320 height=180 playeveryframe=true :)
  Person #5
• (:quicktime http://pages.cpsc.ucalgary.ca/~boyd/gait/g63.mpg width=320 height=180 playeveryframe=true :)
  Person #6

  (:quicktime http://pages.cpsc.ucalgary.ca/~boyd/gait/g53blob.mpg width=320 height=180 playeveryframe=true :)
  blobs for sample sequence. 

  (:quicktime http://pages.cpsc.ucalgary.ca/~boyd/gait/g53scl.mpg width=320 height=180 playeveryframe=true :) 
  'scalar sequence

  (:quicktime http://pages.cpsc.ucalgary.ca/~boyd/gait/g53u.mpg width=320 height=180 playeveryframe=true :)\\

  (:quicktime http://pages.cpsc.ucalgary.ca/~boyd/gait/g53v.mpg width=320 height=180 playeveryframe=true :)\\

  (:quicktime http://pages.cpsc.ucalgary.ca/~boyd/gait/g53mag.mpg width=320 height=180 playeveryframe=true :)\\

  (:quicktime http://pages.cpsc.ucalgary.ca/~boyd/gait/g53.mpg width=320 height=180 playeveryframe=true :)

  (:quicktime http://pages.cpsc.ucalgary.ca/~boyd/gait/g53.mpg width=320 height=180 playeveryframe="true" :)

  (:quicktime http://pages.cpsc.ucalgary.ca/~boyd/gait/g53u.mpg width=320 height=180 playeveryframe="true" :)\\

  (:quicktime http://pages.cpsc.ucalgary.ca/~boyd/gait/g53v.mpg width=320 height=180 playeveryframe="true" :)\\

  (:quicktime http://pages.cpsc.ucalgary.ca/~boyd/gait/g53mag.mpg width=320 height=180 playeveryframe="true" :)\\

  (:quicktime http://pages.cpsc.ucalgary.ca/~boyd/gait/g53u.mpg width=320 height=180 :)
  u

  (:quicktime http://pages.cpsc.ucalgary.ca/~boyd/gait/g53v.mpg width=320 height=180 :)
  v

  (:quicktime http://pages.cpsc.ucalgary.ca/~boyd/gait/g53mag.mpg width=320 height=180 :)
  magnitude

  (:quicktime http://pages.cpsc.ucalgary.ca/~boyd/gait/g53.mpg width=320 height=180 :)

  (:quicktime http://pages.cpsc.ucalgary.ca/~boyd/gait/g53.mpg

1. For each frame of the flow, we compute a set of scalars (see source code) that characterize the shape of the flow in that frame. Examples include x and y coordinates of the moving region (centx and centy), and the aspect ratio of the moving region (aspct).

  || Frequency Spectrum of aspct (20 coefficients) ||  aspct with Sinusoid of Fitted Frequency and Phase ||

http://pages.cpsc.ucalgary.ca/~boyd/gait/features.gif | Feature Vector Plot

  || Raw Time Series for centx, centy, aspct  ||  centx, centy, aspct with Linear Background Removed ||

  || http://pages.cpsc.ucalgary.ca/~boyd/gait/signals1.gif || http://pages.cpsc.ucalgary.ca/~boyd/gait/signals2.gif ||

  || http://pages.cpsc.ucalgary.ca/~boyd/gait/signals3.gif || http://pages.cpsc.ucalgary.ca/~boyd/gait/signals4.gif ||

http://pages.cpsc.ucalgary.ca/~boyd/gait/features.gif | Feature Vector Plot

We tested recognition using a variety of algorithms. The best results were obtained by finding the nearest neighbor from a set of exemplars, vectors of mean feature values for each subject. To get an unbiased estimate of the recognition rate we used a leave-one-out procedure. Using the full feature vector gives a recognition rate of about 90%. If we use only the four best features, daspct, dcenty, vwcenty, and waspct, then the recognition rate went as high as 95%. In comparison, recognition by chance would yield a rate of about 17%.

daspct > dcenty > wcenty > vwcenty > aspct > vwcentx > waspct > centx > uwcentx > wcentx > uwcenty > dcentx

Stereo 3-D Scatter Plot of aspct versus centx and wcentx

The following scatter plots show the features with the greatest variation, centx, wcentx, and aspct:

http://pages.cpsc.ucalgary.ca/~boyd/gait/scatter.gif | Scatter Plot of aspct versus centx

Stereo 3-D Scatter Plot of aspct versus centx and wcentx

centy (y coordinate of the centroid of moving region) is used only as the phase reference signal.

• All features showed significant variations between people.
• The features showing the greatest variation were aspct, dcenty, wcenty, and vwcenty.

1. centx: x coordinate of centroid of moving region
2. wcentx: x coordinate of centroid of moving region weighted by |(u,v)|
3. wcenty: y coordinate of centroid of moving region weighted by |(u,v)|
4. dcentx: wcentx - centx
5. dcenty: wcenty - centy
6. aspct: aspect ratio of moving region
7. waspct: aspect of ratio of moving region weighted by |(u,v)|
8. daspct: aspct - waspct
9. uwcentx: x coordinate of centroid of moving region weighted by |u|
10. uwcenty: y coordinate of centroid of moving region weighted by |u|
11. vwcentx: x coordinate of centroid of moving region weighted by |v|
12. vwcenty: y coordinate of centroid of moving region weighted by |v|

Results

We analyzed the following scalars and their phase features:

1. "centx": x coordinate of centroid of moving region
2. "wcentx": x coordinate of centroid of moving region weighted by |(u,v)|
3. "wcenty": y coordinate of centroid of moving region weighted by |(u,v)|
4. "dcentx": wcentx - centx
5. "dcenty": wcenty - centy
6. "aspct": aspect ratio of moving region
7. "waspct": aspect of ratio of moving region weighted by |(u,v)|
8. "daspct": aspct - waspct
9. "uwcentx": x coordinate of centroid of moving region weighted by |u|
10. "uwcenty": y coordinate of centroid of moving region weighted by |u|
11. "vwcentx": x coordinate of centroid of moving region weighted by |v|
12. "vwcenty": y coordinate of centroid of moving region weighted by |v|

centy (y coordinate of the centroid of moving region) is used only as the phase reference signal.

Analysis of variance, including post hoc testing, indicated the following:

    * All features showed significant variations between people.
    * The features showing the greatest variation were aspct, dcenty, wcenty, and vwcenty. 

The following scatter plots show the features with the greatest variation, centx, wcentx, and aspct:

Scatter Plot of aspct versus centx

Stereo 3-D Scatter Plot of aspct versus centx and wcentx

We tested recognition using a variety of algorithms. The best results were obtained by finding the nearest neighbor from a set of exemplars, vectors of mean feature values for each subject. To get an unbiased estimate of the recognition rate we used a leave-one-out procedure. Using the full feature vector gives a recognition rate of about 90%. If we use only the four best features, daspct, dcenty, vwcenty, and waspct, then the recognition rate went as high as 95%. In comparison, recognition by chance would yield a rate of about 17%.

Recognition was possible for a variety of flow sources varying in spatial resolution. We observed that while the exemplars changed and the features with the greatest variation changed, as long as the parameters for flow computation were kept constant, recognition is successful.

Analysis of variance and our recognition test show that the features have the following approximate significance for recognition: daspct > dcenty > wcenty > vwcenty > aspct > vwcentx > waspct > centx > uwcentx > wcentx > uwcenty > dcentx

  || Raw Time Series for centx, centy, aspct || centx, centy, aspct with Linear Background Removed ||

  || Frequency Spectrum of aspct (20 coefficients) || aspct with Sinusoid of Fitted Frequency and Phase ||

http://pages.cpsc.ucalgary.ca/~boyd/images/som-blckdiag.gif | Data-Flow Diagram

http://pages.cpsc.ucalgary.ca/~boyd/gait/apparatus.gif | Experimental Apparatus

http://pages.cpsc.ucalgary.ca/~boyd/images/som-blckdiag.gif | Data-Flow Diagram

http://pages.cpsc.ucalgary.ca/~boyd/gait/apparatus.gif | Experimental Apparatus

Experiment

To verify that the system is capable of recognition, we sampled the gaits of six people using the apparatus depicted below.

Experimental Apparatus

http://pages.cpsc.ucalgary.ca/~boyd/gait/apparatus.gif

A camera fixed on a tripod points towards a fixed non-reflecting static background. Subjects walk in a circular path such that on one side of the path they pass through the field of view of the camera and pass behind the camera on the other side. Only one subject is in the field of view at any one time. The subjects walk this path for about 15 minutes while the images are recorded on video tape.

Later, we digitize sequences for the six subjects. We discard the first two or three passes for each person and digitize seven sequences for each subject (42 sequences total).

Images from the tape are digitized in 24-bit color at a resolution of 640 by 480 pixels. We resample and crop the images to get black and white images with 320 by 160 pixels.

Click to view an example sequence for each of the six subjects:

• Person #1
• Person #2
• Person #3
• Person #4
• Person #5
• Person #6

The complete data set.

  
  ||border=0
  || http://pages.cpsc.ucalgary.ca/~boyd/gait/signals1.gif || http://pages.cpsc.ucalgary.ca/~boyd/gait/signals2.gif ||
  || raw time series for centx, centy, aspct || centx, centy, aspct with linear background removed ||
  

  
  ||border=0
  || http://pages.cpsc.ucalgary.ca/~boyd/gait/signals3.gif || http://pages.cpsc.ucalgary.ca/~boyd/gait/signals4.gif ||
  || frequency spectrum of aspct (20 coefficients) || aspct with sinusoid of fitted frequency and phase ||
  

1. Phase values are arbitrary and depend on the point in the gait at which the image sequence begins. We select a single signal to be a reference for the others. Phases, with the reference phase subtracted, are features.
2. Selected features combine to form a feature vector for recognition. The plot shows the average feature values and the maximum and minimum values for each subject in our experiment.

http://pages.cpsc.ucalgary.ca/~boyd/gait/features.gif | feature vector plot

The entire process is controlled from a shell script that executes programs to compute the optical flow, calculate the scalar shape descriptions, and analyze the signals for frequency and phase.

• frequency spectrum of aspct (20 coefficients)

http://pages.cpsc.ucalgary.ca/~boyd/gait/signals1.gif http://pages.cpsc.ucalgary.ca/~boyd/gait/signals2.gif

• scalar sequence

1. For each frame of the flow, we compute a set of scalars (see source code) that characterize the shape of the flow in that frame. Examples include x and y coordinates of the moving region (centx and centy), and the aspect ratio of the moving region (aspct).
   View the cropped motion sequence with the the position of (centx,centy) shown as a "+", the |(u,v)|-weighted centroid (wcentx,wcenty) shown as a box, a solid ellipse corresponding to aspct, and a dashed ellipse corresponding to the aspect ratio of the |(u,v)|-weighted moving region, waspct.
   • scalar sequence
   Note how the weighted ellipse (particulary senstive to fast-moving parts of the body) follows the motion of the feet as they accelerate in the gait.

  View plots of some of the signals for the example sequence:

• View MPEG of blobs for sample sequence.

• v,
• |(u,v)|.

(:quicktime http://pages.cpsc.ucalgary.ca/~boyd/gait/g53.mpg:)

1. We compute the optical flow (Bulthoff, Little, and Poggio - source code) of the motion sequence to get n frames of (u,v) data, where u is the x-direction flow and v is the y-direction flow.

• u,

  View MPEGs of flow for sample sequence:

  In the u and v images, red indicates negative flow while green indicates positive. To save computation time, we manually tracked each subject and computed flow only in a box surrounding the person. You can see artifacts from this box in the MPEG images of the flow sequences.

View the cropped motion sequence with the the position of (centx,centy) shown as a "+", the |(u,v)|-weighted centroid (wcentx,wcenty) shown as a box, a solid ellipse corresponding to aspct, and a dashed ellipse corresponding to the aspect ratio of the |(u,v)|-weighted moving region, waspct.

• scalar sequence

Note how the weighted ellipse (particulary senstive to fast-moving parts of the body) follows the motion of the feet as they accelerate in the gait.

View plots of some of the signals for the example sequence:

• raw time series for centx, centy, aspct
• centx, centy, aspct with linear background removed

• frequency spectrum of aspct (20 coefficients)
• aspct with sinusoid of fitted frequency and phase

• feature vector plot

Experiment: an experiment that demonstrates the system

Results: results of the experiment showing successful recognition

//

• |(u,v)|.

//

http://pages.cpsc.ucalgary.ca/~boyd/images/som-blckdiag.gif | Data-Flow Diagram

How the System Works

1. The system begins with a motion sequence of n+1 frames of a person walking.
   • View an MPEG of sample sequence.
2. We compute the optical flow (Bulthoff, Little, and Poggio - source code) of the motion sequence to get n frames of (u,v) data, where u is the x-direction flow and v is the y-direction flow.

View MPEGs of flow for sample sequence:

• u,
• v,
• |(u,v)|.

In the u and v images, red indicates negative flow while green indicates positive. To save computation time, we manually tracked each subject and computed flow only in a box surrounding the person. You can see artifacts from this box in the MPEG images of the flow sequences.

1. Spurious motion in the image background can confound recognition. The aspect ratio signals are particularly sensitive to outliers. To eliminate as many outliers as possible, we compute connected components of regions that are moving and discard the small components. The resulting moving "blobs" have less spurious motion.

• View MPEG of blobs for sample sequence.

1. For each frame of the flow, we compute a set of scalars (see source code) that characterize the shape of the flow in that frame. Examples include x and y coordinates of the moving region (centx and centy), and the aspect ratio of the moving region (aspct).

1. We rearrange the scalars to form one time series for each scalar.

1. The time series are sinusoidal and have a common frequency. We use least-squares linear prediction spectrum analysis (Barrodale and Erikson's method - source code) to find the fundamental frequency and phase for each series. A Matlab example that demonstrates the advantages of this method is available.

1. Phase values are arbitrary and depend on the point in the gait at which the image sequence begins. We select a single signal to be a reference for the others. Phases, with the reference phase subtracted, are features.
2. Selected features combine to form a feature vector for recognition. The plot shows the average feature values and the maximum and minimum values for each subject in our experiment.

• feature vector plot

@]

The Shape of Motion

We have developed a novel vision system that can recognize people by the way they walk. The system computes optical flow for an image sequence of a person walking, and then characterizes the shape of the motion with a set of sinusoidally-varying scalars. Feature vectors composed of the phases of the sinusoids are able to discriminate among people.

System Description

http://pages.cpsc.ucalgary.ca/~boyd/images/som-blckdiag.gif | Data-Flow Diagram

The Shape of Motin

Overview

Our goal is to describe the motion of a moving human figure in order to recognize individuals by variation in the characteristics of the motion description. We begin with a short sequence of images of a moving figure, taken by a static camera, and derive dense optical flow data, (u(x,y),v(x,y)), for the sequence. We determine a range of scale-independent scalar features of each flow image that characterize the spatial distribution of the flow, i.e., the shape of the motion. The scalars are based on various moments of the set of moving points. To characterize the shape of the motion, not the shape of moving points, the points are weighted by |u|, |v|, or |(u,v)|.

We then analyze the periodic structure of these sequences of scalars. All of these sequences share the same fundamental period of the gait, but they differ in phase. Although there are several regularities in the relative phases of the signals, some of the phases show significant statistical variation. Therefore, we are able to use vectors of phase measurements derived for each image sequence to recognize individuals by the shape of their motion.

The representation is model-free, and therefore could be used to characterize the motion of other non-rigid bodies.

System Description

Data-Flow Diagram How the System Works:

As shown in the data-flow diagram (above), the steps in the system are, from top to bottom:

   1. The system begins with a motion sequence of n+1 frames of a person walking.
          * View an MPEG of sample sequence. 

   2. We compute the optical flow (Bulthoff, Little, and Poggio - source code) of the motion sequence to get n frames of (u,v) data, where u is the x-direction flow and v is the y-direction flow.

      View MPEGs of flow for sample sequence:
          * u,
          * v,
          * |(u,v)|. 

      In the u and v images, red indicates negative flow while green indicates positive. To save computation time, we manually tracked each subject and computed flow only in a box surrounding the person. You can see artifacts from this box in the MPEG images of the flow sequences.
   3. Spurious motion in the image background can confound recognition. The aspect ratio signals are particularly sensitive to outliers. To eliminate as many outliers as possible, we compute connected components of regions that are moving and discard the small components. The resulting moving "blobs" have less spurious motion.
          * View MPEG of blobs for sample sequence. 

   4. For each frame of the flow, we compute a set of scalars (see source code) that characterize the shape of the flow in that frame. Examples include x and y coordinates of the moving region (centx and centy), and the aspect ratio of the moving region (aspct).

      View the cropped motion sequence with the the position of (centx,centy) shown as a "+", the |(u,v)|-weighted centroid (wcentx,wcenty) shown as a box, a solid ellipse corresponding to aspct, and a dashed ellipse corresponding to the aspect ratio of the |(u,v)|-weighted moving region, waspct.
          * scalar sequence 
      Note how the weighted ellipse (particulary senstive to fast-moving parts of the body) follows the motion of the feet as they accelerate in the gait.

   5. We rearrange the scalars to form one time series for each scalar.

      View plots of some of the signals for the example sequence:
          * raw time series for centx, centy, aspct
          * centx, centy, aspct with linear background removed 

   6. The time series are sinusoidal and have a common frequency. We use least-squares linear prediction spectrum analysis (Barrodale and Erikson's method - source code) to find the fundamental frequency and phase for each series. A Matlab example that demonstrates the advantages of this method is available.

      View plots of the spectrum analysis for the example sequence:
          * frequency spectrum of aspct (20 coefficients)
          * aspct with sinusoid of fitted frequency and phase 

   7. Phase values are arbitrary and depend on the point in the gait at which the image sequence begins. We select a single signal to be a reference for the others. Phases, with the reference phase subtracted, are features.

   8. Selected features combine to form a feature vector for recognition. The plot shows the average feature values and the maximum and minimum values for each subject in our experiment.
          * feature vector plot 

The entire process is controlled from a shell script that executes programs to compute the optical flow, calculate the scalar shape descriptions, and analyze the signals for frequency and phase.

(:title Shape of Motion:)

We have developed a novel vision system that can recognize people by the way they walk. The system computes optical flow for an image sequence of a person walking, and then characterizes the shape of the motion with a set of sinusoidally-varying scalars. Feature vectors composed of the phases of the sinusoids are able to discriminate among people. Contents

1. Overview: overview of the work
2. System: a description of the recognition system
3. Experiment: an experiment that demonstrates the system
4. Results: results of the experiment showing successful recognition

To Contact the Authors:

    * Jim Little is at the University of British Columbia, Department of Computer Science. Send e-mail to:
      little@cs.ubc.ca

    * Jeff Boyd is at the Department of Computer Science, University of Calgary. Send e-mail to:
      boyd@cpsc.ucalgary.ca

[Next]