The intelligent processing of images of faces is a key resource in building interactive systems with natural and intuitive interfaces. In this presentation we describe two different systems for the analysis/synthesis of facial imagery, carried out by ourselves and colleagues at Interval Research.

The first system, the Magic Morphin Mirror, is an example of real-time, on-line vision processing to detect faces and drive an interactive graphics display. We use multi-modal integration of stereo, color, and pattern cues to robustly segment people and track their faces in crowded, open-background environments. Our first application was an interactive virtual mirror, demonstrated at SIGGRAPH '97, where a user's facial expression was distorted into a surreal effect in real-time.

The second system, Video Rewrite, is an off-line, photo-realistic facial animation system that uses existing footage to create automatically new video of a person mouthing words that she did not speak in the original footage. The system reorders the mouth images in the training footage to match the phoneme sequence of the new audio track. A new automatic morphing technique is used to combine these mouth gestures into the final video sequence. The resulting video combines the dynamics of the original actor's articulations with the mannerisms and setting dictated by the background footage.

These systems and their future extensions hold out the promise of creating fully synthetic characters from video examples, and animating them using real-time, untethered human performance.

This talk describes joint work with Malcolm Slaney, Michele Covell, Gaile Gordon, John Woodfill, Mike Harville and others at Interval Research.