Watch What They Say --Even If They Didn’t
For better or worse, a new video animation technology called Video Rewrite enables computers to automatically put words into someone else’s mouth.
Developed at Interval Research Corp. in Palo Alto, Video Rewrite (https://www.interval.com/papers/1997-012/) instantly creates new video of a person saying things they never said in the original footage. One astoundingly convincing result is a video clip of John F. Kennedy uttering the infamous phrase “Read my lips.”
“I really like the idea of Robert Redford wishing me a happy birthday in a personalized video greeting card,” says Michele Covell, who developed Video Rewrite with colleagues Christoph Bregler and Malcolm Slaney. “I’d also like to dub my favorite film, ‘My Life as a Dog,’ in English.”
Push aside those memories of humorously bad dubs of Godzilla and Bruce Lee movies. Three years in the making, Video Rewrite provides effects of a quality matching the magic of Forrest Gump meeting Kennedy and Nixon, but without the tedious and time-consuming handiwork that went into those scenes.
After a new audio track is recorded by an actor, Video Rewrite uses speech-recognition technology to label the phonemes--the “b” sound in bike, for instance--in the original footage and the new soundtrack. Then the mouth images of the appropriate phonemes from the footage are automatically reordered to match the new soundtrack and are digitally stitched onto the face. Jerky jumps between mouth movements are morphed together while computer-vision technology is used to keep the reanimated mouth in line with the original face.
Interval has no plans to bring the product to market just yet. The Video Rewrite team plans to develop a similar technique to control the movement of facial gestures and then the entire body--Bregler’s ongoing doctoral thesis in UC Berkeley’s computer science department.
