A team from the University of Leeds is building virtual avatars to effectively immortalize pop culture figures
Researchers from the School of Computing at the University of Leeds in the UK have detailed their proof of concept for reanimating characters from TV shows. They have shown how capturing a character’s style of speech, language, and visual appearance can be used to build an interactive avatar and effectively ‘immortalize’ them.
The team obtained audiovisual material from the TV show Friends to demonstrate their concept using the character Joey. They trained a speech synthesizer to detect spoken audio and mute background noise prior to analysis, and used time-aligned transcripts as supervisory labels for training their system. An online multi-face detection and tracking system was used to produce face tracks for all episodes, and the team’s system automatically labeled audio and video with phonetic boundaries, character face bounding boxes with their character name, and facial landmarks, in order to train their generative model.
One episode of Friends was manually labeled with speech and nonspeech for training the speech detector and all other training processes were fully automatic, with the language model trained from sentences containing 5-10 words across all 236 episodes of the show. This led to the creation of a “virtually immortalized” Joey, who could say new sentences in his familiar style. Speech audio was generated and trained from the phonetic labeling, and this was synchronized with the movement of his mouth, while a generated video pasted over his mouth matched what he was saying.
The team plans to improve the rendering of the avatar and extend their model to include interactions between avatars and with real people. This kind of system could also be used to create a natural interface between human and computer, helping to add a face and personality to voice-only assistants such as Apple’s Siri or Amazon’s Alexa.
