
The system enables film editors to change the expression and head position of an actor after filming has finished
New video editing technique could save time and reduce costs for the film industry.
Hyeongwoo Kim from the Max Planck Institute for Informatics explains:
It works by using model-based 3D face performance capture to record the detailed movements of the eyebrows, mouth, nose, and head position of the dubbing actor in a video. It then transposes these movements onto the ‘target’ actor in the film to accurately sync the lips and facial movements with the new audio.
The research is currently at the proof-of-concept stage and is yet to work at real time, however the researchers anticipate the approach could make a real difference to the visual entertainment industry.
Professor Christian Theobalt, from the Max Planck Institute for Informatics, said:
Despite extensive post-production manipulation, dubbing films into foreign languages always presents a mismatch between the actor on screen and the dubbed voice.
Our new Deep Video Portrait approach enables us to modify the appearance of a target actor by transferring head pose, facial expressions, and eye motion with a high level of realism.
Co-author of the paper, Dr Christian Richardt, from the University of Bath’s motion capture research centre CAMERA, adds:
This technique could also be used for post-production in the film industry where computer graphics editing of faces is already widely used in today’s feature films.
A great example is ‘The Curious Case of Benjamin Button’ where the face of Brad Pitt was replaced with a modified computer graphics version in nearly every frame of the movie. This work remains a very time-consuming process, often requiring many weeks of work by trained artists.
Deep Video Portraits shows how such a visual effect could be created with less effort in the future. With our approach even the positioning of an actor’s head and their facial expression could be easily edited to change camera angles or subtly change the framing of a scene to tell the story better.
In addition, this new approach can also be used in other applications, which the authors show on their project website, for instance in video and VR teleconferencing, where it can be used to correct gaze and head pose such that a more natural conversation setting is achieved. The software enables many new creative applications in visual media production, but the authors are also aware of the potential of misuse of modern video editing technology.
Dr Michael Zollhöfer, from Stanford University, explains:
The media industry has been touching up photos with photo-editing software for many years, meaning most of us have learned to take what we see in photos with a pinch of salt. With ever improving video editing technology, we must also start being more critical about the video content we consume every day, especially if there is no proof of origin. We believe that the field of digital forensics should and will receive a lot more attention in the future to develop approaches that can automatically prove the authenticity of a video clip. This will lead to ever better approaches that can spot such modifications even if we humans might not be able to spot them with our own eyes.
To address this, the research team is using the same technology to develop in tandem neural networks trained to detect synthetically generated or edited video at high precision to make it easier to spot forgeries. The authors have no plans to make the software publicly available but state that any software implementing the many creative use cases should include watermarking schemes to clearly mark modifications.
Learn more: AI could make dodgy lip sync dubbing a thing of the past
The Latest on: Artificial intelligence
[google_news title=”” keyword=”artificial intelligence” num_posts=”10″ blurb_length=”0″ show_thumb=”left”]
via Google News
The Latest on: Artificial intelligence
- Meet Archie, the City of Fairfield's artificial intelligence aidon September 29, 2023 at 7:03 pm
Farfield uses artificial intelligence aid on city website 01:35 FAIRFIELD — Artificial intelligence is starting to become a part of everyday life, even by helping to fix potholes among other things in ...
- Artificial intelligence: ‘Heed its warning’on September 29, 2023 at 4:05 pm
Re: “AI wrote this editorial, and it argues that human journalists should keep their jobs” [Sept. 24, Opinion]: The Bing Chat AI-produced editorial was clear, straightforward, well-reasoned and (let’s ...
- Does the Rising Effectiveness of Artificial Intelligence (AI) Make AMD Stock a Buy?on September 29, 2023 at 1:56 pm
Fool.com contributor and finance professor Parkev Tatevosian evaluates AMD ( AMD 0.06%) and its financial position to determine if long-term investors should buy the stock. *Stock prices used were the ...
- Watch out, Apple, an 'iPhone of artificial intelligence' could be in the workson September 29, 2023 at 12:30 pm
Reports indicate that Apple's former Chief Design Officer, Jony Ive, and OpenAI are in talks that could lead to the development of an "iPhone of artificial intelligence." Both parties aim to provide ...
- Cathie Wood Says Robotaxis Are the Investment Opportunity of a Lifetime, and Ark Is Betting Big on 2 Artificial Intelligence (AI) Growth Stockson September 29, 2023 at 3:05 am
Cathie Wood's Ark Invest has sizable positions in Nvidia and Tesla, two companies poised to benefit from the advent of fully autonomous vehicles.
- Artificial Intelligence is invading: Does it have a future on the coast?on September 29, 2023 at 2:30 am
A generative wave of artificial intelligence (AI) is about to hit the coastal region. Mike Berger takes a humorous look at its applications.
- Are we about to see ‘the iPhone of artificial intelligence’?on September 28, 2023 at 8:25 pm
Apple’s former design guru Jony Ive is reportedly in talks with ChatGPT creator OpenAI to build a new hardware device for artificial intelligence.
- National Security Agency is starting an artificial intelligence security centeron September 28, 2023 at 3:01 pm
The National Security Agency is starting an artificial intelligence security center — a crucial mission as AI capabilities are increasingly acquired, developed and integrated into U.S. defense and ...
- Artificial intelligence trainings help teachers learn how to use AI in the classroomon September 28, 2023 at 1:55 pm
Teachers can use AI for administrative tasks that take up a lot of their time, such as writing emails about upcoming school events, scheduling, creating rubrics, or ideas for lesson plans, Schweinle ...
via Bing News