Researchers at MIT, Microsoft, and Adobe have developed an algorithm that can reconstruct an audio signal by analyzing minute vibrations of objects depicted in video. In one set of experiments, they were able to recover intelligible speech from the vibrations of a potato-chip bag photographed from 15 feet away through soundproof glass.
In other experiments, they extracted useful audio signals from videos of aluminum foil, the surface of a glass of water, and even the leaves of a potted plant. The researchers will present their findings in a paper at this year’s Siggraph, the premier computer graphics conference.
“When sound hits an object, it causes the object to vibrate,” says Abe Davis, a graduate student in electrical engineering and computer science at MIT and first author on the new paper. “The motion of this vibration creates a very subtle visual signal that’s usually invisible to the naked eye. People didn’t realize that this information was there.”
Joining Davis on the Siggraph paper are Frédo Durand and Bill Freeman, both MIT professors of computer science and engineering; Neal Wadhwa, a graduate student in Freeman’s group; Michael Rubinstein of Microsoft Research, who did his PhD with Freeman; and Gautham Mysore of Adobe Research.
Reconstructing audio from video requires that the frequency of the video samples — the number of frames of video captured per second — be higher than the frequency of the audio signal. In some of their experiments, the researchers used a high-speed camera that captured 2,000 to 6,000 frames per second. That’s much faster than the 60 frames per second possible with some smartphones, but well below the frame rates of the best commercial high-speed cameras, which can top 100,000 frames per second.
Commodity hardware
In other experiments, however, they used an ordinary digital camera. Because of a quirk in the design of most cameras’ sensors, the researchers were able to infer information about high-frequency vibrations even from video recorded at a standard 60 frames per second. While this audio reconstruction wasn’t as faithful as that with the
high-speed camera, it may still be good enough to identify the gender of a speaker in a room; the number of speakers; and even, given accurate enough information about the acoustic properties of speakers’ voices, their identities.
Read more . . .
The Latest on: Reconstructing audio from video
[google_news title=”” keyword=”Reconstructing audio from video” num_posts=”10″ blurb_length=”0″ show_thumb=”left”]
via Google News
The Latest on: Reconstructing audio from video
- Sound Bites: $3 million in federal aid to repair I-95 bridgeon May 8, 2024 at 5:00 am
The U.S. Department of Transportation has approved $3 million in emergency funding to offset costs for the Norwalk-Fairfield Bridge reconstruction over I-95.
- Ole Miss Fraternity Expels Student for Ape Sound Incidenton May 7, 2024 at 3:02 pm
Pro-Palestine protests have occurred at numerous American universities in recent weeks. During a demonstration at the University of Mississippi, also known as Ole Miss, one Black graduate student ...
- 10 Items That Should Be in Your Caron May 6, 2024 at 4:59 pm
Did Ukraine Get Its Much-Needed Patriot Missiles? 3D Reconstruction of Neanderthal Woman’s Face Created From Skull Scientists sound alarm as growing threat looms over coastal states: 'We are preparing ...
- Alpha Street to close on Monday for major reconstructionon May 3, 2024 at 3:09 pm
A large section of Alpha Street will close on Monday at 6 a.m. and remain closed until major reconstruction work is complete, scheduled for December. On Friday, the City of Owen Sound announced that ...
- Truck driver charged in couple's death, officials say he was streaming Netflix before crashon May 2, 2024 at 10:27 am
A commercial truck driver who was charged in the death of an Iowa couple has been accused of streaming Netflix while driving.
- Scientists reveal the face of a Neanderthal who lived 75,000 years agoon April 30, 2024 at 5:00 pm
They used the contours of the face and skull to guide a reconstruction to understand what she may have looked like. The striking recreation is featured in a new documentary “Secrets of the ...
- Charges: Semi driver was watching Netflix when he caused crash that killed two peopleon April 29, 2024 at 7:50 pm
According to court documents, 55-year-old Billie Joe Grimes was charged with two counts of criminal vehicular homicide and one count of criminal vehicular operation.
- Reconstructing a 3D Flare Around a Black Hole (VIDEO)on April 22, 2024 at 8:04 am
Based on radio telescope data and models of black hole physics, a team led by Caltech has used neural networks to reconstruct a 3D image that shows how explosive flare-ups in the disk of gas ...
- Hanging Lake to release reservation times during continued trail reconstructionon April 22, 2024 at 2:33 am
DENVER (KDVR) — One of Colorado’s most well-known trails is still under reconstruction after being destroyed in a 2020 fire. While Hanging Lake still has a ways to go, limited reservation ...
- Considering Breast Reconstruction After Surgeryon March 6, 2023 at 9:24 pm
But I will say that the majority of ladies are opting for reconstruction. And there are lots of benefits - physical, psychosocial - that data supports for the benefits of breast reconstruction.
via Bing News