Researchers at MIT, Microsoft, and Adobe have developed an algorithm that can reconstruct an audio signal by analyzing minute vibrations of objects depicted in video. In one set of experiments, they were able to recover intelligible speech from the vibrations of a potato-chip bag photographed from 15 feet away through soundproof glass.
In other experiments, they extracted useful audio signals from videos of aluminum foil, the surface of a glass of water, and even the leaves of a potted plant. The researchers will present their findings in a paper at this year’s Siggraph, the premier computer graphics conference.
“When sound hits an object, it causes the object to vibrate,” says Abe Davis, a graduate student in electrical engineering and computer science at MIT and first author on the new paper. “The motion of this vibration creates a very subtle visual signal that’s usually invisible to the naked eye. People didn’t realize that this information was there.”
Joining Davis on the Siggraph paper are Frédo Durand and Bill Freeman, both MIT professors of computer science and engineering; Neal Wadhwa, a graduate student in Freeman’s group; Michael Rubinstein of Microsoft Research, who did his PhD with Freeman; and Gautham Mysore of Adobe Research.
Reconstructing audio from video requires that the frequency of the video samples — the number of frames of video captured per second — be higher than the frequency of the audio signal. In some of their experiments, the researchers used a high-speed camera that captured 2,000 to 6,000 frames per second. That’s much faster than the 60 frames per second possible with some smartphones, but well below the frame rates of the best commercial high-speed cameras, which can top 100,000 frames per second.
Commodity hardware
In other experiments, however, they used an ordinary digital camera. Because of a quirk in the design of most cameras’ sensors, the researchers were able to infer information about high-frequency vibrations even from video recorded at a standard 60 frames per second. While this audio reconstruction wasn’t as faithful as that with the
high-speed camera, it may still be good enough to identify the gender of a speaker in a room; the number of speakers; and even, given accurate enough information about the acoustic properties of speakers’ voices, their identities.
Read more . . .
The Latest on: Reconstructing audio from video
[google_news title=”” keyword=”Reconstructing audio from video” num_posts=”10″ blurb_length=”0″ show_thumb=”left”]
via Google News
The Latest on: Reconstructing audio from video
- Visualizing A Black Hole’s Flares In 3Don April 26, 2024 at 11:49 am
Researchers are trying to develop a better picture of what’s happening in the regions closest to a black hole’s event horizon.
- Wednesday, April 24. Russia’s War On Ukraine: News And Information From Ukraineon April 24, 2024 at 4:38 pm
Ukrainian contemporary artists are represented at both Ukraine’s and Poland’s pavilions at the 60th Venice Biennale international art exhibition, which opened April 20. The Ukrainian Institute’s “Net ...
- Dixie Highway reconstruction project begins April 29on April 24, 2024 at 8:15 am
SAGINAW CO., Mich. (WNEM) - Intermittent lane closures will take place from Junction Road to Birch Run Road starting Monday, April 29, for the Dixie Highway reconstruction project, according to ...
- Reconstructing a 3D Flare Around a Black Hole (VIDEO)on April 22, 2024 at 8:15 am
Based on radio telescope data and models of black hole physics, a team led by Caltech has used neural networks to reconstruct a 3D image that shows how explosive flare-ups in the disk of gas around ...
- A killer confessed his crime in a landmark documentary. But does it really need a sequel?on April 22, 2024 at 12:00 am
Use precise geolocation data and actively scan device characteristics for identification. This is done to store and access information on a device and to provide personalised ads and content, ad and ...
- Reconstruction of 45th Street North in Fargo underwayon April 19, 2024 at 2:01 pm
45th Street is closed from 19th Avenue to 32nd Avenue North for total reconstruction. Once complete, the road will be concrete instead of gravel. Drivers are being detoured to Interstate-29 via ...
- Portage Bay Bridge reconstruction to begin in Auguston April 3, 2024 at 1:08 pm
“Our work to replace the Portage Bay Bridge is an important piece of broader improvements to the SR 520 corridor, one of the busiest across the Puget Sound region.” Skanska will continue ...
- Scientists recreate the face of an ancient Chinese emperoron March 28, 2024 at 8:00 am
By Laura Baisas | Published Mar 28, 2024 11:00 AM EDT The facial reconstruction of Emperor Wu, an ancient Chinese ruler of the Northern Zhou dynasty. Pianpian Wei SHARE A team in China used ...
- US government to pay for Baltimore bridge reconstruction, says Biden – videoon March 26, 2024 at 11:44 am
Joe Biden said the federal government would pay for the reconstruction of a bridge in Baltimore after it collapsed when a container ship collided with it, sending a number of vehicles into the water.
- Considering Breast Reconstruction After Surgeryon March 6, 2023 at 9:24 pm
But I will say that the majority of ladies are opting for reconstruction. And there are lots of benefits - physical, psychosocial - that data supports for the benefits of breast reconstruction.
via Bing News