via Baylor College of Medicine
UMass Amherst, Baylor researchers collaborate on imitating brain’s ‘replay’ ability
Artificial intelligence (AI) experts at the University of Massachusetts Amherst and the Baylor College of Medicine report that they have successfully addressed what they call a “major, long-standing obstacle to increasing AI capabilities” by drawing inspiration from a human brain memory mechanism known as “replay.”
First author and postdoctoral researcher Gido van de Ven and principal investigator Andreas Tolias at Baylor, with Hava Siegelmann at UMass Amherst, write in Nature Communications that they have developed a new method to protect – “surprisingly efficiently” – deep neural networks from “catastrophic forgetting” – upon learning new lessons, the networks forget what they had learned before.
Siegelmann and colleagues point out that deep neural networks are the main drivers behind recent AI advances, but progress is held back by this forgetting.
They write, “One solution would be to store previously encountered examples and revisit them when learning something new. Although such ‘replay’ or ‘rehearsal’ solves catastrophic forgetting,” they add, “constantly retraining on all previously learned tasks is highly inefficient and the amount of data that would have to be stored becomes unmanageable quickly.”
Unlike AI neural networks, humans are able to continuously accumulate information throughout their life, building on earlier lessons. An important mechanism in the brain believed to protect memories against forgetting is the replay of neuronal activity patterns representing those memories, they explain.
Siegelmann says the team’s major insight is in “recognizing that replay in the brain does not store data.” Rather, “the brain generates representations of memories at a high, more abstract level with no need to generate detailed memories.” Inspired by this, she and colleagues created an artificial brain-like replay, in which no data is stored. Instead, like the brain, the network generates high-level representations of what it has seen before.
The “abstract generative brain replay” proved extremely efficient, and the team showed that replaying just a few generated representations is sufficient to remember older memories while learning new ones. Generative replay not only prevents catastrophic forgetting and provides a new, more streamlined path for system learning, it allows the system to generalize learning from one situation to another, they state.
For example, “if our network with generative replay first learns to separate cats from dogs, and then to separate bears from foxes, it will also tell cats from foxes without specifically being trained to do so. And notably, the more the system learns, the better it becomes at learning new tasks,” says van de Ven.
He and colleagues write, “We propose a new, brain-inspired variant of replay in which internal or hidden representations are replayed that are generated by the network’s own, context-modulated feedback connections. Our method achieves state-of-the-art performance on challenging continual learning benchmarks without storing data, and it provides a novel model for abstract level replay in the brain.”
Van de Ven says, “Our method makes several interesting predictions about the way replay might contribute to memory consolidation in the brain. We are already running an experiment to test some of these predictions.”
The Latest Updates from Bing News & Google News
Go deeper with Bing News on:
- AI here, there, everywhere — but it’s just the beginningon March 6, 2021 at 11:00 am
Hey, Google, what’s the temperature in Zone 2,” I say into the darkness. A disembodied voice responds: “The temperature in Zone 2 is 52 degrees.” “Set the heat to 68,” I say, and then I ask the ...
- Fear itself is the real threat to democracy, not tall tales of Chinese AIon March 6, 2021 at 8:03 am
Stoking panic about China’s dominance is just another way for western technology giants to avoid scrutiny and gain power ...
- Taco Bell’s new AI, digital transformation: Friday’s daily briefon March 5, 2021 at 7:01 am
Marketing Land’s daily brief features daily insights, news, tips, and essential bits of wisdom for today’s digital marketer. If you would like to read this before the rest of the internet does, sign ...
- Microsoft And the AI Approach To Creating A Better Company (And World)on March 5, 2021 at 6:56 am
At Microsoft Ignite this year, Mitra Azizirad, Microsoft’s Corporate VP of AI and Innovation, gave one of the most exciting presentations. She opened with Microsoft showcased a robust future at Ignite ...
- Harnessing The Power Of Artificial Intelligence In Accountingon March 5, 2021 at 5:10 am
Instead of fearing technological advancements, CPAs should embrace them and find ways to augment their skills rather than replace them.
Go deeper with Google Headlines on:
Go deeper with Bing News on:
Deep neural networks
- Retrofitting MIT’s deep learning “boot camp” for the virtual worldon March 4, 2021 at 7:11 pm
Graduate students Ava Soleimany and Alexander Amini moved their popular IAP course on deep learning online this year, but still managed to work in some surprises.
- Can Deep Learning Replace Numerical Weather Prediction?on March 3, 2021 at 2:32 am
Numerical weather prediction (NWP) is a mainstay of supercomputing. Some of the first applications of the first supercomputers dealt with climate ...
- Accurate surface ultraviolet radiation forecasting for clinical applications with deep neural networkon March 2, 2021 at 4:00 pm
Surface UV and weather data acquisition. Surface UV radiation, total ozone column, cloud coverage, and aerosol optical depth at 500 nm (AOD500), were collected at the Faculty of S ...
- Featurespace Automated Deep Behavioral Networks protects from card and payments fraudon March 1, 2021 at 3:45 pm
Featurespace introduces Automated Deep Behavioral Networks for the card and payments industry, providing a deeper layer of defense to protect consumers from scams, account takeover, card and payments ...
- Google’s deep learning finds a critical path in AI chipson February 28, 2021 at 10:39 pm
ZDNet spoke with Google Brain director Jeff Dean about how the company is using artificial intelligence to advance its internal development of custom chips to accelerate its software. Dean noted that ...