AI-generated (DALL-E 3) conceptual image depicting light waves passing through a physical system.
Credit: LWE/EPFL
EPFL researchers have developed an algorithm to train an analog neural network just as accurately as a digital one, enabling the development of more efficient alternatives to power-hungry deep learning hardware.
With their ability to process vast amounts of data through algorithmic ‘learning’ rather than traditional programming, it often seems like the potential of deep neural networks like Chat-GPT is limitless. But as the scope and impact of these systems have grown, so have their size, complexity, and energy consumption – the latter of which is significant enough to raise concerns about contributions to global carbon emissions.
And while we often think of technological advancement in terms of shifting from analog to digital, researchers are now looking for answers to this problem in physical alternatives to digital deep neural networks. One such researcher is Romain Fleury of EPFL’s Laboratory of Wave Engineering in the School of Engineering. In a paper published in Science, he and his colleagues describe an algorithm for training physical systems that shows improved speed, enhanced robustness, and reduced power consumption compared to other methods.
“We successfully tested our training algorithm on three wave-based physical systems that use sound waves, light waves, and microwaves to carry information, rather than electrons. But our versatile approach can be used to train any physical system,” says first author and LWE researcher Ali Momeni.
A “more biologically plausible” approach
Neural network training refers to helping systems learn to generate optimal values of parameters for a task like image or speech recognition. It traditionally involves two steps: a forward pass, where data is sent through the network and an error function is calculated based on the output; and a backward pass (also known as backpropagation, or BP), where a gradient of the error function with respect to all network parameters is calculated.
Over repeated iterations, the system updates itself based on these two calculations to return increasingly accurate values. The problem? In addition to being very energy-intensive, BP is poorly suited to physical systems. In fact, training physical systems usually requires a digital twin for the BP step, which is inefficient and carries the risk of a reality-simulation mismatch.
The scientists’ idea was to replace the BP step with a second forward pass through the physical system to update each network layer locally. In addition to decreasing power use and eliminating the need for a digital twin, this method better reflects human learning.
“The structure of neural networks is inspired by the brain, but it is unlikely that the brain learns via BP,” explains Momeni. “The idea here is that if we train each physical layer locally, we can use our actual physical system instead of first building a digital model of it. We have therefore developed an approach that is more biologically plausible.”
The EPFL researchers, with Philipp del Hougne of CNRS IETR and Babak Rahmani of Microsoft Research, used their physical local learning algorithm (PhyLL) to train experimental acoustic and microwave systems and a modeled optical system to classify data like vowel sounds and images. As well as showing comparable accuracy to BP-based training, the method was robust and adaptable – even in systems exposed to unpredictable external perturbations – compared to the state of the art.
An analog future?
While the LWE’s approach is the first BP-free training of deep physical neural networks, some digital updates of the parameters are still required. “It’s a hybrid training approach, but our aim is to decrease digital computation as much as possible,” Momeni says.
The researchers now hope to implement their algorithm on a small-scale optical system, with the ultimate goal of increasing network scalability.
“In our experiments, we used neural networks with up to 10 layers, but would it still work with 100 layers with billions of parameters? This is the next step, and will require overcoming technical limitations of physical systems.”
Original Article: Training algorithm breaks barriers to deep physical neural networks
More from: École Polytechnique Fédérale de Lausanne | Centre National de la Recherche Scientifique | Microsoft Research
The Latest Updates from Bing News
Go deeper with Bing News on:
Analog neural network
- AI Efficiency Breakthrough: How Sound Waves Are Revolutionizing Optical Neural Networks
Researchers have developed a way to use sound waves in optical neural networks, enhancing their ability to process data with high speed and energy efficiency. Optical neural networks may provide the ...
- Intel's Hala Point, the world's largest neuromorphic computer, has 1.15 billion neurons
The Hala Point system's 1,152 Loihi 2 chips enable a total of 1.15 billion artificial neurons, Intel said, "and 128 billion synapses distributed over 140,544 neuromorphic processing cores." That is an ...
- Intel builds world’s largest neuromorphic system
Intel's newest neuromorphic computer is deployed at Sandia National Laboratories. Code-named Hala Point, the brain-inspired system packs 1,152 Loihi 2 processors in a data center chassis the size of a ...
- How Neural Concept’s aerodynamic AI is shaping Formula 1
It's a long way from pedal bikes to Formula 1. But that's precisely the quantum leap that AI-based startup Neural Concept and its co-founder and CEO, ...
- Classical optical neural network exhibits 'quantum speedup'
"These two parts actually perform the operations analog to the quantum gates in the quantum convolutional neural networks," said the scientists. "The convolution part in our network is composed of ...
Go deeper with Bing News on:
AI power consumption
- AI-Powered Phoenix Generation 7 Humanoid Sets New Speed Standards
AI, Sanctuary's latest humanoid, Phoenix Generation 7, made a huge leap in robotics. These new implementations bring a few performance perks ...
- 2 Artificial Intelligence (AI) Chip Stocks to Watch That Aren't Nvidia
Investors are excited by Nvidia's recent announcement about its next-generation AI chips, the Grace Blackwell "superchips." The first iteration, the GB200, will allow for a 25-fold decrease in power ...
- The Untold Story Of AI's Huge Carbon Footprint
We need to increase the power of processors to run new AI models, but at the same time, we need to reduce the energy demand of those processors.
- Lenovo’s New AI Offerings Test Nvidia’s Dominance
Lenovo's new offerings, designed in collaboration with AMD, address the demand for AI compute, while also testing Nvidia’s dominance in enterprise AI.
- U.S. Natural Gas Could Be A Big Winner of The AI Boom
U.S. natural gas producers and shippers could be one of the big winners of the latest advances in Artificial Intelligence ...