via OpenAI
An updated analysis from OpenAI shows how dramatically the need for computational resources has increased to reach each new AI breakthrough.
In 2018, OpenAI found that the amount of computational power used to train the largest AI models had doubled every 3.4 months since 2012.
The San Francisco-based for-profit AI research lab has now added new data to its analysis. This shows how the post-2012 doubling compares with the historic doubling time since the beginning of the field. From 1959 to 2012, the amount of power used doubled every two years, tracking Moore’s Law. This means the resources used today are doubling at a rate seven times faster than before.
This dramatic increase in the resources needed underscores just how costly the field’s achievements have become. Keep in mind that the above graph shows a logarithmic scale. On a linear scale (below), you can more clearly see how compute usage has increased by 300,000-fold in the last seven years.
via OpenAI
The chart also notably does not include some of the most recent breakthroughs, including Google’s large-scale language model BERT, OpenAI’s language model GPT-2, or DeepMind’s StarCraft II-playing model AlphaStar.
In the past year, more and more researchers have sounded the alarm on the exploding costs of deep learning. In June, an analysis from researchers at the University of Massachusetts, Amherst, showed how these increasing computational costs directly translate into carbon emissions.
In their paper, they also noted how the trend exacerbates the privatization of AI research because it undermines the ability for academic labs to compete with much more resource-rich private ones.
In response to this growing concern, several industry groups have made recommendations. The Allen Institute for Artificial Intelligence, a nonprofit research firm in Seattle, has proposed that researchers always publish the financial and computational costs of training their models along with their performance results, for example.
In its own blog, OpenAI suggested policymakers increase funding to academic researchers to bridge the resource gap between academic and industry labs.
Learn more: The computing power needed to train AI is now rising seven times faster than ever before
The Latest Google Headlines on:
Deep learning computational resources
The Latest Bing News on:
Deep learning computational resources
- DeepMind's Perceiver AR: a step toward more AI efficiencyon July 31, 2022 at 8:46 pm
The auto-regressive attention at the heart of the Transformer, and programs like it, becomes a scaling nightmare. A recent DeepMind/Google work proposes a way to put such programs on a diet.
- Supercomputer uncovers new high-entropy alloyson July 27, 2022 at 4:45 am
Alloys are famously more than just the sum of their parts. Steel, for example, which is made by adding a little carbon to iron, revolutionized industry by being much stronger than either of its ...
- Cadence Expands into Molecular Simulation with Acquisition of OpenEye Scientific, a Pioneering Leader in Computational Molecular Designon July 25, 2022 at 7:50 am
Acquisition leverages Cadence’s computational software expertise into molecular modeling and simulation, focused on pharmaceutical and biotechnology drug discovery SAN JOSE, Calif. & SANTA FE, N.M., ...
- Research: Artificial intelligence can fuel racial bias in healthcare, but can mitigate it, tooon July 24, 2022 at 6:18 pm
While some healthcare algorithms exacerbate inequitable medical care, other algorithms can actually close such gaps, a growing body of research shows.
- Alexa and Siri, listen up! Teaching machines to really hear uson July 21, 2022 at 6:56 pm
The implications of new AI voice research go beyond user experience to making AI more efficient, which could change the industry and significantly reduce carbon footprints.
- Deep learning for new alloyson July 20, 2022 at 11:23 pm
When is something more than just the sum of its parts? Alloys show such synergy. Steel, for instance, revolutionized industry by taking iron, adding a little carbon and making an alloy much stronger ...
- Deep learning for new alloyson July 20, 2022 at 3:39 pm
The study, “Composition design of high-entropy alloys with deep sets learning,” was published April 2022 in Npj Computational Materials. The study authors are Jie Zhang, George Kim and Wei ...
- NSTC Report Highlights Challenges, Best Practices Related to Federal Use of Cloud for AI R&Don July 20, 2022 at 1:48 pm
Looking for the latest Government Contracting News? Read about NSTC Report Highlights Challenges, Best Practices Related to Federal Use of Cloud for AI R&D.
- Alexa and Siri, listen up! Research team is teaching machines to really hear uson July 20, 2022 at 10:54 am
University of Virginia cognitive scientist Per Sederberg has a fun experiment you can try at home. Take out your smartphone and, using a voice assistant such as the one for Google's search engine, say ...
- AI Ethics Ringing Alarm Bells About The Looming Specter Of AI Biases At Massive Global Scale, Especially Fueled Via Looming Fully Autonomous Systemson July 13, 2022 at 4:44 am
The Machine Learning or Deep Learning computational pattern matching will simply try ... exasperatingly so for those that lived in those now resource-starved areas. You could assert that the AI pretty ...
The Latest Google Headlines on:
Artificial intelligence
The Latest Bing News on:
Artificial intelligence
- CHIPS Is a Missed Opportunity for Real Securityon August 1, 2022 at 1:15 pm
The need to do something did not produce effective legislation with the CHIPS Act. American technology is still vulnerable, ...
- Innovative AI Platform Increases Efficiency at Tampa General Hospitalon August 1, 2022 at 10:31 am
Using artificial intelligence, Tampa General Hospital (TGH) is streamlining patient navigation within the hospital by improving the efficiency of its patient transport department. Collaborating with ...
- Study finds new hardware offers faster computation for artificial intelligence with much less energyon August 1, 2022 at 6:32 am
Analogue deep learning, a new branch of artificial intelligence, promises faster computation with less energy consumption. The findings of the research were published in the journal ‘Science’.
- Lightmatter Appoints Intel Veteran Ritesh Jain as VP of Engineering, Systems and Packagingon August 1, 2022 at 6:02 am
Lightmatter, the leader in photonic computing, has named its new Vice President of Engineering, Ritesh Jain. He joins Lightmatter after more than 21 years at Intel leading systems and packaging ...
- No code, no problem—we try to beat an AI at its own game with new toolson August 1, 2022 at 6:00 am
And some people even think that an AI has attained sentience. (Spoiler alert: It has not .) And as Ars' Matt Ford recently pointed out here, artificial intelligence may be artificial, but it's not ...
- RAKEZ welcomes American artificial intelligence company Falcons.AIon July 31, 2022 at 5:00 pm
Ras Al Khaimah Economic Zone (RAKEZ) welcomed Falcons.AI, an American company which provides software programmes and consultation services driven by artificial intelligence (AI). With pre-existing ...
- Rather than threaten jobs, artificial intelligence should collaborate with human writerson July 29, 2022 at 7:30 am
The Guardian published an opinion piece written by a program. The artificial intelligence, called GPT-3, is a large language model developed by OpenAI, and it posed a bold question in the headline of ...
- Artificial intelligence can help fleets keep vehicles longer -- saving money and liveson July 29, 2022 at 1:10 am
Artificial intelligence can help with with deep-dive inspections, utilizing easy to operate mobile apps along with standard vehicle equipment like security cameras to routinely document the condition ...
- Submarine Warfare & Artificial Intelligenceon July 26, 2022 at 4:01 am
By Amit Das April 2016, Sea Hunter was launched by the American Navy mentored by DARPA (Defence Advance Research Project Agency) a 40-meter unmanned and completely autonomous warship designed for the ...
- Three Ways Artificial Intelligence And LIDAR Can Be Used In Healthcare Deliveryon July 26, 2022 at 3:30 am
The time is now for healthcare organizations to consider AI and LIDAR to help streamline care, improve outcomes and protect their financial viability.