Two-level training
The main challenge for the researchers was to integrate the key components needed for on-chip learning onto a single neuromorphic chip. “One of the major tasks to solve was, for example, the inclusion of the electrochemical random access memory (EC-RAM) components,” says Van de Burgt. “These are the components that mimic the storage and activation of electrical charges assigned to neurons in the brain.”
The researchers built a two-layer neural network based on EC-RAM components made from organic materials and tested the hardware with an evolution of the widely used training algorithm, backpropagation with gradient descent. “The conventional algorithm is commonly used to improve the accuracy of neural networks, but it is not compatible with our hardware, so we created our own version,” Stevens says.
Additionally, with AI in many domains rapidly becoming an unsustainable source of energy resources, the ability to train neural networks on hardware components for a fraction of the energy cost is a tantalizing possibility for many applications, ranging from ChatGPT to weather forecasting.
The future need
While researchers have demonstrated that the new training approach works, the next logical step is to go further, be bolder, and do better.
“We have shown that this works for a small two-layer network,” says van de Burgt. “We would now like to involve industry and other large research labs so that we can build much larger networks of hardware devices and test them with real data problems.”
This new step would allow the researchers to demonstrate that these systems are very effective for training and operating useful neural networks and AI systems. “We would like to apply this technology in several practical cases,” says Van de Burgt. “My dream is that these technologies become the standard in AI applications in the future.”
Full document details
“Hardware implementation of progressive gradient descent backpropagation for in situ training of multilayer neural networks“, Eveline RW van Doremaele, Tim Stevens, Stijn Ringeling, Simone Spolaor, Marco Fattori and Yoeri van de Burgt, Science Advances, (2024).
Eveline RW van Doremaele and Tim Stevens contributed equally to the research and are both considered first authors of the article.
Tim Stevens currently works as a mechanical engineer at Micro-alignmenta company co-founded by Marco Fattori.