Imagine you are in the middle of an unfamiliar city. Even if your surroundings initially seem unfamiliar, you can explore your surroundings and eventually create a mental map of your environment, where buildings, streets, signs, etc. are located. This ability to build spatial maps in the brain underlies even more advanced types of cognition in humans: for example, there is a theory that language is encoded in a map-like structure in the brain.
Despite all that artificial intelligence and advanced neural networks can do, they cannot build maps from scratch.
“There is a feeling that even the most advanced AI models are not yet truly intelligent,” he explains. Matt Thomsonassistant professor of computational biology and researcher at the Heritage Medical Research Institute. “They don’t solve problems like we do; they can’t prove unproven mathematical results or generate new ideas. We think that’s because they can’t navigate concept space; solving complex problems is like moving through concept space, like navigating. AIs do more of a rote memorization: you give it an input, and it gives you an answer. But it can’t synthesize disparate ideas.”
A new paper from the Thomson Lab reveals that neural networks can be designed to create spatial maps using a type of algorithm called predictive coding. The paper is published in the journal Nature Artificial Intelligence July 18.
Led by graduate student James Gornet, the two built environments in the game Minecraftincorporating complex elements like trees, rivers, and caves. They recorded videos of a player randomly traversing the area and used the video to train a neural network equipped with a predictive coding algorithm. They found that the neural network is able to learn how objects in the Minecraft world are organized relative to each other and were able to “predict” what environments would appear as they moved through space. In addition, the team “opened up” the neural network (the coding equivalent of “checking under the hood”) and found that the representations of the different objects were stored spatially relative to each other – in other words, they saw a map of space. Minecraft environment stored in the neural network.
Neural networks can navigate maps provided to them by human designers, like a self-driving car using GPS, but this is the first time a neural network has been able to create its own map. This ability to store and spatially organize information could eventually help neural networks become more “intelligent,” allowing them to solve truly complex problems in the way humans can.
Gornet is a student in Caltech’s Department of Computational and Neural Systems (CNS), which covers neuroscience, machine learning, mathematics, statistics, and biology.
“The CNS program really gave James a place to do unique work that wouldn’t have been possible anywhere else,” Thomson says. “We’re taking a bio-inspired approach to machine learning that allows us to reverse-engineer brain properties into artificial neural networks, and we hope to learn more about the brain in turn. We have a very receptive community for this type of work here at Caltech.”
The document is titled “Automated construction of cognitive maps with visual predictive coding.” Gornet is first author of the paper. Funding was provided by the David and Lucile Packard Foundation, the Tianqiao and Chrissy Chen Institute for Neuroscience at Caltech, the Heritage Medical Research Institute, and the Chan Zuckerberg Initiative. Matt Thomson is an affiliated faculty member at the Tianqiao and Chrissy Chen Institute for Neuroscience at Caltech.