skip to main content

Information, Geometry, and Physics Seminar

Wednesday, April 10, 2024
4:00pm to 5:00pm
Add to Cal
Linde Hall 310
Predictive coding in neural networks can recover the environment's map
James Gornet, Electrical Engineering Department, Computation and Neural Systems, Caltech,

Mapping is a general mechanism for generating an internal representation of sensory information. While spatial maps facilitate navigation and planning within an environment, mapping is a ubiquitous function that extends beyond visual-spatial mapping. However, it has been unclear how a single mechanism can generate both spatial and non-spatial maps. In this talk, I discuss how predictive coding—by predicting sensory data from past sensory experiences—provides a basic, general mechanism for charting both spatial and non-spatial maps. In this theoretical framework, an agent traverses some environment; the spatial structure of an agent's path is embedded in the sequential structure of agent's video observations. I demonstrate that a neural network that performs predictive coding can construct an implicit spatial map of an environment by assembling information from local paths into a global frame within the neural network's latent space. In addition, the neural network's latent variables generate firing patterns that resemble place fields in the rodent hippocampus. Predictive coding can be performed over any sensory modality that has some temporal sequence. Large language models such as GPT-4, for example, train on causal word prediction, a form of predictive coding, and build internal maps that support generalized reasoning. These results all suggest that predictive coding might provide a unified theory for building representations of information—connecting disparate theories including place cell formation in the hippocampus and human language.

For more information, please contact Mathematics Department by phone at 626-395-4335 or by email at [email protected].