Information, Geometry, and Physics Seminar

Mapping is a general mechanism for generating an internal representation of sensory information. While spatial maps facilitate navigation and planning within an environment, mapping is a ubiquitous function that extends beyond visual-spatial mapping. However, it has been unclear how a single mechanism can generate both spatial and non-spatial maps. In this talk, I discuss how predictive coding—by predicting sensory data from past sensory experiences—provides a basic, general mechanism for charting both spatial and non-spatial maps. In this theoretical framework, an agent traverses some environment; the spatial structure of an agent's path is embedded in the sequential structure of agent's video observations. I demonstrate that a neural network that performs predictive coding can construct an implicit spatial map of an environment by assembling information from local paths into a global frame within the neural network's latent space. In addition, the neural network's latent variables generate firing patterns that resemble place fields in the rodent hippocampus. Predictive coding can be performed over any sensory modality that has some temporal sequence. Large language models such as GPT-4, for example, train on causal word prediction, a form of predictive coding, and build internal maps that support generalized reasoning. These results all suggest that predictive coding might provide a unified theory for building representations of information—connecting disparate theories including place cell formation in the hippocampus and human language.