Memory-Prediction Framework
Memory-Prediction Framework
The memory-prediction framework is a theory of brain function created by Jeff Hawkins and described
in his 2004 book On Intelligence. This theory concerns the role of the mammalian neocortex and its
associations with the hippocampi and the thalamus in matching sensory inputs to stored memory patterns
and how this process leads to predictions of what will happen in the future.
Overview
The theory is motivated by the observed similarities between the brain structures (especially neocortical
tissue) that are used for a wide range of behaviours available to mammals. The theory posits that the
remarkably uniform physical arrangement of cortical tissue reflects a single principle or algorithm which
underlies all cortical information processing. The basic processing principle is hypothesized to be a
feedback/recall loop which involves both cortical and extra-cortical participation (the latter from the
thalamus and the hippocampi in particular).
Consider, for example, the process of vision. Bottom-up information starts as low-level retinal signals
(indicating the presence of simple visual elements and contrasts). At higher levels of the hierarchy,
increasingly meaningful information is extracted, regarding the presence of lines, regions, motions, etc.
Even further up the hierarchy, activity corresponds to the presence of specific objects – and then to
behaviours of these objects. Top-down information fills in details about the recognized objects, and also
about their expected behaviour as time progresses.
The sensory hierarchy induces a number of differences between the various levels. As one moves up the
hierarchy, representations have increased:
Extent – for example, larger areas of the visual field, or more extensive tactile regions.
Temporal stability – lower-level entities change quickly, whereas, higher-level percepts tend
to be more stable.
Abstraction – through the process of successive extraction of invariant features, increasingly
abstract entities are recognized.
The relationship between sensory and motor processing is an important aspect of the basic theory. It is
proposed that the motor areas of the cortex consist of a behavioural hierarchy similar to the sensory
hierarchy, with the lowest levels consisting of explicit motor commands to musculature and the highest
levels corresponding to abstract prescriptions (e.g. 'resize the browser'). The sensory and motor hierarchies
are tightly coupled, with behaviour giving rise to sensory expectations and sensory perceptions driving
motor processes.
Finally, it is important to note that all the memories in the cortical hierarchy have to be learnt – this
information is not pre-wired in the brain. Hence, the process of extracting this representation from the flow
of inputs and behaviours is theorized as a process that happens continually during cognition.
Other terms
Hawkins has extensive training as an electrical engineer. Another way to describe the theory (hinted at in
his book) is as a learning hierarchy of feed forward stochastic state machines. In this view, the brain is
analyzed as an encoding problem, not too dissimilar from future-predicting error-correction codes. The
hierarchy is a hierarchy of abstraction, with the higher level machines' states representing more abstract
conditions or events, and these states predisposing lower-level machines to perform certain transitions. The
lower level machines model limited domains of experience, or control or interpret sensors or effectors. The
whole system actually controls the organism's behavior. Since the state machine is "feed forward", the
organism responds to future events predicted from past data. Since it is hierarchical, the system exhibits
behavioral flexibility, easily producing new sequences of behavior in response to new sensory data. Since
the system learns, the new behavior adapts to changing conditions.
That is, the evolutionary purpose of the brain is to predict the future, in admittedly limited ways, so as to
change it.
Neurophysiological implementation
The hierarchies described above are theorized to occur primarily in mammalian neocortex. In particular,
neocortex is assumed to consist of a large number of columns (as surmised also by Vernon Benjamin
Mountcastle from anatomical and theoretical considerations). Each column is attuned to a particular feature
at a given level in a hierarchy. It receives bottom-up inputs from lower levels, and top-down inputs from
higher levels. (Other columns at the same level also feed into a given column, and serve mostly to inhibit
the activation exclusive representations.) When an input is recognized – that is, acceptable agreement is
obtained between the bottom-up and top-down sources – a column generates outputs which in turn
propagate to both lower and higher levels.
Cortex
These processes map well to specific layers within mammalian cortex. (The cortical layers should not be
confused with different levels of the processing hierarchy: all the layers in a single column participate as
one element in a single hierarchical level). Bottom-up input arrives at layer 4 (L4), whence it propagates to
L2 and L3 for recognition of the invariant content. Top-down activation arrives to L2 and L3 via L1 (the
mostly axonal layer that distributes activation locally across columns). L2 and L3 compare bottom up and
top-down information, and generate either the invariant 'names' when sufficient match is achieved, or the
more variable signals that occur when this fails. These signals are propagated up the hierarchy (via L5) and
also down the hierarchy (via L6 and L1).
Thalamus
To account for storage and recognition of sequences of patterns, a combination of two processes is
suggested. The nonspecific thalamus acts as a 'delay line' – that is, L5 activates this brain area, which re-
activates L1 after a slight delay. Thus, the output of one column generates L1 activity, which will coincide
with the input to a column which is temporally subsequent within a sequence. This time ordering operates
in conjunction with the higher-level identification of the sequence, which does not change in time; hence,
activation of the sequence representation causes the lower-level components to be predicted one after the
other. (Besides this role in sequencing, the thalamus is also active as sensory waystation – these roles
apparently involve distinct regions of this anatomically non-uniform structure.)
Hippocampus
Another anatomically diverse brain structure which is hypothesized to play an important role in hierarchical
cognition is the hippocampus. It is well known that damage to both hippocampi impairs the formation of
long-term declarative memory; individuals with such damage are unable to form new memories of episodic
nature, although they can recall earlier memories without difficulties and can also learn new skills. In the
current theory, the hippocampi are thought of as the top level of the cortical hierarchy; they are specialized
to retain memories of events that propagate all the way to the top. As such events fit into predictable
patterns, they become memorizable at lower levels in the hierarchy. (Such movement of memories down
the hierarchy is, incidentally, a general prediction of the theory.) Thus, the hippocampi continually
memorize 'unexpected' events (that is, those not predicted at lower levels); if they are damaged, the entire
process of memorization through the hierarchy is compromised.
In 2016 Hawkins hypothesized that cortical columns did not just capture a sensation, but also the relative
location of that sensation, in three dimensions rather than two (situated capture), in relation to what was
around it.[1] "When the brain builds a model of the world, everything has a location relative to everything
else" [1] —Jeff Hawkins.
Besides these subjectively satisfying explanations, the framework also makes a number of testable
predictions. For example, the important role that prediction plays throughout the sensory hierarchies calls
for anticipatory neural activity in certain cells throughout sensory cortex. In addition, cells that 'name'
certain invariants should remain active throughout the presence of those invariants, even if the underlying
inputs change. The predicted patterns of bottom-up and top-down activity – with former being more
complex when expectations are not met – may be detectable, for example by functional magnetic resonance
imaging (fMRI).
Although these predictions are not highly specific to the proposed theory, they are sufficiently unambiguous
to make verification or rejection of its central tenets possible. See On Intelligence for details on the
predictions and findings.
The most significant limitation of this theory is its current lack of detail. For example, the concept of
invariance plays a crucial role; Hawkins posits "name cells" for at least some of these invariants. (See also
Neural ensemble#Encoding for grandmother neurons which perform this type of function, and mirror
neurons for a somatosensory system viewpoint.) But it is far from obvious how to develop a mathematically
rigorous definition, which will carry the required conceptual load across the domains presented by
Hawkins. Similarly, a complete theory will require credible details on both the short-term dynamics and the
learning processes that will enable the cortical layers to behave as advertised.
The following models use belief propagation or belief revision in singly connected Bayesian networks.
Hierarchical Temporal Memory (HTM), a model, a related development platform and source
code by Numenta, Inc. (2008).
HtmLib (http://www.numenta.com/phpBB2/download.php?id=130), an alternative
implementation of HTM algorithms by Greg Kochaniak with a number of modifications for
improving the recognition accuracy and speed (2008).
Project Neocortex (https://sourceforge.net/projects/neocortex/), an open source project for
modeling memory-prediction framework (2008).
Saulius Garalevicius' research page (https://web.archive.org/web/20061013134333/htt
p://www.phillylac.org/prediction/), research papers and programs presenting
experimental results with a model of the memory-prediction framework, a basis for the
Neocortex project (2007).
George, Dileep (2005). "A Hierarchical Bayesian Model of Invariant Pattern Recognition in
the Visual Cortex": 1812–1817. CiteSeerX 10.1.1.132.6744 (https://citeseerx.ist.psu.edu/vie
wdoc/summary?doi=10.1.1.132.6744). a paper describing earlier pre-HTM Bayesian model
by the co-founder of Numenta. This is the first model of memory-prediction framework that
uses Bayesian networks and all the above models are based on these initial ideas. Matlab
source code of this model had been freely available for download for a number of years.
Other models
Implementation of MPF (https://web.archive.org/web/20080509193755/http://www.phillylac.o
rg/prediction/2005%2005%20Analysis%20and%20Implementation%20of%20MPF.pdf), a
paper by Saulius Garalevicius describing a method of classification and prediction in a
model that stores temporal sequences and employs unsupervised learning (2005).
M5 (https://web.archive.org/web/20080213005644/http://home.earthlink.net/~gmayhak/M5_h
tm.htm), a pattern machine for Palm OS that stores pattern sequences and recalls the
patterns relevant to its present environment (2007).
BrainGame (https://sourceforge.net/projects/brain-game), open source predictor class which
learns patterns and can be linked to other predictors (2005).
See also
Adaptive resonance theory, a neural network architecture developed by Stephen Grossberg.
Computational neuroscience
Neural Darwinism
Predictive coding
Predictive learning
Sparse distributed memory
References
1. Metz, Cade (October 15, 2018). "A new view of how we think" (https://www.nytimes.com/201
8/10/14/technology/jeff-hawkins-brain-research.html). The New York Times. pp. B1, B4. See:
'Clarity Over a Coffee Cup'
2. Simonite, Tom (April 8, 2015). "IBM tests mobile computing pioneer's controversial Brain
Algorithms" (https://www.technologyreview.com/2015/04/08/11480/ibm-tests-mobile-computi
ng-pioneers-controversial-brain-algorithms/). MIT Technology Review. Retrieved
2015-04-08.
Further reading
Jeff Hawkins (2004), On Intelligence, New York: Henry Holt. Bibliography, Index, 251 pages.
ISBN 0-8050-7456-2
External links
Hierarchical vision algorithm source code & data (http://poggio-lab.mit.edu/codedatasets)
– similar to the Memory-Prediction Framework (from MIT Center for Biological &
Computational Learning (http://poggio-lab.mit.edu))
Group of articles about neuroscience and AI (https://web.archive.org/web/20060510035807/
http://neurosecurity.com/)
– Group of articles and papers supporting Jeff's MPF theory.
MIT Technology Review Monday, February 12, 2007: Building the Cortex in Silicon (http://w
ww.technologyreview.com/read_article.aspx?id=18164&ch=infotech)
Retrieved from "https://en.wikipedia.org/w/index.php?title=Memory-prediction_framework&oldid=1153987808"