Language, Cognition and Neuroscience

ISSN: (Print) (Online) Journal homepage: https://www.tandfonline.com/loi/plcp21

Transforming the neuroscience of language:

estimating pattern-to-pattern transformations of
brain activity

O. Hauk, R. L. Jackson & S. Rahimi

To cite this article: O. Hauk, R. L. Jackson & S. Rahimi (2023): Transforming the neuroscience of
language: estimating pattern-to-pattern transformations of brain activity, Language, Cognition
and Neuroscience, DOI: 10.1080/23273798.2023.2226268

To link to this article: https://doi.org/10.1080/23273798.2023.2226268

© 2023 The Author(s). Published by Informa

UK Limited, trading as Taylor & Francis
Group

Published online: 10 Jul 2023.

Submit your article to this journal

Article views: 318

View related articles

View Crossmark data

Full Terms & Conditions of access and use can be found at

https://www.tandfonline.com/action/journalInformation?journalCode=plcp21
LANGUAGE, COGNITION AND NEUROSCIENCE
https://doi.org/10.1080/23273798.2023.2226268

REGULAR ARTICLE

Transforming the neuroscience of language: estimating pattern-to-pattern

transformations of brain activity
a
O. Hauk , R. L. Jacksona,b and S. Rahimia
a
MRC Cognition and Brain Sciences Unit, University of Cambridge, Cambridge, UK; bDepartment of Psychology, University of York, York, UK

ABSTRACT ARTICLE HISTORY

The cognitive neuroscience of language aims at revealing how linguistic information is Received 14 July 2022
represented and manipulated in the brain to enable communication and meaningful behaviour. Accepted 30 May 2023
An important aspect of the underlying brain processes is the integration and transformation of
KEYWORDS
information across multiple brain systems. In order to understand these processes, a detailed Functional connectivity;
characterisation of brain connectivity is key. For the most accurate characterisation of brain multivariate connectivity;
connectivity, connectivity methods should make use of the full multivariate and multidimensional
multidimensional information available from neuroimaging data. This should include a connectivity; EEG/MEG; fMRI
characterisation of transformations between patterns of activation across brain regions, and in
particular their dependence on stimulus features, task and context. Methods for this type of
analysis in event-related experimental designs have only recently begun to emerge. This paper
describes these novel developments and their potential to transform the neuroscience of
language, with a focus on fMRI and EEG/MEG research.

Introduction
name. A wealth of research has revealed the brain
The neuroscience of language investigates how linguis- regions that are involved in these processes (Binder &
tic information is stored and processed in the brain in Desai, 2011; Friederici & Gierhan, 2013; Hagoort, 2013;
order to allow effective communication through the Jackson, 2021; Poeppel, 2014; Price, 2012; Pulvermuller,
comprehension and production of messages. An essen- 2013), but we are still far from a full understanding of
tial aspect of this endeavour is the transformation of how these regions interact with each other, and in par-
different types of information between different rep- ticular how they transform information among
resentations, such as patterns of brain activation repre- different representations.
senting visual or spoken word forms into the meaning Recent research has begun to explicitly associate
of words and sentences, or patterns of brain activity brain activity with layers of artificial neural networks in
representing our current mental state into motor pro- space (Khaligh-Razavi & Kriegeskorte, 2014) and time
grammes for a verbal utterance or laughter. It is a (Kietzmann, Spoerer, Sorensen, et al., 2019). A range of
common assumption that different aspects of language, methods exists to characterise patterns of activation
such as orthographic, phonological, semantic, and within these layers and compare them with patterns of
articulatory information, are represented in distributed real neural activation in different brain regions, e.g.
patterns of neuronal activity (Hebb et al., 1971; using representational similarity analysis (Higgins et al.,
Hoffman et al., 2018; Mcclelland & Rumelhart, 1985; 2022; Karimi-Rouzbahani et al., 2022; Kriegeskorte,
Rogers et al., 2004; Tomasello et al., 2018). When we 2011; Naselaris et al., 2011). These methods can reveal
use language, we perform transformations among mul- whether similarity structures for patterns of activation
tiple representations of this kind depending on the between layers of neural networks and brain regions
task and context. For example, when naming an object are similar, and therefore likely to represent similar
we transform visual into semantic information, then types of information. For example, the similarity struc-
into phonological information and finally a motor pro- ture in visual brain areas at early latencies reflects low-
gramme that leads to the utterance of the object level object features, which resembles the similarity

CONTACT O. Hauk [email protected] MRC Cognition and Brain Sciences Unit, University of Cambridge, 15 Chaucer Road, Cambridge CB2
7EF, UK
© 2023 The Author(s). Published by Informa UK Limited, trading as Taylor & Francis Group
This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use,
distribution, and reproduction in any medium, provided the original work is properly cited. The terms on which this article has been published allow the posting of the Accepted
Manuscript in a repository by the author(s) or with their consent.
2 O. HAUK ET AL.

structure among patterns in lower layers (i.e. close to the The investigation of brain connectivity is key in this
input) of a deep artificial neural network (Kietzmann, endeavour. However, ‘brain connectivity’ is not a
Spoerer, Sörensen, et al., 2019). In turn, higher layers of simple concept that can be captured in one number. A
the network represent higher-level information about broad range of brain connectivity methods have been
object categories, similar to more anterior temporal proposed, each describing a different type of relation-
regions (e.g. inferior temporal cortex, ITC) in the brain ship among the signals in different brain regions (Basti
at later latencies. However, this does not characterise et al., 2020; Bastos & Schoffelen, 2015; Friston, 2011;
the mapping between nodes in different layers of the Greenblatt et al., 2012; Seth et al., 2015; Valdes-Sosa
network or voxels in different brain regions. It is still an et al., 2011). Until recently, most of these methods
open question whether these pattern transformations focused on univariate signals, i.e. brain activity within
also show similarities between artificial neural network brain regions was pooled across voxels or vertices (com-
and real brain data (illustrated in Figure 1). monly by averaging or principal component analysis).

Figure 1. Illustration of the application of multivariate and multidimensional methods to characterise aspects of artificial or biological
neural network architectures. The example network represents a simple feedforward network with an input layer, a hidden layer and
an output layer. Each layer consists of 10 nodes (putatively corresponding to neurons or cell assemblies). The distribution of activation
levels across all nodes within a layer (illustrated by the size of black circles) represents information at a particular stage of processing.
The blue arrows illustrate connections between nodes of different layers, which transform activation patterns from one layer into
another. In this example, it is assumed that a distributed input pattern is transformed into a sparse output pattern (e.g. a particular
button press) via the hidden layer. Activation and connection strengths were chosen arbitrarily. The network is annotated at the
bottom with different multivariate and multidimensional data analysis methods that characterise different aspects of the network.
While methods to describe patterns within layers or regions as well as the strength of different types of connectivity have already
been established, methods that characterise the transformations of patterns across layers or regions have only just begun to
emerge (Basti et al., 2019; Basti et al., 2020; Rahimi, Jackson, Farahibozorg, & Hauk, 2022).
 LANGUAGE, COGNITION AND NEUROSCIENCE 3

Thus, these methods are only sensitive to covariations of This has mostly been based on anatomical or resting-
overall signal strength between regions. In order to state connectivity. While this can provide an idea
characterise the transformations of brain represen- about possible or likely connection patterns, determin-
tations across regions, we require connectivity ing the actual connectivity for particular stimuli and
methods that take the full multivariate information in tasks requires methods for event-related experimental
each region into account. designs, which have only recently become available
A few methods been proposed for neuroimaging (Basti et al., 2019; Basti et al., 2020; Rahimi, et al.,
analysis that characterise relationships among patterns 2022). Thus, little is known about transformations of dis-
of activations across brain regions (Anzellotti & tributed neural representations in the language domain.
Coutanche, 2018; Basti et al., 2018; Basti et al., 2020; In the following, we will first discuss how these
Geerligs & Henson, 2016). Nevertheless, while these methods could make an impact on the neuroscience
methods can determine whether or not there is a of language. In order to further motivate this approach,
certain relationship among those patterns, they typically we will then summarise previous research on topological
do not describe the transformations of these patterns relationships between brain regions for sensory and
across regions in detail (illustrated in Figure 1). For higher-level brain systems. Finally, we will briefly
example, is every voxel in region X connected to a describe the methods that have been proposed to
specific voxel in region Y? Do multiple voxels from study pattern-to-pattern transformations in fMRI and
region X converge in one voxel in region Y, or vice EEG/MEG data.
versa? Only recently have methods become available
that explicitly estimate the transformations between
Transforming the neuroscience of language
patterns of brain activity (Basti et al., 2018; Rahimi
et al., 2022). Methods of this type have been referred Topographic maps and mappings exist in various brain
to as ‘multi-dimensional connectivity’, i.e. taking into areas at different levels of the processing hierarchy.
account multiple dimensions within regions, in order Some suggestions about their functional role beyond
to distinguish them from ‘multi-variate connectivity’, sensorimotor systems have already been made (Jbabdi
which has usually been used to refer to methods that et al., 2013; Thivierge & Marcus, 2007; Tinsley, 2009;
include more than two regions in the model (Basti Zajzon et al., 2019). How do these concepts relate to
et al., 2020; Rahimi, et al., 2022). the neuroscience of language? How is the transform-
An analysis of pattern transformations is only infor- ation of linguistic information between brain systems
mative if we can expect the mappings or projections reflected in transformations of brain activity patterns
between voxels of different regions to be structured in across brain regions?
a meaningful way. Topographic maps are well estab- It is well-established that patterns of brain activity
lished for sensory brain areas (e.g. Goldman-Rakic, carry information about linguistic stimuli, e.g. about
1988; Pantev et al., 1995; Penfield & Rassmussen, 1950; speech signals (Anumanchipalli et al., 2019; Blank et al.,
Sereno et al., 1994). With respect to connectivity, there 2018; Kocagoncu et al., 2017), written words (Carota
is evidence that topological mappings between et al., 2017; Chan et al., 2011; Devereux et al., 2013;
different regions exist for sensory brain systems, e.g. in Gonzalez Andino et al., 2007), and object categories
visual cortex (Haak et al., 2018; Kaas, 1997). For neuroi- (Cichy et al., 2017; Clarke, 2020; Coutanche & Thomp-
maging researchers the crucial question is whether any son-Schill, 2015; Kietzmann, Spoerer, Sorensen, et al.,
relationships that exist among neurons at the micro- 2019; Kriegeskorte, Mur, Ruff, et al., 2008). The idea
scopic level will be reflected systematically in our neuroi- that conceptual information is represented in distribu-
maging data at the macroscopic level. Studies using ted patterns of brain activation is a core part of current
multivariate pattern analysis (MVPA) or representational neuroscientific approaches to language (Binder &
similarity analysis (RSA) have shown that macroscopic Desai, 2011; Chen et al., 2017; Lambon Ralph et al.,
patterns in individual ROIs carry valuable information 2017; Martin, 2007; Pulvermuller, 2013). However, it is
about stimuli, tasks and brain states (Cichy et al., 2015; not yet known how these patterns are transformed
Kietzmann, Spoerer, Sörensen, et al., 2019; Kriegeskorte, across brain regions, and how these transformations
Mur, and Bandettini, 2008; Stokes et al., 2015). Some are affected by stimulus category, task or context.
authors have suggested that topographic maps and The concepts of topographical maps and mappings
mappings also exist for higher-level cortical areas, e.g. lend themselves to the analogy with channels of infor-
in frontal and parietal regions (Hagler & Sereno, 2006; mation flow in information processing systems. Infor-
Ito et al., 2017; Jbabdi et al., 2013; O’Rawe & Leung, mation comes through different input channels, is
2020; Silver & Kastner, 2009; Thivierge & Marcus, 2007). distributed to the appropriate processing systems
4 O. HAUK ET AL.

where information from different inputs is combined

and abstracted depending on the task at hand, to
finally result in signals to the appropriate output
systems. An obvious problem with psycholinguistic
information is that there is no obvious correspondence
with the physical world and the sensorimotor periphery
of our bodies, such as visual fields corresponding to
areas on the retina, or somatotopic maps corresponding
to different body parts. Information is abstracted away
from its structure in sensory input areas already at very
early stages of processing, such as letter and letter
string processing (Vinckier et al., 2007). These abstract
representations have to be processed in a highly
flexible manner, depending on task demands and
context, i.e. information from different brain systems
has to be integrated. It is therefore unclear whether
we will be able to find frequently occurring sub-struc- Figure 2. Possible pattern transformations involved in a sim-
plified illustration of a word recognition task. The neural pat-
tures of a larger network or of transformations within terns associated with an input word form are transformed into
this network, so-called ‘motifs’ in graph theory, with a a pattern in a multimodal semantic core area. We here
universal functional role (see Figure 5 and the next assume the connectivity of the corresponding brain systems
section for illustrations). In the following section, we to be many-to-many. The pattern in the semantic core area is
will describe studies that have focussed on features of then transformed into a response (e.g. a button press if it
refers to an animal). This likely involves convergent connectivity
transformations that reflect intuitive connectivity pat-
from a distributed semantic representation to a localised motor
terns such as convergence, divergence and one-to-one output. The semantic representation is affected by converging
mappings (Jbabdi et al., 2013; Thivierge & Marcus, top-down signals from multiple brain systems reflecting task
2007; Tinsley, 2009). demands, context and mental state. The semantic represen-
Here, we would also like to highlight a possible tation also involves divergent connectivity to distributed sensor-
feature of transformation that is likely to be omnipresent imotor areas. This connectivity is divergent to multiple modal
brain areas (e.g. visual, auditory, motor cortices), but for each
and at the same time hard to analyse, i.e. many-to-many particular modality may be many-to-many. M2M: many-to-
(M2M) mappings (every voxel can be connected to every many; conv: convergent; div: divergent
other voxel). Distance in semantic space does not
necessarily correspond to distance between neurons or
neuron populations if the code is distributed. At the In Figure 2 we illustrate how pattern transformations
same time, even M2M mappings may still differ system- could be applied to study the transformation of input
atically depending on stimulus features or categories as word forms (e.g. speech and text) to semantic represen-
well as on task and context. While these mappings may tations. We here do not commit to a particular model or
not be decomposable into a finite set of simple motifs, brain areas, and make the simplified assumption that the
the above-mentioned concepts may still prove useful word form will be transformed directly from peripheral
to characterise the sensitivity of M2M mappings to brain areas into a semantic brain system, and will first
experimental manipulations. They may reflect a reach a semantic core area which in turn has connec-
complex mixture of different connectivity patterns, but tions to distributed sensorimotor systems. In order to
analysing some characteristics of this mixture may still allow flexible information retrieval depending on
inform us about the mechanisms of language proces- context and task demands, the semantic system is
sing. We will illustrate this here with two simplified linked to a distributed top-down system. Finally, the
examples. semantic system is linked to an output system, e.g. for
The following examples were created with a focus on a button press or speech act.
clarity, i.e. to illustrate the novel approach to connec- This illustration contains simplifications that will cer-
tivity analysis and its possible application to questions tainly have to be elaborated in the future. For
in the neuroscience of language. While they are inspired example, we only included unidirectional feed-forward
by current neuroscientific theories, we know that the connections, while in reality interactions among brain
reality is much more complex. Hopefully, our new systems including feedback connections is more likely.
methods will be enable us to disentangle some of The terms ‘convergence’ and ‘divergence’ only make
these complexities in the future. sense if a direction of information flow has been
 LANGUAGE, COGNITION AND NEUROSCIENCE 5

specified – a convergent flow in one direction is diver-

gent in the other. For this reason, we simplified our
toy example in order to provide a clear interpretation
of the different pattern transformation.
We cannot expect a simple mapping between word
forms (e.g. strings of letters or phonemes) and semantic
representations in the semantic core area. Letters and
phonemes in different positions are not indicative of
the semantic category or identity of a particular stimu-
lus. Word form and semantic representation are both
distributed within their respective sub-systems. Thus,
the pattern transformation between these two sub-
systems will be M2M.
Furthermore, the same word can have multiple mean-
ings depending on the context (MacGregor et al., 2020;
Rodd, 2020; Rubinstein et al., 1970)}. The semantic infor-
mation retrieved may also depend on the task demands
(Chen et al., 2015; Clarke et al., 2011; Rahimi, et al., 2022;
Rogers et al., 2005). Information about context and task
demands is represented in brain systems distributed
Figure 3. Possible pattern transformations involved in a sim-
across multiple brain areas, likely to include the multiple plified illustration of sentence comprehension. Sentences are
demand system and inferior frontal brain areas (Binder & presented word by word. The neural patterns representing the
Desai, 2011; Hoffman et al., 2018; Jackson, 2021). Connec- meaning of each incoming word are represented by a 4-by-4
tions from this system may converge on the semantic matrix. The pattern transformation (matrix next to black
arrows) transforms each word into an evolving representation
representation system in order to enable the retrieval
of the sentence meaning. We here assume that a predictable
of context- and task-relevant information for an appro- sentence ending (top) results in a sparser final transformation
priate response. At the same time, because the systems as well as sentence representation compared to an unpredict-
are required to be flexible with respect to context and able sentence ending (bottom).
tasks, we cannot expect a simple connectivity pattern
between individual regions. Thus, the transformation of
patterns from the widely distributed top-down system individual input words. The matrices in the second row
to patterns in the semantic system will be generally con- (next to the thick black arrow) illustrate the transform-
vergent as well as M2M. Similarly, the connections from ation of these patterns into an output pattern that rep-
the semantic core system to multiple distributed sensor- resents the current representation of the sentence
imotor areas will be divergent, while connections meaning. Words are coming in one-by-one over time
between the semantic core system and individual sen- (e.g. by fixating them one after the other during
sorimotor regions will be M2M. Finally, the selection of reading, or hearing them in sequence during speech),
a response requires convergent connectivity of distribu- and the representations of every input word are trans-
ted patterns within the semantic system onto a particular formed differently, e.g. depending on preceding
response, e.g. a button press with a specific finger. context and changing expectations, in order to update
It is likely that pattern transformations can change the sentence level meaning.
over time, e.g. as a sentence unfolds word by word. One possibility is that the activation patterns repre-
Figure 3 illustrates how features of pattern transform- senting the input words only depend on the semantic
ations such as their sparsity may be related to exper- content of the words, but the transformations from
imental manipulations such as the predictability of a single word representations to sentence meaning and
word in a sentence. The upper half shows an example the patterns representing sentence meaning are
for a simple three-word sentence with a predictable shaped by context. In the example of a predictable sen-
final word (‘Birds can fly’). We are all aware of most tence ending, the first word alone (‘Birds’) is not predic-
birds’ ability to fly and highly familiar with this tive of the overall sentence meaning. The transformation
concept. Flying is the one prominent ability that birds from single word to sentence meaning is therefore M2M,
are known for. and the sentence meaning representation is still unspe-
The first row of arbitrarily chosen 4-by-4 matrices cified and widely distributed, since many possible rep-
illustrates the patterns representing the concept of the resentations are at least partially activated. With the
6 O. HAUK ET AL.

next incoming word (‘can’), a prediction of the sentence 2009), and therefore transformations can as well. Meta-
meaning can be generated, leading to a sparser pattern bolic neuroimaging methods such as fMRI average
transformation as well as a sparser sentence meaning brain activity over several seconds, and may thus
representation. When the final word arrives (‘fly’), the conflate the effects of feedforward and feedback pro-
prediction is confirmed, leading to an even sparser cesses (Hauk, 2016). This problem is not specific to
pattern transformation and the generation of a simple pattern-based decoding or transformation methods,
and sparse sentence meaning representation. but applies to conventional univariate analyses as well.
For unpredictable sentences (bottom half of Figure 3), The obvious way to address this would be to estimate
the input patterns are similar to those for the predictable pattern transformations using source-estimated EEG/
sentence before. In fact, only the first word differs. MEG data (Rahimi, et al., 2022). Unfortunately, the
Humans are known to be able to do many things, and spatial resolution of these data is fundamentally
flying is usually not the first ability that comes to mind. limited (Hauk et al., 2022; Hämäläinen et al., 1993;
Thus, the pattern transformations and sentence Molins et al., 2008), and the reliable estimation of all-
meaning representations do not become sparser over to-all voxel-to-voxel (or vertex-to-vertex) transform-
time. While the sentence is grammatical and meaningful, ations is currently not possible. Recent developments
it refers to a less familiar concept that requires more with non-cryogenic on-scalp sensors (e.g. ‘OPMs’)
information to make sense, e.g. that humans cannot promise a significant improvement of spatial resolution
naturally fly but need certain machines to do it. of future MEG systems (Boto et al., 2018; Iivanainen
While these examples are highly speculative, they et al., 2017). Furthermore, multidimensional connectivity
show how the concept of pattern transformations methods can be applied to intracranial EEG or electro-
could be used to test specific hypotheses in terms of corticography (ECoG) data, with the caveat that brain
pattern-to-pattern connectivity in neuroimaging data. coverage with these invasive methods is usually very
In the following, we will discuss more ways in which limited (Chen et al., 2016; Rogers et al., 2021). It will
experimental manipulations may affect pattern have to be seen how the ideas and concepts outlined
transformations. above can be tested using spatio-temporal neuroima-
In the above examples, we already discussed the ging methods, with the exciting opportunity to describe
possibility that context and task demands may modulate the temporal dynamics of pattern transformations.
pattern transformations. It seems plausible that a task In the following two sections, we will first motivate
that requires the chain of information processing to con- the endeavour to analyse neural pattern transformation
verge on a specific representation, e.g. an individual based on evidence for topographic mappings in sensory
name or face, should also show converging connectivity and higher-order cortical regions, and then describe
patterns. However, this depends on how semantic analysis methods that have recently been proposed for
specificity of the representations is related to the sparse- this purpose.
ness or complexity of neuronal activity patterns that rep-
resent them. A specific concept may still be represented
The topographic organisation of brain
by a widely distributed pattern. While for concepts such
regions and networks
as famous personalities sparse representations have
been suggested (Quiroga, 2012), semantic represen- Before discussing methods to estimate pattern trans-
tations are generally assumed to be widely distributed, formations from neuroimaging data, we will briefly
possibly including multimodal brain areas (Binder & present evidence for topographic maps and mappings
Desai, 2011; Lambon Ralph et al., 2017; Pulvermuller, between cortical regions, in order to motivate the
2013). Other tasks may require more divergent infor- search for systematic neuronal pattern transformations.
mation flow, e.g. building associations with a specific Topographic maps are well established for sensorimotor
concept (such as ‘name objects related to monkey!’). brain areas (e.g. Goldman-Rakic, 1988; Kaas, 1997;
In our illustrations we assumed that the information Pantev et al., 1995; Patel et al., 2014; Penfield & Rassmus-
flow among brain systems is unidirectional, which sen, 1950; Sereno et al., 1994). For example, visual area
lends itself to the characterisation in terms of transform- V1 is retinotopically organised and contains a complete
ation motifs (such as convergence and divergence). (although distorted) map of the image projected onto
However, feedback connections and recurrent activation the retina (Felleman & Van Essen, 1991). Along the
flow are common in the brain (Clarke et al., 2011; visual hierarchy, the topographical arrangement of
Kietzmann, Spoerer, Sörensen, et al., 2019; Lamme & neurons is preserved although receptive field sizes
Roelfsema, 2000). It has also been stated before that increase (Figure 4(A)). Importantly, this topographic
topographic maps can change over time (Tinsley, arrangement within regions is also reflected in
 LANGUAGE, COGNITION AND NEUROSCIENCE 7

Figure 4. Examples of topographic projections based on anatomic and resting state data. (A) Illustration of retinotopy and topo-
graphic projections between visual areas V1 and V3. The topographical arrangement of neurons is preserved although receptive
field sizes increase. (B) Somatotopy in the supplementary motor area (SMA) revealed by functional connectivity (resting state)
between SMA and primary motor areas. Different colours in the SMA (right) correspond to connectivity with different seeds in
primary motor cortex (left). (C) Resting-state connectivity (fMRI) between inferior parietal sulcus (white grid points). Colour coding
indicates which grid point a voxel correlates most with. Similar gradients (white arrows) are observed in several brain regions. (D)
Comparison between eccentricity maps (left) and resting-state functional connectivity maps (right, as in C with grid on medial
surface of occipital lobe) for fMRI. This figure was published as Figure 2 in Jbabdi et al. (2013), Copyright Elsevier.

topographic structural connectivity between regions, surprising that topographic maps and mappings, where
with neuronal fibres that originate close to each other connectivity between areas preserves the spatial
also ending close to each other (Kaas, 1997). Such topo- arrangements within areas, have also been reported in
graphic connectivity has also been reported between frontal, parietal and temporal areas, with evidence
primary motor areas and the supplementary motor mostly based on structural MRI or resting state fMRI ana-
area (SMA, Figure 4(B)). lyses (Hagler & Sereno, 2006; Ito et al., 2017; Jbabdi et al.,
For sensorimotor regions, a topographic organisation 2013; Patel et al., 2014; Silver & Kastner, 2009; Thivierge &
of neurons and their connections is plausible, as it allows Marcus, 2007) (Figure 4).
efficient groupings of neurons that frequently interact For example, voxels with high resting state corre-
and process related types of information (Kaas, 1997). lations project to voxels that in turn show high resting
The structure of sensorimotor regions ultimately state correlations among themselves. Figure 4(C)
reflects the physical structure of the real world, since shows how similar functional connectivity gradients
visual features that are close to each other are likely to can be observed in different parts of the cortex. Func-
be part of the same object, sound frequencies that are tional resting state connectivity was computed for
close to each other are likely to reflect the same sound seed regions in inferior parietal sulcus (IPS) and the
source, and fingers that are close to each other are whole cortex. The colour coding reflects which seed-
likely to be used together. The processing of information point a voxel correlates most with. The white arrows
in these areas is relatively automatised and rigid. highlight areas where the gradients observed in other
It is less clear whether topographic organisation can cortical areas are similar to those in the original grid of
also be expected for higher level cortical areas. The infor- seed points. Figure 4(D) (right) shows a similar analysis
mation represented in these areas is usually abstract and for seed points in a grid along the medial surface of
needs to be processed flexibly depending on task, the occipital lobe, indicating that these connectivity gra-
context and mental state. There is no obvious spatial dients are a common feature in higher level cortices.
structure that optimally represents this abstract infor- However, the functional relevance of these topo-
mation for flexible information processing. It is therefore graphic arrangements is not yet established. In order
8 O. HAUK ET AL.

Figure 5. Examples of network motifs. (A) Motifs are frequently occurring smaller patterns of connectivity within a larger network.
They may be considered as building blocks of the network serving particular computations of functions. This figure was published as
Figure 14 in Costa et al. (2007), Copyright Taylor & Francis. (B) Possible motifs for pattern-to-pattern connectivity between brain
regions. (a) One-to-one connectivity, (b) Divergence, (c) Convergence. (C) This figure was published as Figure 2 in Tinsley (2009), Copy-
right Elsevier.

to understand the mechanisms within a network, it associations from memory. Tinsley (2009) associated
would be useful to know the functions of at least point-to-point mappings with filtering, divergence with
some of its sub-structures where those can be identified. amplification and convergence with averaging. Thi-
Frequently occurring sub-structures of a larger network vierge and Marcus (2007) highlighted the apparent
are referred to as ‘motifs’ in graph theory (see Figure 5 prevalence of one-to-one mappings across the cortex,
for illustrations). Some general proposals for a role of and propose that these have an essential role in analogi-
certain motifs of topographical connections in higher- cal reasoning. For example, in order to complete the
order cortical regions exist (Jbabdi et al., 2013; Thivierge analogy ‘apple is to fruit as asparagus is to X’, one
& Marcus, 2007; Tinsley, 2009; Zajzon et al., 2019). needs separate feature-to-feature mappings across two
Some motifs lend themselves to an interpretation in semantic categories.
terms of information flow: convergent projections may The authors cited above acknowledge that ideas
reflect the integration of information, e.g. from about the functional role of higher-order topographic
different receptive fields or different sensory areas; connectivity is still speculative, and evidence is scarce
diverging projections may distribute information to and often circumstantial. Most of the evidence stems
different brain systems for different types of parallel pro- from anatomical connectivity and resting state studies.
cessing, e.g. for decision making and retrieving However, in order to determine the functional role of
 LANGUAGE, COGNITION AND NEUROSCIENCE 9

network connectivity and pattern transformations, we representational similarity analysis (RSA (Kriegeskorte,
need to develop paradigms and methods to study Mur, & Bandettini, 2008; Laakso & Cottrell, 2000)),
them in task-based and event-related designs. In the fol- describes how the similarity structure of activation pat-
lowing, we will briefly describe methods that have terns in one region (e.g. the intercorrelation matrix of
recently become available to test for pattern-to-pattern activity patterns for different stimuli) compares to the
relationships between brain regions. similarity structure of patterns in another region (Krie-
geskorte, Mur, & Bandettini, 2008). This method is
based on multivariate pattern information within each
Methods for the estimation of pattern-to-
region, but the connectivity measure is not based on a
pattern transformations
voxel-by-voxel relationship between the regions. In
In this section, we briefly describe which methods are other words, it measures whether the representational
currently available to estimate multidimensional con- structures in two regions are similar or not, but does
nectivity, in order to highlight that transformation- not describe how this structure is mapped from one
based connectivity methods have so far been largely region to another. This can be applied to fMRI data
neglected. We will focus in particular on methods that where activation patterns are first estimated for individ-
are suitable for task-based and event-related designs. ual stimuli, as well as for EEG/MEG on a sample-by-
For more detailed methodological information, see the sample basis.
recent review by Basti et al. (2020) as well as the EEG/ In an approach specific to EEG/MEG the time course
MEG studies by Rahimi, et al. (2022) and (2023). of similarity measures for a seed region is used to
In principle, these methods can be applied to both predict the corresponding time course of the target
fMRI and EEG/MEG data. However, it is well-known region, e.g. using metrics based on Granger-causality
that the BOLD response commonly measured with (Goddard et al., 2016; Karimi-Rouzbahani et al., 2022;
fMRI has very low temporal resolution in the range of Kietzmann, Spoerer, Sorensen, et al., 2019). This can
several seconds, and is therefore rarely analysed in the reveal whether successful decoding of a specific type
frequency domain or on a sample-by-sample basis. In of information in one region at one latency can predict
contrast, EEG/MEG have high temporal resolution in decoding of information in another region at another
the millisecond range and can provide rich information latency, but as before does not estimate the transform-
in the time–frequency domain and on a more fine- ations among patterns.
grained temporal scale, but its limited spatial resolution A method that comes closer to estimating pattern
complicates the interpretation of voxel-to-voxel relation- transformations, Multivariate Pattern Dependence
ships among regions. In the following, we will group (MVPD), determines how well a multivariate time series
different methods into categories depending on what in one region can predict that in another region (Anzel-
type of information they are primarily based on, and lotti et al., 2017; Anzellotti & Coutanche, 2018). In this
will point out where appropriate for which measure- approach, the dimensionality of each multivariate time
ment modality (i.e. fMRI or EEG/MEG) they are most series is first reduced by using PCA and selecting only
suitable. a few dominant principal components. The statistical
A measure that was introduced for fMRI analysis is dependency between the reduced time series can then
Multivariate distance correlation, which reflects linear be tested using linear regression. For example, if p prin-
and nonlinear multidimensional relationships between cipal components are chosen per region, then a square
two time-varying activation patterns (Geerligs & transformation matrix T with dimension p-x-p (i.e. p
Henson, 2016; Székely et al., 2007). It measures columns and p rows) can be estimated. The cross-vali-
whether distances between patterns at different time dated goodness-of-fit for this transformation serves as
points in one region correlate with distances between a measure of functional connectivity between the
patterns in another region. This measure is sensitive to regions. The statistical dependency between the
a larger range of statistical dependencies between reduced time series can also be estimated with nonlinear
time series than a linear correlational approach, but it methods such as neural networks instead of linear
does not provide an estimate for the pattern-to- regression (Anzellotti et al., 2017). In principle this
pattern transformations across regions. method can be applied to fMRI as well as EEG/MEG
Another category of functional connectivity methods data, but to our knowledge it has so far only been
is based on first computing multivariate metrics charac- applied in the fMRI domain. The use of PCA to reduce
terising pattern similarities per region, and then estab- the dimensionality of multidimensional time series
lishing region-to-region relationships for these metrics. requires the definition of a latency range. In the case
For example, representational connectivity, based on of fMRI this prevents its application in event-related
10 O. HAUK ET AL.

designs. While in principle this would be possible for suggested for resting-state fMRI data (Cole et al., 2016;
EEG/MEG, this approach sacrifices temporal resolution, Ito et al., 2017). The transformation matrix between
i.e. it cannot be applied on a millisecond-by-millisecond two regions is estimated based on the correlations of
basis. resting-state activation time-courses for voxel pairs of
Furthermore, the application of PCA per region the two regions. These transformations can then be
removes information about individual voxels, since used to predict activity across regions for task-induced
every component reflects a weighted combination of activation (‘information transfer mapping’). While this
all voxels within each region. Thus, as for represen- provides a pattern-to-pattern transformation matrix, it
tational connectivity and distance correlation, it pro- shares the limitation with MVPD above that it requires
vides a measure of functional connectivity strength but correlations over longer time periods (such as a whole
does not estimate the pattern transformation. Because block of data). It is therefore not well-suited for event-
PCA is applied over time, the method was applied to related experimental fMRI designs and sacrifices tem-
estimate connectivity across the whole datasets in two poral resolution in the case of EEG/MEG.
fMRI experiments, i.e. it did not distinguish connectivity A similar logic has been applied to EEG/MEG connec-
between different tasks or stimuli (Anzellotti et al., 2017). tivity methods, where univariate methods such as
In principle, it is possible to take any unidimensional (lagged) coherence and phase-slope index have been
connectivity measure (e.g. correlation or coherence extended to the multidimensional case by applying
between time series) and apply it for every combination them to all voxel-pairs between two regions and com-
of voxels in two regions, which will result in a pattern-to- puting a summary metric (e.g. the mean). This has for
pattern connection matrix. Such an approach has been example resulted in the multivariate phase-slope index

Figure 6. Estimating features of pattern-to-pattern transformations between brain regions for event-related data. Activation patterns
for individual stimuli are arranged as columns in one matrix per region (X and Y, respectively). The transformation T of patterns from X
to Y can for example be estimated from the linear equation Y = TX using cross-validated ridge regression. Different metrics can be
computed for the transformation T that describe different aspects of the regions’ pattern-to-pattern connectivity. This figure was pub-
lished as Figure 1 in Basti et al. (2019) (PLOS, Open Access).
 LANGUAGE, COGNITION AND NEUROSCIENCE 11

Figure 7. Examples of multidimensional connectivity results. (A) Representational dissimilarity matrices for fMRI patterns for pictures
of objects. Every element within these matrices reflects the dissimilarity between patterns for two individual pictures (on the diagonal
for identical pictures). Left panels: dissimilarity (correlation distance) among the multivariate fMRI activation patterns for inferior tem-
poral cortex (ITC), fusiform face area (FFA) and parahippocampal place area (PPA). Right panels: Same as on the left, but here the
multivariate patterns in ITC, FFA and PPA were estimated from linear transformations of patterns from early visual cortex (EVC).
Some characteristic patterns of results in the left panels are also visible in the right panels, e.g. in the grey boxes for ITC and FFA
that highlight dissimilarity values for face stimuli. Face stimuli do not pop out in the PPA. This shows that pattern transformations
from EVC to higher-level areas capture stimulus-relevant information. This figure was published as Figure 5 in Basti et al. (2019)
(PLOS, Open Access). (B) Inter-regional Connectivity Matrix (ICM) for EEG/MEG source estimates for unidimensional (top left) and multi-
dimensional (bottom right) connectivity. Shown is the statistical contrast between two word recognition tasks with different semantic
processing depths. Each smaller matrix represents a Temporal Transformation Matrix (TTM), i.e. the explained variance for linear trans-
formations of patterns across two regions (x- and y-axes of larger ICM, respectively) and two latencies (x- and y-axes of TTMs, respect-
ively). The multidimensional method produces more statistically significant connectivity (in yellow) than its unidimensional
counterpart. This figure was published as Figure 8 in Rahimi et al. (2023) (Elsevier, Open Access).

(Basti et al., 2018) and the multivariate interaction information about the voxel-to-voxel dependencies
measure (Ewald et al., 2012; Marzetti et al., 2013). So between the regions.
far, these methods have only been used to describe This offers opportunities to define metrics that define
the strength of connectivity between regions, but not particular features of the transformation. Basti et al.
to characterise the pattern transformations. In principle, (2019) proposed two such metrics in addition to the
this information would be available from these methods, goodness-of-fit: (1) sparsity, describing to what degree
since connectivity strengths are estimated for every pair the transformation consists of one-to-one mappings,
of voxels. and (2) pattern deformation, describing the degree to
Only recently have methods been suggested to expli- which the transformation rotates or rescales the patterns
citly estimate the pattern transformations between (Figure 7). They applied this method to event-related
regions for fMRI (Basti et al., 2019) and EEG/MEG data data from an existing fMRI dataset. Instead of estimating
(Rahimi, et al., 2022) (illustrated in Figures 6 and 7). The the transformation matrix across the samples of the con-
approach is similar to MVPD as described above, but it tinuous time series (as in Anzellotti et al. (2017)), they did
is applied to activity patterns without reducing their so across activation patterns for individual stimuli in an
dimensionality through PCA. Thus, the estimated trans- event-related manner that allowed comparisons
formations preserve the dimensionality of the patterns between different stimulus categories (faces and
in the corresponding brain regions. As before, the cross- places). They found that the estimated transformations
validated goodness-of-fit for this transformation can be were surprisingly sparse. It is therefore an interesting
used as a measure of functional connectivity between tool for the investigation of connectivity for different
ROIs. Importantly, in this case the transformation contains types of linguistic stimuli.
12 O. HAUK ET AL.

Future methods developments should also provide exciting to see how this will transform the neuro-
measures that characterise network or transformation science of language.
motifs (see Figure 5), especially with respect to those
highlighted in our previous section and examples
(Figures 2 and 3), i.e. convergence, divergence, one-to- Note
one and many-to-many mappings.1 It will be exciting 1. We would like to provide a novel definition of a ‘diver-
to investigate whether the results for these metrics gence and convergence index’ (DoCoI) here, although
obtained in real neural data resemble those obtained it has not been employed in any of the studies high-
lighted in this manuscript, in order to move this research
in different types of artificial neural networks, as has
forward: A DiCoI can be computed for a target voxel as
been suggested for pattern similarities between brain follows:
regions and network layers in a number of recent
studies (Higgins et al., 2022; Karimi-Rouzbahani et al., (1) Pick the largest connection strength C between
2022; Khaligh-Razavi & Kriegeskorte, 2014; Kietzmann, target voxel and all other relevant voxels.
Spoerer, Sorensen, et al., 2019; Kriegeskorte, 2011; (2) Compute the average difference between C and all
other connection strengths (not including C), and
Naselaris et al., 2011). divide by C.
Importantly, this method of estimating pattern trans- This value will be 1 if all other connections are
formations can be applied to EEG/MEG data on a zero (i.e., connectivity is one-to-one), and zero if all
sample-by-sample basis, which means it is suitable for other connections have the same strength (i.e.,
event-related designs and exploits the full temporal res- highly divergent from target voxel, or highly conver-
gent towards target voxel). Thus:
olution of EEG/MEG data. Not only can it estimate
(3) Define the DiCoI as 1 minus the value obtained from
pattern transformations between pairs of regions, but 2 (such that higher values reflect larger convergence
it can also estimate these transformations for pairs of or divergence).
latencies. For example, one can test how patterns in This index can be computed between all voxels
left anterior temporal lobe at 100 ms are related to pat- within a target ROI and all voxels in other ROIs (or
terns in inferior frontal gyrus at 200 ms, and for all other the whole brain), and averaged across target voxels.
combinations of latencies. This method has recently
been introduced as Time-Lagged Multidimensional
Pattern Connectivity (TL-MDPC, (Rahimi, Jackson, Farahi- Acknowledgments
bozorg, et al., 2023)). For any pair of regions this results For the purpose of open access, the authors have applied a CC
in a temporal transformation matrix (TTM). If this is done BY public copyright license to any Author Accepted Manuscript
for pairs among multiple regions, these TTMs can in turn version arising from this submission.
be arranged into an Inter-regional Connectivity Matrix
(ICM), as illustrated in Figure 6(B) (based on (Rahimi,
Disclosure statement
Jackson, Farahibozorg, et al., 2023)). As for MVPD (Anzel-
lotti et al., 2017), these pattern-to-pattern relationships No potential conflict of interest was reported by the author(s).
can also be estimated using nonlinear methods such
as artificial neural networks. A nonlinear version of TL-
MDPC has recently been reported to produce very
Funding
similar results to its linear counterpart, suggesting that This work was supported by intramural funding from the
linear methods can provide a reasonable and computa- Medical Research Council (MC_UU_00005/18).
tionally efficient approach to multidimensional connec-
tivity (Rahimi et al., 2023).
ORCID
O. Hauk http://orcid.org/0000-0003-0817-6054
Conclusion
Multidimensional connectivity methods have recently
offered us powerful new approaches to investigate
References
the human brain connectome. The ability to estimate Anumanchipalli, G. K., Chartier, J., & Chang, E. F. (2019). Speech
the pattern-to-pattern transformations of brain activity synthesis from neural decoding of spoken sentences.
Nature, 568(7753), 493–498. https://doi.org/10.1038/
across space and time will enable us to study the
s41586-019-1119-1
transformations of information across brain systems Anzellotti, S., Caramazza, A., & Saxe, R. (2017). Multivariate
in more detail. While this will be applicable in all pattern dependence. Plos Computational Biology, 13(11),
areas of cognitive neuroscience, it will be particularly e1005799. https://doi.org/10.1371/journal.pcbi.1005799
 LANGUAGE, COGNITION AND NEUROSCIENCE 13

Anzellotti, S., & Coutanche, M. N. (2018). Beyond functional magnetoencephalography and deep neural networks.
connectivity: Investigating networks of multivariate rep- Neuroimage, 153, 346–358. https://doi.org/10.1016/j.
resentations. Trends in Cognitive Sciences, 22(3), 258–269. neuroimage.2016.03.063.
https://doi.org/10.1016/j.tics.2017.12.002 Cichy, R. M., Ramirez, F. M., & Pantazis, D. (2015). Can visual infor-
Basti, A., Mur, M., Kriegeskorte, N., Pizzella, V., Marzetti, L., & mation encoded in cortical columns be decoded from mag-
Hauk, O. (2018a). Analysing linear multivariate pattern trans- netoencephalography data in humans?. Neuroimage, 121,
formations in neuroimaging data. bioRxiv. https://doi.org/ 193–204. https://doi.org/10.1016/j.neuroimage.2015.07.011
10.1101/497180 Clarke, A. (2020). Dynamic activity patterns in the anterior tem-
Basti, A., Mur, M., Kriegeskorte, N., Pizzella, V., Marzetti, L., & poral lobe represents object semantics. Cognitive
Hauk, O. (2019). Analysing linear multivariate pattern trans- Neuroscience, 11(3), 111–121. https://doi.org/10.1080/
formations in neuroimaging data. PLoS One, 14(10), 17588928.2020.1742678
e0223660. https://doi.org/10.1371/journal.pone.0223660 Clarke, A., Taylor, K. I., & Tyler, L. K. (2011). The evolution of
Basti, A., Nili, H., Hauk, O., Marzetti, L., & Henson, R. N. (2020). meaning: Spatio-temporal dynamics of visual object recog-
Multi-dimensional connectivity: A conceptual and math- nition. Journal of Cognitive Neuroscience, 23(8), 1887–1899.
ematical review. Neuroimage, 221, 117179. https://doi.org/ https://doi.org/10.1162/jocn.2010.21544
10.1016/j.neuroimage.2020.117179 Cole, M. W., Ito, T., Bassett, D. S., & Schultz, D. H. (2016). Activity
Basti, A., Pizzella, V., Chella, F., Romani, G. L., Nolte, G., & flow over resting-state networks shapes cognitive task acti-
Marzetti, L. (2018b). Disclosing large-scale directed func- vations. Nature Neuroscience, 19(12), 1718–1726. https://doi.
tional connections in MEG with the multivariate phase org/10.1038/nn.4406
slope index. Neuroimage, 175, 161–175. https://doi.org/10. Costa, L. D., Rodrigues, F. A., Travieso, G., & Boas, P. R. V. (2007).
1016/j.neuroimage.2018.03.004 Characterization of complex networks: A survey of measure-
Bastos, A. M., & Schoffelen, J. M. (2015). A tutorial review of ments. Advances in Physics, 56(1), 167–242. https://doi.org/
functional connectivity analysis methods and their interpre- 10.1080/00018730601170527
tational pitfalls. Frontiers in Systems Neuroscience, 9, 175. Coutanche, M. N., & Thompson-Schill, S. L. (2015). Creating con-
https://doi.org/10.3389/fnsys.2015.00175 cepts from converging features in human cortex. Cerebral
Binder, J. R., & Desai, R. H. (2011). The neurobiology of semantic Cortex, 25(9), 2584–2593. https://doi.org/10.1093/cercor/
memory. Trends in Cognitive Sciences, 15(11), 527–536. bhu057
https://doi.org/10.1016/j.tics.2011.10.001 Devereux, B. J., Clarke, A., Marouchos, A., & Tyler, L. K. (2013).
Blank, H., Spangenberg, M., & Davis, M. H. (2018). Neural predic- Representational similarity analysis reveals commonalities
tion errors distinguish perception and misperception of and differences in the semantic processing of words and
speech. The Journal of Neuroscience, 38(27), 6076–6089. objects. The Journal of Neuroscience, 33(48), 18906–18916.
https://doi.org/10.1523/JNEUROSCI.3258-17.2018 https://doi.org/10.1523/JNEUROSCI.3809-13.2013
Boto, E., Holmes, N., Leggett, J., Roberts, G., Shah, V., Meyer, S. Ewald, A., Marzetti, L., Zappasodi, F., Meinecke, F. C., & Nolte, G.
S., & Brookes, M. J. (2018). Moving magnetoencephalogra- (2012). Estimating true brain connectivity from EEG/MEG
phy towards real-world applications with a wearable data invariant to linear and static transformations in
system. Nature, 555(7698), 657–661. https://doi.org/10. sensor space. Neuroimage, 60(1), 476–488. https://doi.org/
1038/nature26147 10.1016/j.neuroimage.2011.11.084
Carota, F., Kriegeskorte, N., Nili, H., & Pulvermuller, F. (2017). Felleman, D. J., & Van Essen, D. C. (1991). Distributed hierarch-
Representational similarity mapping of distributional ical processing in the primate cerebral cortex. Cerebral
semantics in left inferior frontal, middle temporal, and Cortex, 1(1), 1–47. https://doi.org/10.1093/cercor/1.1.1-a
motor cortex. Cerebral Cortex, 27(1), 294–309. https://doi. Friederici, A. D., & Gierhan, S. M. (2013). The language network.
org/10.1093/cercor/bhw379 Current Opinion in Neurobiology, 23(2), 250–254. https://doi.
Chan, A. M., Halgren, E., Marinkovic, K., & Cash, S. S. (2011). org/10.1016/j.conb.2012.10.002
Decoding word and category-specific spatiotemporal rep- Friston, K. J. (2011). Functional and effective connectivity: A
resentations from MEG and EEG. Neuroimage, 54(4), 3028– review. Brain Connectivity, 1(1), 13–36. https://doi.org/10.
3039. https://doi.org/10.1016/j.neuroimage.2010.10.073 1089/brain.2011.0008
Chen, L., Lambon Ralph, M. A., & Rogers, T. T. (2017). A unified Geerligs, L., & Henson, R. N. (2016). Functional connectivity and
model of human semantic knowledge and its disorders. structural covariance between regions of interest can be
Nature Human Behaviour, 1(3), 3. https://doi.org/10.1038/ measured more accurately using multivariate distance cor-
s41562-016-0039 relation. Neuroimage, 135, 16–31. https://doi.org/10.1016/j.
Chen, Y., Davis, M. H., Pulvermuller, F., & Hauk, O. (2015). Early neuroimage.2016.04.047
visual word processing is flexible: Evidence from spatiotem- Goddard, E., Carlson, T. A., Dermody, N., & Woolgar, A. (2016).
poral brain dynamics. Journal of Cognitive Neuroscience, 27 Representational dynamics of object recognition:
(9), 1738–1751. https://doi.org/10.1162/jocn_a_00815. Feedforward and feedback information flows. Neuroimage,
Chen, Y., Shimotake, A., Matsumoto, R., Kunieda, T., Kikuchi, T., 128, 385–397. https://doi.org/10.1016/j.neuroimage.2016.
Miyamoto, S., & Lambon Ralph, M. A. (2016). The ‘when’ and 01.006
‘where’ of semantic coding in the anterior temporal lobe: Goldman-Rakic, P. S. (1988). Topography of cognition: Parallel
Temporal representational similarity analysis of electrocorti- distributed networks in primate association cortex. Annual
cogram data. Cortex, 79, 1–13. https://doi.org/10.1016/j. Review of Neuroscience, 11(1), 137–156. https://doi.org/10.
cortex.2016.02.015 1146/annurev.ne.11.030188.001033
Cichy, R. M., Khosla, A., Pantazis, D., & Oliva, A. (2017). Dynamics of Gonzalez Andino, S. L., Grave de Peralta, R., Khateb, A., Pegna,
scene representations in the human brain revealed by A. J., Thut, G., & Landis, T. (2007). A glimpse into your vision.
14 O. HAUK ET AL.

Human Brain Mapping, 28(7), 614–624, https://doi.org/10. Karimi-Rouzbahani, H., Woolgar, A., Henson, R., & Nili, H. (2022).
1002/hbm.20302. http://www.ncbi.nlm.nih.gov/entrez/que Caveats and nuances of model-based and model-free rep-
ry.fcgi?cmd=Retrieve&db=PubMed&dopt=Citation&list_ui resentational connectivity analysis. Frontiers in
ds=17133389 Neuroscience, 16, 755988. https://doi.org/10.3389/fnins.
Greenblatt, R. E., Pflieger, M. E., & Ossadtchi, A. E. (2012). 2022.755988
Connectivity measures applied to human brain electro- Khaligh-Razavi, S. M., & Kriegeskorte, N. (2014). Deep super-
physiological data. Journal of Neuroscience Methods, 207(1), vised, but not unsupervised, models may explain IT cortical
1–16. https://doi.org/10.1016/j.jneumeth.2012.02.025 representation. PLoS Computational Biology, 10(11),
Haak, K. V., Marquand, A. F., & Beckmann, C. F. (2018). e1003915. https://doi.org/10.1371/journal.pcbi.1003915.
Connectopic mapping with resting-state fMRI. Neuroimage, Kietzmann, T. C., Spoerer, C. J., Sörensen, L., Cichy, R. M., Hauk,
170, 83–94. https://doi.org/10.1016/j.neuroimage.2017.06. O., & Kriegeskorte, N. (2019). Recurrence required to capture
075 the dynamic computations of the human ventral visual
Hagler, D. J., & Sereno, M. I. (2006). Spatial maps in frontal and stream. arXiv.
prefrontal cortex. Neuroimage, 29(2), 567–577. https://doi. Kietzmann, T. C., Spoerer, C. J., Sorensen, L. K. A., Cichy, R. M.,
org/10.1016/j.neuroimage.2005.08.058 Hauk, O., & Kriegeskorte, N. (2019). Recurrence is required
Hagoort, P. (2013, July 12). MUC (memory, unification, control) to capture the representational dynamics of the human
and beyond. Frontiers in Psychology, 4, 416. https://doi.org/ visual system. Proceedings of the National Academy of
10.3389/fpsyg.2013.00416 Sciences, 116(43), 21854–21863. https://doi.org/10.1073/
Hämäläinen, M. S., Hari, R., Ilmoniemi, R. J., Knuutila, J., & pnas.1905544116
Lounasmaa, O. V. (1993). Magnetoencephalography - Kocagoncu, E., Clarke, A., Devereux, B. J., & Tyler, L. K. (2017).
theory, instrumentation, and applications to noninvasive Decoding the cortical dynamics of sound-meaning
studies of the working human brain. Reviews of Modern mapping. The Journal of Neuroscience, 37(5), 1312–1319.
Physics, 65(2), 413–497. https://doi.org/10.1103/ https://doi.org/10.1523/JNEUROSCI.2858-16.2016
RevModPhys.65.413 Kriegeskorte, N. (2011). Pattern-information analysis: From
Hauk, O. (2016). Only time will tell - why temporal information stimulus decoding to computational-model testing.
is essential for our neuroscientific understanding of seman- Neuroimage, 56(2), 411–421. https://doi.org/10.1016/j.
tics. Psychonomic Bulletin & Review, 23(4), 1072–1079. neuroimage.2011.01.061
https://doi.org/10.3758/s13423-015-0873-9 Kriegeskorte, N., Mur, M., & Bandettini, P. (2008).
Hauk, O., Stenroos, M., & Treder, M. S. (2022). Towards an objec- Representational similarity analysis - connecting the
tive evaluation of EEG/MEG source estimation methods – branches of systems neuroscience. Frontiers in
The linear approach. Neuroimage, 255, 119177. https://doi. Neuroscience, 2(1), 4, https://doi.org/10.3389/neuro.01.016.
org/10.1016/j.neuroimage.2022.119177 2008. http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=
Hebb, D. O., Lambert, W. E., & Tucker, G. R. (1971). Language, Retrieve&db=PubMed&dopt=Citation&list_uids=19104670
thought and experience. The Modern Language Journal, 55 Kriegeskorte, N., Mur, M., Ruff, D. A., Kiani, R., Bodurka, J., Esteky,
(4), 212–222. https://doi.org/10.2307/323063. H., & Bandettini, P. A. (2008). Matching categorical object
Higgins, C., Vidaurre, D., Kolling, N., Liu, Y., Behrens, T., & representations in inferior temporal cortex of man and
Woolrich, M. (2022). Spatiotemporally resolved multivariate monkey. Neuron, 60(6), 1126–1141, https://doi.org/10.
pattern analysis for M/EEG. Human Brain Mapping, 43(10), 1016/j.neuron.2008.10.043. http://www.ncbi.nlm.nih.gov/
3062–3085. https://doi.org/10.1002/hbm.25835 entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=
Hoffman, P., McClelland, J. L., & Lambon Ralph, M. A. (2018). Citation&list_uids=19109916
Concepts, control, and context: A connectionist account of Laakso, A., & Cottrell, G. (2000). Content and cluster analysis:
normal and disordered semantic cognition. Psychological Assessing representational similarity in neural systems.
Review, 125(3), 293–328. https://doi.org/10.1037/ Philosophical Psychology, 13(1), 47–76. https://doi.org/10.
rev0000094 1080/09515080050002726
Iivanainen, J., Stenroos, M., & Parkkonen, L. (2017). Measuring Lambon Ralph, M. A., Jefferies, E., Patterson, K., & Rogers, T. T.
MEG closer to the brain: Performance of on-scalp sensor (2017). The neural and computational bases of semantic
arrays. Neuroimage, 147, 542–553. https://doi.org/10.1016/j. cognition. Nature Reviews Neuroscience, 18(1), 42–55.
neuroimage.2016.12.048 https://doi.org/10.1038/nrn.2016.150
Ito, T., Kulkarni, K. R., Schultz, D. H., Mill, R. D., Chen, R. H., Lamme, V. A., & Roelfsema, P. R. (2000). The distinct modes of
Solomyak, L. I., & Cole, M. W. (2017). Cognitive task infor- vision offered by feedforward and recurrent processing.
mation is transferred between brain regions via resting- Trends in Neurosciences, 23(11), 571–579, https://doi.org/10.
state network topology. Nature Communications, 8(1), 1016/S0166-2236(00)01657-X. http://www.ncbi.nlm.nih.
1027. https://doi.org/10.1038/s41467-017-01000-w gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=
Jackson, R. L. (2021). The neural correlates of semantic control Citation&list_uids=11074267
revisited. Neuroimage, 224, 117444. https://doi.org/10.1016/ MacGregor, L. J., Rodd, J. M., Gilbert, R. A., Hauk, O., Sohoglu, E.,
j.neuroimage.2020.117444 & Davis, M. H. (2020). The neural time course of semantic
Jbabdi, S., Sotiropoulos, S. N., & Behrens, T. E. (2013). The topo- ambiguity resolution in speech comprehension. Journal of
graphic connectome. Current Opinion in Neurobiology, 23(2), Cognitive Neuroscience, 32(3), 403–425. https://doi.org/10.
207–215. https://doi.org/10.1016/j.conb.2012.12.004 1162/jocn_a_01493
Kaas, J. H. (1997). Topographic maps are fundamental to Martin, A. (2007). The representation of object concepts in the
sensory processing. Brain Research Bulletin, 44(2), 107–112. brain. Annual Review of Psychology, 58(1), 25–45, https://doi.
https://doi.org/10.1016/s0361-9230(97)00094-4 org/10.1146/annurev.psych.57.102904.190143. http://www.
 LANGUAGE, COGNITION AND NEUROSCIENCE 15

ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db= connectivity metric. Neuroimage, 270, 119958. https://doi.

PubMed&dopt=Citation&list_uids=16968210 org/10.1016/j.neuroimage.2023.119958.
Marzetti, L., Della Penna, S., Snyder, A. Z., Pizzella, V., Nolte, G., Rahimi, S., Jackson, R., & Hauk, O. (2023). Identifying nonlinear
de Pasquale, F., & Corbetta, M. (2013). Frequency specific functional connectivity with EEG/MEG using nonlinear time-
interactions of MEG resting state activity within and across lagged multidimensional pattern connectivity (nTL-MDPC).
brain networks as revealed by the multivariate interaction bioRxiv. https://doi.org/10.1101/2023.01.19.524690
measure. Neuroimage, 79, 172–183. https://doi.org/10. Rodd, J. M. (2020). Settling into semantic space: An ambiguity-
1016/j.neuroimage.2013.04.062 focused account of word-meaning access. Perspectives on
Mcclelland, J. L., & Rumelhart, D. E. (1985). Distributed Psychological Science, 15(2), 411–427. https://doi.org/10.
memory and the representation of general and specific 1177/1745691619885860
information. Journal of Experimental Psychology: General, Rogers, T. T., Cox, C. R., Lu, Q., Shimotake, A., Kikuchi, T.,
114(2), 159–188. https://doi.org/10.1037/0096-3445.114.2. Kunieda, T., Miyamoto, S., Takahashi, R., Ikeda, A.,
159 Matsumoto, R., & Lambon Ralph, M. A. (2021). Evidence for
Molins, A., Stufflebeam, S. M., Brown, E. N., & Hamalainen, M. S. a deep, distributed and dynamic code for animacy in
(2008). Quantification of the benefit from integrating MEG human ventral anterior temporal cortex. eLife, 10, 3052.
and EEG data in minimum l(2)-norm estimation. http://doi.org/10.7554/eLife.66276
Neuroimage, 42(3), 1069–1077. https://doi.org/10.1016/j. Rogers, T. T., Hocking, J., Mechelli, A., Patterson, K., & Price, C.
neuroimage.2008.05.064 (2005). Fusiform activation to animals is driven by the
Naselaris, T., Kay, K. N., Nishimoto, S., & Gallant, J. L. (2011). process, not the stimulus. Journal of Cognitive
Encoding and decoding in fMRI. Neuroimage, 56(2), 400– Neuroscience, 17(3), 434–445. https://doi.org/10.1162/
410. https://doi.org/10.1016/j.neuroimage.2010.07.073 0898929053279531
O’Rawe, J. F., & Leung, H. C. (2020). Topographic mapping as a Rogers, T. T., Lambon Ralph, M. A., Garrard, P., Bozeat, S.,
basic principle of functional organization for visual and pre- McClelland, J. L., Hodges, J. R., & Patterson, K. (2004).
frontal functional connectivity. Eneuro, 7(1), ENEURO.0532- Structure and deterioration of semantic memory: A neurop-
19.2019. https://doi.org/10.1523/Eneuro.0532-19.2019. sychological and computational investigation. Psychological
Pantev, C., Bertrand, O., Eulitz, C., Verkindt, C., Hampson, S., Review, 111(1), 205–235, https://doi.org/10.1037/0033-295X.
Schuierer, G., & Elbert, T. (1995). Specific tonotopic organiz- 111.1.205. http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?
ations of different areas of the human auditory cortex cmd=Retrieve&db=PubMed&dopt=Citation&list_uids=1475
revealed by simultaneous magnetic and electric recordings. 6594
Electroencephalography and Clinical Neurophysiology, 94(1), Rubinstein, H., Garfield, L., & Millikan, J. (1970). Homographic
26–40, https://doi.org/10.1016/0013-4694(94)00209-4. entries in the internal lexicon. Journal of Verbal Learning
http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd= and Verbal Behavior, 9(5), 487–494. https://doi.org/10.1016/
Retrieve&db=PubMed&dopt=Citation&list_uids=7530637 S0022-5371(70)80091-3
Patel, G. H., Kaplan, D. M., & Snyder, L. H. (2014). Topographic Sereno, M. I., McDonald, C. T., & Allman, J. M. (1994). Analysis of
organization in the brain: Searching for general principles. retinotopic maps in extrastriate cortex. Cerebral Cortex, 4(6),
Trends in Cognitive Sciences, 18(7), 351–363. https://doi.org/ 601–620. https://doi.org/10.1093/cercor/4.6.601
10.1016/j.tics.2014.03.008 Seth, A. K., Barrett, A. B., & Barnett, L. (2015). Granger causality
Penfield, W., & Rassmussen, T. (1950). The cerebral cortex of analysis in neuroscience and neuroimaging. The Journal of
man. Macmillan. Neuroscience, 35(8), 3293–3297. https://doi.org/10.1523/
Poeppel, D. (2014). The neuroanatomic and neurophysiological JNEUROSCI.4399-14.2015
infrastructure for speech and language. Current Opinion in Silver, M. A., & Kastner, S. (2009). Topographic maps in human
Neurobiology, 28, 142–149. https://doi.org/10.1016/j.conb. frontal and parietal cortex. Trends in Cognitive Sciences, 13
2014.07.005 (11), 488–495. https://doi.org/10.1016/j.tics.2009.08.005
Price, C. J. (2012). A review and synthesis of the first 20 years of Stokes, M. G., Wolff, M. J., & Spaak, E. (2015). Decoding rich
PET and fMRI studies of heard speech, spoken language and spatial information with high temporal resolution. Trends
reading. Neuroimage, 62(2), 816–847. https://doi.org/10. in Cognitive Sciences, 19(11), 636–638. https://doi.org/10.
1016/j.neuroimage.2012.04.062 1016/j.tics.2015.08.016
Pulvermuller, F. (2013). How neurons make meaning: Brain Székely, G. J., Rizzo, M. L., & Bakirov, N. K. (2007). Measuring and
mechanisms for embodied and abstract-symbolic seman- testing dependence by correlation of distances. The Annals
tics. Trends in Cognitive Sciences, 17(9), 458–470. https:// of Statistics, 35(6), 2769–2794. https://doi.org/10.1214/
doi.org/10.1016/j.tics.2013.06.004 009053607000000505
Quiroga, R. Q. (2012). Concept cells: The building blocks of Thivierge, J. P., & Marcus, G. F. (2007). The topographic brain:
declarative memory functions. Nature Reviews From neural connectivity to cognition. Trends in
Neuroscience, 13(8), 587–597. https://doi.org/10.1038/ Neurosciences, 30(6), 251–259. https://doi.org/10.1016/j.
nrn3251 tins.2007.04.004
Rahimi, S., Farahibozorg, S. R., Jackson, R., & Hauk, O. (2022). Tinsley, C. J. (2009). Creating abstract topographic represen-
Task modulation of spatiotemporal dynamics in semantic tations: Implications for coding, learning and reasoning.
brain networks: An EEG/MEG study. Neuroimage, 246, Biosystems, 96(3), 251–258. https://doi.org/10.1016/j.
118768. https://doi.org/10.1016/j.neuroimage.2021.118768 biosystems.2009.03.003
Rahimi, S., Jackson, R., Farahibozorg, S., & Hauk, O. (2023). Time Tomasello, R., Garagnani, M., Wennekers, T., & Pulvermuller, F.
lagged multidimensional pattern connectivity (TL MDPC): (2018). A neurobiologically constrained cortex model of
An EEG/MEG pattern transformation based functional semantic grounding with spiking neurons and brain-like
16 O. HAUK ET AL.

connectivity. Frontiers in Computational Neuroscience, 12. ventral stream: Dissecting the inner organization of the
https://doi.org/10.3389/fncom.2018.00088 visual word-form system. Neuron, 55(1), 143–156, https://
Valdes-Sosa, P. A., Roebroeck, A., Daunizeau, J., & Friston, K. doi.org/10.1016/j.neuron.2007.05.031. http://www.ncbi.
(2011). Effective connectivity: Influence, causality and bio- nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed
physical modeling. Neuroimage, 58(2), 339–361, https://doi. &dopt=Citation&list_uids=17610823
org/10.1016/j.neuroimage.2011.03.058. http://www.ncbi. Zajzon, B., Mahmoudian, S., Morrison, A., & Duarte, R. (2019).
nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed Passing the message: Representation transfer in modular
&dopt=Citation&list_uids=21477655 balanced networks. Frontiers in Computational
Vinckier, F., Dehaene, S., Jobert, A., Dubus, J. P., Sigman, M., & Neuroscience, 13, 79. https://doi.org/10.3389/fncom.2019.
Cohen, L. (2007). Hierarchical coding of letter strings in the 00079