Visual processing from retina to cortex

Essay Sensory Systems I – The Visual System (WT 19/20)

Prof. Dr. Frank Schaeffel
- Based on Lecture 4 -

Corinna Schulz
Student-ID: 5377370
M.Sc. Neural- and Behavioral Sciences
Graduate Training Center of Neurosciences
 2

Visual processing from retina to cortex

“We are so familiar with seeing, that it takes a leap of imagination to realize that there are problems to be
solved. But consider it. We are given tiny distorted upside-down images in the eyes and we see separate
solid objects in surrounding space. From the patterns of stimulation on the retina we perceive the world of
objects and this is nothing short of a miracle.” – Richard L. Gregory, Eye and Brain, 1996

Humans and many other primates have large parts of their brains devoted to vision, more than to any of the
other sensory functions. Vision is also, or maybe because of that, one of the most studied topics in sensory
neuroscience with hierarchical and parallel processing (Stone et al.,1979).
From retina to cortex in a nutshell. Visual perception starts with each eye ‘seeing’ almost the entire visual
field (except the monocular crescent). Axons of the retinal ganglion cells form the optic nerve and carry
information from both hemifields of one eye. At the optic chiasm, a midline crossing point, fibres from the
nasal hemiretina cross over to the contralateral side, while fibres from the temporal hemiretina stay on the
ipsilateral side, forming the optic tract (Kandel et al., 2000, see Appendix A). This hemifield crossing is
present in all animals that have stereopsis, the ability to perceive depth by comparing the images from both
eyes that are slightly different due to their relative horizontal position. The first correct anatomical description
of the relationship between nasal and temporal fibres crossing at the optic chiasm stemmed from an
examination of an injured Civil War soldier in 1970 by Keen and Thomas (Reynolds, 1987). Specifically,
they demonstrated that the vertical meridian that separates nasal from temporal retina is localized at the
macula rather than the optic nerve. The macula is the area of best visual acuity in the centre of our fixation
and the patient of Keen and Thomas was suffering from macular splitting, a hemianopsia resulting in the
loss of vision in half of the visual centre, with a sharp separation of seeing versus non-seeing at the vertical
boundary of the centre of vision. After the optic chiasm, the optic tract carries information from both eyes
but only from one - the contralateral - visual hemifield.
Geniculostriate and superior colliculi pathwaysi. The optic tract has two major targets, the lateral
geniculate nucleus (LGN) of the thalamus and the superior colliculi (SC). While 90% of the ganglion cell
axons terminate in the LGN, from there forming the geniculostriate pathway, around 10% project to the SC
that send signals to control the oculomotor nuclei, regulating lateral eye movements. The geniculostriate
pathway, also called optic radiation, is most important for visual perception. Information from the LGN is
carried through two divisions (upper and lower) along the calcarine fissure to the primary visual cortex (V1)
also called the striate cortex and from there to multiple cortical regions. A lesion in only one optic radiation
leads to quadrantanopia, affecting perception of the visual field in its respective superior or inferior quadrant.
While the geniculostriate pathway is important for conscious perception, the pathway involving the SC can
give insights into the phenomenon blindsight. Patients with lesioned geniculostriate but intact ‘indirect
pathway’ through the pulvinar (thalamic nuclei) to the cortex, do not report conscious perception but are
under certain experimental paradigms able to show residual vision. This means that blindsight patients can
guide their hands or eyes in forced-choice paradigms with high precision to the blind visual field while
denying to consciously perceive the visual targets (Cowey, 2010). Despite controversies around the exact
neurobiological underpinning of blindsight, the SC consistently are shown to be a crucial component
(Kinoshita et al., 2019). I a recent study, Kinoshita et al. (2019) show by pharmacological inhibition the
critical role of SC-pulvinar pathway for visually guided saccades in blindsight monkeys. The SC are an
interesting structure as multiple sensory maps are superimposed in a topographic fashion, i.e. not only
visual but also auditory and somatosensory maps that shift with respect to each other during eye
movements.
A closer look: retinal ganglion cells to LGN. There are different classes of retinal ganglion cells, which
can be differentiated based on their morphology, visual response properties and their projection patterns.
These three classes, the P cells, M cells and intercalated cells, project to three subdivisions of the LGN, the
parvocellular (most dorsal), magnocellular (ventral) and intercalated/kaniocellular layers, respectively.
Since intercalated cells have been difficult to study because of their size and location, not much is known.
However, key differences between the P and M cells have been identified. To understand how visual
information is processed, it is useful to understand the concept of a receptive field. A receptive field (RF)
can be understood as ‘the totality of all stimuli driving a given neuron’ (Mallot, 2013) and can be measured
by correlating neuronal activity with externally measurable stimuli parameter. Looking at the different RF
 3

properties of neurons along the visual stream reveals important processing steps that enable our
perception. It has been found that retinal ganglion cells and LGN neurons have quite similar, so-called
centre-surround, RFs that have ON and OFF subregions, whereby centre and surround regions are mutually
inhibitory (Appendix B, Figure B1). The size of a RF is determined by two factors, the eccentricity, that is its
position from the fovea, and its position in the visual stream such that RFs with a larger distance to the
fovea become progressively larger (Kandel et al., 2000). Selective lesioning (using excitotoxine ibotenic
acid unilaterally) of the magno- (layer 1 or 2) or parvocellular layers (layer 3/5 or 4/6) of the LGN in monkeys
can reveal distinct differences of the magno- and the parvocellular pathways (Kandel et al., 2000). For
example, lesioning the parvocellular layer renders colour vision impossible while lesioning the
magnocellular layer impairs perception of bright and dark patterns with both low spatial frequency and high
temporal frequency. The key differences between M and P cells can be summarized as follows: Firstly, the
RFs of P cells are much smaller than those of M cells, consequently, P cells are more sensitive to higher
spatial frequencies. Secondly, P cells respond in a sustained fashion e.g. to prolonged stimuli such as
colour, whereas M cells respond more transiently, influencing their sensitivity to temporal frequencies.
Thirdly, while most P cells are sensitive to colour, M cells are not, instead M cells are more sensitive to low-
contrasts and black-and-white stimuli. Importantly, the LGN does not combine or integrate the information
from both eyes. Instead, it carries complete information of the contralateral visual field.
What psychophysical experiments reveal about perception. Our visual system is selective to the
orientation and size of retinal images. In psychophysical experiments, Blakemore & Campbell (1969)
demonstrated spatial frequency selective adaptation such that exposure to a high contrast gratings of the
same orientation and spatial frequency lead to a fivefold rise in contrast threshold. Webster et al. (2002)
showed how judgements of image blur depend on the state of adaptation. The blur of an image is
determined by the distribution of spatial contrasts. Importantly, visual cortical neurons are tuned to spatial
contrast distributions as they occur in natural images, described by the 1/f amplitude spectra. This means,
that in natural images for increasing spatial frequency the amplitude decreases in inverse proportion.
Webster et al. (2002) showed that adaptation periods to either blurred or sharpened pictures but also blurred
or sharpened surround strongly biased subsequent judgement of best focus. This highlights how perceived
contrast is to a large extent independent from spatial frequency. Ohlendorf et al. (2011) showed how
adaption to astigmatic defocus can improve visual acuity. While half of their participants watched a movie
that was convolved with an astigmatic point-spread-function, such as in 3D defocus, the other half watched
the movie unfiltered but with astigmatic lenses. Adaptations to both stimulated and real astigmatic defocus,
could improve the visual acuity. Importantly, adaptation transferred to number charts that served as test
stimuli for visual acuity, hence, adaptations were independent of the spatial pattern in the movie, endured
for several minutes and were selective for the axis of astigmatism.
Primary visual cortex and its functional organization. From the LGN information travels to the primary
V1, also called striate cortex because of its heavily myelinated sublamina in layer 4 that are visible as a
stripe (Squire et al., 2012). V1 has six layers but it is layer 4 that mainly receives the input from the LGN.
While the magnocellular pathway terminates in layer 4Cα, the parvocellular layer terminates in layer 4Cß.
Afferents from the intralaminar LGN terminate in layer 2 and 3.
Retinotopy. An important feature of V1 is its retinotopic organization. The surface of V1 is functionally
organized in a map of the visual field, such that neighbouring spots in the visual field are represented by
neighbouring cortical spots. However, this cortical map is distorted by variations in the magnification factor.
Specifically, the central part of the visual field is represented by the largest part of the cortex, with highest
visual acuity, while more eccentric parts are represented by smaller parts of the cortex (Appendix C).
Hoffmann et al. (2011) successfully used high-field fMRI (7T) to improve the retinotopic mapping in humans,
additionally showing its usefulness for ophthalmological research such as studying patients with macular
dysfunction or misrouted optic nerves. How the retina maps onto the cortex can also be nicely visualized
by examining angioscotomas (Adams & Horton, 2003). Angioscotomas are a local form of amblyopia, a
visual deficit in which the brain favours one eye because input from the other eye is not correctly processed.
Blood vessels in the eye cast shadows onto photoreceptors, thereby causing local differences in neural
activity, consequently creating angioscotomas. As the blood vessels restrict normal visual stimulation during
early development, geniculocortical afferents are remodelled in a way that represents the retinal vascular
tree in V1. Adams & Horton (2003) showed successfully how cytochrome oxidase staining can visualize
 4

these local activity differences, showing how the vascular tree in the retina maps onto V1.
Simple and complex cells. While RF size generally increases with the level of processing such that
areas higher in the visual processing stream have larger RFs, their specificity for object features increases
enormously (Freeman & Simoncelli, 2011). One of Hubel & Wiesel's (1962) key findings was that RFs of
cortical neurons show different response properties. While RFs in the retina and LGN are concentric and
contrast-sensitive, RFs of cortical neurons are additionally sensitive to contours. Each V1 neuron receives
input from neighbouring cells in the LGN such that multiple centre-surround RFs are aligned such that they
represent a particular axis of orientation (Appendix B, Figure B2). Importantly, these are emergent
properties of the processing in V1 and not properties of the cortical input itself. Cortical neurons that have
oriented and linear RF properties are called simple cortical cells, they are sensitive to the polarity and the
phase of an image. Gabor functions, sinusoidal multiplied with Gaussian envelopes can model this polarity
and phase sensitivity quite well (Mallot, 2013). One processing step further, we find complex cortical cells,
which lack discrete ON and OFF subregions. Their RF profiles are non-linear as they are invariant to phase
and polarity and respond to edges as long as they have a preferred orientation. They can be modelled by
combining multiple Gabor functions and applying non-linear operations (Mallot, 2013).
Hypercolumns. Another key feature of V1 as discovered by Hubel & Wiese is that neurons with
similar RFs are organized in columns, forming functional modules. Hubel and Wiesel’s describe the striate
cortex to possess (1) Orientation columns, (2) Blobs and (3) Ocular dominance columns (Kandel et al.,
2000; Squire et al., 2012; Ts’o et al., 2009). Using tangential penetration with microelectrodes in monkey
cortex, they found that neurons respond to identical retinal positions with the same axis of orientation,
forming orientation columns. Certain areas where multiple orientation preferences converge are called
pinwheel centres. Yacoub et al. (2008) found that orientation preference is organized radially like a pinwheel
around a single point with a larger number of orientation columns tuned to vertical stimuli (90degree). A full
set of orientation columns makes up the orientation Hypercolumn. Orientation columns are interrupted by
so-called blobs, as visualized by cytochrome oxidase staining. These blobs are not concerned with
orientation but instead with colour. Lastly, orientation columns are divided into ocular dominance columns.
Anatomical staining techniques postmortem in subjects that had lost sight in one eye (J. C. Horton & Hedley-
Whyte, 1984), reveal ocular dominance columns in humans with a size of about 1mm. However, staining
techniques cannot detect orientation columns and only with the availability of modern high-field (7T)
functional magnetic resonance imaging, it was possible to identify orientation columns in human V1 that are
strikingly similar to those reported in monkeys (Yacoub et al., 2008). Taken together, the cortical
computational model, an ‘ice cube’ of roughly 1mm in diameter was described for V1. This module would
be the smallest unit in V1 that could contain all functional and anatomical cell types necessary to understand
perception of a defined part of the visual field: a complete set of orientation columns (orientation
hypercolumn) with one set of ocular dominance columns (i.e. left and right ocular dominance columns) and
several colour processing blobs (Kandel et al., 2000; Appendix D). This idea is certainly appealing and
recent studies extending this concept to other areas such as V2 with hypercolumns for orientation and
binocular disparity that run orthogonal to each other further fosters this idea (Ts’o et al., 2009). However,
the functionality of these columns has been questioned. Horton & Adams (2005) acknowledge the existence
of cortical columns but present the idea that columns as the basic functional entity of the cortex are obsolete.
Columnar structure varies strongly across species without any functional correlate, and even within single
species, there are individuals with and without ocular dominance, again without a known functional
correlate. One such example are squirrel monkeys. Ober 30% of squirrel monkeys do not show ocular
dominance columns and no correlation between a lacking orientation column and stereopsis was found.
Additionally, variation has even been found in single individuals, such that ocular dominance columns are
only present in parts of V1 in squirrel monkeys. Consequently, Horton & Adams (2005) argue that cortical
columns might simply be epiphenomena, that in some species were exploited for purposes after their
emergence but cannot deliver a unifying principle that helps us to explain cortical function.
An intriguing question, going beyond the scope of this essay, is how the brain now ultimately binds various
visual features including colour, form or movement into one percept. One hint to answer this question are
long-range horizontal connection present in each layer of the visual cortex. Specifically, horizontal
connections link columns with the same functional specification (Kandel et al., 2000). Finally, visual
information is always the result of the activity of many neurons, a population code.
 5

References

Adams, D. L., & Horton, J. C. (2003). The Representation of Retinal Blood Vessels in Primate Striate

Cortex. Journal of Neuroscience, 23(14), 5984–5997. https://doi.org/10.1523/JNEUROSCI.23-14-

05984.2003

Cowey, A. (2010). Visual System: How Does Blindsight Arise? Current Biology, 20(17), R702–R704.

https://doi.org/10.1016/j.cub.2010.07.014

Freeman, J., & Simoncelli, E. P. (2011). Metamers of the ventral stream. Nature Neuroscience, 14(9),

1195–1201. https://doi.org/10.1038/nn.2889

Hoffmann, M., Kaule, F., Grzeschik, R., Behrens-Baumann, W., & Wolynski, B. (2011). Retinotope

Kartierung des menschlichen visuellen Kortex mit funktioneller Magnetresonanztomografie –

Grundlagen, aktuelle Entwicklungen und Perspektiven für die Ophthalmologie. Klinische

Monatsblätter für Augenheilkunde, 228(07), 613–620. https://doi.org/10.1055/s-0029-1245625

Horton, J. C., & Hedley-Whyte, E. T. (1984). Mapping of cytochrome oxidase patches and ocular

dominance columns in human visual cortex. Philosophical Transactions of the Royal Society of

London. B, Biological Sciences, 304(1119), 255–272.

Horton, Jonathan C, & Adams, D. L. (2005). The cortical column: A structure without a function.

Philosophical Transactions of the Royal Society B: Biological Sciences, 360(1456), 837–862.

https://doi.org/10.1098/rstb.2005.1623

Hubel, D. H., & Wiesel, T. N. (1962). Receptive fields, binocular interaction and functional architecture in

the cat’s visual cortex. The Journal of Physiology, 160(1), 106-154.2.

Kandel, E. R., Schwartz, J. H., Jessell, T. M., Biochemistry, D. of, Jessell, M. B. T., Siegelbaum, S., &

Hudspeth, A. J. (2000). Principles of neural science (Vol. 4). McGraw-hill New York.

Kinoshita, M., Kato, R., Isa, K., Kobayashi, K., Kobayashi, K., Onoe, H., & Isa, T. (2019). Dissecting the

circuit for blindsight to reveal the critical role of pulvinar and superior colliculus. Nature

Communications, 10(1), 135. https://doi.org/10.1038/s41467-018-08058-0

Mallot, H. A. (2013). Computational neuroscience (Vol. 486). Springer.

Ohlendorf, A., Tabernero, J., & Schaeffel, F. (2011). Neuronal adaptation to simulated and optically-

induced astigmatic defocus. Vision Research, 51(6), 529–534.

https://doi.org/10.1016/j.visres.2011.01.010
 6

Reynolds, T. M. (1987). A nineteenth century view of the optic commissure. Survey of Ophthalmology,

32(3), 214–216. https://doi.org/10.1016/0039-6257(87)90097-x

Squire, L., Berg, D., Bloom, F. E., Du Lac, S., Ghosh, A., & Spitzer, N. C. (2012). Fundamental

neuroscience. Academic Press.

Stone, J., Dreher, B., & Leventhal, A. (1979). Hierarchical and parallel mechanisms in the organization of

visual cortex. Brain Research, 180(3), 345–394. https://doi.org/10.1016/0165-0173(79)90010-9

Ts’o, D. Y., Zarella, M., & Burkitt, G. (2009). Whither the hypercolumn? The Journal of Physiology,

587(12), 2791–2805. https://doi.org/10.1113/jphysiol.2009.171082

Webster, M. A., Georgeson, M. A., & Webster, S. M. (2002). Neural adjustments to image blur. Nature

Neuroscience, 5(9), 839–840. https://doi.org/10.1038/nn906

Yacoub, E., Harel, N., & Ugurbil, K. (2008). High-field fMRI unveils orientation columns in humans.

Proceedings of the National Academy of Sciences of the United States of America, 105(30),

10607–10612. https://doi.org/10.1073/pnas.0804110105

*Content not explicitly referenced is taken from the lecture series Visual System, specifically Lecture 4 by
Prof. Dr. Frank Schaeffel that follows Kandel et al. (2000), Chapter 27.
 7

Appendix A

Figure taken from Kandel et al. (2000), Chapter 25, Figure 25-12.
 8

Appendix B

Figure B1: Simple receptive fields in V1. Figure taken from Hubel & Wiesel (1962)

Figure B2: Visual receptive fields. Figure taken from (Mallot, 2013), Chapter 2: Receptive Fields and the
Specificity of Neuronal Firing
 9

Appendix C

Figure C: Eccentricity. Figure taken from Kandel et al. (2000), Chapter 27, Figure 27-9.
 10

Appendix D

Figure D: A cortical computational model. Figure taken from Kandel et al. (2000), Chapter 25,
Figure 25-13.

i
Third pathway, the pupillary pathway, less important for perception but important for the pupillary reflex relays light
signals via the midbrain to the Edinger-Westphal nucleus and the ciliary ganglion to control the smooth muscle of the
pupillary sphincter and the muscle controlling the lens. As pupils are usually coupled, they can reveal the location of
damages. By comparing ipsi- and contralateral pupil responses it can be determined whether the lesion is efferent or
afferent.