Interpreting fMRI data: maps, modules and dimensions

Monkey studies

Studies in monkeys with fMRI and extracellular recordings have revealed striking similarities between the monkey and human ventral visual pathways. As in humans, fMRI in monkeys has revealed regions of the ventral visual pathway that are selective for a few object categories, including faces and bodies^45,46. Electrophysiological recordings have recently confirmed the clustering of single-neuron response properties that underlie this fMRI selectivity⁴⁷: in some subregions in the inferior temporal (IT) cortex almost all neurons respond more strongly to faces than to other objects. A distributed pattern of selectivity is also found for other object categories⁴⁶.

In addition, category membership seems to be the main determinant of the population response in the IT. In a recent study in which monkeys were shown images of objects from many natural categories, the population of IT neurons was highly selective for category membership, especially for animate objects like faces and bodies⁴⁸. More indirect evidence for category selectivity is the observation that single IT neurons are more selective for shape features that are useful for object categorization (‘non-accidental properties’) than for other shape features (‘metric properties’)⁴⁹. Thus, the stimulus property that seems to be associated with the strongest selectivity in single cells in the IT cortex is object category.

Finally, IT neurons also show selectivity for other features, including object shape^50,51, viewpoint⁵², position^53,54 and size⁵³. These results are in accordance with the findings from fMRI adaptation studies in humans. Overall, there seems to be a high degree of similarity in the neural mechanisms that underlie face and object processing in monkeys and in humans, as well as in the sensitivity for object properties that define category membership.

Over the years several controversies have arisen concerning the interpretation of category-selective regions in the brain. First, how distinct are these ‘regions’? Are they discrete modules or are they parts of a continuous selectivity map? Second, is ‘stimulus category’ really what these regions are selective for? In the following sections we tackle both questions, drawing on findings from the primary sensory cortex in animals, where similar questions have been addressed in great detail.

Distinction between maps and modules

The word ‘map’ is generally used to refer to a gradient of selectivities along the cortical sheet. By contrast, ‘module’ — in the context of brain function — refers to the clustering of selectivities in discrete regions, with clear selectivity discontinuities at the boundaries of these regions. For now we will stick to this simple functional definition and not commit to further, as yet unresolved, questions about the size of a module (except for the notion that it is bigger than a column⁵⁵) or further anatomical criteria. Note that modularity in cognitive science is a more complex, multi-faceted and theoretically committed concept⁵⁶ than in neuroscience. Nevertheless, the two meanings of modularity are not completely distinct, and the existence of discrete brain regions with clear functional boundaries invites the question of whether the more extensive criteria for modularity in cognitive science also hold for these regions.

How can we empirically distinguish between maps and (brain) modules as defined here? Experimentally this enterprise requires a continuous variation of a stimulus parameter, for example, stimulus position, and an investigation of how the peak of activation shifts along the cortical surface in response to the varying stimulus parameter. If the continuous variation is associated with a gradual shift in the peak of activation, we have found a map. If the continuous variation is associated with discrete jumps in the peak of activation, we have evidence for a module. Note, however, that this method applies only to stimulus dimensions that vary along continual, or at least ordinal, scales. In addition, to provide convincing evidence of a map, the variable that determines the map must explain the strength of selectivity in each region. In the following sections we review the evidence for maps and modules in primary sensory cortices and then apply these insights to the ventral visual pathway.

Maps and modules in primary sensory cortex

Is there evidence for maps and modules in primary sensory cortex, and can the two types of organization be distinguished? The retinotopic organization in the primary visual cortex (area v1 in the striate cortex; see FIG. 1a) is a prototypical example of a map: the preferred stimulus position changes smoothly across the cortical surface^57–59. The situation is less clear, however, for other stimulus parameters, such as orientation. Gradual variation of stimulus orientation produces a gradual shift of orientation preference in numerous v1 subregions⁶⁰, but these regions are separated by singularities (pinwheel centres) in which the orientation preference shifts abruptly^61–63. This pattern is sometimes referred to as a mosaic-like map⁶⁴. Computational analyses have shown that the discontinuities that are found in a mosaic-like map are unavoidable whenever multiple stimulus properties (such as orientation, direction of motion and spatial frequency) are mapped onto the two-dimensional cortical sheet^58,65–67. We will not use the term ‘module’ to refer to this kind of mosaic-like map for two reasons. First, the pinwheels where orientation preference changes abruptly are local exceptions in what is otherwise a smooth map. Second, in contrast to our definition of a module, the pinwheels do not divide the cortex into discontinuous regions that have no relationship in preferred values across their boundaries; instead, the preferred stimulus values depend on the direction in which the pinwheel is crossed. Note that this conclusion is based on the functional response properties in the cortex, and the relevance of certain relationships between functional properties, such as colour selectivity, and cytoarchitectonic features, such as ‘blobs’ and ‘interblobs’, is as-yet unclear^68–70.

Does any primary sensory region contain functional modules? At first glance, the somatosensory cortex of animals with whiskers or similar organs comprised of discrete units, appears to be a plausible candidate. Indeed, the first-order cortical somatosensory representations in these animals are discontinuous. Typical examples are the barrels in the rat somatosensory cortex and the layout of the nasal appendages of the star-nosed mole rat^71,72. However, even though the barrel cortex is sometimes cited as an example of columnar structure, it is in fact analogous to the retinotopic visual cortex⁵⁵ because a barrel constitutes a first-order representation of a whisker that is isomorphic with the structure of the receptor organ. Furthermore, a clear spatial relationship holds across barrels, with the barrel array on the cortex reflecting the whisker array on the snout. Thus, the barrel cortex has an ordinal (although not continuous) mapping of whiskers. Finally, the selectivity in each barrel is as strong as one would expect given the stimulus property that is mapped across barrels: given that each barrel represents one whisker, it is not surprising that neurons in a barrel will respond only to stimulation of that particular whisker. From this perspective the barrel cortex contains a map, not modules. More generally, there is no evidence for modules (as defined here) in sensory cortex.

Maps and modules in the ventral visual pathway

Should category-selective regions in the ventral visual pathway be regarded as stand-alone modules that are specialized for the recognition of a special category of objects^6,73, or should they be regarded as part of a larger topographical organization that encompasses most or all of these regions^13,31,74,75? As mentioned above, the best way to answer this question is to investigate how the pattern of selectivity in these regions shifts with a gradual change in object properties. Given that object category seems to be important for the organization of the object-vision pathway, it is object category that should change gradually. However, the investigation of this idea encounters two problems. First, in contrast to the stimulus properties that are represented in v1, category membership is not linked to variation in a simple physical parameter. Although the term ‘object category’ is intuitively clear, no analytic approach or computational model offers a convincing parameterization of ‘object category’. So what can we do without a physical standard for object category? The solution that has been adopted in numerous studies of categorization is to derive the complex properties by which objects are represented from the behaviour of humans when they rate the similarity between objects^76–78 (FIG. 2). FIGURE 2a shows images of twelve objects in a spatial arrangement that reflects the physical pixel-based similarity between the images (as accurately as is possible with two dimensions). This spatial configuration is strikingly different from that which most closely reflects the similarity between these images as judged by a human observer (FIG. 2b). These judgments suggest that the relevant dimension in humans’ ‘mental object space’ is object category. Thus, object category dominates not only the functional organization in the object-vision pathway but also perceptual similarity. One fMRI study found good correspondence between the rated similarity among objects and the degree of overlap among their representations in the object-vision pathway⁷⁹. This study illustrated how a detailed analysis of how humans perceive objects can provide at least a partial solution to the lack of a simple parameterization.

Another problem remains, however. In all stimulus sets that have been used in fMRI research, category membership was a discontinuous variable. The stimulus set shown in FIG. 2b is a representative example: there are several faces in this set, but there is no gradual morphing of a face into an exemplar of another category. This problem might be unsolvable, because the mental object space seems to be only locally continuous. Within each object category, individual objects’ shapes can be changed to move parametrically from one exemplar to another^50,80–85: morphing the faces of two members of a species or of members of two different species is relatively straightforward, as the corresponding features in the two faces are immediately obvious (FIG. 3). However, the mental object space has sharp discontinuities that coincide with the boundaries between categories, and morphing across these boundaries is not straightforward. For example, what are the corresponding features of a hand and a face? What would a hand–face morph look like, and what are the odds of seeing such a morph in real life? This simple example suggests that the category ‘faces’ has relatively sharp boundaries. The same applies to other categories, with the exception of categories of objects that have similar shapes (for example, arms and legs, or bottles and vases). Thus, a few special cases excluded, category membership is an inherently discontinuous variable.

Returning to our original question, does the object-vision pathway contain a large-scale map, or does it contain a set of independent modules? We have seen above that in the barrel cortex even discrete stimulus parameters, such as different whiskers, can be represented in a continuous map: even though both whiskers and cortical barrels are discrete units, neighbouring whiskers are nevertheless mapped onto neighbouring barrels in the cortex. This represents the ordinal characteristic of a map. Similarly, even though faces and other objects constitute discontinuous categories in visual object space, and even though the corresponding cortical regions have relatively sharp boundaries, as is required of modules⁸⁶, these regions can still be part of a map if there is any systematic relationship between their relative positions. Indeed, such a relationship has been suggested: high-level visual cortex exhibits a weak centre–periphery organization³⁰, and it has been argued that the relative location of category-selective regions in this eccentricity map corresponds to the eccentricity at which the preferred stimuli of these regions are typically seen. These considerations raise the question of whether even a face-selective ‘module’ such as the FFA can be considered to be part of a larger topographic object category map.

Other data argue against this view of continuous functional organization in the ventral visual pathway, however. First, the relatively weak retinotopic organization in high-level visual cortex does not seem to be strong enough to explain the much stronger category selectivity that is observed. The retinotopic organization might have a role in determining where selectivity for a category will be found, but it does not seem sufficiently strong to function as the sole argument against modules. Further research is needed to establish whether there is another variable, or combination of variables (see below), that links the regions that have strong category selectivity. Second, it is not yet known whether category-selective regions such as the FFA and the LOC differ from one another in their cytoarchitecture or connectivity; if such differences exist and exceed the simple wiring differences of, for example, barrels, this would challenge the view that these regions are parts of a larger object-category map.

Finally, we use a minimal definition of a ‘module’ here, referring only to the functional neuroanatomical characteristics of an area — namely the strength of its functional specificity and the sharpness of its boundaries. More elaborate definitions of a ‘module’ in cognitive science include a list of additional properties, such as mandatory processing and a characteristic ontogeny^56,87. Although the evidence is not conclusive for any of these properties, the FFA might satisfy some of them, and such findings could challenge the idea of a large-scale map. First, the FFA and the nearby LOC differ not only in terms of their selectivity, but also in terms of the computations that they conduct on their preferred stimuli. For example, the FFA responds similarly during discrimination of faces on the basis of face parts and on the basis of the spacing between parts. By contrast, the LOC responds much more strongly to part-based than spacing-based discrimination of both faces and houses⁸⁸. Second, recent evidence suggests that the FFA and the PPA develop on a different timescale to the rest of the ventral visual pathway, so they seem to have a characteristic ontogeny⁸⁹.

In sum, the category-selective regions in the ventral visual cortex can be considered part of a larger topographic organization that reflects the characteristics of the mental space of objects; however, further studies are needed to differentiate this perspective from a discontinuous, modular view of the ventral visual pathway.

Basic properties in the primary visual cortex

What functional properties are represented in primary sensory cortex? For some aspects of coding the answer is simple and uncontested. An obvious example is the first-order representation of receptor arrays (such as retinotopic maps) that determine the large-scale organization of primary sensory cortices. Although the fine characteristics of these maps (for example, their magnification factor, local smoothness and scatter) have been discussed in many studies, there is little debate about what these maps represent (for example, visual-field position in the case of retinotopy).

However, controversy arises once the clear link to the receptor array is lost and the maps reflect higher-order properties of the stimulus. The first demonstrations of more fine-scale functional organization in v1 were maps of ocular dominance¹⁰ and orientation preference^90,91. Later studies provided evidence of clustering of several other functional-response properties in at least some species, for example, spatial frequency and direction of motion⁶⁴. The traditional approach has been to consider these maps as feature maps with overlapping regions that give rise to selectivity for particular feature combinations^58,92. However, this interpretation was a consequence of the experimental approach that was used: the different feature maps were discovered one by one, typically by mapping one feature at a time and averaging across all values of the other features.

By contrast, recent studies have included multidimensional stimulus manipulations. Using this approach, the map of orientation preference in v1 was found to depend heavily on several other stimulus properties, such as bar length, direction and speed⁹³. These results cast doubt on the multiple-feature-map interpretation. Instead, it has been proposed that these multiple maps might arise from the mapping of only one property — spatiotemporal energy⁹³. According to this view, the finding of independent maps for multiple features is an artefact of using stimuli that vary in only one feature at a time. This discussion illustrates that, even for seemingly simple functional properties, it is not an easy task to find the ‘basic’ dimensions that most parsimoniously describe a regions’ functional organization.

Basic properties in the ventral visual cortex

Even more controversy exists regarding how to describe the functional organization in the ventral visual cortex. Until now we have described it in terms of object category, and most studies that have targeted the object-vision pathway have presented exemplars from a wide variety of ‘everyday’ object categories. However, object category is potentially confounded by various other factors, such as shape characteristics, the way in which the stimuli are processed (for example, part-based versus holistic processing), semantic information and retinal eccentricity. In early studies on high-level visual cortex these properties were manipulated jointly, but recent studies have begun to isolate specific variables. So far, however, no individual variable has been able to explain a substantial proportion of the observed category selectivity. We next review the evidence for a variety of such candidate variables.

First, could high-level visual areas simply be selective for shape characteristics? In support of this view, unfamiliar artificial-object categories and relatively simple patterns elicit selective responses in the object- and face-selective cortex in humans^94,95 (FIG. 4). Furthermore, it has been shown that single cells and columns in the monkey IT cortex are selective for moderately complex features^96,97; that neurons in v4 and in the posterior IT cortex are tuned for simple shape characteristics^98,99; and that IT neurons exhibit gradual tuning in simple shape spaces^50,51. Finally, single-cell recordings and fMRI studies have provided evidence of selectivity for object parts and non-accidental properties (for example, symmetry, parallelism and collinearity) in the ventral temporal cortex^49,100, as predicted by structural description theories of object recognition^101,102.

These findings indicate that part of the functional specificity for familiar objects and faces might be due to differences in basic shape characteristics across categories. However, several lines of evidence limit the role of shape properties. First, the selectivity for unfamiliar shapes is weaker than that for familiar objects and faces. The pattern of selectivity is distributed and weak in individual voxels, at least for the unfamiliar-object classes that have been tested. For example, in a recent study with high spatial resolution, in the scattered regions that showed the most selectivity the strength of the regions’ response to their least-preferred novel objects was approximately two thirds of the response to a preferred novel object⁹⁵. By contrast, the maximum selectivity for some familiar objects is far stronger, even at lower spatial resolution: the FFA, the PPA and the EBA all respond at least two to three times more strongly to their preferred object class (faces, houses and bodies, respectively) than to a wide range of non-preferred object classes²¹. Second, the strong and indistinguishable response of the body-selective regions to, for example, hands and legs, which have very different shapes, indicates that shape alone cannot account for all aspects of the selectivity of these regions^12,26,46. Thus, simple shape characteristics can explain only part of the functional organization that exists in cortical responses to familiar-object classes. Third, it is possible that apparent selectivities for simple shape features are by-products of selectivities for more complex objects. In particular, the aforementioned studies that demonstrated selectivity for shape features cannot rule out that this selectivity is actually caused by a tuning for the whole shape of complex, familiar objects. Indeed, computational work has shown that empirical data that seem to favour an explanation in terms of part- or feature-based selectivity can be mimicked when simple shapes are presented to a hierarchical model with units that are tuned for the whole shape of complex objects¹⁰³. Thus, some tuning and clustering for simple shape features is to be expected, even if the actual shape or object characteristics that determine tuning and clustering are more complex in nature.

Could functional organization be driven not by physical stimulus attributes but by the neural processing that is triggered by each stimulus¹⁰⁴? According to this view, any object that engages a given process could strongly activate the relevant region. This idea has been mostly put forward for face-selective regions, based on evidence that faces are processed more holistically than other types of object^105–108. The hypothesis that the face-selective cortex is not actually selective for faces per se, but rather for the neural processing that is triggered by our extensive expertise with faces, has been tested with other object categories for which some individuals have expertise^109–111 — for example, cars and birds in car experts and ornithologists, respectively. However, recent evidence casts doubt on the idea that holistic processing occurs for any expert object category other than faces¹¹², and all fMRI studies of expertise that investigated both the FFA and the LOC found that any increased responses to expert categories were larger in the LOC than in the FFA^112,113. Further, as mentioned above, the FFA response to faces is largely unaffected by task participation, and when humans are induced to process non-face stimuli in a face-like fashion the face area does not respond strongly⁸⁸. This means that either the face-selective responses are not caused by process differences or that the holistic processing of faces is so automatic and mandatory that there is no way to dissociate it from viewing a face. Finally, the hypothesis applies only to faces, and is not sufficiently specific to address the full pattern of functional specificity for a wider range of object categories. Thus, there is little evidence that category-selective responses are due only to differences in types of processing.

It has also been proposed that the association of object categories with non-visual information might determine part of the organization in the ventral visual pathway^114,115. For example, the distribution of neural specificity for ‘tools’ or manipulable objects in the ventral visual pathway might be driven by the connectivity of some ventral regions with other brain regions that are involved in the coding of actions. Similarly, one might speculate that the neural specificity for faces in the fusiform gyrus is related to the functional connectivity that exists between this region and brain regions that are involved in affective reactions, such as the amygdala, or social cognition, such as the temporo-parietal junction¹¹⁶. Likewise, the lateralization of the visual word-form area in the left hemisphere might be caused by the left-hemisphere lateralization of language processing¹⁹. The evidence indicating that functional connectivity is a general organizing principle in the ventral visual pathway is currently mostly circumstantial, but this is a question that warrants further research.

One series of studies has suggested that there is a systematic relationship between the localization of category-selective regions and a weak eccentricity map in the object-selective cortex^30,31. Object categories that tend to be seen at particular eccentricities evoke selective activity in these eccentricity bands. For example, faces, which are mostly foveated, activate regions that have a strong preference for foveal stimulation, rather than regions that are activated by scenes and houses. This relationship might explain the location of some category-selective regions in high-level visual cortex, although it does not by itself explain strong category selectivity (which is much stronger than the eccentricity biases).

In sum, many studies have tried to define more precisely what aspects of objects and faces drive functional specificity in the occipitotemporal cortex. Simple and quantitatively defined stimulus properties, such as shape characteristics, explain some of the observed specificity, but object category remains the most parsimonious criterion by which to explain functional organization in the object-vision pathway.

Object category as a basic property

Despite the parsimony of using one concept, object category, this criterion remains unattractive because of its subjectivity. Two routes may lead to a more systematic understanding of the selectivity for object category.

Computational modelling might provide a more mechanistic way to describe the functional organization in high-level visual cortex. This has been illustrated by the fresh perspective that computational modelling has afforded of the distinction between part-based and configural processing of objects and faces¹¹⁷. Computational evidence has also revealed that image fragments of intermediate complexity are more informative for object categorization than fragments of low or high complexity^118,119, suggesting that neural coding in object-selective areas might be based on such intermediate-complexity shape features. However, a computational description of cortical organization in the ventral pathway remains a distant goal, because current neurophysiologically plausible models cannot yet predict the strong category selectivity that exists in the object-vision pathway^48,120: for example, the dominance of object category in the selectivities of monkey IT neurons is not predicted by ‘standard’ hierarchical models⁴⁸. Similar discrepancies might exist in the human brain¹⁰⁰.

Adding learning processes to computational models might increase their power to explain the dominance of object category as an organizing principle in high-level visual cortex. Visual experience changes the visual processing of objects in various ways, and it might be an important factor for the development of the sensitivity to category distinctions and the emergence of category-selective regions. Part of this learning might occur in a bottom-up, unsupervised manner¹²¹. In addition, supervised category learning can change the perception and visual processing of objects¹¹⁸, leading to a biased processing of relevant dimensions^122–126. Furthermore, learning to discriminate objects within categories changes the pattern of selectivity across the object-vision pathway⁹⁵. Thus, learning about categories and their members might be responsible for some of the functional organization that exists with regards to object category.