Paired neuron recordings in the prefrontal and inferotemporal cortices reveal that spatial selection precedes object identification during visual search

Results

Data were collected from two monkeys (M1 and M2) performing the covert visual search task shown in Fig. 1. On each trial, a target was drawn from a pool of 20 targets previously associated with a left or right lever turn (Fig. 1A). The monkeys’ task was to maintain fixation on a central stimulus and report the identity of the learned target object among distractors with a manual lever turn (Fig. 1B). A cue array of colored rings was presented 235 ms before the presentation of the search array. One object of the search array appeared in each of the cue array rings. In half of the trials, all of the rings were green (neutral cue trials). In the remaining trials, the location of the target object was indicated with a red ring (cued trials).

Based on the IT neuron’s responses, we identified a preferred target object and a nonpreferred target object associated with opposite lever turn directions, and four distractor objects that elicited similar responses to the nonpreferred target using a passive viewing task (SI Text). On each trial, three of the four identified distractors were chosen at random. On 75% of the trials, the preferred target and the nonpreferred target had an equal chance of being chosen as the target of the search array. On 25% of the trials, the target was chosen at random from the 18 remaining objects in the target pool. These control trials were used to discourage the monkeys from adapting search strategies specific to the preferred and nonpreferred target objects chosen for that session (18).

We simultaneously recorded the activity of single neurons in FEF and IT during 24 recording sessions in two monkeys (M1: 20 neuron pairs, M2: 4 neuron pairs). More information about the locations at which the neuronal recordings were performed is provided in Methods and Fig. S1. The behavioral performance of the two monkeys was similar. Overall, the monkeys performed correctly on 94% of cued trials and 92% of neutral cue trials. Median reaction time across sessions was faster on cued trials than on neutral cue trials (mean ± SD; neutral cue trials: 352 ± 56 ms; cued trials: 333 ± 56 ms), and this difference was significant (paired t test: P < 0.001). Performance accuracy and reaction times from the control trials with random target images did not differ significantly from that obtained from the preferred and nonpreferred target trials during each of the 24 recording sessions (SI Text). This result indicates that the monkeys’ behavioral report was guided by an object recognition process rather than by the detection of low-level features specific to the preferred and nonpreferred target objects.

Neuron activity in FEF and IT was recorded simultaneously to compare the time course of spatial selection and object identification during the same trials. The term “spatial selection” refers to the neuronal activity in FEF that discriminates whether a target or a distractor is in the neuron’s response field (10). “Object identification” refers to the neuronal activity in IT that differentiates between the preferred versus the nonpreferred target (18).

For the 24 recording sessions, the FEF neuron reliably signaled target location and the IT neuron reliably signaled target identity. The relationship between object identification in IT and the selection of the location of that object in FEF can be observed on neutral cue trials. Fig. 2 plots the average activity of an example neuron in neutral cue trials when the preferred target (Fig. 2 A and B) and the nonpreferred target (Fig. 2 C and D) is presented at each of the four possible target locations. After the presentation of the search array, the activity of the IT neuron differentiated which object was in the RF by exhibiting a robust response for the preferred object in the contralateral hemifield (Fig. 2B) and the absence of a response for the nonpreferred object at the same locations (Fig. 2D). The activity of the FEF neuron signaled the location of the target object by exhibiting greater activity when the target was in the RF than when a distractor was in the RF, regardless of which object was the target (Fig. 2 A and C). Another example FEF–IT pair is shown in Fig. S2. The response of the IT neuron shown in Fig. S2 was insensitive to spatial location even though the timing, size, and eccentricity of the search and cue arrays were the same as for the neuron shown in Fig. 2.

For each neuron pair, we compared the time of object identification by the IT neuron to the time of spatial selection by the simultaneously recorded FEF neuron (Fig. 2 E and F). Because the responses of many IT neurons vary with location (15, 16, 19), we structured this analysis so that we would obtain the earliest possible time of object identification by the IT neuron. The time of object identification in IT was calculated during trials in which the preferred target was presented at the array location that elicited the greatest and earliest response from the IT neuron, and was defined as the first time when the preferred target produced a significantly greater response than the nonpreferred target (Fig. 2F). For the analysis of spatial selection in FEF, we were constrained by the spatial extent of the FEF neuron’s receptive field, which was limited to one or sometimes two array locations. The time of spatial selection in FEF was defined as the first time the FEF neuron’s activity during trials in which the preferred target was inside the receptive field was significantly greater than the activity during trials in which the preferred target was outside of the receptive field (Fig. 2E). The target locations for which the FEF and IT activity was analyzed did not match in 8 of the recording sessions from monkey 1 and in one session from monkey 2. The results from these 9 sessions did not differ from the results from the remaining 15 recording sessions in which the best locations matched. The FEF and IT neuron pair shown in Fig. 2 exhibited the strongest responses to the preferred target at the same location in the search array. For this neuron pair, spatially selective activity in the FEF neuron became significant at 120 ms (Fig. 2E) and object selective activity became significant at 134 ms (Fig. 2F) following the presentation of the search array.

The time course of object identification and spatial selection on cued and neutral cue trials can be seen in the pooled average activity from all 24 recording sessions shown in Fig. 3 A–D. The average differences in activity on cued and neutral cue trials that define the spatially selective response in FEF and the object selective response in IT are compared in Fig. 2 E and F. On cued trials, a red ring cued the location of the target. Because activity in FEF signaled the presence of the red cue ring in the receptive field (11), the time of spatial selection on cued trials measures the selection of the cue stimulus and occurs well before the appearance of the target array (Fig. 3A).

A comparison of the means ± SDs of spatial selection times in FEF, object identification times in IT, and reaction times on cued and neutral cue trials is plotted in Fig. 3G. The average time of spatial selection (mean ± SD) measured from FEF activity on cued trials was −139 ± 21 ms relative to the presentation of the search array. The average time of spatial selection in FEF on neutral cue trials was 116 ± 24 ms after the presentation of the search array. Reaction times were, on average, 18.5 ms faster on cued trials than on neutral cue trials. This indicates that spatial attention was directed to the spatial cue. The time of object identification in IT on cued trials was 18.3 ms faster than on neutral trials (cued trials: 133 ± 26 ms; neutral cue trials: 151 ± 31 ms). The nearly equivalent difference demonstrates that the presence of an attentional cue affects the latency of object identification in IT and the reaction time difference can be attributed to the time it takes to form an object representation in IT.

For the remainder of this report we concentrate on neutral cue trials, trials that produced only target-related spatial selection in FEF neuron activity (Fig. 3B). From the activity recorded on neutral cue trials we were able to compare the time course of object identification for the preferred target in IT to the time course of spatial selection elicited by that target in FEF. Overall, spatial selection in FEF preceded object identification in IT by 34 ± 27 ms (Fig. 3G). After the presentation of the search array, some IT neurons had an initial nonselective visual response (Fig. S2). It may be argued that across the population of IT neurons, object identification occurs at the time of IT visual onset. Considering this possibility, we compared the IT visual onset times evoked by the presentation of the search array (relative to the activity before the search array presentation) to the times of FEF spatial selection. IT visual onset times occurred after FEF selected the location of the target to be identified (P = 0.02; SI Text).

Reaction times varied from session to session, most likely because the stimuli used in each session were tailored to the IT neuron’s response. This variation in the combinations of targets and distractor stimuli changed the difficulty of the task from day to day. This variability allowed us to examine the temporal relationships between reaction time, object identification in IT, and spatial selection in FEF across recording sessions. Both the spatial selection times in FEF and the object identification times in IT were correlated with the median reaction times across sessions (Fig. 4A; FEF vs. reaction time: r = 0.42, P = 0.04, IT vs. reaction time: r = 0.52, P = 0.009). This suggests that the neural activity in FEF and in IT contributed to the object identification process leading to the behavioral report in our task. The stronger correlation to reaction time for IT than for FEF may indicate that neuronal activity in IT represents a processing stage that has a greater influence over the object identification process than does FEF.

Next, we performed a correlation analysis between the spatial selection times obtained from FEF and the object identification times obtained from IT using activity recorded on the same trials (Fig. 4B). The best locations for the FEF neuron and the IT neuron matched for 15 of the 24 FEF–IT neuron pairs, as they do for the FEF–IT neuron pair shown in Fig. 2. For these 15 neuron pairs (M1: 12 pairs; M2: 3 pairs), the strongest object selective response in IT was obtained at a search array location inside the FEF neuron’s receptive field. These 15 neurons are indicated by the solid data points in Fig. 4B. For an additional five neuron pairs (M1: 4 pairs; M2: 1 pair) the FEF neurons’ receptive field overlapped with a location at which the preferred object elicited a response from the IT neuron that was nearly as strong as when that object was at the IT neuron’s best location. The object selection times used for these IT neurons were, on average, 12.5 ms later than those obtained from the best location and are indicated by the open data points in Fig. 4B. We could not obtain object selection times from both the IT neuron and the FEF neuron from the same trials for the remaining 4 sessions, so they were omitted from the correlation analysis. For the 20 neuron pairs with overlapping receptive fields, the spatial selection times in FEF were correlated with object identification times in IT (r = 0.59, P = 0.007). The correlation was significant when we included only the 15 neuron pairs that exhibited the best responses at the same array locations (r = 0.65, P = 0.009). For the 15 recording sessions in which the best locations matched, the spatial selection in FEF occurred, on average, 35 ± 25 ms before object identification in IT. It is important to note that the objects used in the task differed from day to day and the monkeys could have used different strategies to find the target. Nevertheless, all of the data points in Fig. 4B are above the identity line, indicating that spatial selection in FEF occurred before object identification in IT on the same trials in every experimental session.

It could be argued that the correlation between spatial selection time in FEF and object identification time IT (Fig. 4B) is coincidental, a product of the correlations with reaction time and variations in task difficulty across sessions (Fig. 4A). To address this question, we examined whether there was a correlation between the spatial selection time in FEF and object identification time in IT on individual trials within recording sessions. Simultaneous activity recorded in FEF and IT was analyzed to obtain a spatial selection time and an object identification time for the same trial. The single trial selection time analysis is discussed and illustrated in Fig. S3. We performed a partial correlation analysis on the single trial timing results that removed differences in timing across sessions (Fig. 4C). We included only those sessions in which we obtained a spatial selection time from the FEF neuron and an object identification time from the IT neuron for at least 10 simultaneously recorded trials. This yielded a total of 290 trials from 14 sessions (M1: 10 sessions; M2: 4 sessions). The partial correlation analysis removed the across-session correlation (shown in Fig. 4B) by subtracting the session means from the spatial selection time and object identification time obtained from each single trial. There was a strong correlation between the times of spatial selection in FEF and object identification in IT on single trials (M1: r = 0.25, P < 0.0001; M2: r = 0.42, P < 0.0003; both: r = 0.28, P < 0.00001). A similar result was obtained when selection times from all sessions, including those with <10 simultaneously recorded single trials, were included in the partial correlation analysis (n = 321, r = 0.22, P < 0.0001). In addition, we performed a single trial Pearson’s correlation analysis separately for each session. The correlation coefficients were >0 for all 14 sessions (mean r = 0.33 ± 0.06), of which 4 were significant (M1: 2 sessions; M2: 2 sessions; P < 0.05).

Discussion

We simultaneously recorded neural signals in FEF and IT representing two distinct but related cognitive processes leading to a behavioral report of visual object identification during covert visual search without eye movements. The selective signals observed in each area were consistent with those that have been described previously. FEF activity localizes the target object to be identified and is correlated with spatial attention (11, 20), and IT activity represents object identity and is correlated with object recognition (13, 18). The emergence of spatially selective signals in FEF and object representations in IT in our task was correlated with variations in the time it took the monkeys to report target identity across recording sessions with varying levels of difficulty (Fig. 4A). This is also consistent with the results of previous studies (18, 21). The correlations with reaction time suggest that the neuronal signals observed in both of these areas play a role in the target identification process. It should be noted that the correlations to reaction time do not necessarily demonstrate a causal relationship between FEF and IT to choice behavior. In our view, the activity recorded simultaneously in FEF and IT allows us access to the outcome of the brain processes responsible for spatial selection and object recognition, respectively.

Recently we demonstrated that the magnitude of FEF activity is correlated with the speed and accuracy of object identification and inactivation of FEF produced deficits in object identification (11). Here we analyzed concurrent activity recorded in FEF and IT on the same trials providing insight about the neural processes leading to object identification that cannot be addressed from the analysis of either signal alone. The emergence of object representations in IT and the time of selection of that object’s location in FEF on the same trials are correlated across recording sessions (Fig. 4B) and on individual trials within recording sessions (Fig. 4C). The task used in this study did not explicitly require a spatial component; the monkeys reported target object identity. Nevertheless, visual spatial selection signals in FEF were present about 35 ms before object identification signals emerged in IT. This suggests that spatial selection is an integral part of the object recognition process and is consistent with the “early selection” hypothesis of attentional selection, that an object must be within the focus of spatial attention to bind simple visual features (e.g., color, shape) into a complex object representation (22–26).

We found significant correlations between the time of neuronal object identification and the monkeys’ reaction times across recording sessions whereas other studies have found little covariation between reaction time and IT neuronal activity (27–29). In these other studies, the task was relatively easy; the monkeys identified a single stimulus at the center of the screen. In contrast, we required the monkeys to perform a difficult search task in which the monkeys had to covertly search for the object among distractors and identify it; and on any given trial any one of 20 targets could appear. Consistent with our study, a recent study also found a relationship between object-related activity in IT and reaction time in monkeys performing a near threshold object recognition task (18). We hypothesize that differences in task difficulty can explain why some studies observe a correlation between reaction time and IT activity and others do not.

The results of our study seem to be at odds with the results of studies by Chelazzi and colleagues (30, 31). In these studies, neurons in IT were recorded in monkeys performing a memory-guided visual search task in which a cue stimulus was followed, after a delay, by a search array with two or more stimuli. The animal was rewarded for making a saccade to the target stimulus in the search array that matched the cue stimulus. The target stimulus could be “good” or “poor” depending on how effective it was in driving the neuron’s activity. They showed that there was strong early activation for the good stimulus even when it was a distractor. The activity of the IT neuron then changed to signal the poor stimulus as the poor stimulus was selected as the target for the saccade. Their results are consistent with the “late selection” hypothesis in which there is parallel processing of both the target and distractor stimuli before the selection of the target stimulus that matched the previous sample.

Our results and those of Chelazzi et al. (31) can be reconciled by considering differences in the tasks and by the fact that they did not simultaneously record spatially selective activity in an area such as FEF that indexes the outcome of the spatial selection process. Also, the memory-guided search task used by Chelazzi et al. (31) relied on visual working memory. The monkeys were shown the item to search for at the beginning of every trial, and the IT neurons exhibited elevated activity related to this knowledge. Visual working memory for the target object was not a factor in our task.

On the basis of Chelazzi et al. (31), Hamker (32) proposed the reentry model of visual search in which spatial selection by FEF neurons is required for object detection and identification (32). Predictions of the reentry model are in line with our finding that the times of spatial selection in FEF and object identification in IT are correlated on single trials (Fig. 4C). Hamker (32) constrained this model with the assumption that neurons in IT compete in parallel and that the spatial signal in FEF resolves this competition, which results in the perception of the target. We found that the neural representation of the preferred target object’s identity in IT emerges after spatial selection has taken place.

It has been suggested that there are different types of selection for different cognitive processes (6–8). It is possible that the matching of an object representation held in working memory requires an object representation in IT and is thereby consistent with a late selection mechanism. We showed that the emergence of these object representations in IT require spatial selection signals and are therefore consistent with an early selection mechanism. This hypothesis predicts that distractors in our task are not recognizable unless they are first selected by spatial selection processes. Whether this prediction holds true will require further experiments. But this view is consistent with the results from a study by Sheinberg and Logothetis (17). Although their study could not determine whether spatial selection or object identification happens first, they showed that object representations emerge in IT when viewing complex visual scenes only when the preferred object becomes the target for a saccade.

The purpose of this study was to determine the order of cognitive events leading to object recognition in a complex visual scene. We chose to record neuronal activity in FEF and IT because they are known to represent cognitive processes related to spatial selection and object identification, respectively. FEF forms an interconnected network with brain regions such as the lateral intraparietal area and the superior colliculus that exhibit similar patterns of spatially selective signals during visual search tasks (33–35). We hypothesize that the results would be similar if spatial selection times had been ascertained from neuronal activity recorded in these other structures.

Object identity and category can be read out from the activity of IT neurons by classifier models with great precision (36)—something not generally found for “earlier” visual areas (e.g., V1, V4). The active disambiguation of competing object representations is much more closely related to activity in IT than in earlier visual areas (37). More recent work has shown that choices in the face of uncertainty link IT activity to recognition decisions in a paradigm quite similar to the one used in this task (18). IT activity has been linked to the representation of knowledge in an object association task (38). When monkeys make saccades to a target to be identified in an active search task (17) neurons fire before the saccade is initiated signaling the noticing of the preferred stimulus. Nevertheless, we cannot be certain that the time of object identification measured in IT activity is the time at which the animal has consciously “recognized” the target object. Our paper sheds light on the order of cognitive events leading up to object recognition, but does not clearly link IT activity to conscious recognition of objects.

In summary, we show that during a visual search task in which the monkeys were rewarded for reporting target identity, not location, activity in the brain specifies where a target object is before identifying what it is. Our results are consistent with those of a recent psychophysical study that showed that attentional selection is a necessary prerequisite for target identification (39). The spatial selection process indexed by FEF activity could be guided by a preattentive low-level visual feature analysis or by an endogenous serial search process. Regardless of what guides the spatial selection process, our results provide a neural basis for early selection theories that hypothesize that the selection of an object’s location is necessary for its identification. The spatial attentional selection represented in FEF may facilitate the perceptual binding of features to object representations in IT (23–26).

Methods

Supplementary Material