Dissociable effects of surprise and model update in parietal and anterior cingulate cortex

Behavioral Confirmation of Task Strategy.

The logic of including one-off trials was to dissociate the effects of surprise and behavioral reprogramming from the process of updating an internal model. On both update and one-off trials, we expected participants to be surprised by the target location and to engage motor reprogramming to cancel the planned saccade and plan a saccade to the new target location. However, updating should only occur on the update trials, because participants were explicitly informed that one-off trials had no predictive value relating to future trials.

Model-Based Measures of Surprise and Updating.

The RT data suggest that participants performed the task as instructed [i.e., they used prior knowledge of the distribution of target locations to facilitate speeded eye movements (because time to arrive at the target was longer when the target location was unexpected)] and, furthermore, that participants updated their internal model, as instructed, on update trials but not on one-off trials, as indicated by the data in Fig. 2A. They also indicated that whereas the production of the saccadic response within trials was slowed for all surprising target locations (linking surprise and behavioral reorienting), the processing of the target location (indexed by dwell time) was specifically affected by updating.

To give a formal account of the two processes (surprise/reorienting and updating), we wished to analyze the behavioral and neural correlates of two information theoretical measures, I_S (surprise) and D_KL (updating), as described in the Introduction. These measures must be calculated with reference to a model of the state of the environment; for example, the I_S is the log probability of an observed data point, given some model. Specifically, we wished to calculate the I_S and D_KL on each trial with reference to the participants’ internal model of the target distribution.

Participants’ internal models of the targets’ distribution could be expected to differ from the true (generative) distribution of target locations because they observed a limited number of data points and data were observed sequentially; hence, for example, in the first few trials after a change of distribution, the internal model would be based on relatively few data points.

To obtain an estimate of a participant’s beliefs (i.e., the state of the participant’s internal model of the environment) on a trial-to-trial basis, we constructed a normative Bayesian learner. The model is described in detail in Methods; however, briefly, its key features were that it updated its estimate of the underlying distribution using data only from non–one-off trials and the estimated distribution on each trial was updated with the new data point using Bayes’ rule. Two features of the model merit emphasis here.

First, no updating occurred on one-off trials (equivalently, one might say the learning rate was set to zero on one-off trials). This was in accordance with the instructions given to participants (that the locations of one-off targets did not predict the location of future targets). Behavioral evidence suggests participants complied with these instructions, because RTs on the trials following one-off trials were not slowed, as they should have been if participants had erroneously updated their model on the one-off trial. However, it is worth noting that if participants did update on one-off trials, this could only lead to a false-negative result in terms of detecting update-specific brain activation because it would reduce the contrast between update and nonupdate trials.

Second, on the first trial of a new run, the prior in force before the update was first “blanked” (replaced with a uniform distribution); hence, the new posterior model for the update trial (on which the prior for trial +1 was based) was obtained by updating from a uniform distribution using Bayes’ rule. This “blanking” step was introduced to reflect a feature of the task design, of which participants were explicitly informed: that the location of one “block” of target dots and the next block (after an update trial) were totally independent, such that expectations about target locations based on previous targets should be disregarded.

The model was Bayes’ optimal, implying that it returned the best possible estimate of the underlying distribution based on the data points it was given (the same data points that human participants observed) (20). Furthermore, the model was optimal in that it was supplied with correct assumptions about the structure of the world. For example, the model correctly assumed that when an update occurred, the new distribution of target locations did not depend on the previous distribution; this reflected the true form of the generative process by which target locations were chosen in the experiment. Therefore, the internal model generated by our Bayesian learning algorithm represents an upper bound on the accuracy with which any participant could have estimated the underlying distribution.

Distinct Brain Networks for Surprise and Updating.

Our behavioral results suggested that although the within-trial behavior (saccadic RT) was affected by both surprise and updating, stimuli that elicited updating (high D_KL) were processed differently: (i) because behavior on subsequent trials supported the hypothesis that participants updated on update but not one-off trials; (ii) because of increased dwell times on update trials; and (iii) because the D_KL produced an opposite effect on pupil dilation to the I_S.

To determine the neural correlates of surprise and updating, we collected fMRI data from the same group of 17 participants for whom behavior was presented while performing the task. Note that eye tracking data were acquired during the fMRI session and that these data show a very similar pattern of RT effects to the data reported above (replications of Figs. 2 and 3, using the eye tracking data from the fMRI session, are provided in Fig. S4); however, pupillometry was not possible with the in-scanner setup. Details of fMRI data acquisition, preprocessing, and analysis are described in Methods.

We began by analyzing the pattern of activity associated with surprise and updating across the whole brain, by means of a GLM analysis implemented using the Centre for Functional MRI of the Brain (FMRIB) software library (28). The task was modeled in terms of four model-derived regressors: main effect of task (i.e., all trials, modeled as a delta function at the time of the target appearance) and three information theoretical parameters, the prior entropy (the predictability of the environment), the I_S of the observed target location (surprise), and the D_KL of the posterior from the prior (updating). Quantitative definitions of these regressors are given in Methods.

We observed a spatial dissociation between activity correlated with surprise (I_S) and activity correlated with updating (D_KL); no significant activity associated with prior entropy was observed (in line with the lack of behavioral effects relating to this parameter).

Surprise (I_S) was strongly correlated with activity in the posterior parietal cortex, with the peak activation in the superior parietal lobule (SPL) and extending along the IPS. In contrast, updating (D_KL) was strongly correlated with activity in the anterior cingulate cortex (ACC), particularly the rostral cingulate motor area (rCMA) and extending into the ventral part of the adjacent presupplementary motor area (pre-SMA). These spatially distinct effects, which were the only effects to survive multiple comparisons correction from the whole-brain analysis, are shown in Figs. 5A and 6A. For illustrative purposes, the uncorrected statistical maps from which the corrected statistics were drawn are shown in Fig. S5. Note that all results presented in the main text are corrected for multiple comparisons.

Although we had specific parametric predictions about brain activity based on the I_S and D_KL, as a confirmatory analysis, we also analyzed the effects of different trial types: update trials + one-off trials (both types evoke surprise/behavioral reorienting) and update trials − one-off trials (to isolate behavioral reorienting). The maps obtained are very similar to those obtained using our parametric regressors (Fig. S6). Region of interest (ROI) effects using the trial type and parametric regressors are compared in Fig. S7.

To characterize these effects further, we extracted and analyzed the fMRI signal from anatomically defined ROIs as follows.

IPS, Surprise, and Saccadic Reprogramming.

We observed activity in the IPS (area IPS3) that was parametrically related to the degree of surprise elicited by a stimulus (I_S); hence, stimuli that occurred in low-probability locations (given an optimal model of the environment) were associated with high levels of activity in the SPL.

This activity can be understood in terms of the need to reprogram planned eye movements in response to surprising target locations. Both surprise and posterior parietal activity were correlated with longer RTs, suggesting a link between behavioral reprogramming or spatial remapping (41, 42) and parietal activity. This remapping may be described as a within-trial effect because it is concerned with producing or reprogramming a behavioral response to the unexpected stimulus itself, whether or not predictions are updated for future trials.

Saccadic Reprogramming and Updating Expectations in Parietal Cortex.

Although it is widely agreed that the posterior parietal cortex is active when either movements or covert attention is redirected or reoriented from one location to another (47–52), there is disagreement about which regions within the parietal cortex are most important for redirecting attention and eye movements (44). One possibility is that different parietal regions are concerned with different aspects of redirection (53).

A closer analysis of anatomically defined ROIs in the parietal cortex suggests that the computational functions defined in this study may differentiate two regions of the parietal cortex, both of which have been linked to saccadic planning. Region IPS3, a putative homolog of macaque region LIP, which has a generally motoric connection profile, was activated when surprise necessitated saccadic reprogramming. In contrast, posterior IPS/IPL region PGp, a putative homolog of macaque region 7a, was more strongly activated during updating. Human PGp and macaque area 7a are connected to the ACC, the main region activated during updating in the present study (38, 39).

This functional specialization within the parietal saccadic system is reminiscent of a long-standing observation from the neurological literature (16, 17) that lesions of the IPS and IPL produce different deficits in patients. Lesions of the IPS are associated with impairments in the ability to redirect the eyes, as in Balint syndrome (18), or the covert focus of attention, as in extinction (4). In contrast, spatial neglect is associated with damage to the IPL (16, 19). Neglect, unlike Balint syndrome and extinction, cannot be characterized as an inability to reorient to surprising events; rather, it seems to reflect a persistent inability to orient to the neglected field in the first place, which could be characterized as a distorted spatial prior.

ACC Is Specifically Involved in Updating.

We observed activity in the ACC that was specific to trials on which the internal model was updated. ACC activity was not modulated by surprising stimuli that did not cause updating.

Although the ACC has previously been associated with the reorienting of attention and eye movements, and deficits therein, its specific role has remained unclear (31, 54). At the same time, little in the way of theory has been offered to relate the role of the ACC in reorienting to the role it clearly has in error monitoring and reward prediction.

The current findings suggest a specific computational function for the ACC: It is involved in updating internal models to facilitate future information processing. This mechanistic theory of ACC function suggests a possible framework in which observations that the ACC is involved in reorienting and attention may be reconciled with the abundant evidence for it having a role in reward processing, learning, exploration, and foraging.

Errors and Updating in the ACC.

A major line of research on the ACC has focused on observations that the ACC is active in situations when participants make errors. It has long been known that a negative-going potential, the error- or feedback-related negativity, is evoked from the region of the ACC (45) when participants make errors in psychological tasks or are presented with feedback on their performance (46). However, these findings are not incompatible with a role for the ACC in updating of internal models; indeed, it could be argued that the functional role of error- or feedback-related signals is to facilitate the updating of internal models from which future action is generated (55). This interpretation is supported by the findings that activity in ACC cells is particularly strong in instrumental tasks (56) and, furthermore, that the magnitude of the error related negativity is proportional to the degree to which participants modify their behavior on future trials (45, 57).

Exploration, Updating, and Estimation Uncertainty.

A separate line of research has linked ACC activity to exploratory behavior. The ACC is more active on experimental trials when participants explore alternative options rather than exploiting a known source of reward (58, 59). Furthermore, the ACC is active during foraging, when participants decide to “forage” (seek alternative options) as opposed to choosing from among the options immediately presented to them (60). Note that this role of the ACC in exploring and engaging in alternative actions is quite distinct from that of another frontal area, the orbitofrontal cortex, in updating the value or significance of specific stimuli (61, 62).

Although exploration and foraging may appear to be only loosely related to learning and updating, there is a computational concept that relates the two functions: control of estimation uncertainty. Estimation uncertainty (8) is uncertainty about the parameters of the environment. It is distinct from expected uncertainty (or risk), which characterizes the uncertainty or stochasticity that is inherent within a state of the environment and would persist even if the observer were certain about the parameters of the environment. In contrast, estimation uncertainty is not inherent in the environment but reflects the agent’s knowledge of the environment and can be reduced if the agent has the opportunity to make further observations of the environment. Computationally, the level of estimation uncertainty affects the ability of an internal model to learn (or be updated), because a model that is totally certain about the state of the world should not learn anything new (9, 26); this idea is embodied in learning theory by the concept of “associability” (63).

Although estimation uncertainty is driven up by unexpected observations as in the present study (9), it can also be argued that estimation uncertainty should be elevated during foraging and exploration, because in exploring, the observer deliberately seeks new information; therefore, his neural networks should be in a state to accept new learning (11).

The pattern of activity of the ACC in learning tasks could support an interpretation of its role in terms of controlling estimation uncertainty. In a one-armed bandit task (10), it was observed that the ACC was more active in response to new observations when the environment was volatile (and estimation uncertainty was hence high). Furthermore, recent recordings from rat ACC (59) support the hypothesis that the ACC is active when estimation uncertainty should be elevated. In a two-armed bandit task, after the reward probability associated with different actions changed, there was a radical shift in the pattern of activity across ACC neurons; this shift occurred at the start of a period of exploratory behavior rather than during the acquisition of new information per se, suggesting that the ACC was active at the point at which estimation uncertainty increased (when a model of the environment was abandoned in favor of “knowing nothing”). This result is particularly closely related to the present study, in which participants were instructed to disregard their previous experience as soon as the onset of a new experimental run was signaled; this was represented in our model by the application of a uniform probability distribution over all possible target locations at the start of each run.

Conclusions and Implications.

Although updating and surprise are closely linked and are often correlated in learning paradigms, they are conceptually distinct processes. We observed distinct neural and behavioral responses correlated with surprise (I_S) and updating (D_KL). Our results suggest that although the SPL may play a particular role in spatial remapping and the reprogramming of actions within a trial, the ACC is only active when internal models are to be updated, in the context of change in the underlying environment. The results define distinct computational roles for two regions that have previously been shown to be involved in reorienting and reprogramming of saccades. Hence, they may inform models of attention and motor control, as well as models of neurological phenomena linked with failure to reorient, such as extinction and Balint syndrome. Conversely, the computational functions of the ACC and posterior parietal cortex in the present task may offer a framework in which the involvement of these regions in a variety of other paradigms can be explained.

Participants.

Participants were 17 healthy volunteers age range 21–32 years, 7 females). All participants gave written informed consent in accordance with the National Health Service Office for Research Ethics Committees (ref 07/Q1603/11).

Model.

The model was a normative Bayesian learner implemented numerically in MATLAB (MathWorks). It estimated the probability density distribution from which each target location, α_t, was drawn, based on the current and previous observations (i.e., α_1:t), by assigning probabilities to putative parameters of the underlying Gaussian distribution p(μ_t, σ_t | α_1:t).

The posterior probabilities assigned to each parameter pair p(μ_t, σ_t | α_1:t) were calculated as follows. If the trial type is one-off, no updating occurs; that is,

• Otherwise, probabilities are updated using Bayes’ rule:

where the variable denotes dependence of the prior on trial type as follows. If the trial type is update, a uniform prior over () is used:

where the denominator 360 × 50 represents the number of possible values for ( probed in the numerical implementation of the model. If the trial type is expected (i.e., any trial that is not one-off or update), then the prior on trial t is obtained without modification from the posterior on trial t − 1:

Full details are given in SI Methods.

A prior probability density function for the subsequent trial, t+1, was obtained from the posterior on trial t, and expressed as a function of target location (rather than parameters as follows:

The probabilities of each trial type occurring on trial t + 1, p(one-off_t+1), p(update_t+1), and p(expected_t+1) were simply set to the true probability of each trial type in the generative model [i.e., p(one-off_t+1) = 1/4, p(update_t+1) = 1/15, and p(expected_t+1) = 1 − (p(update_t+1) + p(one-off_t+1)].

Output of the model is illustrated in Figs. S9 and S10.

Information Theoretic Regressors.

For the analyses presented in Fig. 3 (analysis of saccadic RTs), Fig. 4 (analysis of pupil area), and the fMRI whole-brain analysis and ROI analysis presented in Figs. 5 and 6, respectively, we used a GLM analysis with information theoretic regressors based on the learning model.

The four task regressors were main effect of task, entropy of the model, I_S, and D_KL. Each regressor used in fMRI analysis was defined as a series of 300 delta functions (events of 0.1 s in duration) at the times of target appearance (for 300 trials), weighted by task parameters and convolved with the hemodynamic response function. The weighting of each regressor on each trial was defined as follows.

All trials took value 1 as the main effect of task. Entropy of the model on trial t was expressed as

where p(α_t+1|α_1:t) is defined analogously as in Eq. 8. The I_S of trial t is defined as

where defined analogously as in Eq. 8. The D_KL on trial t is defined as

where is defined analogously as in Eq. 8 and is defined analogously to the definition of in Eq. 8:

The output of the model and resulting regressors are illustrated in Figs. S9 and S10.

The shared variance between each pair of regressors was relatively low; the R² value for each pair was as follows: prior entropy and I_S/R² = 0.14; prior entropy and D_KL/R² = 0.14; and I_S and D_KL/R² = 0.15.