Placing prediction into the fear circuit

Modulation of learning by expectation: Prediction errors as teaching signals

Pavlovian fear conditioning depends on the potentiation of CS input synapses onto LA neurons [3, 4]. Such plasticity is triggered by afferent pathways that transmit US related information to LA neurons. Many different CSs can come to elicit fear responses after being paired with an aversive US, so it is natural to regard these pathways as carrying a teaching signal which instructs learning, and synaptic plasticity, across CS – US pairings. Aversive USs may act as teaching signals to trigger plasticity at CS input synapses to the LA, at least in part, by causing depolarization and action potential firing in LA neurons while CS inputs are active [9, 10]. There is reason to believe that the strength of this teaching signal is not invariant; rather, it is modulated by the expectation of the US during each learning trial. Several lines of evidence show that Pavlovian fear conditioning is more effective when the CS is paired with an unexpected US than with an expected US [11 – 14]. For example, the acquisition of fear to a CS is negatively accelerated across learning trials, so that fear of a CS increases most during early CS – US pairings (when the US is unexpected) and least during later pairings (when the CS has come to predict the US). To explain such findings, learning theories have posited that fear conditioning is not instructed by a simple sensory representation of the US, but instead by an error signal measuring the difference between the US actually present and that expected. In the following sections, we briefly review three types of error signals that have been proposed by formal learning theories.

Instruction of fear acquisition by aversive prediction errors

During fear conditioning, memories for the CS - US association are thought to be stored by synaptic plasticity in LA neurons, and studies have shown that LA neurons respond preferentially to an unexpected rather than expected US [20, 21]. This suggests that LA neurons may receive instructive teaching inputs that encode an aversive prediction error signal, but if so, from where might this teaching signal derive?

Several studies suggest that instructive prediction error signals may arise from the midbrain periaqueductal gray (PAG), a structure that has been implicated in the expression of fear behavior (Box 3) as well as the regulation of aversive stimulus processing. Direct stimulation of PAG neurons can serve as a US in the absence of a peripheral shock to support fear conditioning [22], supporting the view that the PAG is positioned to play a role in instructing associative plasticity during fear conditioning. In a recent study [21] (Figure 1), neurons were recorded from LA and PAG neurons during Pavlovian fear conditioning in rats, and the PAG was inactivated while recording from LA neurons. Pharmacological inhibition of PAG neurons during fear acquisition prevented learning. Critically, shock US-evoked responding in LA and PAG neurons were modulated by expectation. Across the course of auditory CS – shock US pairings, US-evoked responses in LA and PAG neurons declined, concomitant with an increase in expression of conditioned fear responses. This suggests that US-evoked responses of LA and PAG neurons declined as the shock became expected and prediction error declined. This interpretation was supported by the finding that in well-trained rats, neurons recorded in the LA and PAG responded more to the US when it was presented by itself (that is, unexpectedly) than when it was signalled by the CS, and thus expected to occur. Finally, there was evidence that the expectancy modulated US-evoked response in LA neurons is relayed from the PAG to the LA, because PAG inactivation reduced US-evoked responding in LA neurons. US-evoked depolarization and action potential firing in LA neurons are likely to be critical components of the instructive signal that triggers plasticity at CS input synapses during fear conditioning (9, 10). Thus these data suggest a plausible neural mechanism for limiting learning when the CS predicts the US.

Endogenous opioids acting at μ-opioid receptors (MOR) in the ventrolateral PAG (vlPAG) are candidate receptors for mediating this mechanism. MOR antagonism augments acquisition of fear learning by removing limits on the ability of an expected US to condition fear [23 - 26]. A similar augmentation of fear conditioning has been observed in human subjects. Specifically, functional magnetic imaging resonance (fMRI) recordings in subjects who underwent fear conditioning revealed that opioid receptor antagonism during the experiment prevented the diminution of amygdala blood oxygen level dependent (BOLD) responses typically observed across repeated CS – US pairings [27]. Studies using more complex behavioral training paradigms in rats have shown that activation of MORs in the vlPAG reduces the effectiveness of an expected US as a reinforcer during fear learning [28, 29]. For example, associative blocking tasks have been used to study the role of vlPAG MOR in learning about expected versus unexpected events. In one such study [28] (Figure 1), rats were first trained to fear a CS “A” [ie. CS(A)], via pairings with a shock US . In the second stage of the experiment (Stage II), rats were trained to fear two compounds. One compound consisted of CS(A) + CS(B) paired with the shock US . The second consisted of CS(C)+CS(D) paired with the shock US. The prediction error during the CS(AB)-shock pairings was low, since CS(A) had been previously paired with the shock USin Stage I, and hence, the shock was expected in Stage II. By contrast, the prediction error during CS(CD)-shock pairings was high because neither CS(C) nor CS(D) had previously been paired with the shock US in Stage I, and hence the shock was unexpected during Stage II. Rats were later tested for fear reactions to CS(B) and CS(D). Control rats showed evidence for associative blocking so that fear was less to CS(B) than to CS(D). That is, the prior fear learning about CS(A) blocked later fear learning to CS(B). This pattern of learning was prevented by antagonising MORs in the vlPAG during Stage II of the experiment .Taken taken, these data suggest that the PAG may be part of the teaching signal pathway for fear learning, instructing LA associative plasticity. Furthermore, this function depends, at least in part, on vlPAG MORs.

It is worth considering whether these data also permit selection between the different error signals proposed by formal learning theories. Electrophysiological data show that the US-evoked population response in PAG is largest on the first CS-US pairing trial [21]. This result does not easily permit selection between different error signals because most theories predict that such signals decline across CS – US pairings. Behavioral data show that vlPAG MOR contributions to associative blocking are observed with a single Stage II conditioning trial [29]. This finding is more consistent with error signals described by Rescorla-Wagner and TD models (causing variations in US processing) than the error signals described by the Pearce-Hall model (causing variations in CS processing). Nonetheless, some amygdala neurons encode Pearce-Hall type attentional signals [20, 30, 31] and there is behavioural evidence that amygdala NMDA receptors regulate an attentional or salience signal during fear learning [32]. Significant theoretical effort is being devoted to the development of hybrid associative models, which would allow for both US and CS error signals in Pavlovian learning [33, 34]. It will be of interest to determine whether this effort permits more parsimonious explanation of neuronal activity during fear conditioning.

Instruction of fear extinction by negative prediction errors

For aversive conditioning, a negative prediction error is defined as a signal which is generated when the actual US (λ) is less than expected (∑V). This error can be generated by increasing expectation (∑V) (e.g., overexpectation) or by decreasing US intensity (λ). The simplest example is fear extinction, when a CS which was previously paired with an aversive US is subsequently presented alone in the absence of the US. This negative prediction error instructs loss of fear during extinction training.

Fear extinction depends on the LA and the prefrontal cortex (PFC), where activation of NMDA receptors, their associated intracellular signalling cascades, and synaptic plasticity is crucial to extinction learning and memory storage [3, 35 - 37]. If opioid signalling in the PAG contributes to negative feedback regulation during fear learning (as indicated above), then behavioral fear extinction and the plasticity upon which it depends might also be influenced by PAG opioids. Recent studies support this. Systemic [38, 39] or vlPAG microinjections [40 – 42] of MOR antagonists prevent fear extinction learning. Conversely, fear extinction learning can be facilitated by infusions of a peptidase inhibitor that reduces catabolism of vIPAG enkephalins [43]. Moreover, vlPAG infusions of MOR antagonists not only impair extinction learning, they also prevent the normal increase in phosphorylation of the extracellular related kinase/mitogen activated protein kinase (ERK/MAPK) observed in the PFC and amygdala during extinction learning [42] and which has been shown to be critical for fear extinction memory consolidation [44 - 46]. Thus, the vlPAG regulates synaptic plasticity in the LA and PFC during fear extinction learning.

Another line of evidence that opioid receptors, although not necessarily those in the midbrain, are important for learning not to fear is derived from clinical studies. Exposure therapies for human anxieties are modelled on fear extinction training from animal conditioning studies. Just as opioid receptors are critical for fear reduction by extinction training in animal conditioning studies, so too are they important for the therapeutic benefit of exposure therapies for human clinical anxiety. Administrations of opioid antagonists prior to exposure based treatments for simple phobias reduce the efficacy of these treatments [47 – 49]. Moreover, exogenous opiates administered in the hours to days following a traumatic event can reduce development of post-traumatic stress disorder (PTSD) [50].

The Rescorla-Wagner and TD learning rules posit that learning is instructed by a signed error signal. If fear extinction is instructed by this error signal, then neurons encoding prediction errors might be expected to decrease their firing rates during omission of an expected aversive US. Electrophysiological data do not show evidence for phasic responses—inhibitory or excitatory—in firing of PAG or LA neurons upon omission of an expected aversive US [20, 21]. It is possible that negative prediction errors for fear learning are encoded by other neurons. Likewise, it is possible that PAG neurons signal an unsigned error as described by Pearce-Hall, although behavioural data using single trial blocking studies are inconsistent with this. Another possible explanation is simply that the negative component of the fear error signal is difficult to detect. This negative error signal may be smaller in magnitude and more distributed across time than the phasic negative error signals observed in the reward learning system. The signed TD prediction error is the time derivative of expected future reinforcement [12], and in the case of fear learning, this would be the time derivative of fear itself (since fear may be regarded as the expectation of an aversive US). Fear, unlike anticipated reward, is rarely fleeting. Hence, if amygdala plasticity is instructed by a TD-like error signal encoding the time derivative of fear, then rapid phasic responses to the omission of an expected aversive stimulus might not be observed. Instead, smaller and slower signals might instruct the gradual decline of fear during the transition from danger to safety.

Circuit level mechanisms for predicting danger

Computation of aversive prediction error, and expectancy-modulation of US-teaching signals, can be achieved via a negative feedback circuit [51 - 53] (Figure 2). This requires convergence between efferents of the CS system and afferents of the US system. Extrapolating to fear conditioning [54, 55], output of the conditioned fear system corresponds to the expected outcome of the conditioning trial because it carries information about the outcome of previous CS – US pairings (-ΣV), whereas transmission in the somatosensory system conveys information about the actual aversive US on the current trial (λ).

vlPAG is an important locus of neuroanatomical convergence in the pathways conveying information about the actual aversive US (λ) and the expected aversive US (ΣV). vlPAG is a target of ascending nociceptive pathways conveying information about aversive USs present on a conditioning trial [56]. vlPAG also receives extensive projections from PFC and amygdala neurons whose activity determines expression of conditioned fear. Medial central nucleus (CeAm) projection neurons are excited during expression of conditioned fear [57-59]. The CeAm → vlPAG projection could convey information about the expected aversive US (-ΣV). The PAG also receives projections from the PFC, including prelimbic, infralimbic, and orbital regions [60 – 63], regions implicated in conditioned fear expression [64]. Cortical projections to the PAG play important roles in response selection and coping during stress [62, 65], for example, as a consequence of exposure to inescapable shock [66, 67], and could convey information about the expected aversive US.

The circuit level mechanism(s) by which US processing is inhibited during fear conditioning have not been carefully studied and are largely unknown. CeA neurons are mainly inhibitory and their output is thought to inhibit neurons at efferent target sites [57]. CeA stimulation produces inhibition and excitation of different populations of PAG neurons and these effects are partially blocked by local infusion of a MOR antagonist [68]. CeA-mediated MOR activation may directly dampen or attenuate US processing in vlPAG, so that the ascending output of PAG neurons codes for the error signal (λ – ΣV) [55]. A different possibility is that CeA-mediated recruitment of PAG neurons activates descending antinociceptive circuits which attenuate US processing at the level of the spinal and trigeminal dorsal horn [54, 69, 70]. The relationship between the roles of PAG in encoding a fear teaching signal and fear expression is unclear. The available evidence favours the possibility that these roles are partially distinct. Learning in response to PAG teaching signals can be pharmacologically dissociated from fear expression [28, 29]. Moreover, different populations of PAG neurons have been described [71], including in response to electrical or chemical stimulation of the CeA [68].

Although the PAG likely transmits US teaching signal information to the LA, it does send direct projections there [61]. Thus, an indirect pathway is almost certainly involved. One pathway involves the midline and intralaminar thalamus. Projections from the vlPAG terminate throughout the midline thalamus [72]. These, in turn, project widely to the medial PFC, the orbital frontal cortex (OFC), anterior cingulate (Cg), and rostral agranular insular cortex (RAIC) [72 - 75] and include direct and indirect (from the PFC) [75, 76] projections to amygdala. These projections, part of the ‘medial’ pain pathway [77], are important candidates for conveying US teaching signals to LA. Findings from human neuroimaging as well as rodent functional neuroanatomical studies support this (Figure 3). BOLD signals recorded in the human PFC (notably the insula, orbital, and cingulate), are related to the magnitude of prediction error during fear learning [e.g., 78 – 80]. Moreover, the US-evoked BOLD signal in midline thalamus, PFC, and amygdala diminishes across CS – US pairings [27; 82]. There is a negative correlation between US-evoked BOLD signal in these regions and the self-report of US expectancy [82]. In rodents, unexpected, but not expected, CS – aversive US pairings produce activation of midline thalamic regions as well as the PFC and LA [83], including in identified midline thalamus → PFC projection neurons [83].

These findings suggest that midline thalamic → PFC pathway activity may convey an aversive US teaching signal, however, there is currently little additional behavioural evidence that supports this possibility. Lesion studies have implicated midline thalamus in fear learning. Posterior intralaminar thalamus or parietal insular cortex lesions impair fear acquisition in rats, consistent with a role for these regions in aversive US processing [84], but conflicting effects have been reported in other studies[85, 86]. Likewise, several studies have implicated PFC in fear learning. Lesions of rodent dorsal PFC, encompassing prelimbic (PL) and cingulate (Cg) cortex, can augment acquisition of conditioned fear [87] as can lesions of the RAIC [88]. Pharmacological activation of glutamate receptors [89] or electrical stimulation [90] as a US in rodent Cg is sufficient to support fear learning, whereas pharmacological inhibition or antagonism of glutamate receptors in the Cg can retard or prevent such learning [89, 91]. These effects could be due to modulation of the US teaching signal. However, other interpretations are possible and conflicting findings have been reported [86, 88, 90 - 92]. A recent demonstration that reversible inactivation of rodent dorsomedial PFC (dmPFC), encompassing caudal Cg and dorsal PL, prevents associative blocking of fear learning [83] and so restores the effectiveness of an expected US as an aversive reinforcer, provides important evidence in support of this. However, further work is needed to understand these circuit level mechanisms. In particular, studies are needed which use behavioral paradigms (e.g., associative blocking [Figure 1]) that permit clear isolation of the contribution of prediction error to fear learning and involve selective, reversible modulation of discrete cortical regions during such learning. Recently developed optogenetic techniques (eg. [10, 93, 94]) could allow a more temporally precise tool with which to parse the contribution of these brain regions to specific events occurring during fear learning.