Optimality and some of its discontents: Successes and shortcomings of existing models for binary decisions

2.1 Leaky competing accumulators and drift-diffusion processes

In the simplest LCA model, appropriate for 2AFC tasks, two units with activity levels (x₁, x₂) represent pools of neurons selectively responsive to the stimuli [75]. They mutually inhibit via input-output or frequency current (f-I) functions that express neural activity (e.g., short-term firing rates) in terms of inputs. These include constant currents representing mean stimulus levels and i.i.d. Wiener processes modeling noise that pollute the stimuli and/or enter the local circuit from other brain regions. The dynamics are governed by the following stochastic differential equations:

where the state variables x_j denote unit activities (spike rates), γ, β are the leak and inhibition rates and μ_j, σ are the means and standard deviation of the noisy stimuli. A decision is recorded when the first activity x_j(t) exceeds a fixed threshold x_j,th, adjustment of which is used to tune the speed-accuracy tradeoff. Similar effects may be obtained by changing baseline activity or initial conditions, mechanisms that also appear likely, and to which we return in §3.2. See [64, 32, 75] for background on related connectionist networks, and [49] on the equivalence of different integrator models.

The function f(·) characterizing neural response is typically sigmoidal:

and if the gain g and bias b are set appropriately, Eqns. (1–2) without noise (σ = 0) can have one or two stable equilibria, separated by a saddle point. These correspond to two “choice attractors” in the noisy system, and if γ and β are sufficiently large, a one-dimensional, attracting curve exists that contains the three equilibria and orbits connecting them: see [24, Fig. 2] and [10]. Hence, after rapid transients have decayed following stimulus onset, the dynamics relax to that of a nonlinear Ornstein-Uhlenbeck (OU) process [10, 63]. The dominant terms are found by linearizing Eqns. (1–2) and subtracting to yield one equation for the difference x = x₁ − x₂:

In §3.5 we note that models of networks of spiking neurons can be reduced to nonlinear LCAs, providing a theoretical path from biophysical detail to analytically-tractable linear models.

2.2 An optimal speed-accuracy tradeoff

If leak and inhibition are balanced (β = γ), and initial data are unbiased, appropriate when stimuli appear with equal probability and have equal reward value, Eq. (4) becomes a DD process

where A = μ₁ − μ₂ denotes the drift rate. Responses are given when x, identified with the logarithmic likelihood ratio, first crosses a threshold ±x_th; if A > 0 then crossing of +x_th corresponds to a correct response and crossing −x_th to an incorrect one. The error rate and mean decision time, quantifying accuracy and speed, are

see [13] and [4, Appendix]. Note that the parameters A, σ and x_th reduce to two quantities η ≡ (A/σ)² (signal-to-noise ratio (SNR), having units of inverse time), and η ≡ |x_th/A| (threshold-to-drift ratio, the first passage time for the noise-free process x(t) = At).

If SNR and mean response-to-stimulus interval D_RSI remain constant across each block of trials, block durations are fixed, and rewards are insensitive to time, then optimality is achieved by maximising reward rate; that is, average accuracy divided by average trial duration:

Here T₀ is that part of the reaction time due to non-decision-related (e.g., sensory and motor) processing. Since T₀ and η also typically remain (approximately) constant for each participant, we may substitute Eqns. (6) into (7) and maximize RR for fixed η, T₀ and D_RSI, obtaining a unique threshold-to-drift ratio θ = θ_op for each pair (η,D_tot):

Inverting the relationships (6) to obtain

the parameters θ_op, η in Eq. (8) can be replaced by the performance measures, p(err) and 〈DT〉, yielding a unique, parameter-free relationship describing the speed-accuracy tradeoff that maximizes RR:

Eq. (10) defines an optimal performance curve (OPC) [4], shown in Fig. 1(a) Each point (p(err), 〈DT〉/D_tot) along the OPC corresponds to an optimal threshold-to-drift ratio θ_op specified by Eq. (8) which, in turn, is associated with a particular decision time and error rate combination that maximize RR: any other threshold, lower or higher, associated with responses that are faster or slower (diamonds in Fig. 1(a)), yields smaller net rewards. Different points on the curve represent θ_op’s and corresponding speed-accuracy trade-offs for different values of η (i.e., task difficulty) and D_tot (i.e., task timing). Critically, the shape of the OPC itself is parameter free. Thus, it can be used to assess performance with respect to optimality and compare this across conditions, tasks, and even individuals, irrespective of differences in task difficulty or timing.

The OPC’s shape may be intuitively understood by observing that very noisy stimuli (η ≈ 0) contain little information, so that, if they are equally likely, it is optimal to choose at random, giving p(err) = 0.5 and 〈DT〉 = 0 (cf. SNR = 0.1 at the right of Fig. 1(a)). At its left, as η → ∞, stimuli become so easy to discriminate that both 〈DT〉 and p(err) approach zero (cf. SNR = 100). For intermediate SNRs it is advantageous to accumulate evidence for just long enough (e.g., SNRs = 1 and 10). Thus, to maximize rewards, optimal performance is achieved by balancing speed versus accuracy, and not by systematically favoring accuracy. Human data appear to present a challenge to this model, as we now consider.

2.3 Experimental evidence: Failures to optimize

Two 2AFC experiments [4, 5] have directly tested whether humans optimize reward rate in accord with the OPC derived above. In the first, 20 participants viewed motion stimuli [9] and were rewarded for each correct discrimination. The experiment was divided into 7-minute blocks with different response to stimulus intervals in each block. In the second, 60 participants discriminated if the majority of 100 locations on a static display were filled with stars or empty. Blocks lasted for 4 minutes, and two difficulty conditions were used. In all cases participants were instructed to maximize their total earnings, and unrewarded practice blocks were administered before testing began.

As noted above, one of the appeals of the OPC is that it is independent of the parameters defining the DD process, with D_tot entering only as the denominator in Eq. (10). This allows data to be pooled across all participants and task conditions. That is, findings concerning actual performance can be compared with optimal performance irrespective of task difficulty, timing or individual differences. For the experiments of [4, 5], 〈DT〉’s were estimated by fitting the DD model to reaction time distributions, the 0 − 50% error rate range was divided into 10 bins, and 〈DT/D_tot〉 were computed for each bin by averaging over those results and conditions with error rates in that bin. This yields the open (tallest) bars in Fig. 1(b); the shaded bars derive from similar analyses restricted to subgroups of participants ranked by their total rewards accrued over all blocks of trials and conditions.

The top 30% group clearly performs close to the OPC, supporting the DD model as an account of human decision making performance, and the conjecture that (at least some) participants can achieve near-optimal performance. Nonetheless, a majority achieve substantially lower total scores. Interestingly, this is due to longer decision times and greater accuracy than are required to maximize reward rate [5]. This particular pattern of deviations from optimality raises two possibilities: 1) Participants seek to optimize some other criterion of performance, such as accuracy, in place of, or in addition to, maximizing monetary reward. 2) Participants seek to maximize reward, but fall short in a systematic way due to specific constraint(s) on performance and/or the influence of other cognitive factor(s). We address these possibilities in the following sections.

2.4 A preference for accuracy?

There is a longstanding literature that suggests humans favor accuracy over speed in reaction time tasks (e.g., [52]). This could explain the observations in Fig. 1(b), insofar as decision times that are longer than optimal are typically associated with more accurate responses. Participants may try to maximize accuracy in addition to (or even instead of) rewards, as postulated in [46, 6]. This can be formalized by combining accuracy with reward rate in at least two ways [4]:

The first (RA) subtracts a fraction of error rate from RR of Eq. (7); the second (RR_m) penalizes errors by reducing previous winnings. Critically, both include a free parameter q ∈ (0, 1) that specifies the relative weight placed on accuracy. Increasing q drives the OPC upward [4, Fig. 13], consistent with the empirical observations shown in Fig. 1(b), suggesting that participants may indeed include accuracy in their objective function. One reason for this may be that they (incorrectly) assume that errors are explicitly penalized, at least early during their experience with the task. This possibility is addressed by further empirical findings discussed below.

However, there are at least two alternative accounts of the data that preserve the assumption that participants seek primarily to optimize rewards. One is that timing uncertainty corrupts estimates of reward rate, systematically biasing performance toward longer decision times. The other is that participants factor into their estimates of reward rate the costs associated with fine-grained adjustment of parameters (such as response threshold), the advantages of which may be small relative to those costs. We consider each of these in turn, and compare them with accuracy-based models in their abilities to account for empirical data.

2.5 Robust decisions in the face of uncertainty?

In the analyses of §2.2 and §2.4 it is assumed that participants maximize an objective function for the DD model, given the exact task parameters. However, it is unlikely that participants can perfectly infer these. For example, RR depends on inter-trial delays as well as SNR. Delays, in particular, may be hard to estimate. Information-gap theory [3] allows parameters to lie within a bounded uncertainty set, and uses a maximin strategy to identify parameters that optimize a worst case scenario.

Interval timing studies [12] indicate that time estimates are normally distributed around the true duration with a standard deviation proportional to it [26]. This prompted the assumption in [85] that the estimated delay D_tot lies in a set U_p(α_p; D̃_tot) = {D_tot > 0 : |D_tot − D̃_tot| ≤ α_p D̃_tot}, of size proportional to the actual delay D̃_tot, with presumed level of uncertainty α_p analogous to the coefficient of variation in scalar expectancy theory [26]. In place of the optimal threshold of Eq. (8), the maximin strategy selects the threshold θ_MM that maximizes the worst RR that can occur for D_tot ∈ U_p(α_p; D̃_tot). For uncertainties in delay, it predicts a one-parameter family of maximin performance curves (MMPCs) that are scaled versions of the OPC (10):

where γ ≡ D̃_tot/D_tot [85]. Like the objective functions (11) that emphasize accuracy, these curves also predict longer mean decision times than the OPC (10). Uncertain SNRs can be treated similarly, yielding families of MMPCs that differ from both the OPC Eq. (10) and MMPC for timing uncertainty Eq. (12), rising to peaks at progressively smaller p(err) as uncertainty increases. An alternative strategy, also investigated in [85], yields robust-satisficing performance curves (RSPCs) that provide poorer fits and thus are not discussed here.

Fig. 2 shows data fits to the parameter-free OPC, the objective functions of Eq. (11), to MMPCs for timing uncertainty and SNR, and to RSPCs for timing uncertainty. While there is little difference among fits to the top 30%, data from the middle 60% and lowest 10% subgroups exhibit patterns that distinguish among the theories. Maximum likelihood computations show that MMPCs for uncertainties in delays provide the best fits, followed by RSPCs for uncertainties in delays and RA [85]. These findings indicate that optimizing reward rate given timing uncertainty leads to slower and more accurate decisions than those predicted by the OPC, which assumes perfect knowledge of delays. Greater accuracy can be considered a consequence of maximizing reward rate under uncertainty, and not the objective of optimization.

2.6 Experimental evidence: Practice and timing uncertainty

To test whether deviations from the OPC can be better explained by an emphasis on accuracy or by timing uncertainty, Balci et al. [2] conducted a 2AFC experiment with motion stimuli encompassing a range of discriminabilities (moving dot displays with 0%, 4%, 8%, 16% and 32% coherences [9], kept fixed in each block), and administered interval timing tests in parallel [12]. 17 participants completed at least 13 sessions in each condition, insuring the greatest likelihood of achieving optimal performance by providing participants with extensive training, and affording the opportunity to examine practice effects [19, 53]. Four main results emerged, three of which we now summarize; the fourth is described in §3.2.

First, average performance converges toward the OPC with increasing practice. Fig. 3(a) shows the mean normalized decision times (dots) for five error bins averaged over sessions 1, 2–5, 6–9 and 10–13. Performance during the final two sets of sessions is indistinguishable from the OPC for higher coherences, but decision times remain significantly above the OPC for the two lowest coherences.

Second, the accuracy-weighted objective function RR_m of Eq. (11) is superior to the OPC in fitting decision times across the full range of error rates during the early sessions, with accuracy weight decreasing monotonically through sessions 1−9 and thereafter remaining at q ≈ 0.2 (not shown here, see [2, Fig. 9]), suggesting that participants may indeed favor accuracy early during training, but that this diminishes with practice. This is consistent with the idea that participants come to such tasks expecting that errors are associated with explicitly negative outcomes (e.g., immediate losses, as may be common in real world settings), rather than being limited to opportunity costs (as assumed by the OPC). With practice, they may adjust their beliefs to accord with the fact that errors incur only opportunity costs.

Third, as shown in Fig 4, deviations from the OPC are stably related to timing ability. Throughout training, the timing inaccuracy of participants, independently assessed by their coefficients of variation in a peak-interval timing task [56], is significantly positively correlated with their distance from the OPC [2]. Furthermore, this relationship provides a better account of deviations from the OPC than emphasis on accuracy (as indexed by the weight parameter q in the objective function RR_m). This suggests that participants may choose decision parameters that maximize reward rate, taking into account their timing uncertainty as discussed in the previous section.

Collectively, these results support the hypothesis that, given sufficient practice, humans can learn to approach optimal performance (maximizing rewards) by devaluing accuracy, with a deviation from optimality that is inversely related to timing ability. Nevertheless, MMPCs based on timing uncertainty fall short of fully capturing performance, especially for the two lowest coherences: Fig. 3(a). Thus, even after accounting for timing uncertainty, reward rates are suboptimal according to the DD model. This suggests that other factors may be involved.

One of these is persistent variability across trials. The extended DDM accounts for this by drawing drift rates A and starting points x(0) from distributions on each trial [60, 4], thus allowing slow and fast errors respectively. The former may explain deviations from the OPC seen for high error rates in Fig. 3(a). However, as yet there is no principled way to determine which factor (variability in drift vs. starting point) should dominate. Moreover, each additional parameter brings with it a family of OPCs (as in §§2.4–2.5), and analytical expressions are not available for those derived from extended DDMs. Both [2] and [70] (cf. §3.2 below) contain extended DDM fits in their optimality analyses, but we are unaware of any systematic studies of the effects of deviations from optimality due to such variability. In contrast, considering the costs of cognitive control may provide a normative account of these effects, to which we now turn.

3.1 The cost of control

It has long been recognized that cognitive control is associated with mental effort [54]. Accordingly, the effort associated with adjusting parameters as conditions change (i.e., engagement of control) may register as a cost. The fourth finding of Balci et al. [2] supports the hypothesis that participants may take account of such costs, and weigh them against benefits that would accrue from optimizing performance for each task condition. The OPCs of Fig 3 assume that the decision threshold was optimized for the coherence (SNR) in each block. However, block-by-block adjustments would engage cognitive control that may have been perceived as effortful. Accordingly, especially if the benefits of such adjustments were small, participants may have sought a single threshold that does best over all conditions, relieving them of the need to estimate coherences and adjust thresholds from block to block.

To assess this possibility, Balci et al. [2] computed the single threshold that was optimal when applied over all coherence conditions. Fig. 3(b) shows that the resulting curve fits the full range of performance for the later sessions (6–13), suggesting that, at least with practice, participants came to adopt a single threshold that maximized reward across all conditions. Consistent with this strategy, it was found that rewards for this single threshold differed minimally from those accrued using the individual thresholds, optimized for the different conditions. Thus, the cost-benefit tradeoff appears to have weighed against exercising control on a condition-by-condition basis, and in favor of a single threshold. Hence, allowing for presumed costs of control, participants may have indeed been optimizing performance.

The analysis above highlights the need for an independent assessment of the costs of control. Recent work has begun to quantify costs associated with cognitive control [8], and to incorporate these in formal analyses of behavioral performance (e.g., [73]). Insofar as such costs offset the rewards associated with performance, a better understanding of these, and factoring them into analyses of behavior, promises to permit more accurate assessments of the extent to which people perform optimally. That said, there is increasing evidence that, under many circumstances, people do actively adjust decision parameters in response to changes in task conditions [55, 40, 45, 31, 42, 34]. Information needed to make such adjustments can come from several sources, including prior experience (i.e., expectations), the current stimulus, and assessments of the outcome of performance itself (e.g., accuracy, speed, processing conflict, confidence, etc.).

Recent studies have begun to address the mechanisms responsible for making such adjustments, with a particular focus on how they are driven by performance outcomes [7, 35, 81, 11, 69, 39]. However, few of these studies have involved normative models that explicitly address the extent to which adjustments in decision parameters serve optimization. Below, we review two areas of recent work that attempt to do this: one addresses trial-to-trial adjustments based on prior knowledge, and the other considers the dynamics of adjustment within individual trials under stimulus variability.

3.2 Prior expectations and trial-to-trial adjustments

Given prior information on the probability of observing each stimulus in a 2AFC task, a DD process can be optimized by appropriately shifting the initial condition; biased rewards favoring one response over the other can be accommodated in a similar manner [4]. Comparisons of these predictions with human behavioral data were carried out in [70]. As in the experiments described above, stimulus discriminability and other conditions were fixed over each block of trials. On average participants achieved 97 − 99% of maximum reward, and some performed essentially optimally.

A related study of monkeys used a cued-response paradigm with a fixed stimulus presentation period that relieves the need for a speed-accuracy tradeoff, but in which motion coherences varied randomly from trial to trial, and differences in reward contingencies for the two responses were signaled before each trial. The monkeys came within 0.5% and 2% respectively of maximum possible rewards by adjusting the balance between preferring the higher reward and accurately assessing the noisy stimulus [24]. This implies appropriate shifts in the psychometric (accuracy vs. discriminability) function depending on coherence, which can be effected by either biasing drift rates, or by shifting initial conditions. That the latter is more likely was established in [62] by fitting LIP recordings. Behavioral studies of humans discriminating low aspect ratio rectangles also revealed near-optimal shifts in initial conditions in response to changes in reward bias [25].

In the above cases, participants were shown to perform (near) optimally based on their estimates of stimulus or reward probabilities, when explicitly informed of them. Humans also exhibit adjustment effects in response to patterns of repetitions and alternations that necessarily occur in random sequences, presumably based on a general expectation that locally observed patterns will persist or recur. Models that address RT and accuracy effects reflective of such adjustments appear in [14, 38, 37, 30]. These models further support the view that participants adapt by shifting the initial conditions of the decision process. While such adjustments typically degrade performance in response to truly random sequences (they are suboptimal), the ability to extract patterns in natural situations is advantageous, since it allows people to adapt prior beliefs to better match context-specific, stationary (or slowly changing) environments, once they have learned relevant statistics. This possibility is explored in [83].

3.3 Stimulus variability and within-trial adjustments

The work described in the previous subsection indicates that participants make trial-to-trial adjustments in decision parameters, especially when patterns are present or apparent in the sequence of stimuli. Modeling work has also suggested that participants can adapt decision parameters within trials, especially when task conditions vary from trial to trial. For example, in 2AFC experiments with mixed motion coherences it has been proposed that slow decisions, typically associated with low SNR and lack of confidence, may be accelerated by a monotonically-rising “urgency” signal corresponding to increasing drift rate, a proposal supported by fits to LIP recordings [33]. A recent analysis of DD models applied to variable discriminability stimuli shows that drift rate adjustments are suboptimal compared to setting initial conditions [76], but also that human participants appear to combine the two strategies in a similar manner for both fixed and variable discriminability, in contrast to the claims that drift rate bias should dominate in the latter case [33], and that this is necessary for optimality [80].

The Eriksen flanker task [23] has also been used to study within-trial dynamics of processing and attentional control (e.g., [31, 18, 7]). In this task participants carry out a standard 2AFC on a central stimulus, flanked by stimuli that either invite the same response (compatible) or its alternative (incompatible). Accuracy is lower for incompatible trials and, plotted against RT, displays a dip below 50% followed by a rise to the same asymptote as for compatible trials: see [31, Fig. 1]. The network model of [18] suggested that these effects could be explained by dynamic shifts of attention within each trial. In this model separate units accumulate bottom-up perceptual evidence from the central and flanker stimuli while subject to top-down attentional influences [18, Fig. 8], so that processing of the central stimulus eventually prevails. Subsequently, [43] showed that shifting attention can be captured by a time varying drift rate A(t) in a one-dimensional OU model. Representing A(t) as a sum of exponentials, it was possible to fit accuracy/RT data and RT distributions with fewer parameters than the network of [18].

A normative interpretation has been provided by a Bayesian analysis that models task performance as dynamic updating of the joint probability distribution for stimulus identity and trial type (compatible/incompatible) [82]. Updating Bayesian posteriors may seem unrelated to the model of [18] or its reduction [43], but for Gaussian stimulus probabilities with sufficiently large variance, it can be well-approximated by a set of uncoupled DD processes that allow analytical estimation of the evolving posteriors, and hence of accuracy and RT distributions [44]. These also fit the data of [18], suggesting both that participants adjust decision parameters within each trial, and that the network of [18] represents a neurally-plausible model of this (near-) optimal computation.

3.4 Exploration, exploitation, and neuromodulation

Thus far, we have focused on studies that examine performance in the context of a particular task. While the decision parameters optimal for that task may vary over stimuli, more general task features have remained constant (e.g., the domain of the stimuli, responses and rewards, the relevant set of stimulus-response mappings, etc.). Under such stationary conditions, good performance implies exploitation of a well-defined strategy. However, real world conditions can change, requiring exploration of different possible goals (i.e., what new tasks should be performed). This typically involves foregoing immediate rewards in the service of greater ones over the longer term. Balancing this tradeoff involves a higher level form of optimization. While there is no general solution to this problem, a growing body of research seeks to understand how organisms, including humans, manage the exploitation/exploration tradeoff [17].

One theory proposes that the neuromodulatory brainstem nucleus locus coeruleus (LC) contributes to regulating this balance, by dynamically modulating neural responsivity through the phasic release of norepinephrine (NE) [66, 74, 1]. According to this Adaptive Gain Theory, transient increases in LC-NE activity promote exploitation by increasing the gain of task-relevant processes, and thereby optimize performance of the current task [10, 68]. In contrast, sustained (and therefore task-indiscriminate) increases in LC-NE activity favor off-task processes, promoting exploration. Shifts between these modes are driven by ongoing assessments of task utility, such that sustained increases favor exploitation, while prolonged decreases favor exploration [1]. A closely-related proposal suggests that fluctuations in uncertainty can similarly drive the LC-NE system [84].

These mechanisms were implemented in a reinforcement learning (RL) model of reversal learning [48]. This study showed that including the LC-NE gain regulation mechanism and a simple cortical performance monitoring mechanism in a standard RL model dramatically improved the system’s ability to adapt to changes in reinforcement contingencies, and provided a qualitative fit to both behavioral data and direct recordings from the LC in monkeys performing a reversal task [16]. Following reversals, the combined effects of gain modulation and RL allowed fast transitions from exploration to exploitation, whereas with RL alone such transitions were associated with substantially higher ERs that persisted for longer. Subsequent studies in humans have used the observation that pupil diameter closely tracks LC activity [1] to demonstrate that engagement of the LC-NE system conforms to predictions about its role in regulating the explore/exploit tradeoff [22, 27, 36, 51]. Further work remains to more rigorously test the extent to which this is achieved optimally, under suitably constrained conditions.

3.5 Physiological constraints?

Most of the theoretical work reviewed above is based on relatively abstract models. While these have provided important insights into constraints on optimality at the systems level, they do not address constraints that may arise from the underlying neural circuits. Explorations of these have begun by relating abstract models to biophysical aspects of neural function. For example, spiking-neuron models can be reduced in dimension by averaging over populations of cells [79, 21], allowing them to include the effects of synaptic time constants and neurotransmitters such as NE [20].

The resulting nonlinear differential equations are more complex than the LCA Eqs. (1–2), and unlike the optimal DD process of Eq. (5), they can possess multiple stable states [79, Figs. 4–5], [21, Fig. 13]. Even when there is an attracting curve, nonlinear dynamics can cause suboptimal integration, and deviations from the curve can blur the decision thresholds [77, Fig. 7]. Adjustments in baseline activity and gain can keep accumulators in near-linear dynamical ranges [66, 15], but the nonlinear effects suggest that there are physiological obstructions to optimality, especially when task conditions span a wide range. Improvements of low-dimensional models, derived from biophysically-detailed neural networks and coupled with multi- unit and multi-area recordings, may help determine whether and how such constraints cause significant deviations from optimality.