Binding of sights and sounds: Age-related changes in multisensory temporal processing

1.1. Temporal aspects of multisensory processing

The benefits of the combined use of information from several senses have been revealed in numerous studies, and include enhancements in signal detection, speeded motor responses, and improved speech in noise performance (Frens, Van Opstal, & Van der Willigen, 1995; Grant & Seitz, 2000; Hughes, Reuter-Lorenz, Nozawa, & Fendrich, 1994; Lovelace, Stein, & Wallace, 2003; Sumby & Pollack, 1954). In addition, a host of psychophysical illusions reveal the continual and compelling interplay between the senses. For example, pairing distinct and discordant auditory and visual speech cues (e.g., an auditory /ba/ with a visual /ga/) can result in report of an intermediary and novel percept (e.g., /tha/ or /da/) (McGurk & MacDonald, 1976). Illusory percepts can also be generated with highly reduced multisensory stimuli, as evidenced by the fact that the presentation of a single visual flash accompanied by two tone pips can result in report of multiple flashes (Shams, Kamitani, & Shimojo, 2000).

Where examined, these behavioral benefits and perceptual illusions have been shown to be critically dependent on the temporal and spatial structure of the paired stimuli, with the strength of multisensory interactions declining as a function of increasing spatial and/or temporal disparity. In the temporal realm, numerous studies have suggested the presence of a temporal window of multisensory integration, or a range of interstimulus intervals over which multisensory stimuli are highly likely to be bound into a single perceptual event (Dixon & Spitz, 1980; Shams, Kamitani, & Shimojo, 2002; van Wassenhove, Grant, & Poeppel, 2007). The boundaries of these temporal binding processes have been delineated by quantifying the perseverance and magnitude of multisensory effects (e.g., speeded motor reaction times, psychophysical illusions, reports of simultaneity) as the time interval between the presentation of the constituent multisensory stimuli is lengthened (Colonius & Diederich, 2004; Dixon & Spitz, 1980; Koppen & Spence, 2007; Munhall, Gribble, Sacco, & Ward, 1996; Shams et al., 2002; van Wassenhove et al., 2007). Studies have indicated that multisensory interactions are reduced with increasing asynchrony and that the rate of decay of integrative effects is asymmetric; the slope of the left side of the window is steeper indicating that asynchrony is more readily detected for stimulus pairings when the auditory cue is presented first.

In the search for the neural correlates for these multisensory behavioral and perceptual phenomena, human electrophysiological and imaging studies have revealed a similar temporal dependency. Synchronous presentation of auditory and visual speech produces decreases in the latency of early cortical auditory evoked potentials (van Wassenhove, Grant, & Poeppel, 2005). Furthermore, combined audiovisual stimuli produce greater gamma band oscillatory activity when presented at smaller audiovisual stimulus onset asynchronies (SOAs) (Senkowski, Talsma, Grigutsch, Herrmann, & Woldorff, 2007), which is suggested to be reflective of multisensory binding (Schneider, Debener, Oostenveld, & Engel, 2008; Senkowski, Schneider, Tandler, & Engel, 2009). Imaging studies have identified a distributed network of cortical and subcortical regions involved in multisensory integration whose activation profiles reveal a strong temporal dependency (Bushara, Grafman, & Hallett, 2001; Dhamala, Assisi, Jirsa, Steinberg, & Kelso, 2007; Noesselt et al., 2007; Powers, Hevey, & Wallace, Unpublished results). For example, blood oxygenation level dependent (BOLD) signals, an indirect measure of neuronal activity revealed in fMRI studies, have been found to be increased in the superior temporal sulcus (STS) and auditory and visual cortices in response to coincident audiovisual stimuli (Noesselt et al., 2007). Conversely, BOLD decreases have been observed in these areas in response to asynchronous stimulus pairs (Noesselt et al., 2007).

1.2. Age effects on multisensory task performance

While the bulk of human multisensory research has focused on adults, the development of various aspects of multisensory integration has been examined in human infants, largely using paradigms that track the duration of gaze maintenance (i.e., preferential looking). Behavioral work has indicated that infants as young as four months of age show the ability to detect tempo and synchrony, and that temporal invariants (amodal cues – available to both auditory and visual senses) assist in discerning what stimuli are produced by a unitary event (Bahrick, 1983, 1987, 1988; Lewkowicz, 1986, 1992, 1996, 2000; Spelke, 1979). Furthermore, Lewkowicz (1996) has shown that when compared to adults, infants have a larger temporal window for binding asynchronous visual and auditory stimuli, suggesting that they perceptually fuse temporally disparate multisensory stimulus pairs that are not fused in adults.

While no studies have examined multisensory performance between older children and adults on temporally based tasks, work has reported immature multisensory processing abilities in pre-adolescents and adolescents relative to adults on a variety of non-temporally based tasks in the audiovisual realm and in other sensory modalities (Barutchu et al., 2010; Gori, Del Viva, Sandini, & Burr, 2008; Massaro, 1984; McGurk & MacDonald, 1976; Tremblay et al., 2007). Studies examining the McGurk illusion in children and adults reported fewer perceived illusions (instances of multisensory integration) in younger participants and found that when responses were dominated by a single modality (i.e., unfused trials), children relied more heavily on the auditory input (McGurk & MacDonald, 1976; Massaro, 1984; Tremblay et al., 2007). While the influence of vision on audiovisual processing appears to increase with age for speech, performance on a measure employing basic stimuli (i.e., flash beep illusion) reportedly did not change with age (Tremblay et al., 2007). Like the speech studies, work in other sensory modalities (i.e., visual–haptic, visual–proprioception) has identified compelling immaturities in multisensory integration (Gori et al., 2008; Nardini, Jones, Bedford, & Braddick, 2008). Performance differences were noted in children 5–10 years of age and adults on a visual and haptic (active touch) size discrimination task (Gori et al., 2008). Whereas adults and older children were found to weight visual information more heavily in making size estimations, the judgments of the youngest children were more influenced by the haptic information. This and other data suggest that middle childhood (i.e., 8–10 years) represents an important transitional period for the maturation of multisensory processing. For a brief review on age-related differences in statistical optimality of multisensory integration and more evidence for delays in the emergence of interactions, see Ernst (2008).

1.3. Characterization of multisensory temporal processing in children: rationale for the current study

As highlighted, research has identified age-related differences in multisensory abilities and has established the concept of a plastic multisensory temporal binding window that changes with ontogeny. However, no studies have systematically documented differences in the temporal aspects of audiovisual integration between children and adults. The goal of the current study was to characterize multisensory temporal function in children and adults using an audiovisual simultaneity judgment task previously used by our group to assess the multisensory temporal binding window in adults (as well as its malleability, see Powers, Hillock, & Wallace, 2009). This measure enables us to directly compare multisensory temporal function in younger and older participants.

2.1. Subjects

Typically developing children (n = 13; 11 males; mean age = 10.7 years) and adults (n = 14; 6 males; mean age = 26.6 years) were recruited to participate in the study. All participants and parents/guardians of minors were consented and assented prior to study participation in accordance with an approved protocol of the Vanderbilt Institutional Review Board (IRB). All subjects had normal hearing (pure tone thresholds less than 25 dB HL at octave frequencies from 250 to 8000 Hz) and good visual acuity with or without correction (Snellen criterion of 20/25 – 2 [2 or fewer errors per eye]) and average or above-average intelligence. All children completed the Kaufmann Brief Intelligence Test, second edition (K-BIT II), which provides an estimate of intellectual ability. No formal intelligence screening was completed on adults. The adult group was comprised of college educated individuals and undergraduate students at Vanderbilt University with no history of learning difficulties (per self report). Hearing and vision screenings were completed at the start of the session and intelligence testing followed completion of the multisensory task.

2.2. Stimuli and experimental design

A point of subjective simultaneity (PSS) judgment task (adapted from Fujisaki, Shimojo, Kashino, & Nishida, 2004) was administered to each participant twice (i.e., assessments 1 and 2). Subjects were seated in a quiet, dimly lit room approximately 48 cm from a high refresh-rate computer monitor (ViewSonic E70fB, 120 Hz). A white crosshair fixation marker (1 cm × 1 cm) was situated in the center of a black background on the computer screen for the duration of the experiment. Auditory (8 ms duration, 1800 Hz tone burst, 99.2 peak dB SPL [unweighted]) and visual (8 ms duration, white ring flash subtending 15° of space, outer diameter = 12.4 cm, inner diameter = 6.0 cm, area = 369.8 cm²) stimulus pairs were presented in a randomly interleaved fashion at the following visual–auditory SOAs: 0, ±50, ±100, ±150, ±200, ±250, ±350 and ±450 ms (Fig. 1). The auditory stimulus was presented via Etymotic Research ER-3A insert earphones and auditory and visual stimulus durations and SOAs were verified externally with an oscilloscope. Whereas positive values represent visual stimuli leading auditory stimuli, negative values represent the opposite. A total of 330 responses were collected during each assessment (22 samples × 15 SOAs). Each new trial was initiated 1 s after the participant logged his/her response to the previous trial. On the infrequent occasion that a participant failed to log his/her response within approximately 5 s of stimulus presentation, he/she was prompted to make a response by the experimenter. Total test time for the multisensory task was approximately 25 min, including breaks. Stimulus delivery and data logging were controlled by E-prime 2.0 (2.0.1.109).

Instructions were read aloud for both groups and the task was initiated by the experimenter (see Appendices 1 and 2 for adult and child instructions). For the children, the task objectives were placed in the context of a story that distinguished the auditory–visual communication styles of boy and girl lightning bugs. Following the story, children answered a circumscribed list of questions to ensure proper understanding of the task and the capacity to distinguish between the bug images. Behavioral judgments (i.e., simultaneous or asynchronous) were recorded by pressing buttons labeled with numbers (1 or 4, adults) or lightning bug images (blue [male] or red [female], children) on a response box (Psychology Software Tools Response Box Model 200A). Bug images had characteristics other than color to aid in distinguishing their gender (as screenings did not include testing for color vision deficiency). Responses were counterbalanced across participants such that the buttons associated with simultaneity and non-simultaneity were reversed for half of the subjects. A follow-up experiment in which a novel group of adults (n = 8) were instructed using the story-based technique employed with the children revealed no significant effect of instruction method on responses compared to those of 8 randomly selected adults from our experimental group.

2.3. Data analysis and temporal window derivation

Differences in window size and simultaneity judgment at sampled SOAs were assessed between groups and across assessments using independent samples’ t-tests and repeated-measures analysis of variance (rmANOVA). To correct for dependence among the repeated measures within subjects, Greenhouse–Geisser corrections were performed. Because no significant effect of session was observed for measures of window size (Supplement 1), the data was collapsed across assessments. However, for analysis of within-session effects (e.g., effects resulting from fatigue or task-learning), responses were divided between the first and second halves of the assessments.

The mean probability of simultaneity report (i.e., number of simultaneous responses as a function of total responses) was calculated at each SOA for all participants. Points were fit to create a distribution which served as the basis for determining the size of the temporal binding window. Sigmoids were generated to the discretely sampled points from each half of the distribution (left: −450 to 0 ms, right: 0 to +450 ms) for all subjects. Distributions were produced from a two-by-two matrix comprised of interpolated y values (probability of simultaneity report) at x values (time points) ranging from −600 to +600 ms in 0.1 ms increments. The criterion used to characterize the size of the multisensory temporal binding window for each subject was the width of this distribution at 3/4 maximum.

3.1. Defining group differences in window size

The mean multisensory temporal binding window for the 10 and 11 year old children (413 ms) was found to be significantly wider than that of the adults (299 ms) (t(25) = 3.945, p = 0.001; Table 1 and Fig. 2). Although several children had window sizes comparable to the adult mean, more than a third of the children had windows in excess of 450 ms, a value never seen in adults. In general, children were found to have more symmetric distributions when compared with adults, as illustrated by the comparison in two representative subjects shown in Fig. 3.

3.2. Defining group differences in the probability of simultaneity judgment

Although the comparison of mean window size provided a global measure of group differences in the multisensory temporal binding window, additional analyses were performed to further delineate the temporal structure and nature of the age-related differences in participants’ reports of simultaneity/asynchrony. A repeated-measures ANOVA with a within-subject factor of SOA and a between-subject factor of group revealed a significant main effect of group and SOA and an SOA by group interaction, p < 0.01, all tests (Table 2). These results indicate group differences in simultaneity judgment that differ across SOA conditions, or distinctions between groups at select SOAs.

Follow-up contrasts (i.e., independent samples’ t-tests) were performed to identify which SOAs contributed to these group differences. A Bonferroni correction was applied to reduce the risk of elevation of familywise error due to the number of comparisons (i.e., 15), resulting in a more stringent criterion value for significance (p < 0.0033). Equal variance between groups was assumed for all comparisons unless otherwise noted based on results from Levene’s test for equality of variances. Significant differences in the likelihood of simultaneity report were observed for the following conditions: A350V, A250V, A200V, A150V (Fig. 4, Table 3) – moderate to moderately long SOAs in which the onset of the auditory stimulus preceded the onset of the visual stimulus. A marginally significant difference was noted between groups at the most extreme auditory leading visual SOA (A450V), p = 0.004. In contrast, no differences in simultaneity report were observed in the objectively simultaneous condition, at short stimulus onset asynchronies (e.g., −50 ms, −100 ms) in which the auditory stimulus was leading, or in any of the conditions in which the visual stimulus preceded the auditory stimulus.

3.3. Summary of between and within assessment group effects

In an effort to assess possible fatigue and procedural or perceptual learning effects, subjects’ responses were compared across the two assessments (Fig. 5). These pseudocolor plots serve to reinforce the difference between children and adults in the size of the temporal window (C–A, bottom contrast – note the difference being more pronounced on the left side of the distributions), as well as showing equivalent performance between the two assessments (A1–A2, right contrast).

Despite the lack of differences across assessments for either group, additional analyses were performed to assay more rapid within-session changes. In this analysis, windows were derived from averaged responses collected during the first half of assessments 1 and 2 and compared to those collected in the latter half of the assessments. In adults, no significant within assessment effects were observed (p > 0.05). In contrast, windows derived from children’s responses on the last half of trials were larger than those from the first half of responses, t(12) = −2.496, p = 0.028 (Table 4). Mean window size in children differed by 79 ms, whereas in adults this difference was only 12 ms.

To determine whether the widened distributions exhibited by children during the latter part of each experiment were driving the group differences, window sizes derived from the earlier and later trials were compared across groups. Results of independent samples’ t tests revealed significant group differences in both the beginning and end of assessments 1 and 2 (Table 5). While these findings suggest that children fatigued more quickly than adults, illustrated by window widening in the latter half of the assessments, the enlargement of window size was still evident in the earliest trials. Thus, while group differences may be more pronounced in the latter portion of the assessments, age effects are apparent throughout the experimental task suggesting that differential fatigue and/or learning effects cannot fully account for group differences.