Defining MCI in the Framingham Heart Study Offspring: Education vs. WRAT-based norms

Introduction

Mild cognitive impairment (MCI) is a transitional clinical stage between normal aging and mild dementia. Its diagnosis requires evidence suggestive of cognitive decline, by self or informant report, as well as by impaired performance on neuropsychological tests based on published norms or on changes in a given individual over time. It also requires that the cognitive decline be insufficient to cause the functional deficits required for a diagnosis of dementia.^1,2 In many studies, evidence of cognitive change from a premorbid level is inferred based on low test scores during a single cognitive screening; it is not directly demonstrated by decline over multiple testing sessions.³

An elderly person with low premorbid cognitive ability, and resultant low test scores, should not be classified as having developed MCI if his scores do not represent a substantial decline from his own baseline. To avoid such misclassifications, diagnostic methods need to account for premorbid functioning. One method relies on clinical judgment, rather than strict cutoff scores, to determine the likelihood that a participant's performance on a given test represents a decline from his or her premorbid level of functioning.^4,5 Information such as the participant's educational and occupational background, and scores on cognitive tests designed to measure premorbid abilities, can help shape the clinician's judgment. The most common method, however, uses age and education-based norms when determining the threshold for MCI-level performance on a given test. This threshold is often set at 1.5 SDs below the reference-group mean.^1,3,6–13 Depending on the characteristics of the population being studied, years of education (expressed as a continuous variable), or the attainment of a specific benchmark (i.e. high school diploma), may be used to determine the `education-based norms.'

While convenient in large-scale epidemiological studies, the use of education-based norms can be problematic. Educational attainment can be influenced by many non-cognitive factors, including income, race, immigrant status, and gender, especially in the current generation of elderly persons.^14,15 Moreover, quantity of education is not synonymous with quality.¹⁶ For individuals attending poorly-funded and/or segregated schools, the `years of education' measure could overestimate the amount of learning that took place. Manly et al<sup>16</sup> and O'Bryant et al<sup>17</sup> found this to be the case when they compared racially diverse participants' reported years of education to their scores on the Wide Range Achievement Test: Third Edition (WRAT-3) Reading subtest,<sup>18</sup> a word identification test that has been validated as a measure of premorbid functioning. Many participants scored lower on the WRAT-3 than their years of education would have predicted. There are also cohort effects that the `years of education' measure does not capture. For instance, an average 9th grade education in 1950 was different from an average 9th grade education in 2010, in its scope and rigor; this fact complicates the application of cross-generational education based cut-points to identify sub-optimal performance.

For highly-educated individuals, or those with considerable cognitive pursuits after they completed their formal education, a score in the `normal' range defined using an education cut-point could mask a significant decline in their own premorbid abilities. In a study of highly intelligent elderly participants, Rentz et al<sup>14</sup> found that utilizing cut-off scores based on participants' performance on the American version of the National Adult Reading Test (AMNART)<sup>19</sup> yielded significantly more-sensitive diagnoses of progressive cognitive decline than using education-based cut-offs. For most participants, the AMNART-based thresholds for `cognitively intact' scores were much higher than the education-based ones. Thus, actual impairment was better identified among the highly intelligent when using the AMNART-based thresholds. Manly et al¹⁶ argued that scores on tests of reading performance, such as the WRAT-3 and the AMNART, measure educational experience more accurately than years of education, and thus constitute a better estimate of the skills and information required to satisfactorily complete neuropsychological tests. The use of education-based norms with a broad range of educational attainment across study samples may contribute to the widely divergent estimates of MCI's prevalence and time course in the literature. It may also partly account for the high percentage of prevalent MCI cases that “improve” or “revert to normal” at subsequent visits, at rates ranging from 10% to 50%^20,21 for different subtypes of MCI as persons with randomly `low' scores regress towards the mean.

The purpose of this study is to examine how MCI case identification is differentially impacted when using education-based versus reading ability-based norms in a community-based young-old sample. The reliability and validity of MCI categorization will be examined through rates of conversion back to normal and progression to mild dementia. To our knowledge, this is the first large-scale, population-based study to use an objective test of premorbid abilities as the basis for the cut-off scores used to determine a participant's MCI status at a single cognitive assessment.

Methods

The Framingham Heart Study is a longitudinal, prospective, community-based study of risk factors for cardiovascular and cerebrovascular disease, with initial enrollment of 5209 Original participants in 1948. Their Offspring (n=5124) came into the study in 1971, and each participant had at least one biological parent in the first-generation FHS Original cohort or was the spouse of a participant with an Original cohort parent. Detailed information about demographics and recruitment is provided elsewhere; briefly, all participants have a structured medical history, physical examination and laboratory testing done once every 2–4 years, and since 1982 these assessments have included a Folstein Mini-Mental Status Examination (MMSE).^22,23 Medical records from hospitalizations and office visits are requested at the time of each physical exam.

As part of a longitudinal study of brain aging, 2607 FHS Offspring participants had an initial (1999–2005) comprehensive neuropsychological evaluation, and results from the assessments of 2506 participants were used to calculate FHS-specific norms. 96 of the participants excluded from the norms were excluded due to prevalent clinical dementia, stroke, or other neurological illnesses, such as multiple sclerosis or brain tumors, that could impact cognitive test scores. For this study, participants who did not meet our minimum age criterion of at least 65 years were also excluded (n=1571). The final study sample included 935 FHS Offspring participants (mean age = 72 ± 5, 53% women) who were free of dementia and stroke at their initial assessment. The corresponding MMSE was administered within 0.9 ±1.0 years of their initial NP assessment. The median score was 29, with an interquartile range of 28 – 30. Of the 935 participants, 535 completed follow-up neuropsychological evaluations, which were conducted between August 2005 and December 2009. Testing took place at the study site as well as at participants' places of residence, including private houses and nursing homes. The study was given full IRB approval. All participants received an explanation of their role in the study, gave their consent to participate, and signed informed consent forms.

Results

Figure 1 shows the distribution of WRAT scores amongst the four education groups. The boxplots display the 25^th and 75^th percentile WRAT scores, and the whiskers display the minimum and maximum WRAT scores. There is considerable variability both between and within the groups. For instance, the WRAT score in the 75^th percentile of the No HS Diploma group, 45, was higher than the 25^th percentile score of the HS Diploma group, 43. The maximum possible score on the WRAT is a 57.

Table 1 shows the number of participants classified as cognitively intact or MCI, based on their performance on tests of Memory (MCI-M), Executive Function (MCI-EF), Abstraction (MCI-Ab), Language (MCI-L), and Visuospatial Function (MCI-VS).¹ The MCI participants were divided into three groups: those classified as MCI using both education-based and WRAT-based norms, those classified as MCI using education-based norms only, and those classified as MCI using WRAT-based norms only. The percent discordant represents the percentage of participants classified as MCI by only one of the two methods. It was calculated for each test by adding the education-only and WRAT-only groups and then dividing this sum by the total number of participants. Due to the higher discordant percentages for Memory and Abstraction and the results of the incident dementia analyses (see Table 3), additional analyses stratified these discordant calculations by the four education groups. For Memory, the discordant percentages for the No HS Diploma, HS Diploma, Some College, and College Graduate groups were, 8.2%, 2.2%, 5.3%, and 5.8%, respectively. For Abstraction, the discordant percentages for the four groups were, 6.7%, 2.5%, 4.8%, and 5.4%, respectively. Discordance between MCI defined using WRAT-based norms versus education-based norms was greatest for participants with no HS diploma and least for those with HS diplomas but no further education.

Table 2 shows, for the 535 participants who also completed a follow-up neuropsychological assessment, NP 2 (average time interval between assessments = 6 years), their MCI status at the two visits, using the education and WRAT-based classification methods. The last column shows the percentage of participants whose cognitive status “improved,” or “reverted to normal,” over time: Their test scores classified them as MCI at the first assessment but did not meet performance criteria for MCI at the second assessment. Neither method was superior at minimizing the amount of participants who “revert to normal” from the first visit to the second.

Of the 935 participants, 835 had sufficient follow-up to allow the investigators to assess the development of subsequent dementia. 29 (3.5%) developed definite, probable or possible Alzheimer's dementia over a mean follow-up of 6 years (range = 5 – 11). Table 3 shows the hazard ratios (HR) for developing AD, comparing participants with and without MCI-level scores at their first cognitive assessment. Results are shown for each of the cognitive domains, and using the education and WRAT methods. Participants with MCI-M had a significantly increased risk of developing dementia, using both the education (HR = 14.38 p < .001) and WRAT (HR = 14.46, p < .001) methods. Participants with MCI-EF also had a significantly increased risk of developing dementia, with slightly higher HRs using the WRAT method (HR = 5.57, p <.001) than the education method (HR = 4.49, p = .002). For Abstraction, using the education method, participants with MCI-Ab did not have a significantly increased risk of developing dementia (HR = 1.19, p = .814). However, participants with MCI-Ab using the WRAT method had a significantly increased risk of developing dementia (HR = 3.20, p = .033). Participants with MCI-L and MCI-VS did not have a significantly increased risk of developing dementia, regardless of whether the WRAT or education method was used.

To further compare the utility of the education and WRAT methods in predicting dementia, additional analyses were conducted on the Abstraction test scores. To more accurately distinguish the educational attainment of participants with higher education degrees, the highest level, College Graduate, was split into two groups, College Graduate Only and Postgraduate Degree. The norms were recalculated based on five education levels, and the survival analyses were conducted again. While the HRs for MCI-level Abstraction scores were higher when using five levels (HR=1.74, p=0.369) than four (HR=1.19, p=.814), participants with MCI-level Abstraction scores using five levels of education-based norms still did not have a significantly increased risk of developing dementia.

Discussion

The goal of this study was to determine if using WRAT-based norms to define MCI yields different groups of participants than using education-based norms, and if one method demonstrates an advantage over the other. The results showed some differences, particularly for the Memory, Abstraction, and Language tests, for which 4.4%,4.2%, and 4.5% of participants, respectively, were classified as MCI by one method but not the other. Although the magnitude of this difference may seem small, when focusing just on MCI participants, rather than the entire sample, the difference is more pronounced. For instance, of the 95 participants with MCI-Ab by any method, 56 were classified as MCI according to both methods, 21 by education alone, and 18 by WRAT alone. This means that, for populations that are expected to have higher rates of MCI (for instance, oldest-old or memory clinic samples), there may be a greater discordance between the two methods. One explanation for why MCI-EF showed the least discordance is that the executive function abilities measured by the Trails B test are not strongly correlated with measures of general intelligence^29–31 or years or education.³² Factors other than premorbid abilities/educational exposure appear to strongly influence individual differences in Trails B performance, and neither the education nor the WRAT method fully account for such differences.

For MCI-M and MCI-Ab, the greatest levels of discordance were found for the lowest (No HS Diploma) and highest (College Graduate) levels of education. This could be attributed to a high potential for a divergence between premorbid abilities and actual educational attainment in the case of the No HS Diploma group, and a wide range of educational experiences in the College Graduate group. Because school-based testing for learning disabilities was uncommon in the current generation of older Americans, we could not collect valid data on the incidence of learning disabilities in our cohort. However, anecdotal reports from our participants suggest that the No HS Diploma group included many participants with self-reported learning difficulties who struggled in high school, as well as those who were highly capable of completing high school but left before graduation for other reasons, such as military service or to work and support their families. We would expect to see very different neuropsychological test scores, as well as WRAT scores, from these types of participants, even though their level of educational attainment is identical. The College Graduate group included those with bachelor's degrees alone, as well as those with doctorates. WRAT estimates of ability may be especially likely to diverge from educational attainment for participants with very low or high levels of education.

The mere presence of discordance does not imply the superiority of one method over the other. To evaluate the WRAT and education-based methods, it is necessary to examine how accurately they identified those with true MCI: MCI which was more likely to progress to dementia. Results from the incident dementia analyses suggest an advantage for the WRAT method. Using the WRAT method, the HRs were either slightly or substantially higher than the HRs using the education method, for all five tests. The advantage of the WRAT was strongest for MCI-Ab: participants identified using the WRAT had a significantly increased risk of developing dementia, but those identified based on education did not. The WRAT advantage continued to hold when five levels of education were created by dividing the highly educated into two groups, suggesting that the advantage cannot merely be attributed to differences in years of education amongst college graduates (i.e. those with and without postgraduate education).

Our findings of considerable WRAT-score variability amongst participants with the same level of education help explain these results. It seems as though the premorbid abilities of the very low and very high scorers in each education level were not accurately captured by their years of education. For instance, the maximum WRAT score for the No HS Diploma group was a 54, which is the 75th percentile score for the College Graduate group. Our findings on discrepancies between WRAT and education echo those of other researchers,^14,16,17 and they provide a rationale for why the WRAT is important. Each educational group consists of individuals with unique intellectual strengths, weaknesses, and experiences, both inside and outside the classroom, and this complexity cannot be conveyed by educational attainment alone. The Similarities test, a subtest of the WAIS, requires participants to think abstractly and then articulate a concise response. It is a high g-loading test: it is strongly correlated with other measures of intellectual ability.³³ It may be especially responsive to premorbid abilities and educational experience. Educational exposure boosts one's performance on the test, as the abstract, scientific thinking required for high scores may be best learned in academic environments.³³ It follows that, if the WRAT more accurately captures participants' premorbid abilities and educational quality, using the WRAT would help identify appropriate cut-off scores for this test.

Using both methods, a high percentage of participants with MCI-level scores at baseline “improved” or “reverted to normal” over time: Their scores from the second assessment were better than 1.5 SDs below the reference-group mean. The percentages ranged from 42% to 87%, depending on the test and method. Practice effects may have contributed to the high percentage of participants whose scores improved from the first visit to the second, a shortcoming common to many longitudinal studies of cognition. These effects, however, are likely attenuated by the average interval of 6 years between assessments, and the progressive nature of MCI. They are unlikely to lead to incorrect classification of MCI as normal at the second assessment. For most participants, the best explanation for their “improvement” is that they were incorrectly classified as MCI at baseline. The underlying cause of their low initial test scores was probably not MCI, but rather another factor, such as anxiety, depression, low motivation, inattention, or a lower/higher level of premorbid functioning than their education or WRAT scores predicted. Other studies have found high rates of “impaired” test scores in cognitively intact elderly participants, particularly with non-memory tests, suggesting that neuropsychological tests may be oversensitive in the elderly.^{13,20,21,34,35}

Similar percentages of participants “improved” using both the WRAT and education method, suggesting that neither was superior at minimizing the problem of false positives. It is worth noting that some participants classified as MCI did not attend the second assessment but were classified as demented at follow-up based on other sources of information, such as medical records or neurology exams; if these participants had attended the second assessment, the percentage that “improved” may have been smaller.

Weaknesses of this study include the lack of racial diversity, the high number of participants (42.8%) who did not complete the second neuropsychological assessment, and the high percentage of participants who “reverted to normal” at their second assessment. The latter percentage was much higher than has been found by most other MCI studies. It is important to note, however, that this study was designed to investigate a specific methodological issue. The subjective components of the MCI criteria proposed by Petersen et al¹ were not implemented here. This study used a psychometric method based on impairment on a single test. Jak et al²⁰ found the single-test method to be the least reliable method for defining MCI, especially for nonamnestic tests, as many participants classified this way scored in the normal range at follow-up. Loewenstein et al²¹ and Snitz et al¹³ also found that single-test impairment failed to distinguish between normal aging and substantial, progressive cognitive impairment. For studies aiming to maximize the specificity of their MCI diagnoses, we caution against using the single-test impairment method. Prior research has identified poor memory and executive function test scores as strong predictors of subsequent dementia,^36–38 and results from this study supported our hypothesis that they would be the best indicators of genuine MCI. Performance on the Hooper Visual Organization Test was the least effective at identifying MCI.

One criticism of the WRAT as a measure of premorbid functioning is that it underestimates the abilities of those with weak reading skills but strengths in areas like math or science. (This is especially true of those with dyslexia, who can have an overall profile of high intelligence with isolated deficits in reading-related abilities.³⁹) This is a fair criticism, but word identification is one of few cognitive abilities that does not rapidly decline in MCI.^24–26 Because MCI often affects one's capacity to exercise skills such as visuospatial reasoning and complex psychomotor speed,⁴⁰ many tests used to assess intelligence in young adults are ill-suited for use in the elderly.

The use of a community-based cohort is a strength of this study. Rather than relying upon published norms from other studies, we were able to use normative data from our own cohort. The prospective study design also allowed surveillance for incident dementia in an established dementia-free cohort. The mean follow-up of 6 years between the baseline assessment and final diagnosis of dementia status is equal to, or longer than, that of most MCI studies.^{5,9–12,35–37}

Our findings indicate that the WRAT may be a better surrogate marker of premorbid abilities to use for determining cut-off scores for MCI diagnoses when compared with the commonly-used measure of educational attainment. Rentz et al¹⁴ has shown the benefits of using a word identification test to help identify cognitive impairment amongst the highly intelligent; this study supports using the WRAT in a study sample with a wide range of intellectual abilities. The WRAT can help improve diagnostic accuracy while adding only a few minutes of test administration, and it is particularly useful in assessing participants who lack previous testing data. Our results suggest that the WRAT should be considered for use as a complement to the standard measure of years of education.