How precise are tests used to diagnose learning disability, progressive brain disease or impairment from head injury?

Timothy Salthouse, PhD, a noted cognitive psychologist at the University of Virginia, has demonstrated that giving a test only once isn’t enough to get a clear picture of someone’s mental functioning. It appears that repeating tests over a short period may give a more accurate range of scores, improving diagnostic workups.

Salthouse gave 16 common cognitive and neuropsychological tests to evenly divided participants (90 in the first, 1600 in the second) into groups of ages 18-39, 50-59 and 60-97 years old. In both studies, the variation between someone’s scores on the same test given three times over two weeks was as big as the variation between the scores of people in different age groups. It’s as if on the same test, someone acted like a 20-year-old on a Monday, a 45-year-old the following Friday, and a 32-year-old the following Wednesday. This major inconsistency raises questions about the worth of single, one-time test scores.

“I don’t think many people would have expected that the variability would be this large, and apparent in a wide variety of cognitive tests – not simply tests of speed or alertness,” says Salthouse.

Psychologists frequently use tests of vocabulary, word recall, spatial relations, pattern comparison and the like to understand normal function and diagnose impairment. Experts use the scores to differentiate between diagnoses, detect changes in level of functioning or to give a diagnosis in the first place. Where scores fall relative to standardized cutoffs affects treatment, insurance, education plans and more. Yet the apparent fuzziness of one-time assessments could make it hard to tell whether someone is truly impaired, or truly improving or worsening, instead of showing normal short-term fluctuation.

Accordingly, Salthouse has come to believe that everyone has a range of typical performances, a one-person bell curve. Any given test will net a performance somewhere along that curve, as when a hitter’s good and bad days are factored into a seasonal batting average. Some persons’ scores would hew more closely to their average, but for those who have high internal variation, classification based on one assessment could be way off the mark.

Salthouse says it may be time to view cognitive abilities as a distribution of many potential levels of performance instead of as one stable short-term level. He proposes the use of a “measurement burst” procedure that bases understanding on several parallel assessments within a relatively short period. Results gained in this manner are likely to be more stable, offering a better basis for calibrating individual change.

Before any procedural updates, Salthouse says, “More will have to be learned about this phenomenon and the conditions under which it operates.” Multiple assessments involve more time and expense but may be necessary, he notes, to distinguish short-term fluctuation from true ability level. In addition, psychologists would have to develop new test norms and truly equivalent versions of the same test.

Finally, Salthouse believes that measures of within-person variability could be a useful diagnostic marker in their own right. For example, he and other cognitive psychologists are discussing whether wilder fluctuations within one person’s test scores are an early warning of mental decline.

Source: Implications of Within-Person Variability in Cognitive and Neuropsychological Functioning for the Interpretation of Change, Timothy A. Salthouse, PhD, University of Virginia; Neuropsychology, Vol. 21, No. 4.