ORIGINAL RESEARCH
Reliability of a Computerized Neurocognitive Test in Baseline Concussion Testing of High School Athletes James MacDonald, MD, MPH*† and Drew Duerson, MD*
Objective: Baseline assessments using computerized neurocognitive tests are frequently used in the management of sport-related concussions. Such testing is often done on an annual basis in a community setting. Reliability is a fundamental test characteristic that should be established for such tests. Our study examined the test–retest reliability of a computerized neurocognitive test in high school athletes over 1 year.
Design: Repeated measures design.
factors that might affect test performance, and (3) identifying the ideal time interval to repeat baseline testing in high school athletes.
Clinical Relevance: Computerized neurocognitive tests are used frequently in high school athletes, often within a model of baseline testing of asymptomatic individuals before the start of a sporting season. This study adds to the evidence that suggests in this population such testing may lack sufficient reliability to support clinical decision making.
Setting: Two American high schools.
Key Words: AxonSports, computerized cognitive assessment tool, intraclass correlation coefficient, psychometrics
Participants: High school athletes (N = 117) participating in
(Clin J Sport Med 2015;25:367–372)
American football or soccer during the 2011-2012 and 2012-2013 academic years.
Interventions: All study participants completed 2 baseline computerized neurocognitive tests taken 1 year apart at their respective schools. The test measures performance on 4 cognitive tasks: identification speed (Attention), detection speed (Processing Speed), one card learning accuracy (Learning), and one back speed (Working Memory).
Main Outcome Measures: Reliability was assessed by measuring the intraclass correlation coefficient (ICC) between the repeated measures of the 4 cognitive tasks. Pearson and Spearman correlation coefficients were calculated as a secondary outcome measure. Results: The measure for identification speed performed best (ICC = 0.672; 95% confidence interval, 0.559-0.760) and the measure for one card learning accuracy performed worst (ICC = 0.401; 95% confidence interval, 0.237-0.542). All tests had marginal or low reliability. Conclusions: In a population of high school athletes, computerized neurocognitive testing performed in a community setting demonstrated low to marginal test–retest reliability on baseline assessments 1 year apart. Further investigation should focus on (1) improving the reliability of individual tasks tested, (2) controlling for external
Submitted for publication October 19, 2013; accepted June 12, 2014. From the *Department of Pediatrics, Division of Sports Medicine, Nationwide Children’s Hospital, Columbus, Ohio; and †The Ohio State University College of Medicine, Columbus, Ohio. The authors report no conflicts of interest. Dr. MacDonald is supported in his research through an American Medical Society for Sports Medicine “Young Investigator” Grant. Corresponding Author: James MacDonald, MD, MPH, Department of Pediatrics, Division of Sports Medicine, Nationwide Children’s Hospital, 5680 Venture Dr., Dublin, OH 43017 ([email protected]). Copyright © 2014 Wolters Kluwer Health, Inc. All rights reserved.
Clin J Sport Med Volume 25, Number 4, July 2015
INTRODUCTION The fourth international consensus statement on concussion in sport defines concussion as a brain injury characterized by a complex pathophysiological process affecting the brain and caused by a direct blow to the head or an indirect blow elsewhere on the body, transmitting an impulsive force to the head.1 From 2001 to 2005, there were an estimated 502 000 emergency department (ED) visits for concussions; about half were identified as sport-related concussions.2 Sports are a frequent cause of concussion especially in youth: among individuals aged 15 to 24 years, sports are second only to motor vehicle accidents as the leading cause of concussions.3 Concussions now represent nearly 15% of all high school sport-related injuries.4 Computerized neurocognitive testing plays an increasingly important role in the contemporary management of sport-related concussions. These tests have been emphasized in the diagnosis and management of concussions because of the potential unreliability of athletes’ self-report of symptoms.5 The fourth international consensus statement recognizes neuropsychological testing, including computerized neurocognitive testing, as a clinical aid in making return to play decisions, although it also found insufficient evidence to recommend widespread testing.1 Because of their availability and convenience, computerized paradigms interpreted by well-trained personnel, as opposed to formal pen-and-paper neuropsychological tests, have become the predominant method for assessing cognitive function in athletes at risk for concussion.6 A study of more than 1000 concussions in high school athletes published in 2011 found that computerized neurocognitive tests were used in the assessment of 41.2% of the cases.4 Neurocognitive tests have often been promoted with the specific purpose of supporting return to play decisions within www.cjsportmed.com |
367
Copyright © 2015 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
MacDonald and Duerson
Clin J Sport Med Volume 25, Number 4, July 2015
a model of “baseline testing”: athletes participating in highrisk sports take such a test preseason, and the results can be used comparatively during follow-up testing for athletes who sustain a concussion during the season. This model, although appealing, has critics.7,8 The ideal interval to repeat baseline testing for computerized neurocognitive tests has not been established.9 The Centers for Disease Control and Prevention recommend that most components of baseline testing be repeated annually, although it also states that neurocognitive testing may need to be repeated as infrequently as every 2 years.10 It is the authors’ experience that most high school sports districts that provide baseline testing repeat these tests for athletes on an annual, preseason basis; and that such testing is most often conducted in a large group setting in the schools themselves or a similar venue, with little control placed on environmental stimuli. There are other potential problems with computerized neurocognitive tests. Randolph et al11 argue that the most critical psychometric property of a computerized neurocognitive test is its test–retest reliability. In tests with low or marginal reliability, any difference between the baseline score and the score on retesting may reflect the error of retesting rather than any true change in the athlete.12 In their review of several neurocognitive tests, they found that most failed to meet adequate reliability criteria for clinical decision making. Kirkwood et al7 note that what reliability studies do exist use short periods where tests may be expected to be more stable; they report that few studies span periods such as the weeks or months that may pass between baseline and postinjury testing. Moreover, testing environment seems to negatively affect performance especially in younger athletes.13 Other critiques of the tests have focused on the published studies that have largely been conducted by the test developers themselves, giving rise to concerns for bias.14 Although reliability testing has been done for adult athletes using computerized neurocognitive tests,15,16 there is a relative lack of similar testing for high school athletes. Register-Mihalik et al17 looked at a population of 20 college students (20 6 0.79 years) and 20 high school students (16 6 0.86 years) and measured test–retest reliability using Immediate Post-Concussion Assessment and Cognitive Testing over 3 visits separated by 24 to 72 hours. Test–retest reliabilities reported largely failed to meet adequate reliability criteria for clinical decision making. Segalowitz et al12 looked at a small population of 15 girls and 14 boys (mean age 15.4 years) and measured test–retest reliability over 7 days using Automated Neuropsychological Metrics testing. They reported mixed results, with an aggregate measure showing “robust reliability,” but individual subtests showing weaker correlations. In our review of the literature, we found no other studies looking at the test–retest reliability of a computerized neurocognitive test in a strictly high school population. We specifically found no study of test–retest reliability conducted over the course of a year. Given the above concerns, there is a further need to independently evaluate the psychometric properties of computerized neurocognitive tests, especially as they are typically administered in an adolescent population. There is also a need to determine whether the common
Baseline assessments for these athletes used the Axon Sports Computerized Cognitive Assessment Tool (CCAT), a computerized neurocognitive test (AxonSports, Wausau, Wisconsin).18 Axon Sports is an online version of CogState Sport, a computerized neurocognitive test that has been used since 2002.19 Testing was performed in groups in a large public space at the school. For each school, the location of testing remained the same for each study year, and Axon Sports was accessed with the same desktop computers each study year. Data were collected prospectively as part of an ongoing study for validity testing of a novel reaction time device published elsewhere in the literature.20 The Axon Sports CCAT reports performance in 4 cognitive tasks, each of which uses a simple paradigm in which a playing card initially appears face down and then suddenly turns face-up, awaiting a response from the testtaker.19 The 4 tasks are as follows: Attention (identification speed), using a choice reaction time paradigm; Processing Speed (detection speed), using a simple reaction time paradigm; Learning (one card learning accuracy), using visual recognition memory; and Working Memory (one back speed), using a one back paradigm.19
368
Copyright © 2014 Wolters Kluwer Health, Inc. All rights reserved.
| www.cjsportmed.com
practice of administering such tests on an annual, preseason basis is an ideal time frame. We set out to assess the test–retest reliability of 4 cognitive tasks measured by a computerized neurocognitive test in a population of high school athletes who were baseline tested at their respective schools on 2 occasions, 1 year apart. Our null hypothesis was that the measured test reliability would be adequate to support clinical decision making.
METHODS Study Subjects Students participating in American football, boys’ soccer and girls’ soccer at 2 high schools in central Ohio whose athletes are cared for by the authors’ institution were included. Baseline computerized neurocognitive testing was conducted in the summer of 2011 and again in 2012. The precise timing of testing was such that all study participants took their 2012 test between 50 and 52 weeks after their 2011 test. Any athlete who finished a testing session with an invalid baseline test despite repeated attempts was excluded. All athletes who completed a valid baseline test in both 2011 and in 2012 (N = 117) were included for final statistical analysis. As a secondary outcome of interest, athletes who sustained a concussion in their 2011 sport and who were baseline tested again in 2012 (N = 9) were identified to perform subgroup analyses (Figure 1). All concussions were diagnosed using consensus guidlelines1 by the authors themselves or other physicians specializing in pediatric sports medicine working in the authors’ same institution. At the time of 2012 testing, these athletes had all been medically cleared to play.
Assessments
Copyright © 2015 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
Clin J Sport Med Volume 25, Number 4, July 2015
Reliability of a Neurocognitive Test in Athletes
FIGURE 1. Inclusion and exclusion of eligible participants.
Axon Sports baseline reports used by clinicians typically label the tasks “Attention,” “Processing Speed,” “Learning,” and “Working Memory,” but in the research literature, the terms “identification speed,” “detection speed,” “one card learning accuracy,” and “one back speed” have been used, respectively, when studying Axon Sports. We routinely use the latter convention in this report.
Statistical Analysis Test-retest reliability is measured as an association between the scores on any test taken by the same participant on 2 different occasions over some time interval. Historically, correlations such as a Pearson coefficient (r) or a Spearman coefficient (rs) (the latter if data were not normally distributed) were calculated to measure this association. In contemporary psychometrics, such measures are thought prone to seriously exaggerate the impression of reliability.21 The intraclass correlation coefficient (ICC) is now thought to be better suited to the assessment of reliability and has been used in other studies similar to this one, although Pearson and Spearman coefficients are often reported concurrently.15,16,22 We calculated a 2-way random effects analysis of variance ICC to estimate the test–retest reliability of each of the 4 tasks measured by the Axon Sports CCAT. We assessed these data for normality using the Kolmogorov–Smirnov test. Correlations for any normally distributed data were also measured using the Pearson product moment correlation coefficient (r); for data that were not normally distributed, Spearman rho (rs) was used. We repeated ICC calculations for both the subgroup who had been concussed in 2011 and those who had remained concussion free in their sport (N = 108). Copyright © 2014 Wolters Kluwer Health, Inc. All rights reserved.
We used previously published criteria for qualitatively rating the levels of test reliability: test–retest reliability coefficients greater than 0.9 are considered very high; 0.80 to 0.89 are considered high; 0.70 to 0.79 are considered adequate; 0.60 to 0.69 are marginal; and 0.59 or below are low.23 The ICC values which are “very high” (greater than 0.9) are preferred for clinical decision making.11 Anastasi24 recommends 0.60 as the minimal acceptable ICC value for clinical use. IBM SPSS statistics version 21.0 (SPSS, Inc, Chicago, Illinois) was used for all analyses. Statistical significance was set at a = 0.05.
Ethical Considerations Informed consent was obtained in all cases. The study was approved by the Institutional Review Board of Nationwide Children’s Hospital.
RESULTS Demographic data for the 117 study participants are reported in Table 1. There was a preponderance of male athletes (72.6%). The mean age was 16.1 6 1.0 years (Table 1). The ICC and Pearson or Spearman correlation coefficients of 2 test variables demonstrated marginal reliability: identification speed (ICC = 0.672; rs = 0.624) and one back speed (ICC = 0.664; r = 0.664). The remaining 2 test variables demonstrated low reliability: detection speed (ICC = 0.554; rs = 0.520) and one card learning accuracy (ICC = 0.401; rs = 0.504). All correlation coefficients were found to be statistically significant at P , 0.001 (Table 2). Results for the subgroup analyses performed on the concussed and nonconcussed cohorts of the study group are reported in Table 3. For the concussed athletes, the only test www.cjsportmed.com |
369
Copyright © 2015 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
Clin J Sport Med Volume 25, Number 4, July 2015
MacDonald and Duerson
TABLE 1. Characteristics of Study Participants Participants
Gender, Male/Female, n (%)
Handedness, Right/Left, n (%)
Age, Mean 6 SD (Range), y*
Football Players, n (%)
Boys’ Soccer, n (%)
Girls’ Soccer, n (%)
117
85/32 (72.6/27.4)
107/10 (91.5/8.5)
16.1 6 1.0 (14-18)
37 (31.6)
48 (41.0)
32 (27.4)
*Age calculated as of 2012 testing date.
variable that was statistically significant was the one back speed, which demonstrated marginal reliability (ICC = 0.622; P , 0.027). All other findings were not statistically significant.
DISCUSSION We identified a need to independently determine the reliability of a computerized neurocognitive test when used in baseline assessments of high school athletes at risk for concussion, conducted 1 year apart. Our results demonstrate low to marginal reliability measures for the 4 cognitive tasks measured by the Axon Sports CCAT, with ICC values ranging from 0.401 to 0.672. None of the measures were found to have “very high” reliability, an ICC value of 0.9 or above, the characteristic preferred for clinical decision making.11 The ICC values for 2 cognitive tasks (detection speed and one card learning accuracy) were below a recommended minimally acceptable value for clinical use.24 The reliability measures for the entire study population and the nonconcussed individuals were not appreciably different (Tables 2 and 3). For the analyses done on the concussed individuals in our study cohort, the results were statistically significant only for one back speed (Table 3). This cognitive task was of only marginal reliability in the entire study group and the concussed and nonconcussed subgroups. Sustaining a concussion in one’s sport during a 1-year interval between baseline testing may negatively affect test reliability. Our study was not powered to examine this possibility, and our concussed cohort contained only a small number of such individuals (N = 9). There is debate over this issue, with literature showing that concussion history may affect the measured test–retest reliability of computerized neurocognitive tests25 and literature finding that concussion history is not predictive of performance.26 Further study is indicated.
Our findings correspond to previous results reported in the literature. Register-Mihalik et al17 reported ICC values of 0.12 to 0.72 in their young study populations using a different computerized neurocognitive test. Cole et al,15 studying an older military population and looking at 4 different computerized neurocognitive tests including CogState Sport, found ICC values ranging from 0.22 to 0.83. These authors concluded that their results were “. consistent with reliabilities reported in the literature and are lower than desired for clinical decision making.” Reliability testing studies of computerized neurocognitive tests, ours included, have typically found ICC values which fail to meet criteria preferred for clinical decision making. Our results should be considered in the light of several limitations. There was a preponderance of male athletes in our study population. There is some evidence that there are gender differences both in response to concussion27 and in performance on computerized neurocognitive tests,28 and so our relative lack of female athletes may have biased our results. There was no attempt at each testing point to obtain a comprehensive medical history, including any interval change in baseline health status. Computerized neurocognitive test results can be sensitive to underlying conditions, such as learning disorders or attention deficit hyperactivity disorder.29 There was, likewise, no attempt to assess baseline variables at the time of testing that could impact performance, such as sleep. Sleep quantity and quality can have an effect on baseline concussion assessment.30,31 We did not control for this, and this may have affected our results. The ideal timing, frequency, and type of neurocognitive test have not been determined.9 Our findings suggest that the typical time frame to consider baseline retesting may need to be rethought. In our study, test–retest reliability was conducted over the course of 1 year, the time frame for baseline retesting in most school districts with which the authors are familiar. Our results suggest that test results may not be
TABLE 2. Intraclass Correlation Coefficient and Associated 95% Confidence Interval, and Pearson/Spearman Correlation Coefficient, for Each Axon Sports Task Axon Sports CCAT Task (N = 117)
ICC*
ICC 95% Confidence Interval
r/rs*
Identification speed Detection speed One card learning accuracy One back speed
0.672 0.554 0.401 0.664
0.559-0.760 0.415-0.668 0.237-0.542 0.549-0.754
0.624‡ 0.520‡ 0.504‡ 0.664§
Reliability† Marginal Low Low Marginal
*Correlations are significant, P , 0.001 level. †Qualitative descriptions as defined in Methods.23 ‡Spearman (rs) correlation coefficient. §Pearson (r) correlation coefficient.
370
| www.cjsportmed.com
Copyright © 2014 Wolters Kluwer Health, Inc. All rights reserved.
Copyright © 2015 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
Clin J Sport Med Volume 25, Number 4, July 2015
Reliability of a Neurocognitive Test in Athletes
TABLE 3. Reliability as Measured by ICC for Each Axon Sports Task, for Concussed and Nonconcussed Individuals Axon Sports CCAT Task
ICC for Nonconcussed Cohort (N = 108)
P
Identification speed Detection speed One care learning accuracy One back speed
0.690† 0.566† 0.405† 0.672†
,0.0001 ,0.0001 ,0.0001 ,0.0001
Reliability* Marginal Low Low Marginal
ICC for Concussed Cohort (N = 9)
P
0.452 0.382 0.295 0.622†
,0.095 ,0.138 ,0.204 ,0.027
Reliability* Low Low Low Marginal
*Qualitative descriptions as defined in Methods.23 †Correlations are significant.
sufficiently stable over the course of a year for use in clinical decision making. Future avenues of research should also identify what tasks in specific batteries of computerized neurocognitive tests have the highest reliability; the results for those tasks should then be used to guide clinical decision making. It may also be that baseline testing, properly done, will require rigorous control of testing environment and proper assessments of medical history and issues such as previous night’s sleep. There is a lack of rigorous psychometric assessment of the computerized neurocognitive testing done on young athletes. Our study begins to redress this situation and we believe is the largest to date, but more focused research should be conducted on the use of such testing in high school athlete populations, given the large number of concussions this population sustains and the widespread use of the tests.
CONCLUSIONS Sport-related concussion is a significant and growing problem in high school sports. Baseline computerized neurocognitive testing is increasingly being used in the diagnosis and management of this injury. Our findings contribute to the literature indicating that further study is needed to improve the reliability of these tests in high school athletes and to identify the ideal time frame for retesting.
ACKNOWLEDGMENTS The authors would like to acknowledge the generous assistance provided by Jason Cromer of Axon Sports in accessing test data and answering the authors’ questions; and Tina Lepley and Leslie McCann of Nationwide Children’s Hospital for their administrative help. REFERENCES 1. McCrory P, Meeuwisse W, Aubry M, et al. Consensus statement on concussion in sport—the 4th International Conference on Concussion in Sport, held in Zurich, November 2012. Clin J Sport Med. 2013;23: 89–117. 2. Bakhos LL, Lockhart GR, Myers R, et al. Emergency department visits for concussion in young child athletes. Pediatrics. 2010;126:e550–e556. 3. Marar M, McIlvain N, Fields S, et al. Epidemiology of concussions among United States high school athletes in 20 sports. Am J Sports Med. 2012;40:747–755. 4. Meehan WP, d’Hemecourt P, Collins C, et al. Assessment and management of sport-related concussions in United States high schools. Am J Sports Med. 2011;39:2304–2310.
Copyright © 2014 Wolters Kluwer Health, Inc. All rights reserved.
5. Van Kampen DA, Lovell MR, Pardini JE, et al. The ‘value added’ of neurocognitive testing after sports-related concussion. Am J Sports Med. 2006;34:1630–1635. 6. Meehan WP, d’Hemecourt P, Collins CL, et al. Computerized neurocognitive testing for the management of sport-related concussions. Pediatrics.2012;129:38–44. 7. Kirkwood MW, Randolph C, Yeates KO. Returning pediatric athletes to play after concussion: the evidence (or lack thereof) behind baseline neuropsychological testing. Acta Pediatr. 2009;98:1409–1411. 8. Randolph C, Kirkwood MW. What are the real risks of sport-related concussion, and are they modifiable? J Int Neuropsychol Soc. 2009;15: 512–520. 9. Harmon KG, Drezner J, Gammons M. American Medical Society for Sports Medicine position statement: concussion in sport. Clin J Sport Med. 2013;23:1–18. 10. Centers for Disease Control and Prevention. FAQs about baseline testing among young athletes. http://www.cdc.gov/concussion/sports/baseline_ test.html. Accessed October 8, 2013. 11. Randolph C, McCrea M, Barr WB. Is neuropsychological testing useful in the management of sport-related concussion? J Athl Train. 2005;40: 139–152. 12. Segalowitz SJ, Mahaney P, Santesso DL, et al. Retest reliability in adolescents of a computerized neuropsychological battery used to assess recovery from concussion. NeuroRehabilitation. 2007;22:243–251. 13. Lichetenstein JD, Scolaro MR, Schatz P. Smaller groups and more supervision may be necessary for baseline testing in younger athletes. Am J Sports Med. 2014;42:479–484. 14. Halstead ME, Walter KD. Sport-related concussion in children and adolescents. Pediatrics. 2010;126:597–615. 15. Cole WR, Arrieux JP, Schwab K, et al. Test-retest reliability of four computerized neurocognitive assessment tools in an active duty military population. Arch Clin Neuropsychol. 2013;28:732–742. 16. Broglio SP, Ferrara MS, Macciocchi SN, et al. Test-retest reliability of computerized concussion assessment programs. J Athl Train. 2007;42: 509–514. 17. Register-Mihalik JK, Kontos DL, Guskiewicz KM, et al. Age related differences and reliability on computerized and paper-and-pencil neurocognitive assessment batteries. J Athl Train. 2012;47:297–305. 18. Axon Sports Company Web site. https://www.axonsports.com/. Accessed October 8, 2013. 19. Darby D. Validity of Axon Sports Computerized Cognitive Assessment Tool (CCAT). http://theconcussionblog.files.wordpress.com/ 2011/02/axon-sports-ccat-cmo-validity-letter.pdf. Accessed October 8, 2013. 20. MacDonald J, Wilson J, Young J, et al. Evaluation of a simple test of reaction time in the diagnosis and management of concussions in high school athletes. Clin J Sport Med. 2012;22:187–188. 21. McDowell I, ed. Measuring Health: A Guide to Rating Scales and Questionnaires. 3rd ed. New York, NY: Oxford University Press; 2006. 22. Collie A, Maruff P, Makdissi M, et al. CogSport: reliability and correlation with conventional cognitive tests used in postconcussion medical evaluations. Clin J Sport Med. 2003;13:28–32. 23. Lezak MD, Howieson DB, Bigler ED, et al. Neuropsychological Assessment. 5th ed. New York, NY: Oxford University Press; 2012. 24. Anastasi A. Psychological Testing. 6th ed. New York, NY: Macmillan; 1998.
www.cjsportmed.com |
371
Copyright © 2015 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
MacDonald and Duerson
Clin J Sport Med Volume 25, Number 4, July 2015
25. Moser RS, Schatz P, Jordan B. Prolonged effects of concussion in high school athletes. Neurosurgery. 2005;57:300–306. 26. Broglio SP, Ferrara MS, Piland SG, et al. Concussion history is not a predictor of computerized neurocognitive performance. Br J Sports Med. 2006;40:802–805. 27. Dick RW. Is there a gender difference in concussion incidence and outcomes? Br J Sports Med. 2009;43:i46–i50. 28. Gur RC, Richard J, Calkins ME, et al. Age group and sex differences in performance on a computerized neurocognitive battery in children age 8–21. Neuropsychology. 2012;26:251–265.
29. Elbin RJ, Kontos AP, Kegel N, et al. Individual and combined effects of LD and ADHD on computerized neurocognitive concussion test performance: evidence for separate norms. Arch Clin Neuropsychol. 2013;28: 476–484. 30. Mihalik JP, Lengas E, Register-Mihalik JK, et al. The effects of sleep quality and sleep quantity on concussion baseline assessment. Clin J Sport Med. 2013;23:343–348. 31. McClure DJ, Zuckerman SL, Kutscher SJ, et al. Baseline neurocognitive test results in sports-related concussions: the importance of a prior night’s sleep. Am J Sports Med. 2014;42:472–478.
372
Copyright © 2014 Wolters Kluwer Health, Inc. All rights reserved.
| www.cjsportmed.com
Copyright © 2015 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
Reliability of a Computerized Neurocognitive Test in Baseline Concussion Testing of High School Athletes James MacDonald, MD, MPH*† and Drew Duerson, MD*
Objective: Baseline assessments using computerized neurocognitive tests are frequently used in the management of sport-related concussions. Such testing is often done on an annual basis in a community setting. Reliability is a fundamental test characteristic that should be established for such tests. Our study examined the test–retest reliability of a computerized neurocognitive test in high school athletes over 1 year.
Design: Repeated measures design.
factors that might affect test performance, and (3) identifying the ideal time interval to repeat baseline testing in high school athletes.
Clinical Relevance: Computerized neurocognitive tests are used frequently in high school athletes, often within a model of baseline testing of asymptomatic individuals before the start of a sporting season. This study adds to the evidence that suggests in this population such testing may lack sufficient reliability to support clinical decision making.
Setting: Two American high schools.
Key Words: AxonSports, computerized cognitive assessment tool, intraclass correlation coefficient, psychometrics
Participants: High school athletes (N = 117) participating in
(Clin J Sport Med 2015;25:367–372)
American football or soccer during the 2011-2012 and 2012-2013 academic years.
Interventions: All study participants completed 2 baseline computerized neurocognitive tests taken 1 year apart at their respective schools. The test measures performance on 4 cognitive tasks: identification speed (Attention), detection speed (Processing Speed), one card learning accuracy (Learning), and one back speed (Working Memory).
Main Outcome Measures: Reliability was assessed by measuring the intraclass correlation coefficient (ICC) between the repeated measures of the 4 cognitive tasks. Pearson and Spearman correlation coefficients were calculated as a secondary outcome measure. Results: The measure for identification speed performed best (ICC = 0.672; 95% confidence interval, 0.559-0.760) and the measure for one card learning accuracy performed worst (ICC = 0.401; 95% confidence interval, 0.237-0.542). All tests had marginal or low reliability. Conclusions: In a population of high school athletes, computerized neurocognitive testing performed in a community setting demonstrated low to marginal test–retest reliability on baseline assessments 1 year apart. Further investigation should focus on (1) improving the reliability of individual tasks tested, (2) controlling for external
Submitted for publication October 19, 2013; accepted June 12, 2014. From the *Department of Pediatrics, Division of Sports Medicine, Nationwide Children’s Hospital, Columbus, Ohio; and †The Ohio State University College of Medicine, Columbus, Ohio. The authors report no conflicts of interest. Dr. MacDonald is supported in his research through an American Medical Society for Sports Medicine “Young Investigator” Grant. Corresponding Author: James MacDonald, MD, MPH, Department of Pediatrics, Division of Sports Medicine, Nationwide Children’s Hospital, 5680 Venture Dr., Dublin, OH 43017 ([email protected]). Copyright © 2014 Wolters Kluwer Health, Inc. All rights reserved.
Clin J Sport Med Volume 25, Number 4, July 2015
INTRODUCTION The fourth international consensus statement on concussion in sport defines concussion as a brain injury characterized by a complex pathophysiological process affecting the brain and caused by a direct blow to the head or an indirect blow elsewhere on the body, transmitting an impulsive force to the head.1 From 2001 to 2005, there were an estimated 502 000 emergency department (ED) visits for concussions; about half were identified as sport-related concussions.2 Sports are a frequent cause of concussion especially in youth: among individuals aged 15 to 24 years, sports are second only to motor vehicle accidents as the leading cause of concussions.3 Concussions now represent nearly 15% of all high school sport-related injuries.4 Computerized neurocognitive testing plays an increasingly important role in the contemporary management of sport-related concussions. These tests have been emphasized in the diagnosis and management of concussions because of the potential unreliability of athletes’ self-report of symptoms.5 The fourth international consensus statement recognizes neuropsychological testing, including computerized neurocognitive testing, as a clinical aid in making return to play decisions, although it also found insufficient evidence to recommend widespread testing.1 Because of their availability and convenience, computerized paradigms interpreted by well-trained personnel, as opposed to formal pen-and-paper neuropsychological tests, have become the predominant method for assessing cognitive function in athletes at risk for concussion.6 A study of more than 1000 concussions in high school athletes published in 2011 found that computerized neurocognitive tests were used in the assessment of 41.2% of the cases.4 Neurocognitive tests have often been promoted with the specific purpose of supporting return to play decisions within www.cjsportmed.com |
367
Copyright © 2015 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
MacDonald and Duerson
Clin J Sport Med Volume 25, Number 4, July 2015
a model of “baseline testing”: athletes participating in highrisk sports take such a test preseason, and the results can be used comparatively during follow-up testing for athletes who sustain a concussion during the season. This model, although appealing, has critics.7,8 The ideal interval to repeat baseline testing for computerized neurocognitive tests has not been established.9 The Centers for Disease Control and Prevention recommend that most components of baseline testing be repeated annually, although it also states that neurocognitive testing may need to be repeated as infrequently as every 2 years.10 It is the authors’ experience that most high school sports districts that provide baseline testing repeat these tests for athletes on an annual, preseason basis; and that such testing is most often conducted in a large group setting in the schools themselves or a similar venue, with little control placed on environmental stimuli. There are other potential problems with computerized neurocognitive tests. Randolph et al11 argue that the most critical psychometric property of a computerized neurocognitive test is its test–retest reliability. In tests with low or marginal reliability, any difference between the baseline score and the score on retesting may reflect the error of retesting rather than any true change in the athlete.12 In their review of several neurocognitive tests, they found that most failed to meet adequate reliability criteria for clinical decision making. Kirkwood et al7 note that what reliability studies do exist use short periods where tests may be expected to be more stable; they report that few studies span periods such as the weeks or months that may pass between baseline and postinjury testing. Moreover, testing environment seems to negatively affect performance especially in younger athletes.13 Other critiques of the tests have focused on the published studies that have largely been conducted by the test developers themselves, giving rise to concerns for bias.14 Although reliability testing has been done for adult athletes using computerized neurocognitive tests,15,16 there is a relative lack of similar testing for high school athletes. Register-Mihalik et al17 looked at a population of 20 college students (20 6 0.79 years) and 20 high school students (16 6 0.86 years) and measured test–retest reliability using Immediate Post-Concussion Assessment and Cognitive Testing over 3 visits separated by 24 to 72 hours. Test–retest reliabilities reported largely failed to meet adequate reliability criteria for clinical decision making. Segalowitz et al12 looked at a small population of 15 girls and 14 boys (mean age 15.4 years) and measured test–retest reliability over 7 days using Automated Neuropsychological Metrics testing. They reported mixed results, with an aggregate measure showing “robust reliability,” but individual subtests showing weaker correlations. In our review of the literature, we found no other studies looking at the test–retest reliability of a computerized neurocognitive test in a strictly high school population. We specifically found no study of test–retest reliability conducted over the course of a year. Given the above concerns, there is a further need to independently evaluate the psychometric properties of computerized neurocognitive tests, especially as they are typically administered in an adolescent population. There is also a need to determine whether the common
Baseline assessments for these athletes used the Axon Sports Computerized Cognitive Assessment Tool (CCAT), a computerized neurocognitive test (AxonSports, Wausau, Wisconsin).18 Axon Sports is an online version of CogState Sport, a computerized neurocognitive test that has been used since 2002.19 Testing was performed in groups in a large public space at the school. For each school, the location of testing remained the same for each study year, and Axon Sports was accessed with the same desktop computers each study year. Data were collected prospectively as part of an ongoing study for validity testing of a novel reaction time device published elsewhere in the literature.20 The Axon Sports CCAT reports performance in 4 cognitive tasks, each of which uses a simple paradigm in which a playing card initially appears face down and then suddenly turns face-up, awaiting a response from the testtaker.19 The 4 tasks are as follows: Attention (identification speed), using a choice reaction time paradigm; Processing Speed (detection speed), using a simple reaction time paradigm; Learning (one card learning accuracy), using visual recognition memory; and Working Memory (one back speed), using a one back paradigm.19
368
Copyright © 2014 Wolters Kluwer Health, Inc. All rights reserved.
| www.cjsportmed.com
practice of administering such tests on an annual, preseason basis is an ideal time frame. We set out to assess the test–retest reliability of 4 cognitive tasks measured by a computerized neurocognitive test in a population of high school athletes who were baseline tested at their respective schools on 2 occasions, 1 year apart. Our null hypothesis was that the measured test reliability would be adequate to support clinical decision making.
METHODS Study Subjects Students participating in American football, boys’ soccer and girls’ soccer at 2 high schools in central Ohio whose athletes are cared for by the authors’ institution were included. Baseline computerized neurocognitive testing was conducted in the summer of 2011 and again in 2012. The precise timing of testing was such that all study participants took their 2012 test between 50 and 52 weeks after their 2011 test. Any athlete who finished a testing session with an invalid baseline test despite repeated attempts was excluded. All athletes who completed a valid baseline test in both 2011 and in 2012 (N = 117) were included for final statistical analysis. As a secondary outcome of interest, athletes who sustained a concussion in their 2011 sport and who were baseline tested again in 2012 (N = 9) were identified to perform subgroup analyses (Figure 1). All concussions were diagnosed using consensus guidlelines1 by the authors themselves or other physicians specializing in pediatric sports medicine working in the authors’ same institution. At the time of 2012 testing, these athletes had all been medically cleared to play.
Assessments
Copyright © 2015 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
Clin J Sport Med Volume 25, Number 4, July 2015
Reliability of a Neurocognitive Test in Athletes
FIGURE 1. Inclusion and exclusion of eligible participants.
Axon Sports baseline reports used by clinicians typically label the tasks “Attention,” “Processing Speed,” “Learning,” and “Working Memory,” but in the research literature, the terms “identification speed,” “detection speed,” “one card learning accuracy,” and “one back speed” have been used, respectively, when studying Axon Sports. We routinely use the latter convention in this report.
Statistical Analysis Test-retest reliability is measured as an association between the scores on any test taken by the same participant on 2 different occasions over some time interval. Historically, correlations such as a Pearson coefficient (r) or a Spearman coefficient (rs) (the latter if data were not normally distributed) were calculated to measure this association. In contemporary psychometrics, such measures are thought prone to seriously exaggerate the impression of reliability.21 The intraclass correlation coefficient (ICC) is now thought to be better suited to the assessment of reliability and has been used in other studies similar to this one, although Pearson and Spearman coefficients are often reported concurrently.15,16,22 We calculated a 2-way random effects analysis of variance ICC to estimate the test–retest reliability of each of the 4 tasks measured by the Axon Sports CCAT. We assessed these data for normality using the Kolmogorov–Smirnov test. Correlations for any normally distributed data were also measured using the Pearson product moment correlation coefficient (r); for data that were not normally distributed, Spearman rho (rs) was used. We repeated ICC calculations for both the subgroup who had been concussed in 2011 and those who had remained concussion free in their sport (N = 108). Copyright © 2014 Wolters Kluwer Health, Inc. All rights reserved.
We used previously published criteria for qualitatively rating the levels of test reliability: test–retest reliability coefficients greater than 0.9 are considered very high; 0.80 to 0.89 are considered high; 0.70 to 0.79 are considered adequate; 0.60 to 0.69 are marginal; and 0.59 or below are low.23 The ICC values which are “very high” (greater than 0.9) are preferred for clinical decision making.11 Anastasi24 recommends 0.60 as the minimal acceptable ICC value for clinical use. IBM SPSS statistics version 21.0 (SPSS, Inc, Chicago, Illinois) was used for all analyses. Statistical significance was set at a = 0.05.
Ethical Considerations Informed consent was obtained in all cases. The study was approved by the Institutional Review Board of Nationwide Children’s Hospital.
RESULTS Demographic data for the 117 study participants are reported in Table 1. There was a preponderance of male athletes (72.6%). The mean age was 16.1 6 1.0 years (Table 1). The ICC and Pearson or Spearman correlation coefficients of 2 test variables demonstrated marginal reliability: identification speed (ICC = 0.672; rs = 0.624) and one back speed (ICC = 0.664; r = 0.664). The remaining 2 test variables demonstrated low reliability: detection speed (ICC = 0.554; rs = 0.520) and one card learning accuracy (ICC = 0.401; rs = 0.504). All correlation coefficients were found to be statistically significant at P , 0.001 (Table 2). Results for the subgroup analyses performed on the concussed and nonconcussed cohorts of the study group are reported in Table 3. For the concussed athletes, the only test www.cjsportmed.com |
369
Copyright © 2015 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
Clin J Sport Med Volume 25, Number 4, July 2015
MacDonald and Duerson
TABLE 1. Characteristics of Study Participants Participants
Gender, Male/Female, n (%)
Handedness, Right/Left, n (%)
Age, Mean 6 SD (Range), y*
Football Players, n (%)
Boys’ Soccer, n (%)
Girls’ Soccer, n (%)
117
85/32 (72.6/27.4)
107/10 (91.5/8.5)
16.1 6 1.0 (14-18)
37 (31.6)
48 (41.0)
32 (27.4)
*Age calculated as of 2012 testing date.
variable that was statistically significant was the one back speed, which demonstrated marginal reliability (ICC = 0.622; P , 0.027). All other findings were not statistically significant.
DISCUSSION We identified a need to independently determine the reliability of a computerized neurocognitive test when used in baseline assessments of high school athletes at risk for concussion, conducted 1 year apart. Our results demonstrate low to marginal reliability measures for the 4 cognitive tasks measured by the Axon Sports CCAT, with ICC values ranging from 0.401 to 0.672. None of the measures were found to have “very high” reliability, an ICC value of 0.9 or above, the characteristic preferred for clinical decision making.11 The ICC values for 2 cognitive tasks (detection speed and one card learning accuracy) were below a recommended minimally acceptable value for clinical use.24 The reliability measures for the entire study population and the nonconcussed individuals were not appreciably different (Tables 2 and 3). For the analyses done on the concussed individuals in our study cohort, the results were statistically significant only for one back speed (Table 3). This cognitive task was of only marginal reliability in the entire study group and the concussed and nonconcussed subgroups. Sustaining a concussion in one’s sport during a 1-year interval between baseline testing may negatively affect test reliability. Our study was not powered to examine this possibility, and our concussed cohort contained only a small number of such individuals (N = 9). There is debate over this issue, with literature showing that concussion history may affect the measured test–retest reliability of computerized neurocognitive tests25 and literature finding that concussion history is not predictive of performance.26 Further study is indicated.
Our findings correspond to previous results reported in the literature. Register-Mihalik et al17 reported ICC values of 0.12 to 0.72 in their young study populations using a different computerized neurocognitive test. Cole et al,15 studying an older military population and looking at 4 different computerized neurocognitive tests including CogState Sport, found ICC values ranging from 0.22 to 0.83. These authors concluded that their results were “. consistent with reliabilities reported in the literature and are lower than desired for clinical decision making.” Reliability testing studies of computerized neurocognitive tests, ours included, have typically found ICC values which fail to meet criteria preferred for clinical decision making. Our results should be considered in the light of several limitations. There was a preponderance of male athletes in our study population. There is some evidence that there are gender differences both in response to concussion27 and in performance on computerized neurocognitive tests,28 and so our relative lack of female athletes may have biased our results. There was no attempt at each testing point to obtain a comprehensive medical history, including any interval change in baseline health status. Computerized neurocognitive test results can be sensitive to underlying conditions, such as learning disorders or attention deficit hyperactivity disorder.29 There was, likewise, no attempt to assess baseline variables at the time of testing that could impact performance, such as sleep. Sleep quantity and quality can have an effect on baseline concussion assessment.30,31 We did not control for this, and this may have affected our results. The ideal timing, frequency, and type of neurocognitive test have not been determined.9 Our findings suggest that the typical time frame to consider baseline retesting may need to be rethought. In our study, test–retest reliability was conducted over the course of 1 year, the time frame for baseline retesting in most school districts with which the authors are familiar. Our results suggest that test results may not be
TABLE 2. Intraclass Correlation Coefficient and Associated 95% Confidence Interval, and Pearson/Spearman Correlation Coefficient, for Each Axon Sports Task Axon Sports CCAT Task (N = 117)
ICC*
ICC 95% Confidence Interval
r/rs*
Identification speed Detection speed One card learning accuracy One back speed
0.672 0.554 0.401 0.664
0.559-0.760 0.415-0.668 0.237-0.542 0.549-0.754
0.624‡ 0.520‡ 0.504‡ 0.664§
Reliability† Marginal Low Low Marginal
*Correlations are significant, P , 0.001 level. †Qualitative descriptions as defined in Methods.23 ‡Spearman (rs) correlation coefficient. §Pearson (r) correlation coefficient.
370
| www.cjsportmed.com
Copyright © 2014 Wolters Kluwer Health, Inc. All rights reserved.
Copyright © 2015 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
Clin J Sport Med Volume 25, Number 4, July 2015
Reliability of a Neurocognitive Test in Athletes
TABLE 3. Reliability as Measured by ICC for Each Axon Sports Task, for Concussed and Nonconcussed Individuals Axon Sports CCAT Task
ICC for Nonconcussed Cohort (N = 108)
P
Identification speed Detection speed One care learning accuracy One back speed
0.690† 0.566† 0.405† 0.672†
,0.0001 ,0.0001 ,0.0001 ,0.0001
Reliability* Marginal Low Low Marginal
ICC for Concussed Cohort (N = 9)
P
0.452 0.382 0.295 0.622†
,0.095 ,0.138 ,0.204 ,0.027
Reliability* Low Low Low Marginal
*Qualitative descriptions as defined in Methods.23 †Correlations are significant.
sufficiently stable over the course of a year for use in clinical decision making. Future avenues of research should also identify what tasks in specific batteries of computerized neurocognitive tests have the highest reliability; the results for those tasks should then be used to guide clinical decision making. It may also be that baseline testing, properly done, will require rigorous control of testing environment and proper assessments of medical history and issues such as previous night’s sleep. There is a lack of rigorous psychometric assessment of the computerized neurocognitive testing done on young athletes. Our study begins to redress this situation and we believe is the largest to date, but more focused research should be conducted on the use of such testing in high school athlete populations, given the large number of concussions this population sustains and the widespread use of the tests.
CONCLUSIONS Sport-related concussion is a significant and growing problem in high school sports. Baseline computerized neurocognitive testing is increasingly being used in the diagnosis and management of this injury. Our findings contribute to the literature indicating that further study is needed to improve the reliability of these tests in high school athletes and to identify the ideal time frame for retesting.
ACKNOWLEDGMENTS The authors would like to acknowledge the generous assistance provided by Jason Cromer of Axon Sports in accessing test data and answering the authors’ questions; and Tina Lepley and Leslie McCann of Nationwide Children’s Hospital for their administrative help. REFERENCES 1. McCrory P, Meeuwisse W, Aubry M, et al. Consensus statement on concussion in sport—the 4th International Conference on Concussion in Sport, held in Zurich, November 2012. Clin J Sport Med. 2013;23: 89–117. 2. Bakhos LL, Lockhart GR, Myers R, et al. Emergency department visits for concussion in young child athletes. Pediatrics. 2010;126:e550–e556. 3. Marar M, McIlvain N, Fields S, et al. Epidemiology of concussions among United States high school athletes in 20 sports. Am J Sports Med. 2012;40:747–755. 4. Meehan WP, d’Hemecourt P, Collins C, et al. Assessment and management of sport-related concussions in United States high schools. Am J Sports Med. 2011;39:2304–2310.
Copyright © 2014 Wolters Kluwer Health, Inc. All rights reserved.
5. Van Kampen DA, Lovell MR, Pardini JE, et al. The ‘value added’ of neurocognitive testing after sports-related concussion. Am J Sports Med. 2006;34:1630–1635. 6. Meehan WP, d’Hemecourt P, Collins CL, et al. Computerized neurocognitive testing for the management of sport-related concussions. Pediatrics.2012;129:38–44. 7. Kirkwood MW, Randolph C, Yeates KO. Returning pediatric athletes to play after concussion: the evidence (or lack thereof) behind baseline neuropsychological testing. Acta Pediatr. 2009;98:1409–1411. 8. Randolph C, Kirkwood MW. What are the real risks of sport-related concussion, and are they modifiable? J Int Neuropsychol Soc. 2009;15: 512–520. 9. Harmon KG, Drezner J, Gammons M. American Medical Society for Sports Medicine position statement: concussion in sport. Clin J Sport Med. 2013;23:1–18. 10. Centers for Disease Control and Prevention. FAQs about baseline testing among young athletes. http://www.cdc.gov/concussion/sports/baseline_ test.html. Accessed October 8, 2013. 11. Randolph C, McCrea M, Barr WB. Is neuropsychological testing useful in the management of sport-related concussion? J Athl Train. 2005;40: 139–152. 12. Segalowitz SJ, Mahaney P, Santesso DL, et al. Retest reliability in adolescents of a computerized neuropsychological battery used to assess recovery from concussion. NeuroRehabilitation. 2007;22:243–251. 13. Lichetenstein JD, Scolaro MR, Schatz P. Smaller groups and more supervision may be necessary for baseline testing in younger athletes. Am J Sports Med. 2014;42:479–484. 14. Halstead ME, Walter KD. Sport-related concussion in children and adolescents. Pediatrics. 2010;126:597–615. 15. Cole WR, Arrieux JP, Schwab K, et al. Test-retest reliability of four computerized neurocognitive assessment tools in an active duty military population. Arch Clin Neuropsychol. 2013;28:732–742. 16. Broglio SP, Ferrara MS, Macciocchi SN, et al. Test-retest reliability of computerized concussion assessment programs. J Athl Train. 2007;42: 509–514. 17. Register-Mihalik JK, Kontos DL, Guskiewicz KM, et al. Age related differences and reliability on computerized and paper-and-pencil neurocognitive assessment batteries. J Athl Train. 2012;47:297–305. 18. Axon Sports Company Web site. https://www.axonsports.com/. Accessed October 8, 2013. 19. Darby D. Validity of Axon Sports Computerized Cognitive Assessment Tool (CCAT). http://theconcussionblog.files.wordpress.com/ 2011/02/axon-sports-ccat-cmo-validity-letter.pdf. Accessed October 8, 2013. 20. MacDonald J, Wilson J, Young J, et al. Evaluation of a simple test of reaction time in the diagnosis and management of concussions in high school athletes. Clin J Sport Med. 2012;22:187–188. 21. McDowell I, ed. Measuring Health: A Guide to Rating Scales and Questionnaires. 3rd ed. New York, NY: Oxford University Press; 2006. 22. Collie A, Maruff P, Makdissi M, et al. CogSport: reliability and correlation with conventional cognitive tests used in postconcussion medical evaluations. Clin J Sport Med. 2003;13:28–32. 23. Lezak MD, Howieson DB, Bigler ED, et al. Neuropsychological Assessment. 5th ed. New York, NY: Oxford University Press; 2012. 24. Anastasi A. Psychological Testing. 6th ed. New York, NY: Macmillan; 1998.
www.cjsportmed.com |
371
Copyright © 2015 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
MacDonald and Duerson
Clin J Sport Med Volume 25, Number 4, July 2015
25. Moser RS, Schatz P, Jordan B. Prolonged effects of concussion in high school athletes. Neurosurgery. 2005;57:300–306. 26. Broglio SP, Ferrara MS, Piland SG, et al. Concussion history is not a predictor of computerized neurocognitive performance. Br J Sports Med. 2006;40:802–805. 27. Dick RW. Is there a gender difference in concussion incidence and outcomes? Br J Sports Med. 2009;43:i46–i50. 28. Gur RC, Richard J, Calkins ME, et al. Age group and sex differences in performance on a computerized neurocognitive battery in children age 8–21. Neuropsychology. 2012;26:251–265.
29. Elbin RJ, Kontos AP, Kegel N, et al. Individual and combined effects of LD and ADHD on computerized neurocognitive concussion test performance: evidence for separate norms. Arch Clin Neuropsychol. 2013;28: 476–484. 30. Mihalik JP, Lengas E, Register-Mihalik JK, et al. The effects of sleep quality and sleep quantity on concussion baseline assessment. Clin J Sport Med. 2013;23:343–348. 31. McClure DJ, Zuckerman SL, Kutscher SJ, et al. Baseline neurocognitive test results in sports-related concussions: the importance of a prior night’s sleep. Am J Sports Med. 2014;42:472–478.
372
Copyright © 2014 Wolters Kluwer Health, Inc. All rights reserved.
| www.cjsportmed.com
Copyright © 2015 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
