http://informahealthcare.com/bij ISSN: 0269-9052 (print), 1362-301X (electronic) Brain Inj, 2014; 28(13–14): 1726–1733 ! 2014 Informa UK Ltd. DOI: 10.3109/02699052.2014.947618
ORIGINAL ARTICLE
Practice effects and test–re-test reliability of the Five Digit Test in patients with stroke over four serial assessments En-Chi Chiu1, Chia-Lin Koh1, Chia-Yin Tsai2, Wen-Shian Lu3,4, Ching-Fan Sheu5, I-Ping Hsueh1,6, & Ching-Lin Hsieh1,6 1
School of Occupational Therapy, College of Medicine, National Taiwan University, Taipei, Taiwan, 2Department of Physical Medicine and Rehabilitation, E-Da Hospital, Kaohsiung, Taiwan, 3School of Occupational Therapy, Chung Shan Medical University, Taichung, Taiwan, 4 Occupational Therapy Room, Chung Shan Medical University Hospital, Taichung, Taiwan, 5Institute of Education, National Cheng Kung University, Tainan, Taiwan, and 6Department of Physical Medicine and Rehabilitation, National Taiwan University Hospital, Taipei, Taiwan Abstract
Keywords
Objectives: To investigate practice effect and test–re-test reliability of the Five Digit Test (FDT) over four serial assessments in patients with stroke. Design: Single-group repeated measures design. Methods: Twenty-five patients with stroke were administered the FDT in four consecutive assessments every 2 weeks. The FDT contains four parts with five indices: ‘basic measures of attention and processing speed’, ‘selective attention’, ‘alternating attention’, ‘ability of inhibition’ and ‘ability of switching’. Results: The five indices of the FDT showed trivial-to-small practice effects (Cohen’s d ¼ 0.03– 0.47) and moderate-to-excellent test–re-test reliability (intra-class correlation coefficient ¼ 0.59– 0.97). Practice effects of the five indices all appeared cumulative, but one index, ‘basic measures of attention and processing speed’, reached a plateau after the second assessment. The minimum and maximum values of the 90% confidence interval (CI) of reliable change index modified for practice (RCIp) for this index were [17.6, 11.2]. Conclusions: One of five indices of the FDT reached a plateau, whose minimum and maximum values of the 90% CI RCIp are useful to determine whether the change in an individual’s score is real. However, clinicians and researchers should be cautious when interpreting the test results of these four indices over repeated assessments.
Attention, executive functions, practice effect, reliable change index modified for practice, stroke
Introduction Cognitive impairments, such as attention deficits and executive dysfunctions, occur frequently in patients with stroke [1]. Cognitive impairment may both impede functional recovery and diminish the potential benefits of rehabilitation [2]. Attention deficits in patients with stroke are associated with poor balance, which may increase the risk of fall [3]. Executive dysfunctions could contribute to the inability of performing activities of daily living (ADL) and influence a patient’s return-to-work and post-stroke productivity [4, 5]. Therefore, evaluating attention and executive functions in patients with stroke is critical for both clinicians and researchers to develop treatment plans and conduct outcome studies. The Five Digit Test (FDT) measures attention and executive functions [6, 7]. The FDT has been designed as an alternative tool for the Stroop Color Word Test (SCWT). Three advantages of the FDT compared to the SCWT are: (1)
Correspondence: I-Ping Hsueh, 4F, No. 17, Xu-Zhou Road, Taipei 100, Taiwan. Tel: +886-2-33668174. Fax: +886-2-23511331. E-mail: iping@ ntu.edu.tw
History Received 19 September 2012 Revised 20 April 2014 Accepted 20 July 2014 Published online 4 September 2014
it is appropriate for speakers of any language who are familiar with Arabic numerals; (2) it is appropriate for both competent readers and illiterate subjects (provided they know at least the Arabic numerals 1–5); and (3) the cognitive requirements for performing the tasks are low, which involves only naming and counting of 1–5 digits or stars [8]. Therefore, the FDT could be considered for application in patients with stroke. The FDT is composed of four parts, each of which contains 50 items. Each item is a group of 1–5 digits or stars printed inside a frame. The digits can either be named as the number printed in the frame (two 5 s are read as ‘five’) or counted by the amount of digits in the frame (two 5 s are counted as ‘two’). In part 1, the subject names the digits. In part 2, the subject counts the stars. In part 3, the subject counts the amount of digits and avoids naming the number. In part 4, the subject needs to switch between counting and naming the digits, depending on the thickness of the frame (Appendix). The items of all four parts require verbal responses. Investigating the practice effect and test–re-test reliability of a cognitive measure is important for repeated assessments. Practice effect is defined as improvements in performance over repeated testing due to the influence of the previous experience in performing the same test [9, 10]. Practice effect
Practice effects and reliability of the FDT
DOI: 10.3109/02699052.2014.947618
1727
contains two primary phases over multiple assessments. Accumulation of the practice effect happens in the early stage of assessments (cumulative phase) and then the practice effect becomes stable to achieve a plateau phase [11, 12]. The practice effect of a cognitive measure is greatly influenced by the number of assessments [13] and is most pronounced in the first three or four assessments [14]. Although no studies have examined the practice effect of the SWCT in patients with stroke, several cognitive measures have demonstrated practice effects in patients with stroke. These measures include the Symbol Digit Modalities Test, the Conners’ Continuous Performance Test II and the Digit Vigilance Test [15, 16]. The reliable change index (RCI) calculates a change in score beyond measurement error, which reflects that the change in an individual’s score is real [17]. A reliable change index modified for practice (RCIp) is the RCI corrected, which takes practice effect into account. Test–re-test reliability refers to the level of consistency on the assumption that the behaviour being scored does not change over repeated assessments [18]. These above-mentioned psychometric properties are essential for interpreting re-test results in both clinical and research settings. The FDT has shown satisfactory convergent validity with the SCWT in elderly people [19]. However, to the authors’ knowledge, no study has examined psychometric properties of the FDT in patients with stroke and other populations. Therefore, this study aimed to (1) examine the practice effect of the FDT over four serial assessments, explore the phases of practice effects and calculate RCIp; and (2) examine test–retest reliability of the FDT in patients with stroke.
Institutional Review Board of each hospital. The following criteria were used to recruit the participants: (1) diagnosis of cerebral haemorrhage or cerebral infarction; (2) aged over 18 years; (3) onset of stroke more than 6 months prior to the first evaluation; (4) ability to follow instructions, as assessed by a score 4 1 on two items (i.e. reading ability [READ] and comprehension and memory [CMD]) from the Cognitive Abilities Screening Instrument, Chinese version (CASI C-2.0) [20]; (5) lack of visual or auditory deficits; (6) lack of visual spatial neglect, as assessed by a score 4 51 on the Star Cancellation Test [21]; (7) ability to read Arabic numerals; and (8) ability to give informed consent in person. The criteria for exclusion included: (1) inability to give verbal responses because of severe aphasia; (2) experiencing recurrent stroke during the study; (3) presence of concurrent diseases that could influence cognitive ability, such as dementia, Parkinsonism, major depression or schizophrenia; and (4) having interventions for attention or executive functions.
Methods
Five Digit Test (FDT)
Participants
The FDT contains five indices representing ability of attention and executive functions. Three attention indices include ‘basic measures of attention and processing speed’, ‘selective attention’ and ‘alternating attention’. Two indices representing ability of executive functions include ‘ability of inhibition’ and ‘ability of switching’ [6, 7]. The formulae for scoring these five indices were calculated as follows:
Forty-two outpatients with stroke were recruited from the Department of Physical Medicine and Rehabilitation in five hospitals in northern Taiwan between August 2009 and July 2011. Figure 1 shows the total number of initial participants and the number of participants whose data were not analysed for their respective reasons. This study was approved by the
Procedures At baseline (Time 1), participants were screened by four measures, the Barthel Index (BI), the READ, the CMD and the Star Cancellation Test, and then participants were administered by the FDT. The FDT was administered again after 2 weeks (Time 2), 4 weeks (Time 3) and 6 weeks (Time 4). The participants were assessed in a quiet room to avoid any distractions that could affect their performance. Measures
Basic measures of attention and processing speed ¼ RT of part 1 þ RT of part 2 Selective attention ¼ RT of part 3
42 participants were recruited
Reasons why data from 17 participants were not analyzed 1. Time interval between any of test and retest over the four assessments was more than 15 days (n=9) 2. Participants could not complete all four assessments (n=8)
Alternating attention ¼ RT of part 4 Ability of inhibition ¼ RT of part 3 ðRT of part 1 þ RT of part 2Þ=2 Ability of switching ¼ RT of part 4 ðRT of part 1 þ RT of part 2Þ=2 where RT is the reaction time (in seconds). A lower score indicates better performance. Barthel Index (BI)
Data from 25 participants were analyzed
Figure 1. Flow chart depicting the number of participants whose data were not analysed.
The BI assesses 10 basic ADL items, including feeding, transfer, grooming, toileting, bathing, ambulation, stair climbing, dressing, bowel control and bladder control. The total score of the BI ranges from 0–20. The BI has been demonstrated to have good reliability and validity in patients
1728
E.-C. Chiu et al.
with stroke [22]. In this study, the BI was used to determine participants’ disability levels [23]. READ and CMD The READ and the CMD are two items of the CASI C-2.0. The CASI C-2.0 is a 20-item screening tool for measuring cognition. The READ measures reading ability by showing the participants a card with words ‘close eyes’ and testing whether the participants can perform this action. The CMD measures comprehension, memory and execution of the instructions by giving the participants three phrases of verbal instructions. The score of the READ and CMD ranges from 0–3 [24]. The CASI C-2.0 has good internal consistency, test–re-test reliability, sensitivity and specificity in the elderly population [25]. Star Cancellation Test The Star Cancellation Test is the most sensitive sub-test of the Behavioural Inattention Test [26]. It consists of 56 small stars, 52 large stars, 13 letters and 10 words arranged randomly on the test sheet. The subject is instructed to cross out all of the small stars on the test sheet. The two small stars in the middle are used for demonstration. The total score range is 0–54. A score of 51 indicates unilateral spatial neglect [27]. The Star Cancellation Test has excellent test–re-test reliability and good convergent validity in patients with stroke [28, 29]. Data analysis Practice effect The presence of the practice effect was examined through three methods. First, two characteristics of the practice effect (i.e. cumulative phase and plateau phase) were examined over the four assessments. The mean value of each assessment was used to evaluate practice effect in the cumulative phase. Cumulative phase was observed when the mean values (reaction time) of post-tests were lower than those of pretests. Cohen’s d and one-way repeated measures analysis of variance (ANOVA) were used to determine a plateau phase. For reaching a plateau phase, the following two criteria had to be met: (1) two stable consecutive Cohen’s ds, wherein two consecutive values of Cohen’s d were lower or equal to the previous values (i.e. d of Time 2–3 d of Time 1–2, d of Time 3–4 d of Time 2–3); and (2) a non-significant ANOVA ( ¼ 0.01) in three consecutive time points (e.g. Time 2 to Time 4). If an ANOVA revealed non-significant difference from Time 2 to Time 4, then a plateau phase was reached at Time 2. Second, the repeated measures ANOVA ( ¼ 0.01) was examined over the four assessments. Post hoc analysis ( ¼ 0.01) was conducted to compare differences between pairs of testing times, whether or not the ANOVA results showed significant differences. Third, effect size (Cohen’s d) was calculated to estimate the magnitudes of the practice effects in six pairs of time points (i.e. Time 1–2, Time 2–3, Time 3–4, Time 1–3, Time 1–4 and Time 2–4). The sample size was small in this study, which might have resulted in nonsignificant difference for the ANOVA results. Therefore, examining the characteristics of practice effect and effect size
Brain Inj, 2014; 28(13–14): 1726–1733
were two primary methods to determine the presence of practice effect. Two criteria of determining practice effect were: (1) the cumulative phase was observed and d was 0.01–0.19, indicating a trivial practice effect; and (2) the cumulative phase was observed and d was 0.20–0.49, indicating a small practice effect. Reliable change index modified for practice (RCIp) The minimum and maximum RCIp with 90% confidence interval (CI) were chosen across three adjacent assessments to provide the conservative criteria for interpreting score changes in a patient with stroke. The 90% CI RCIp was calculated as shown below [30]. A negative 90% CI RCIp value indicates that the RT of the post-test was shorter than that of the pre-test in the five indices. 90% CI RCIp ¼ mean practice effect 1:645 SEdiff ð1Þ
SEdiff ¼
qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 2ðSEMÞ2
ð2Þ
pffiffiffiffiffiffiffiffiffiffiffiffiffiffi 1 rxx
ð3Þ
SEM ¼ SD1
where mean practice effect is the average of the difference between the two test scores (1); SEdiff is the standard error of the differences (2); SEM is the standard error of measurement; SD1 is the standard deviation of the first score of adjacent tests; and rxx is the intra-class correlation coefficient value (3). Test–re-test reliability Test–re-test reliability was estimated using the intra-class correlation coefficient (ICC2,1) across four assessments, on the basis of a two-way random-effect model with an absolute agreement type. The following criteria were used for ICC values: 50.39, poor reliability; 0.40–0.59, moderate reliability; 0.60–0.79, good reliability; and 0.80–1.00, excellent reliability [31].
Results The mean age of the 25 participants was 57.9 years and75% were male. The time since onset of stroke was over 9 months for the participants (i.e. participants were in a stable state of neurological recovery) [32, 33]. All participants obtained scores of 3 on the READ and CMD, respectively. The demographic data and other clinical characteristics of the participants are provided in Table I The mean scores of the four sessions are shown in Table II Practice effect The mean values (reaction time) of the five indices over the four assessments generally showed a decreasing trend (Table II and Figure 2), demonstrating cumulative phases. Only the ‘basic measures of attention and processing speed’ index fitted the two criteria of reaching a plateau phase at Time 2: (1) d of Time 2–3 5d of Time 1–2 and d of Time 3–4 ¼ d of Time 2–3; and (2) an ANOVA revealed
Practice effects and reliability of the FDT
DOI: 10.3109/02699052.2014.947618
ability of inhibition and ability of switching, respectively (Table III).
non-significant difference from Time 2 to Time 4. The other four indices displayed unstable Cohen’s ds in three adjacent assessments, not fitting the first criterion for reaching a plateau phase. For the result of ANOVA, only one index (selective attention) was statistically significant (p ¼ 0.001) (Table III). Post-hoc analysis was statistically significant in Time 1–3 in the selective attention index (p ¼ 0.007) and in Time 1–4 in three indices, such as selective attention, alternating attention and ability of switching (p 0.002). The Cohen’s ds of six pairs of time points were 0.04–0.17, 0.08–0.30, 0.09–0.45, 0.03–0.29 and 0.06–0.47 in basic measures of attention and processing speed, selective attention, alternating attention,
Reliable change index modified for practice (RCIp) The parameters for calculating the 90% CI RCIp are shown in Table IV. The minimum and maximum 90% CI RCIp across three adjacent assessments for the five indices were listed as follows: basic measures of attention and processing speed [17.6, 11.2]; selective attention [19.9, 12.8]; alternating attention [58.1, 40.6]; ability of inhibition [19.6, 16.6]; and ability of switching [58.1, 42.1]. Test–re-test reliability Across the four serial assessments, three attention indices showed moderate-to-excellent ICC values (0.75–0.97) (Table II). Two indices representing ability of executive functions showed moderate-to-good ICC values (0.59–0.70).
Table I. Demographic and clinical characteristics of the participants (n ¼ 25). Characteristic Age (years; mean ± SD) Gender, n (%) Male Female Education, n (%) Elementary school Middle school High school Vocational school University Stroke type, n (%) Haemorrhagic Ischaemic Side of hemiparesis, n (%) Left Right Time since onset (month; mean ± SD) Barthel Index (mean ± SD, range) Star Cancellation Test (mean ± SD, range)
Discussion
57.9 ± 11.3
Practice effect
18 (72.0) 7 (28.0) 4 1 8 4 8
1729
The cumulative phases were observed in the five indices, indicating that practice effects result in gradual improvements in score with an increasing number of assessments. All five FDT indices exhibited trivial practice effects in Time 1–2. However, small practice effects were shown in Time 2–3, Time 3–4, Time 1–3, Time 2–4 or Time 1–4, except the ‘basic measures of attention and processing speed’ index. Small practice effects indicate that the patients with stroke may develop test strategies or recall of test items over repeated assessments. Therefore, users should consider practice effects after the third or fourth assessments when interpreting the results of these four indices. The ‘basic measures of attention and processing speed’ index reached a plateau phase at the second assessment, indicating that practice effect was stabilized and the impact of
(16.0) (4.0) (32.0) (16.0) (32.0)
11 (44.0) 14 (56.0) 11 (44.0) 14 (56.0) 33.4 ± 17.0 17.4 ± 3.1, 12–20 53.5 ± 0.8, 52–54
SD, standard deviation.
Table II. Mean and SD at four assessments and test–re-test reliability (n ¼ 25).
Index Basic measures of attention and processing speed Selective attention Alternating attention Ability of inhibition Ability of switching
Time 1 M ± SD
Time 2 M ± SD
Time 3 M ± SD
Time 4 M ± SD
75.0 ± 40.1 56.3 ± 23.3 87.5 ± 35.2 18.8 ± 12.5 50.0 ± 25.8
71.8 ± 35.9 54.3 ± 25.0 84.1 ± 41.9 18.4 ± 12.2 48.2 ± 33.1
70.4 ± 33.8 50.8 ± 19.8 80.5 ± 42.5 15.6 ± 8.7 45.3 ± 34.3
69.0 ± 31.9 49.6 ± 21.3 71.8 ± 34.1 15.1 ± 12.7 37.3 ± 27.7
ICC (95% CI) 0.97 0.91 0.75 0.70 0.59
(0.94, (0.83, (0.60, (0.54, (0.40,
0.98) 0.95) 0.86) 0.84) 0.76)
SD, standard deviation; CI, confidence interval; ICC, intra-class correlation coefficient (estimation of four assessments). Table III. Repeated measures ANOVA and Cohen’s d over four assessments. Cohen’s d (p value) Index Basic measures of attention and processing speed Selective attention Alternating attention Ability of inhibition Ability of switching
F3,75 (p value)
Time 1–2
Time 2–3
Time 3–4
Time 1–3
Time 2–4
Time 1–4
4.5 (0.023)
0.08 (0.055)
0.04 (0.279)
0.04 (0.189)
0.12 (0.046)
0.08 (0.064)
0.17 (0.018)
5.9 3.2 2.2 2.1
0.08 0.09 0.03 0.06
0.16 (0.049) 0.09 (0.409) 0.26* (0.118) 0.09 (0.530)
0.06 (0.488) 0.23* (0.112) 0.04 (0.768) 0.26* (0.154)
0.25 (0.007) 0.18 (0.294) 0.29* (0.097) 0.15 (0.497)
0.20* 0.32* 0.26* 0.35*
0.30* 0.45* 0.29* 0.47*
*Small effect size (0.20 d50.50).
(0.001) (0.046) (0.093) (0.129)
(0.269) (0.575) (0.863) (0.768)
(0.031) (0.042) (0.086) (0.075)
(0.001) (5 0.001) (0.029) (0.002)
1730
E.-C. Chiu et al.
Brain Inj, 2014; 28(13–14): 1726–1733
practice effect was reduced for further assessments. Plateau phases were not observed for the other four indices, indicating that practice effects were not stable over the four assessments. A possible explanation for these results is related to task
complexity Parts 1 and 2 are simpler tasks of the FDT. Part 3 is more complex and part 4 is the most difficult task. This study showed that noticeable practice effects occurred at the third and fourth assessments in the indices containing parts 3
Figure 2. Trend of the mean values of five indices over four assessments (M ± SD).
Practice effects and reliability of the FDT
DOI: 10.3109/02699052.2014.947618
1731
Table IV. Parameters of reliable change index (n ¼ 25). Index Basic measures of attention and processing speed Selective attention Alternating attention Ability of inhibition Ability of switching
Adjacent assessments Time Time Time Time Time Time Time Time Time Time Time Time Time Time Time
1–2 2–3 3–4 1–2 2–3 3–4 1–2 2–3 3–4 1–2 2–3 3–4 1–2 2–3 3–4
ICC (95% CI) 0.98 0.98 0.99 0.94 0.92 0.92 0.71 0.87 0.75 0.66 0.66 0.74 0.52 0.78 0.61
(0.94, (0.96, (0.97, (0.86, (0.82, (0.82, (0.45, (0.74, (0.51, (0.37, (0.37, (0.49, (0.15, (0.56, (0.30,
0.99) 0.99) 0.99) 0.97) 0.97) 0.96) 0.86) 0.94) 0.88) 0.84) 0.83) 0.88) 0.75) 0.90) 0.80)
SEM
SEdiff
6.2 4.7 4.0 6.0 7.0 5.7 18.8 14.9 21.2 7.3 7.2 4.5 18.0 15.6 21.5
8.8 6.6 5.7 8.4 10.0 8.1 26.6 21.1 30.0 10.3 10.2 6.3 25.4 20.0 30.4
90% CI RCIp 17.6, 12.3, 10.7, 15.8, 19.9, 14.6, 47.1, 38.4, 58.1, 17.3, 19.6, 10.9, 43.5, 39.2, 58.1,
11.2 9.5 7.9 11.9 12.8 12.2 40.4 31.2 40.6 16.6 13.8 10.0 40.0 33.2 42.1
ICC, intra-class correlation coefficient (estimation of two consecutive time points); CI, confidence interval; SEM, standard error of measurement; SEdiff, standard error of the differences; RCIp, reliable change index modified for practice.
and 4, respectively. The more complex the task of the FDT is, the more delayed the practice effect is likely to occur. Although plateau phases were not observed in these four indices, this study provides the trends of practice effects within four assessments. These trends could help users to consider the impact of practice effects, depending on the number of assessments required. Reliable change index modified for practice (RCIp) The minimum and maximum 90% CI RCIp were calculated for the five indices of the FDT in patients with stroke. A plateau phase was reached for the index of ‘basic measures of attention and processing speed’. Therefore, a score change beyond its minimum and maximum 90% CI RCIp can help users interpret a patient’s score change as a real change (improvement or deterioration) with 90% certainty. The other four indices that did not reach plateau phases indicate that the minimum and maximum 90% CI RCIp for these indices may continue to change after four assessments. The minimum and maximum 90% CI RCIp of these four indices could be used to interpret a patient’s score change as a real change within only four repeated assessments. Test–re-test reliability Good-to-excellent test–re-test reliability was demonstrated for the three attention indices and moderate-to-good test–re-test reliability was demonstrated for the two indices representing ability of executive functions across four sessions. These results indicate that the FDT has moderate-to-excellent level of scoring consistency over repeated assessments. As a whole, the FDT has acceptable test–re-test reliability in patients with stroke. Study limitations Four limitations of this study were noted. First, it re-assessed the FDT over a specific time interval of 2 weeks. The length of the time interval may influence the size of the practice effect [14]. Further studies could examine the practice effect at different time intervals (e.g. over 1 week, 1 month, 3
months, 6 months and even 1 year) in order to comprehensively determine the practice effect of the FDT. Clinicians could select the results of a time interval based on how often they measured patients with stroke. Second, plateau phases were not reached for four indices in this study within four assessments. One possible method to achieve a plateau phase is to increase the familiarity of tasks using more practice items [34]. The users could increase the 10 practice items for each part of the original FDT to, for example, 30 items (each part is expected to take 1–3 minutes to do a 30-item practice). However, further studies may need to examine whether plateau phases can be reached within four assessments or less using 30 practice items. Third, the criteria for reaching a plateau phase were determined by researchers in this study. Further studies might use other methods for ascertaining a plateau phase and provide further validation of the findings. Fourth, the sample size of this study was small, which may limit the explanations of practice effect (e.g. ANOVA results). It was found that the ANOVA results showed non-significant differences in two indices, alternating attention (p ¼ 0.046) and ability of switching (p ¼ 0.129). However, the results of post-hoc analyses in Time 1–4 showed significant differences in these two indices (p50.001 and p ¼ 0.002, respectively). The reason was that the practice effects were cumulative over multiple testing. In this study, the repeated measures ANOVA might have under-estimated the practice effect of the FDT. Thus, future studies with a larger sample size and a higher number of testing (44) are needed to examine practice effects of the FDT.
Conclusions In summary, this study found trivial-to-small practice effects and moderate-to-excellent test–re-test reliability of the FDT. A plateau phase was reached in the ‘basic measures of attention and processing speed’ index. The minimum and maximum values of the 90% CI RCIp for this index are useful for both clinicians and researchers to determine whether the patient’s score change is beyond measurement error and practice effect. However, a plateau was not reached for the other four indices. Therefore, clinicians and researchers
1732
E.-C. Chiu et al.
should be cautious when interpreting the test results of these four indices over repeated assessments.
Acknowledgements This study was supported by a research grant from the E-Da Hospital (EDAHT-101013).
Declaration of interest The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the paper.
References 1. Stephens S, Kenny RA, Rowan E, Allan L, Kalaria RN, Bradbury M, Ballard CG. Neuropsychological characteristics of mild vascular cognitive impairment and dementia after stroke. International Journal of Geriatric Psychiatry 2004;19:1053–1057. 2. Donovan NJ, Kendall DL, Heaton SC, Kwon S, Velozo CA, Duncan PW. Conceptualizing functional cognition in stroke. Neurorehabilitation and Neural Repair 2008;22:122–135. 3. Stapleton T, Ashburn A, Stack E. A pilot study of attention deficits, balance control and falls in the subacute stage following stroke. Clinical Rehabilitation 2001;15:437–444. 4. Ownsworth T, Shum D. Relationship between executive functions and productivity outcomes following stroke. Disability and Rehabilitation 2008;30:531–540. 5. Mok VC, Wong A, Lam WW, Fan YH, Tang WK, Kwok T, Hui AC, Wong KS. Cognitive impairment and functional outcome after stroke associated with small vessel disease. Journal of Neurology, Neurosurgery, and Psychiatry 2004;75:560–566. 6. Verdejo-Garcı´a A, Pe´rez-Expo´sito M, Schmidt-Rı´o-Valle J, Ferna´ndez-Serrano MJ, Cruz F, Pe´rez-Garcı´a M, Lo´pez-Belmonte G, Martı´n-Matillas M, Martı´n-Lagos JA, Marcos A, Campoy C. Selective alterations within executive functions in adolescents with excess weight. Obesity 2010;18:1572–1578. 7. Sedo MA. ‘‘5-Digit’’ Stroop: Mini-verbal exploration of reactions, decisions, and response delays. Natick, MA: Mini Verbal Tests; 2010. 8. Hsieh SL, Tori CD. Normative data on cross-cultural neuropsychological tests obtained from Mandarin-speaking adults across the life span. Archives of Clinical Neuropsychology 2007;22:283–296. 9. Lezak MD, Howieson DB, Loring DW. Neuropsychological Assessment. 4th edn. Oxford, New York: Oxford University Press; 2004. 10. Bartels C, Wegrzyn M, Wiedl A, Ackermann V, Ehrenreich H. Practice effects in healthy adults: A longitudinal study on frequent repetitive cognitive testing. BMC Neuroscience 2010;11:118. 11. Noble CE, Alcock WT, Farese FJ. Habit reversal under differential instructions in compound trial-and-error learning. The Journal of Psychology 1958;46:253–264. 12. Catroppa C, Anderson V. Recovery in memory function in the first year following TBI in children. Brain Injury 2002;16:369–384. 13. Eonta SE, Carr W, McArdle JJ, Kain JM, Tate C, Wesensten NJ, Norris JN, Balkin TJ, Kamimori GH. Automated neuropsychological assessment metrics: Repeated assessment with two military samples. Aviation, Space, and Environmental Medicine 2011;82: 34–39. 14. Beglinger LJ, Gaydos B, Tangphao-Daniels O, Duff K, Kareken DA, Crawford J, Fastenau PS, Siemers ER. Practice effects and the use of alternate forms in serial neuropsychological testing. Archives of Clinical Neuropsychology 2005;20:517–529.
Brain Inj, 2014; 28(13–14): 1726–1733
15. Koh C-L, Lu W-S, Chen H-C, Hsueh I-P, Hsieh J-J, Hsieh C-L. Test-retest reliability and practice effect of the Oral-format Symbol Digit Modalities Test in patients with stroke. Archives of Clinical Neuropsychology 2011;26:356–363. 16. Chen HC, Koh CL, Hsieh CL, Hsueh IP. Test-re-test reliability of two sustained attention tests in persons with chronic stroke. Brain Injury 2009;23:715–722. 17. Matthey S. Calculating clinically significant change in postnatal depression studies using the Edinburgh Postnatal Depression Scale. Journal of Affective Disorders 2004;78:269–272. 18. James C, Mackenzie L, Capra M. Test-retest reliability of the manual handling component of the WorkHab functional capacity evaluation in healthy adults. Disability and Rehabilitation 2010;32: 1863–1869. 19. Sedo MA, DeCristoforo L. All-language verbal tests free from linguistic barriers. Revista Espan˜ola de Psicologia 2001;3:68–82. 20. Lin KN, Wang PN, Liu CY, Chen WT, Lee YC, Liu HC. Cutoff scores of the cognitive abilities screening instrument, Chinese version in screening of Dementia. Dementia and Geriatric Cognitive Disorders 2002;14:176–182. 21. Wilson B, Cockburn J, Halligan P. Behavioural inattention test manual. Titchfield, UK: Thames Valley Test Company; 1987. 22. Hsueh IP, Lee MM, Hsieh CL. Psychometric characteristics of the Barthel activities of daily living index in stroke patients. Journal of the Formosan Medical Association 2001;100:526–532. 23. Kwon S, Hartzema AG, Duncan PW, Min-Lai S. Disability measures in stroke: Relationship among the Barthel Index, the Functional Independence Measure, and the Modified Rankin Scale. Stroke 2004;35:918–923. 24. Teng EL, Lin K-N, Liu H-C. Cognitive Abilitites Screening Instrument (CASI version C-2.0). Taipei: Taipei Veterans General Hospital; 1993. 25. Lu R, Luo Z, Liu X. The reliability and validity of Cognitive Abilities Screening Instrument C-2.0 used in elderly people in Chengdu. Chinese Journal of Clinical Psychology 2000;2:70–72. 26. Halligan P, Wilson B, Cockburn J. A short screening test for visual neglect in stroke patients. International Disability Studies 1990;12: 95–99. 27. Tanaka T, Sugihara S, Nara H, Ino S, Ifukube T. A preliminary study of clinical assessment of left unilateral spatial neglect using a head mounted display system (HMD) in rehabilitation engineering technology. Journal of Neuroengineering and Rehabilitation 2005; 2:31. 28. Bailey MJ, Riddoch MJ, Crome P. Evaluation of a test battery for hemineglect in elderly stroke patients for use by therapists in clinical practice. NeuroRehabilitation 2000;14:139–150. 29. Agrell BM, Dehlin OI, Dahlgren CJ. Neglect in elderly stroke patients: A comparison of five tests. Psychiatry and Clinical Neurosciences 1997;51:295–300. 30. Raymond PD, Hinton-Bayre AD, Radel M, Ray MJ, Marsh NA. Assessment of statistical change criteria used to define significant change in neuropsychological test performance following cardiac surgery. European Journal of Cardio-thoracic Surgery 2006;29: 82–88. 31. Bushnell CD, Johnston DC, Goldstein LB. Retrospective assessment of initial stroke severity: Comparison of the NIH Stroke Scale and the Canadian Neurological Scale. Stroke 2001;32:656–660. 32. Green JB. Brain reorganization after stroke. Topics in Stroke Rehabilitation 2003;10:1–20. 33. Brodal A. Self-observations and neuroanatomical considerations after a stroke. Brain 1973;96:675–694. 34. Barnett JH, Robbins TW, Leeson VC, Sahakian BJ, Joyce EM, Blackwell AD. Assessing cognitive function in clinical trials of schizophrenia. Neuroscience and Biobehavioral Reviews 2010;34: 1161–1177.
DOI: 10.3109/02699052.2014.947618
Appendix: Examples of the four parts of the Five Digit Test
Practice effects and reliability of the FDT
1733
Copyright of Brain Injury is the property of Taylor & Francis Ltd and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.
ORIGINAL ARTICLE
Practice effects and test–re-test reliability of the Five Digit Test in patients with stroke over four serial assessments En-Chi Chiu1, Chia-Lin Koh1, Chia-Yin Tsai2, Wen-Shian Lu3,4, Ching-Fan Sheu5, I-Ping Hsueh1,6, & Ching-Lin Hsieh1,6 1
School of Occupational Therapy, College of Medicine, National Taiwan University, Taipei, Taiwan, 2Department of Physical Medicine and Rehabilitation, E-Da Hospital, Kaohsiung, Taiwan, 3School of Occupational Therapy, Chung Shan Medical University, Taichung, Taiwan, 4 Occupational Therapy Room, Chung Shan Medical University Hospital, Taichung, Taiwan, 5Institute of Education, National Cheng Kung University, Tainan, Taiwan, and 6Department of Physical Medicine and Rehabilitation, National Taiwan University Hospital, Taipei, Taiwan Abstract
Keywords
Objectives: To investigate practice effect and test–re-test reliability of the Five Digit Test (FDT) over four serial assessments in patients with stroke. Design: Single-group repeated measures design. Methods: Twenty-five patients with stroke were administered the FDT in four consecutive assessments every 2 weeks. The FDT contains four parts with five indices: ‘basic measures of attention and processing speed’, ‘selective attention’, ‘alternating attention’, ‘ability of inhibition’ and ‘ability of switching’. Results: The five indices of the FDT showed trivial-to-small practice effects (Cohen’s d ¼ 0.03– 0.47) and moderate-to-excellent test–re-test reliability (intra-class correlation coefficient ¼ 0.59– 0.97). Practice effects of the five indices all appeared cumulative, but one index, ‘basic measures of attention and processing speed’, reached a plateau after the second assessment. The minimum and maximum values of the 90% confidence interval (CI) of reliable change index modified for practice (RCIp) for this index were [17.6, 11.2]. Conclusions: One of five indices of the FDT reached a plateau, whose minimum and maximum values of the 90% CI RCIp are useful to determine whether the change in an individual’s score is real. However, clinicians and researchers should be cautious when interpreting the test results of these four indices over repeated assessments.
Attention, executive functions, practice effect, reliable change index modified for practice, stroke
Introduction Cognitive impairments, such as attention deficits and executive dysfunctions, occur frequently in patients with stroke [1]. Cognitive impairment may both impede functional recovery and diminish the potential benefits of rehabilitation [2]. Attention deficits in patients with stroke are associated with poor balance, which may increase the risk of fall [3]. Executive dysfunctions could contribute to the inability of performing activities of daily living (ADL) and influence a patient’s return-to-work and post-stroke productivity [4, 5]. Therefore, evaluating attention and executive functions in patients with stroke is critical for both clinicians and researchers to develop treatment plans and conduct outcome studies. The Five Digit Test (FDT) measures attention and executive functions [6, 7]. The FDT has been designed as an alternative tool for the Stroop Color Word Test (SCWT). Three advantages of the FDT compared to the SCWT are: (1)
Correspondence: I-Ping Hsueh, 4F, No. 17, Xu-Zhou Road, Taipei 100, Taiwan. Tel: +886-2-33668174. Fax: +886-2-23511331. E-mail: iping@ ntu.edu.tw
History Received 19 September 2012 Revised 20 April 2014 Accepted 20 July 2014 Published online 4 September 2014
it is appropriate for speakers of any language who are familiar with Arabic numerals; (2) it is appropriate for both competent readers and illiterate subjects (provided they know at least the Arabic numerals 1–5); and (3) the cognitive requirements for performing the tasks are low, which involves only naming and counting of 1–5 digits or stars [8]. Therefore, the FDT could be considered for application in patients with stroke. The FDT is composed of four parts, each of which contains 50 items. Each item is a group of 1–5 digits or stars printed inside a frame. The digits can either be named as the number printed in the frame (two 5 s are read as ‘five’) or counted by the amount of digits in the frame (two 5 s are counted as ‘two’). In part 1, the subject names the digits. In part 2, the subject counts the stars. In part 3, the subject counts the amount of digits and avoids naming the number. In part 4, the subject needs to switch between counting and naming the digits, depending on the thickness of the frame (Appendix). The items of all four parts require verbal responses. Investigating the practice effect and test–re-test reliability of a cognitive measure is important for repeated assessments. Practice effect is defined as improvements in performance over repeated testing due to the influence of the previous experience in performing the same test [9, 10]. Practice effect
Practice effects and reliability of the FDT
DOI: 10.3109/02699052.2014.947618
1727
contains two primary phases over multiple assessments. Accumulation of the practice effect happens in the early stage of assessments (cumulative phase) and then the practice effect becomes stable to achieve a plateau phase [11, 12]. The practice effect of a cognitive measure is greatly influenced by the number of assessments [13] and is most pronounced in the first three or four assessments [14]. Although no studies have examined the practice effect of the SWCT in patients with stroke, several cognitive measures have demonstrated practice effects in patients with stroke. These measures include the Symbol Digit Modalities Test, the Conners’ Continuous Performance Test II and the Digit Vigilance Test [15, 16]. The reliable change index (RCI) calculates a change in score beyond measurement error, which reflects that the change in an individual’s score is real [17]. A reliable change index modified for practice (RCIp) is the RCI corrected, which takes practice effect into account. Test–re-test reliability refers to the level of consistency on the assumption that the behaviour being scored does not change over repeated assessments [18]. These above-mentioned psychometric properties are essential for interpreting re-test results in both clinical and research settings. The FDT has shown satisfactory convergent validity with the SCWT in elderly people [19]. However, to the authors’ knowledge, no study has examined psychometric properties of the FDT in patients with stroke and other populations. Therefore, this study aimed to (1) examine the practice effect of the FDT over four serial assessments, explore the phases of practice effects and calculate RCIp; and (2) examine test–retest reliability of the FDT in patients with stroke.
Institutional Review Board of each hospital. The following criteria were used to recruit the participants: (1) diagnosis of cerebral haemorrhage or cerebral infarction; (2) aged over 18 years; (3) onset of stroke more than 6 months prior to the first evaluation; (4) ability to follow instructions, as assessed by a score 4 1 on two items (i.e. reading ability [READ] and comprehension and memory [CMD]) from the Cognitive Abilities Screening Instrument, Chinese version (CASI C-2.0) [20]; (5) lack of visual or auditory deficits; (6) lack of visual spatial neglect, as assessed by a score 4 51 on the Star Cancellation Test [21]; (7) ability to read Arabic numerals; and (8) ability to give informed consent in person. The criteria for exclusion included: (1) inability to give verbal responses because of severe aphasia; (2) experiencing recurrent stroke during the study; (3) presence of concurrent diseases that could influence cognitive ability, such as dementia, Parkinsonism, major depression or schizophrenia; and (4) having interventions for attention or executive functions.
Methods
Five Digit Test (FDT)
Participants
The FDT contains five indices representing ability of attention and executive functions. Three attention indices include ‘basic measures of attention and processing speed’, ‘selective attention’ and ‘alternating attention’. Two indices representing ability of executive functions include ‘ability of inhibition’ and ‘ability of switching’ [6, 7]. The formulae for scoring these five indices were calculated as follows:
Forty-two outpatients with stroke were recruited from the Department of Physical Medicine and Rehabilitation in five hospitals in northern Taiwan between August 2009 and July 2011. Figure 1 shows the total number of initial participants and the number of participants whose data were not analysed for their respective reasons. This study was approved by the
Procedures At baseline (Time 1), participants were screened by four measures, the Barthel Index (BI), the READ, the CMD and the Star Cancellation Test, and then participants were administered by the FDT. The FDT was administered again after 2 weeks (Time 2), 4 weeks (Time 3) and 6 weeks (Time 4). The participants were assessed in a quiet room to avoid any distractions that could affect their performance. Measures
Basic measures of attention and processing speed ¼ RT of part 1 þ RT of part 2 Selective attention ¼ RT of part 3
42 participants were recruited
Reasons why data from 17 participants were not analyzed 1. Time interval between any of test and retest over the four assessments was more than 15 days (n=9) 2. Participants could not complete all four assessments (n=8)
Alternating attention ¼ RT of part 4 Ability of inhibition ¼ RT of part 3 ðRT of part 1 þ RT of part 2Þ=2 Ability of switching ¼ RT of part 4 ðRT of part 1 þ RT of part 2Þ=2 where RT is the reaction time (in seconds). A lower score indicates better performance. Barthel Index (BI)
Data from 25 participants were analyzed
Figure 1. Flow chart depicting the number of participants whose data were not analysed.
The BI assesses 10 basic ADL items, including feeding, transfer, grooming, toileting, bathing, ambulation, stair climbing, dressing, bowel control and bladder control. The total score of the BI ranges from 0–20. The BI has been demonstrated to have good reliability and validity in patients
1728
E.-C. Chiu et al.
with stroke [22]. In this study, the BI was used to determine participants’ disability levels [23]. READ and CMD The READ and the CMD are two items of the CASI C-2.0. The CASI C-2.0 is a 20-item screening tool for measuring cognition. The READ measures reading ability by showing the participants a card with words ‘close eyes’ and testing whether the participants can perform this action. The CMD measures comprehension, memory and execution of the instructions by giving the participants three phrases of verbal instructions. The score of the READ and CMD ranges from 0–3 [24]. The CASI C-2.0 has good internal consistency, test–re-test reliability, sensitivity and specificity in the elderly population [25]. Star Cancellation Test The Star Cancellation Test is the most sensitive sub-test of the Behavioural Inattention Test [26]. It consists of 56 small stars, 52 large stars, 13 letters and 10 words arranged randomly on the test sheet. The subject is instructed to cross out all of the small stars on the test sheet. The two small stars in the middle are used for demonstration. The total score range is 0–54. A score of 51 indicates unilateral spatial neglect [27]. The Star Cancellation Test has excellent test–re-test reliability and good convergent validity in patients with stroke [28, 29]. Data analysis Practice effect The presence of the practice effect was examined through three methods. First, two characteristics of the practice effect (i.e. cumulative phase and plateau phase) were examined over the four assessments. The mean value of each assessment was used to evaluate practice effect in the cumulative phase. Cumulative phase was observed when the mean values (reaction time) of post-tests were lower than those of pretests. Cohen’s d and one-way repeated measures analysis of variance (ANOVA) were used to determine a plateau phase. For reaching a plateau phase, the following two criteria had to be met: (1) two stable consecutive Cohen’s ds, wherein two consecutive values of Cohen’s d were lower or equal to the previous values (i.e. d of Time 2–3 d of Time 1–2, d of Time 3–4 d of Time 2–3); and (2) a non-significant ANOVA ( ¼ 0.01) in three consecutive time points (e.g. Time 2 to Time 4). If an ANOVA revealed non-significant difference from Time 2 to Time 4, then a plateau phase was reached at Time 2. Second, the repeated measures ANOVA ( ¼ 0.01) was examined over the four assessments. Post hoc analysis ( ¼ 0.01) was conducted to compare differences between pairs of testing times, whether or not the ANOVA results showed significant differences. Third, effect size (Cohen’s d) was calculated to estimate the magnitudes of the practice effects in six pairs of time points (i.e. Time 1–2, Time 2–3, Time 3–4, Time 1–3, Time 1–4 and Time 2–4). The sample size was small in this study, which might have resulted in nonsignificant difference for the ANOVA results. Therefore, examining the characteristics of practice effect and effect size
Brain Inj, 2014; 28(13–14): 1726–1733
were two primary methods to determine the presence of practice effect. Two criteria of determining practice effect were: (1) the cumulative phase was observed and d was 0.01–0.19, indicating a trivial practice effect; and (2) the cumulative phase was observed and d was 0.20–0.49, indicating a small practice effect. Reliable change index modified for practice (RCIp) The minimum and maximum RCIp with 90% confidence interval (CI) were chosen across three adjacent assessments to provide the conservative criteria for interpreting score changes in a patient with stroke. The 90% CI RCIp was calculated as shown below [30]. A negative 90% CI RCIp value indicates that the RT of the post-test was shorter than that of the pre-test in the five indices. 90% CI RCIp ¼ mean practice effect 1:645 SEdiff ð1Þ
SEdiff ¼
qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 2ðSEMÞ2
ð2Þ
pffiffiffiffiffiffiffiffiffiffiffiffiffiffi 1 rxx
ð3Þ
SEM ¼ SD1
where mean practice effect is the average of the difference between the two test scores (1); SEdiff is the standard error of the differences (2); SEM is the standard error of measurement; SD1 is the standard deviation of the first score of adjacent tests; and rxx is the intra-class correlation coefficient value (3). Test–re-test reliability Test–re-test reliability was estimated using the intra-class correlation coefficient (ICC2,1) across four assessments, on the basis of a two-way random-effect model with an absolute agreement type. The following criteria were used for ICC values: 50.39, poor reliability; 0.40–0.59, moderate reliability; 0.60–0.79, good reliability; and 0.80–1.00, excellent reliability [31].
Results The mean age of the 25 participants was 57.9 years and75% were male. The time since onset of stroke was over 9 months for the participants (i.e. participants were in a stable state of neurological recovery) [32, 33]. All participants obtained scores of 3 on the READ and CMD, respectively. The demographic data and other clinical characteristics of the participants are provided in Table I The mean scores of the four sessions are shown in Table II Practice effect The mean values (reaction time) of the five indices over the four assessments generally showed a decreasing trend (Table II and Figure 2), demonstrating cumulative phases. Only the ‘basic measures of attention and processing speed’ index fitted the two criteria of reaching a plateau phase at Time 2: (1) d of Time 2–3 5d of Time 1–2 and d of Time 3–4 ¼ d of Time 2–3; and (2) an ANOVA revealed
Practice effects and reliability of the FDT
DOI: 10.3109/02699052.2014.947618
ability of inhibition and ability of switching, respectively (Table III).
non-significant difference from Time 2 to Time 4. The other four indices displayed unstable Cohen’s ds in three adjacent assessments, not fitting the first criterion for reaching a plateau phase. For the result of ANOVA, only one index (selective attention) was statistically significant (p ¼ 0.001) (Table III). Post-hoc analysis was statistically significant in Time 1–3 in the selective attention index (p ¼ 0.007) and in Time 1–4 in three indices, such as selective attention, alternating attention and ability of switching (p 0.002). The Cohen’s ds of six pairs of time points were 0.04–0.17, 0.08–0.30, 0.09–0.45, 0.03–0.29 and 0.06–0.47 in basic measures of attention and processing speed, selective attention, alternating attention,
Reliable change index modified for practice (RCIp) The parameters for calculating the 90% CI RCIp are shown in Table IV. The minimum and maximum 90% CI RCIp across three adjacent assessments for the five indices were listed as follows: basic measures of attention and processing speed [17.6, 11.2]; selective attention [19.9, 12.8]; alternating attention [58.1, 40.6]; ability of inhibition [19.6, 16.6]; and ability of switching [58.1, 42.1]. Test–re-test reliability Across the four serial assessments, three attention indices showed moderate-to-excellent ICC values (0.75–0.97) (Table II). Two indices representing ability of executive functions showed moderate-to-good ICC values (0.59–0.70).
Table I. Demographic and clinical characteristics of the participants (n ¼ 25). Characteristic Age (years; mean ± SD) Gender, n (%) Male Female Education, n (%) Elementary school Middle school High school Vocational school University Stroke type, n (%) Haemorrhagic Ischaemic Side of hemiparesis, n (%) Left Right Time since onset (month; mean ± SD) Barthel Index (mean ± SD, range) Star Cancellation Test (mean ± SD, range)
Discussion
57.9 ± 11.3
Practice effect
18 (72.0) 7 (28.0) 4 1 8 4 8
1729
The cumulative phases were observed in the five indices, indicating that practice effects result in gradual improvements in score with an increasing number of assessments. All five FDT indices exhibited trivial practice effects in Time 1–2. However, small practice effects were shown in Time 2–3, Time 3–4, Time 1–3, Time 2–4 or Time 1–4, except the ‘basic measures of attention and processing speed’ index. Small practice effects indicate that the patients with stroke may develop test strategies or recall of test items over repeated assessments. Therefore, users should consider practice effects after the third or fourth assessments when interpreting the results of these four indices. The ‘basic measures of attention and processing speed’ index reached a plateau phase at the second assessment, indicating that practice effect was stabilized and the impact of
(16.0) (4.0) (32.0) (16.0) (32.0)
11 (44.0) 14 (56.0) 11 (44.0) 14 (56.0) 33.4 ± 17.0 17.4 ± 3.1, 12–20 53.5 ± 0.8, 52–54
SD, standard deviation.
Table II. Mean and SD at four assessments and test–re-test reliability (n ¼ 25).
Index Basic measures of attention and processing speed Selective attention Alternating attention Ability of inhibition Ability of switching
Time 1 M ± SD
Time 2 M ± SD
Time 3 M ± SD
Time 4 M ± SD
75.0 ± 40.1 56.3 ± 23.3 87.5 ± 35.2 18.8 ± 12.5 50.0 ± 25.8
71.8 ± 35.9 54.3 ± 25.0 84.1 ± 41.9 18.4 ± 12.2 48.2 ± 33.1
70.4 ± 33.8 50.8 ± 19.8 80.5 ± 42.5 15.6 ± 8.7 45.3 ± 34.3
69.0 ± 31.9 49.6 ± 21.3 71.8 ± 34.1 15.1 ± 12.7 37.3 ± 27.7
ICC (95% CI) 0.97 0.91 0.75 0.70 0.59
(0.94, (0.83, (0.60, (0.54, (0.40,
0.98) 0.95) 0.86) 0.84) 0.76)
SD, standard deviation; CI, confidence interval; ICC, intra-class correlation coefficient (estimation of four assessments). Table III. Repeated measures ANOVA and Cohen’s d over four assessments. Cohen’s d (p value) Index Basic measures of attention and processing speed Selective attention Alternating attention Ability of inhibition Ability of switching
F3,75 (p value)
Time 1–2
Time 2–3
Time 3–4
Time 1–3
Time 2–4
Time 1–4
4.5 (0.023)
0.08 (0.055)
0.04 (0.279)
0.04 (0.189)
0.12 (0.046)
0.08 (0.064)
0.17 (0.018)
5.9 3.2 2.2 2.1
0.08 0.09 0.03 0.06
0.16 (0.049) 0.09 (0.409) 0.26* (0.118) 0.09 (0.530)
0.06 (0.488) 0.23* (0.112) 0.04 (0.768) 0.26* (0.154)
0.25 (0.007) 0.18 (0.294) 0.29* (0.097) 0.15 (0.497)
0.20* 0.32* 0.26* 0.35*
0.30* 0.45* 0.29* 0.47*
*Small effect size (0.20 d50.50).
(0.001) (0.046) (0.093) (0.129)
(0.269) (0.575) (0.863) (0.768)
(0.031) (0.042) (0.086) (0.075)
(0.001) (5 0.001) (0.029) (0.002)
1730
E.-C. Chiu et al.
Brain Inj, 2014; 28(13–14): 1726–1733
practice effect was reduced for further assessments. Plateau phases were not observed for the other four indices, indicating that practice effects were not stable over the four assessments. A possible explanation for these results is related to task
complexity Parts 1 and 2 are simpler tasks of the FDT. Part 3 is more complex and part 4 is the most difficult task. This study showed that noticeable practice effects occurred at the third and fourth assessments in the indices containing parts 3
Figure 2. Trend of the mean values of five indices over four assessments (M ± SD).
Practice effects and reliability of the FDT
DOI: 10.3109/02699052.2014.947618
1731
Table IV. Parameters of reliable change index (n ¼ 25). Index Basic measures of attention and processing speed Selective attention Alternating attention Ability of inhibition Ability of switching
Adjacent assessments Time Time Time Time Time Time Time Time Time Time Time Time Time Time Time
1–2 2–3 3–4 1–2 2–3 3–4 1–2 2–3 3–4 1–2 2–3 3–4 1–2 2–3 3–4
ICC (95% CI) 0.98 0.98 0.99 0.94 0.92 0.92 0.71 0.87 0.75 0.66 0.66 0.74 0.52 0.78 0.61
(0.94, (0.96, (0.97, (0.86, (0.82, (0.82, (0.45, (0.74, (0.51, (0.37, (0.37, (0.49, (0.15, (0.56, (0.30,
0.99) 0.99) 0.99) 0.97) 0.97) 0.96) 0.86) 0.94) 0.88) 0.84) 0.83) 0.88) 0.75) 0.90) 0.80)
SEM
SEdiff
6.2 4.7 4.0 6.0 7.0 5.7 18.8 14.9 21.2 7.3 7.2 4.5 18.0 15.6 21.5
8.8 6.6 5.7 8.4 10.0 8.1 26.6 21.1 30.0 10.3 10.2 6.3 25.4 20.0 30.4
90% CI RCIp 17.6, 12.3, 10.7, 15.8, 19.9, 14.6, 47.1, 38.4, 58.1, 17.3, 19.6, 10.9, 43.5, 39.2, 58.1,
11.2 9.5 7.9 11.9 12.8 12.2 40.4 31.2 40.6 16.6 13.8 10.0 40.0 33.2 42.1
ICC, intra-class correlation coefficient (estimation of two consecutive time points); CI, confidence interval; SEM, standard error of measurement; SEdiff, standard error of the differences; RCIp, reliable change index modified for practice.
and 4, respectively. The more complex the task of the FDT is, the more delayed the practice effect is likely to occur. Although plateau phases were not observed in these four indices, this study provides the trends of practice effects within four assessments. These trends could help users to consider the impact of practice effects, depending on the number of assessments required. Reliable change index modified for practice (RCIp) The minimum and maximum 90% CI RCIp were calculated for the five indices of the FDT in patients with stroke. A plateau phase was reached for the index of ‘basic measures of attention and processing speed’. Therefore, a score change beyond its minimum and maximum 90% CI RCIp can help users interpret a patient’s score change as a real change (improvement or deterioration) with 90% certainty. The other four indices that did not reach plateau phases indicate that the minimum and maximum 90% CI RCIp for these indices may continue to change after four assessments. The minimum and maximum 90% CI RCIp of these four indices could be used to interpret a patient’s score change as a real change within only four repeated assessments. Test–re-test reliability Good-to-excellent test–re-test reliability was demonstrated for the three attention indices and moderate-to-good test–re-test reliability was demonstrated for the two indices representing ability of executive functions across four sessions. These results indicate that the FDT has moderate-to-excellent level of scoring consistency over repeated assessments. As a whole, the FDT has acceptable test–re-test reliability in patients with stroke. Study limitations Four limitations of this study were noted. First, it re-assessed the FDT over a specific time interval of 2 weeks. The length of the time interval may influence the size of the practice effect [14]. Further studies could examine the practice effect at different time intervals (e.g. over 1 week, 1 month, 3
months, 6 months and even 1 year) in order to comprehensively determine the practice effect of the FDT. Clinicians could select the results of a time interval based on how often they measured patients with stroke. Second, plateau phases were not reached for four indices in this study within four assessments. One possible method to achieve a plateau phase is to increase the familiarity of tasks using more practice items [34]. The users could increase the 10 practice items for each part of the original FDT to, for example, 30 items (each part is expected to take 1–3 minutes to do a 30-item practice). However, further studies may need to examine whether plateau phases can be reached within four assessments or less using 30 practice items. Third, the criteria for reaching a plateau phase were determined by researchers in this study. Further studies might use other methods for ascertaining a plateau phase and provide further validation of the findings. Fourth, the sample size of this study was small, which may limit the explanations of practice effect (e.g. ANOVA results). It was found that the ANOVA results showed non-significant differences in two indices, alternating attention (p ¼ 0.046) and ability of switching (p ¼ 0.129). However, the results of post-hoc analyses in Time 1–4 showed significant differences in these two indices (p50.001 and p ¼ 0.002, respectively). The reason was that the practice effects were cumulative over multiple testing. In this study, the repeated measures ANOVA might have under-estimated the practice effect of the FDT. Thus, future studies with a larger sample size and a higher number of testing (44) are needed to examine practice effects of the FDT.
Conclusions In summary, this study found trivial-to-small practice effects and moderate-to-excellent test–re-test reliability of the FDT. A plateau phase was reached in the ‘basic measures of attention and processing speed’ index. The minimum and maximum values of the 90% CI RCIp for this index are useful for both clinicians and researchers to determine whether the patient’s score change is beyond measurement error and practice effect. However, a plateau was not reached for the other four indices. Therefore, clinicians and researchers
1732
E.-C. Chiu et al.
should be cautious when interpreting the test results of these four indices over repeated assessments.
Acknowledgements This study was supported by a research grant from the E-Da Hospital (EDAHT-101013).
Declaration of interest The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the paper.
References 1. Stephens S, Kenny RA, Rowan E, Allan L, Kalaria RN, Bradbury M, Ballard CG. Neuropsychological characteristics of mild vascular cognitive impairment and dementia after stroke. International Journal of Geriatric Psychiatry 2004;19:1053–1057. 2. Donovan NJ, Kendall DL, Heaton SC, Kwon S, Velozo CA, Duncan PW. Conceptualizing functional cognition in stroke. Neurorehabilitation and Neural Repair 2008;22:122–135. 3. Stapleton T, Ashburn A, Stack E. A pilot study of attention deficits, balance control and falls in the subacute stage following stroke. Clinical Rehabilitation 2001;15:437–444. 4. Ownsworth T, Shum D. Relationship between executive functions and productivity outcomes following stroke. Disability and Rehabilitation 2008;30:531–540. 5. Mok VC, Wong A, Lam WW, Fan YH, Tang WK, Kwok T, Hui AC, Wong KS. Cognitive impairment and functional outcome after stroke associated with small vessel disease. Journal of Neurology, Neurosurgery, and Psychiatry 2004;75:560–566. 6. Verdejo-Garcı´a A, Pe´rez-Expo´sito M, Schmidt-Rı´o-Valle J, Ferna´ndez-Serrano MJ, Cruz F, Pe´rez-Garcı´a M, Lo´pez-Belmonte G, Martı´n-Matillas M, Martı´n-Lagos JA, Marcos A, Campoy C. Selective alterations within executive functions in adolescents with excess weight. Obesity 2010;18:1572–1578. 7. Sedo MA. ‘‘5-Digit’’ Stroop: Mini-verbal exploration of reactions, decisions, and response delays. Natick, MA: Mini Verbal Tests; 2010. 8. Hsieh SL, Tori CD. Normative data on cross-cultural neuropsychological tests obtained from Mandarin-speaking adults across the life span. Archives of Clinical Neuropsychology 2007;22:283–296. 9. Lezak MD, Howieson DB, Loring DW. Neuropsychological Assessment. 4th edn. Oxford, New York: Oxford University Press; 2004. 10. Bartels C, Wegrzyn M, Wiedl A, Ackermann V, Ehrenreich H. Practice effects in healthy adults: A longitudinal study on frequent repetitive cognitive testing. BMC Neuroscience 2010;11:118. 11. Noble CE, Alcock WT, Farese FJ. Habit reversal under differential instructions in compound trial-and-error learning. The Journal of Psychology 1958;46:253–264. 12. Catroppa C, Anderson V. Recovery in memory function in the first year following TBI in children. Brain Injury 2002;16:369–384. 13. Eonta SE, Carr W, McArdle JJ, Kain JM, Tate C, Wesensten NJ, Norris JN, Balkin TJ, Kamimori GH. Automated neuropsychological assessment metrics: Repeated assessment with two military samples. Aviation, Space, and Environmental Medicine 2011;82: 34–39. 14. Beglinger LJ, Gaydos B, Tangphao-Daniels O, Duff K, Kareken DA, Crawford J, Fastenau PS, Siemers ER. Practice effects and the use of alternate forms in serial neuropsychological testing. Archives of Clinical Neuropsychology 2005;20:517–529.
Brain Inj, 2014; 28(13–14): 1726–1733
15. Koh C-L, Lu W-S, Chen H-C, Hsueh I-P, Hsieh J-J, Hsieh C-L. Test-retest reliability and practice effect of the Oral-format Symbol Digit Modalities Test in patients with stroke. Archives of Clinical Neuropsychology 2011;26:356–363. 16. Chen HC, Koh CL, Hsieh CL, Hsueh IP. Test-re-test reliability of two sustained attention tests in persons with chronic stroke. Brain Injury 2009;23:715–722. 17. Matthey S. Calculating clinically significant change in postnatal depression studies using the Edinburgh Postnatal Depression Scale. Journal of Affective Disorders 2004;78:269–272. 18. James C, Mackenzie L, Capra M. Test-retest reliability of the manual handling component of the WorkHab functional capacity evaluation in healthy adults. Disability and Rehabilitation 2010;32: 1863–1869. 19. Sedo MA, DeCristoforo L. All-language verbal tests free from linguistic barriers. Revista Espan˜ola de Psicologia 2001;3:68–82. 20. Lin KN, Wang PN, Liu CY, Chen WT, Lee YC, Liu HC. Cutoff scores of the cognitive abilities screening instrument, Chinese version in screening of Dementia. Dementia and Geriatric Cognitive Disorders 2002;14:176–182. 21. Wilson B, Cockburn J, Halligan P. Behavioural inattention test manual. Titchfield, UK: Thames Valley Test Company; 1987. 22. Hsueh IP, Lee MM, Hsieh CL. Psychometric characteristics of the Barthel activities of daily living index in stroke patients. Journal of the Formosan Medical Association 2001;100:526–532. 23. Kwon S, Hartzema AG, Duncan PW, Min-Lai S. Disability measures in stroke: Relationship among the Barthel Index, the Functional Independence Measure, and the Modified Rankin Scale. Stroke 2004;35:918–923. 24. Teng EL, Lin K-N, Liu H-C. Cognitive Abilitites Screening Instrument (CASI version C-2.0). Taipei: Taipei Veterans General Hospital; 1993. 25. Lu R, Luo Z, Liu X. The reliability and validity of Cognitive Abilities Screening Instrument C-2.0 used in elderly people in Chengdu. Chinese Journal of Clinical Psychology 2000;2:70–72. 26. Halligan P, Wilson B, Cockburn J. A short screening test for visual neglect in stroke patients. International Disability Studies 1990;12: 95–99. 27. Tanaka T, Sugihara S, Nara H, Ino S, Ifukube T. A preliminary study of clinical assessment of left unilateral spatial neglect using a head mounted display system (HMD) in rehabilitation engineering technology. Journal of Neuroengineering and Rehabilitation 2005; 2:31. 28. Bailey MJ, Riddoch MJ, Crome P. Evaluation of a test battery for hemineglect in elderly stroke patients for use by therapists in clinical practice. NeuroRehabilitation 2000;14:139–150. 29. Agrell BM, Dehlin OI, Dahlgren CJ. Neglect in elderly stroke patients: A comparison of five tests. Psychiatry and Clinical Neurosciences 1997;51:295–300. 30. Raymond PD, Hinton-Bayre AD, Radel M, Ray MJ, Marsh NA. Assessment of statistical change criteria used to define significant change in neuropsychological test performance following cardiac surgery. European Journal of Cardio-thoracic Surgery 2006;29: 82–88. 31. Bushnell CD, Johnston DC, Goldstein LB. Retrospective assessment of initial stroke severity: Comparison of the NIH Stroke Scale and the Canadian Neurological Scale. Stroke 2001;32:656–660. 32. Green JB. Brain reorganization after stroke. Topics in Stroke Rehabilitation 2003;10:1–20. 33. Brodal A. Self-observations and neuroanatomical considerations after a stroke. Brain 1973;96:675–694. 34. Barnett JH, Robbins TW, Leeson VC, Sahakian BJ, Joyce EM, Blackwell AD. Assessing cognitive function in clinical trials of schizophrenia. Neuroscience and Biobehavioral Reviews 2010;34: 1161–1177.
DOI: 10.3109/02699052.2014.947618
Appendix: Examples of the four parts of the Five Digit Test
Practice effects and reliability of the FDT
1733
Copyright of Brain Injury is the property of Taylor & Francis Ltd and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.
