The Journal of Pain, Vol 15, No 1 (January), 2014: pp 32-39 Available online at www.jpain.org and www.sciencedirect.com
Minimally Clinically Significant Differences for Adolescents With Chronic Pain—Variability of ROC-Based Cut Points Gerrit Hirschfeld, Julia Wager, Pia Schmidt, and Boris Zernikow German Paediatric Pain Centre, Children’s and Adolescents’ Hospital, Datteln, Germany; and Department of Children’s Pain Therapy and Paediatric Palliative Care, School of Medicine, Faculty of Health, Witten/Herdecke University, Witten, North Rhine-Westphalia, Germany.
Abstract: Assessing if an individual patient has achieved clinically meaningful changes in pain intensity is a core aspect in the evaluation of pain treatments. The aim of the present study was to define minimally clinically significant differences (MCSDs) for the numerical rating scale (0–10 NRS) in adolescents with chronic pain. Data from 153 adolescents who completed an inpatient treatment were analyzed. MCSDs were defined as those cut points that yielded an optimal balance between sensitivity and specificity with regard to patients’ global impression of change. The variability of the empirically defined cut points was quantified using bootstrapping. Our results show that raw changes of 1 NRS point and percent changes of 12.5% can be considered MCSDs both within the full sample and within various subsamples of patients. Applying the MCSDs developed for adults to pediatric patients yielded extremely low sensitivities; for example, only 22% of the children who described global improvement met the 50% decrease in pain criterion. Studies evaluating chronic pain treatments for adolescents should use MCSDs that are specifically developed for this group of patients. Raw changes of 1 point and 12.5% on the 0 to 10 NRS should be considered clinically meaningful. On a methodological level, we call for more systematic studies aimed at defining MCSDs that also address the variability of cut point estimates so as to foster the integration of findings. Perspective: Many studies are aimed at empirically defining cut points for clinically relevant pain using receiver operating characteristic-based methods. For the first time, we apply these methods to children and show that even when taking into account the variability of the method, cut points specific for children are needed. ª 2014 by the American Pain Society Key words: Pain, optimal cut points, receiver operating characteristic, bootstrap.
A
core aspect in the evaluation of pain therapies is the assessment whether a change in pain intensity can be considered clinically meaningful.12 Such an assessment requires reliable measures for pain intensity and standards for their interpretation, that is, cut points that define minimally clinically significant differences
Received February 15, 2013; Revised August 1, 2013; Accepted September 13, 2013. No funding was received for this study. The authors declare that they have no conflict of interest, including specific financial interests, relationships, or affiliations relevant to the study. Address reprint requests to Gerrit Hirschfeld, PhD, German Paediatric Pain Centre and Department of Children’s Pain Therapy and Paediatric Palliative Care, Children’s and Adolescents’ Hospital, Datteln, Witten/ Herdecke University, Dr.-Friedrich-Steiner Str. 5, 45711 Datteln, Germany. E-mail: [email protected] 1526-5900/$36.00 ª 2014 by the American Pain Society http://dx.doi.org/10.1016/j.jpain.2013.09.006
32
(MCSDs). The reliability and validity of scales such as the 11-point numerical rating scale (0–10 NRS)23 or faces pain scale21 has been established in children and adolescents, but there are no agreed-upon MCSDs for use in pediatric chronic pain. Following the landmark study by Farrar and colleagues,6 patients’ global impressions of change (PGIC) are used as anchors and evaluated against possible cut points by means of receiver operating characteristic (ROC) methods. This allows defining those cut points as MCSD that best differentiate between patients who report that their pain intensity has improved and patients who report that their pain intensity is unchanged or worsened. In the present paper, an ROC method is utilized for the first time to establish MCDS for the 0 to 10 NRS for adolescents with chronic pain. Previous research has identified conflicting standards to interpret change on the 0 to 10 NRS for adolescents.
Hirschfeld et al
Methods to Define Minimally Clinically Significant Change Used in the Farrar et al Study on a Pooled Adult Sample and Studies in Children and Adolescents
Table 1.
STUDY FIRST AUTHOR, YEAR 6
SAMPLE
SCALES
SETTING
PAIN
METHODS
PGIC
TARGET PGIC CATEGORIES
STATISTIC
MCSD
‘‘Much improved’’ or ‘‘very much improved’’ vs other categories Comparison between ‘‘a bit better’’ and ‘‘a bit worse’’
ROC (balance between sensitivity and specificity) for raw and percent change Difference between changes in ‘‘bit better’’ and ‘‘bit worse’’ groups Median of pre – post differences Median (and mean) of pre–post differences Mean of pre–post differences
1.74/30%
Farrar, 2001
Adults; n = 2,724
Pregabalin in diabetic neuropathy
NRS (0–10)
7-point: ‘‘very much improved’’ to ‘‘very much worse’’
Powell, 200115
Adolescents (8–15 years); n = 73
Emergency department
VAS (100 mm)
5-point: ‘‘heaps better’’ to ‘‘heaps worse’’
Bulloch, 20022
Adolescents (9.8 6 3.15 years); n = 121
Emergency department
FPS (0–6) CAS (100 mm)
5-point: ‘‘much less’’ to ‘‘much worse’’
Adolescents (5–12 years); n = 126 Adolescents (7–16 years); n = 113
Emergency department
CAS (100 mm)
5-point: ‘‘much less’’ to ‘‘much worse’’
Mean for each category of PGIC
Postoperatively
NRS (0–10)
5-point: ‘‘a lot better’’ to ‘‘a lot worse’’
Comparison between ‘‘a little better’’ and ‘‘a little worse’’
McConahay, 200711 Voepel-Lewis, 201222
‘‘Little less’’ pain
Mean
10 mm
1 face 20 mm (17 mm) 24 mm
1
Abbreviations: VAS, visual analog scale; FPS, faces pain scale; CAS, color analog scale.
The Journal of Pain
33
34
MCSD in Pediatric Chronic Pain
The Journal of Pain
Table 2.
Core Demographic Characteristics VARIABLE
Age (mean [SD]) Gender (female n (%]) Main location (n [%]) Head Back/Extremities Abdomen Days missed school in past 3 month (mean [SD]) Pediatric pain disability index (range 12–60) (mean [SD])
VALUE 15.54 (1.83) 115 (75%) 85 (56%) 39 (25%) 29 (19%) 15 days (21) 37 (9)
Four previous studies aimed to empirically establish MCSDs for pain intensity in adolescents with acute pain in the emergency department or postoperatively.2,11,15,22 In addition to using a variety of methods to assess pain intensity, they also differed in their use and interpretation of the global impression of change and statistical procedure (see Table 1). A recent study suggested a change of 1 on the 0 to 10 NRS as an MCSD for adolescents with acute pain,22 which is considerably different from the 1.74 cut point suggested for adults with chronic pain.6 Two problems make it hard to apply existing ‘‘pediatric’’ MCSDs to pediatric chronic pain. First, the pediatric studies all dealt with acute pain and changes within several hours. Second, these studies did not access sensitivity or specificity of the changes in pain intensity with regard to PGIC. As a result, it is unclear how well these cut points are able to differentiate between improved and unimproved pediatric chronic pain patients. The main aim of the present study was to define MCSDs for adolescents with chronic pain using the same methodology that was used to define MCSDs in adults. In addition to reporting the MCSDs as raw changes, we also calculated MCSDs for percent change as these seem more stable.4 Our study also had 2 secondary aims. First, we wanted to test whether possible deviations between the MCSDs for pediatric22 and adult6 samples are due to chance fluctuations or reflect reliable differences. Second, we wanted to test whether MCSDs are also feasible in subgroups of patients with different temporal pain qualities (constant vs intermittent) and patients with the same pain location (head vs abdomen vs musculoskeletal).
local ethics committee of the university hospital. Of all 204 patients, 51 (25%) had to be excluded because they did not provide the pain intensity or PGIC ratings after completion of treatment necessary for the analysis. Core demographic information for the 153 adolescents is given in Table 2.
Measures The 2 variables central to this paper were pain intensity and PGIC. Pain intensity ratings were collected at admission and on discharge. At both time points, children were asked to indicate their worst pain in the preceding 7 days (‘‘When you were experiencing the main pain, how strong was this pain mostly during the past 7 days?’’ [‘‘Wenn du deine Hauptschmerzen hattest, wie stark waren diese Schmerzen dann meistens in den letzten 7 Tagen’’]) on the 11-point NRS. The reliability and validity of this scale is well established in adolescents.23 PGIC in pain intensity were assessed at discharge. Five possible response categories were given, ranging from ‘‘much more intense pain’’ to ‘‘much less intense pain.’’ In order to describe the sample and perform additional subgroup analyses, we also collected data on the main pain location, constant pain, days absent from school, and pain-related disability.9
Data Analysis The data analysis for raw changes (pre – post) and percent changes ([pre – post]/pre) proceeded in 4 steps: describing pain intensity and PGIC ratings, defining MCSDs, performing bootstrap, and analyzing subgroups. First, change from pre- to posttreatment 0 to 10 NRS was calculated for each patient. These differences were
Methods Subjects In this study, data from all adolescents who completed a 3-week multimodal inpatient pain therapy at the German Paediatric Pain Centre in 2012 were retrospectively analyzed. This is a secondary analysis of an ongoing project to assess the effectiveness of the treatment. Patients as well as their parents provided written informed consent for data collection, electronic data storage, and data analysis when they first visited the German Paediatric Pain Centre as approved by the
Figure 1. NRS change from baseline to endpoint for raw change (left) and percent change (right). Error bars indicate 95% CI.
Hirschfeld et al
The Journal of Pain
35
Figure 2. ROC curves for NRS changes for raw change (left) and percent change (right). Broken lines represent chance classification. averaged within the 5 PGIC categories and plotted together with their confidence interval. Second, the MCSDs were defined empirically using the ROC-based method suggested by Farrar and colleagues.6 Specifically, the 2 positive PGIC categories (ie, improved, much improved) were combined into a single ‘‘meaningful change category.’’ The area under the curve (AUC) was calculated to estimate the overall accuracy of the change scores with regard to this category. AUC values are interpreted according to the standards developed by Swets20; values between .5 and .7 are considered ‘‘rather low’’; values between .7 and .9 are considered ‘‘useful for some purposes’’; and values higher than .9 are considered ‘‘rather high.’’ The sensitivity and specificity for all possible cut points in 0 to 10 NRS changes were computed, and the cut point associated with the smallest absolute difference between sensitivity and specificity was defined as MCSD. Third, the variability of MCSDs was quantified in order to test whether these are systematically different from the MCSDs developed for adults. Specifically, a nonparametric bootstrap with 1,000 repetitions was used to yield sufficiently stable estimates for the vari-
ability of MCSD.3 For each of the 1,000 randomly chosen pseudo-samples, the MCSDs were determined using the ROC-based method described above. For each pseudo-sample, MCSDs were recorded. The resulting distributions of MCSDs were inspected for feasible values with the help of descriptive statistics using 5% as a cut point. Fourth, the bootstrapping analysis was replicated in homogeneous subgroups of patients, that is, patients with the same temporal pain quality (constant vs intermittent) and patients with the same pain location. This was motivated by research into optimal cut points for mild, moderate, or severe pain that found several alternative cut points for groups of patients with different pain locations.8 For example, Jensen and colleagues found different cut points for phantom limb vs back pain.10 Importantly, differences between groups may be found once the variability of the optimal cut points is taken into account.7 The software R16 and the ROCR package19 was used for data analysis.
Performance of Different Cut Points for Raw Change
CUT POINT
Table 3. CUT POINT 10 8 7 6 5 4 3 2 1* 0 1 2 3 5
SENSITIVITY
SPECIFICITY
.01 .02 .04 .06 .12 .21 .32 .45 .71 .88 .97 .98 .99 1
.98 .98 .98 .95 .95 .95 .89 .82 .63 .28 .07 .04 0 0
NOTE. Positive cut points indicate improvement. *Optimal cut point according to the criterion of equal sensitivity and specificity.
Table 4. Performance of Different Cut Points for Percent Change
50 44.44 42.86 40 37.5 33.33 28.57 25 22.22 20 16.67 14.29 12.5* 11.11 10 0
SENSITIVITY
SPECIFICITY
.22 .23 .28 .29 .31 .36 .41 .42 .44 .51 .55 .62 .68 .7 .71 .88
.95 .95 .95 .95 .89 .89 .89 .81 .81 .79 .74 .72 .65 .63 .63 .28
NOTE. Positive cut points indicate improvement. In order to save space, extreme cut points (cut points > 50% and cut points < 0%) were truncated. *Optimal cut point according to the criterion of equal sensitivity and specificity.
36
The Journal of Pain
MCSD in Pediatric Chronic Pain
Figure 3. Frequency at which the cut points were identified as optimal for raw change (left) and percent change (right). Broken lines represent 5% of the pseudo-samples.
Results
Definition of MCSDs
At discharge most patients reported that their pain was ‘‘improved’’ (n = 68; 44%) or ‘‘much improved’’ (n = 28; 18%), some indicated that their pain did not change (n = 50; 33%), and very few indicated that their pain became ‘‘worse’’ (n = 7; 5%). On average, the raw 0 to 10 NRS reduced by 1.29 6 2.35 raw points and 15.74 6 39.30%. The levels of change in the 0 to 10 NRS for each PGIC categories are presented in Fig 1 for the change in raw points (left) and percent (right).
Raw and percent changes in the 0 to 10 NRS were systematically related to PGIC categories (Fig 2). The overall accuracy of the change scores as indexed by the AUC was .70 (95% confidence interval [CI] = .61–.77) for raw change and .70 (95% CI = .61–.78) for percent change, indicating that changes in 0 to 10 NRS yield predictions that are ‘‘useful for some purposes.’’20 Inspecting the possible cut points for the raw change showed that a difference score of 1 yielded the best balance between sensitivity (.71) and specificity (.63; Table 3). That is, only raw changes that are 1 point or larger can be considered clinically meaningful. Using 2 as a cut point resulted in considerably lower sensitivity (.45) and higher specificity (.82).
Figure 4. Results of the bootstrap analysis for subgroups with
Figure 5. Results of the bootstrap analysis for subgroups with
constant vs intermittent pain. Frequency at which the cut points were identified as optimal for raw change (left) and percent change (right). Broken lines represent 5% of the pseudo-samples.
different pain locations (head, extremities, abdomen). Frequency at which the cut points were identified as optimal for raw change (left) and percent change (right). Broken lines represent 5% of the pseudo-samples.
Description of Pain Intensity and PGIC Ratings
Hirschfeld et al With regard to percent change, a difference of 12.5 yielded the best balance between sensitivity (.68) and specificity (.65; Table 4). That is, only changes that are 12.5% or larger can be considered clinically meaningful. Using 50% as a cut point resulted in very low sensitivity (.22) and high specificity (.95) compared to adolescents’ PGIC.
Variability of MCSDs The analysis of the results in the bootstrapped pseudosamples revealed that MCSDs for raw changes were identified more consistently than MCSDs for percent changes. The cut point of 1 on the 0 to 10 NRS that was identified as MCSD in the whole sample was also identified in 986 of the total 1,000 pseudo-samples, and the alternative cut point 2 was identified in the remaining 14 pseudosamples (Fig 3, left). The results for the percent change showed a larger variability (Fig 3 right); the cut point 12.5% was identified as MCSD in 432 of the pseudosamples. Several other cut points were also identified in more than 50 of the pseudo-samples; specifically, the 3 cut points of 10%, 11.1%, and 14.3% were identified as MCSDs in 116, 125, and 299 of the pseudo-samples.
Results in Homogeneous Subgroups We repeated these analysis in several subgroups, the first by frequency (constant vs intermittent pain) and the second by main pain location (head vs abdomen vs extremities). The analysis for patients with constant vs intermittent pain showed that the raw change of 1 point was most frequently identified as MCSD in both subgroups (Fig 4). However, for adolescents with intermittent pain, a cut point of 2 was also identified as MCSD in more than 5% of the pseudo-samples. For the percent change, the results also showed less stable MCSD estimations for adolescents with intermittent pain. For adolescents with constant pain, the MCSDs ranged between 10% and 14.3%, whereas for adolescents with intermittent pain, values between 0% and 25% were identified as MCSDs. The analysis of subgroups based on main pain location showed large agreement between adolescents with chronic headache and musculoskeletal pain (Fig 5). For these 2 groups, 1 was identified as MCSD in most pseudo-samples. In adolescents with abdominal pain, 2 was identified as MCSD in the majority (70%) of the pseudo-samples, whereas 1 was identified as MCSD in only 29% of the pseudo-samples. The MCSD for percent change also indicates that the most frequent MCSD for adolescents with headache or musculoskeletal pain was 12.5% whereas the most frequent MCSD for adolescents with abdominal pain was 20%. Again the 12.5% cut point was also identified as MCSD in more than 5% of the pseudo-samples.
Discussion The present study investigated empirically defined MCSDs for pain intensity measured with the 0 to 10 NRS in adolescents with chronic pain after inpatient
The Journal of Pain
37
treatment. Using a methodology similar to the one established in adults, we identified 1 and 12.5% as MCSDs for adolescents with chronic pain. Most importantly, these are lower than the cut points developed for adults with chronic pain6 but similar to those defined for adolescents with acute pain.22 This was also true when taking into account the variability of this procedure using bootstrapping. Furthermore, MCSDs that were developed for adults with chronic pain, that is, reduction by 1.74 raw points and 30%,6 result in low detection rates and underestimate the effectiveness of treatments in children. Comparison of homogeneous subgroups revealed that different cut points were identified as optimal, but the MCSDs developed for the whole sample were feasible in all different subgroups. Thus, we argue for using ‘‘pediatric’’ MCSDs that are different from the ‘‘adult’’ MCSDs but similar in diverse pediatric chronic pain conditions. In the following, we discuss the lack of standardization of procedures to establish cut points for meaningful change and the relevance of estimating the variability of cut points. Our results demonstrate that specialized cut points need to be utilized when interpreting changes in pain ratings by children with chronic pain. Use of the cut points for raw and percent change developed for adolescents yielded very low sensitivities. That is, the majority of patients who after therapy describe their pain as ‘‘less intense’’ or ‘‘much less intense’’ do not meet the criteria developed for adults. Furthermore, when taking into account the variability of the method to define MCSDs, we were able to show that these differences are not due to chance fluctuations but rather reflect real differences between the groups. Our results agree with a recent study on MCSDs for acute pain in children and adolescents.22 In contrast to this, most primary research papers and reviews rely on 50% pain reduction as a criterion for improvement.5 The IMMPACT guideline has defined 10–20% decrease as minimally important, 30% as moderately important, and 50% as substantial4 but offers no recommendation for children. The PedIMMPACT guideline12 does not recommend a specific cut point for pain intensity. In light of the present data on MCSD for chronic pain and earlier results of MCSDs for acute pain in pediatric samples,22 it seems prudent to adopt these guidelines. At present, it is unclear what exactly brings about these differences in comparison to the initial study by Farrar et al. In addition to the different age, the underlying disease and treatment differed. Most patients in the Farrar et al study had neuropathic pain, whereas the majority of patients in the present sample had headache or musculoskeletal pain.6 Looking closely at the reductions in pain intensity for different underlying diseases in the Farrar et al study (see Fig 2), it seems that, for example, patients with osteoarthritis who reported improvement reduced their pain ratings much less than patients with peripheral diabetic neuropathies. In line with this, a recent study that analyzed data from patients with osteoarthritis found that 1 point is a reasonable MCSD for this group.17 Second, the type of treatment (analgesic vs psychotherapy) may also impact the expected
38
The Journal of Pain
MCSD in Pediatric Chronic Pain
treatment effect, which serves as an internal standard against which changes are evaluated. It seems likely that patients who take an active role in their treatment are more likely to perceive small improvements in pain intensity as meaningful. Clearly more research is needed to untangle the different factors that contribute to differences in MCSDs. The inspection of different subgroups of adolescents with chronic pain revealed that the MCSDs emerged with slightly different frequencies. For example, patients with constant pain seem to exhibit more stable MCSDs compared to those with intermittent pain. However, in a study of mild, moderate, and severe pain, a group with constant pain also exhibited highly variable cut points.8 Thus, it is unlikely that systematic differences underlie these differences in MCSDs. If one were interested in the relation between PGIC and changes in pain intensity ratings, a correlational analysis would be more adept than an analysis of MCSDs. For us, the subgroup analysis indicated that bootstrapping supports the idea that the MCSDs defined for the whole group are also feasible in all subgroups. On a methodological level, we call for taking into account the variability of the methods to define optimal cut points. Research into mild, moderate, and severe pain has already demonstrated that differences in optimal cut points may be due to chance fluctuations.8,18 Even though those studies use analyses of variance to define optimal cut points, similar problems arise with the ROCbased methods that have been used to define MCSDs. We suggest a 5% criterion to judge whether an existing cut point is feasible for a subgroup of patients. That is, as long as a preexisting cut point is found in 5% of the bootstrapped pseudo-samples, the data should be interpreted as supporting the preexisting cut point. It is important to note that this is procedure is highly conservative in the sense that it makes it hard to establish specialized cut points. We believe that this is in line with the logic of null hypothesis significance testing13 with the associated alpha level of 5%, according to which the null hypothesis is rejected only if less than 5% of the samples would yield a result more extreme than the one found. This bias against novel results is an important safeguard against supporting novel cut points. We believe that such a strategy may be a core aspect toward the harmonization of the interpretation for meaningful change.
Several limitations need to be taken into account when interpreting the results of this study. First, the sample size was relatively small compared to studies in adults.6 This may affect the variability of the cut points and calls for a replication in larger samples. Second, even though we tried to replicate the methods used in adults as closely as possible, we did not collect daily pain ratings, which could have been averaged, but rather asked for a weekly average. Hence, only rounded difference scores are possible so that fewer alternative cut points were tested. This might explain the seemingly larger variability in percent change scores. Third, it is important to note that pain is a multimodal construct, and other aspects such as quality of life and functional interference should be taken into account as well. But although these are only reported in subsets of studies, pain intensity is a core marker found in all studies.14 Fourth, the overall AUC values as well as sensitivity and specificity are somewhat smaller than those reported for adults. This may be due to the fact that Farrar and colleagues averaged several pain ratings to calculate pretreatment and posttreatment differences. Such an averaging most likely increases the reliability of the pain ratings and in turn enhances the probability of finding meaningful relations to other constructs. As averaging pretreatment or posttreatment pain ratings is unusual in studies of pediatric chronic pain, and studies into MCSDs, we believe that our methods give an accurate estimate of AUC that can be expected in typical studies. The main conclusion to be drawn from the present study is that raw changes of 1 point or more than 12.5% on the 0 to 10 NRS should be used as a criterion for clinically meaningful reduction in pain intensity. MCSDs for the intensity of chronic pain developed for adults are of little utility in adolescents and underestimate the number of patients that attain a meaningful improvement. On a more general level, we have shown that MCSDs are easy to calculate for a given sample but hard to integrate across different studies. A major problem in integrating the existing results is the lack of explicit assessment of variability in most studies. Instead of arguing for a complete overhaul of the idea of cut points,1 we suggest to improve on the methods we use to arrive at agreed-upon cut points. Specifically, we suggest using bootstrapping and the 5% criterion for the reporting and integration of MCSDs.
References
Kerns RD, Ader DN, Brandenburg N, Burke LB, Cella D, Chandler J, Cowan P, Dimitrova R, Dionne R, Hertz S, Jadad AR, Katz NP, Kehlet H, Kramer LD, Manning DC, McCormick C, McDermott MP, McQuay HJ, Patel S, Porter L, Quessy S, Rappaport BA, Rauschkolb C, Revicki DA, Rothman M, Schmader KE, Stacey BR, Stauffer JW, von Stein T, White RE, Witter J, Zavisic S: Interpreting the clinical importance of treatment outcomes in chronic pain clinical trials: IMMPACT recommendations. J Pain 9:105-121, 2008
1. Birnie KA, McGrath PJ, Chambers CT: When does pain matter? Acknowledging the subjectivity of clinical significance. Pain 153:2311-2314, 2012 2. Bulloch B, Tenenbein M: Assessment of clinically significant changes in acute pain in children. Acad Emerg Med 9: 199-202, 2002 3. Diaconis P, Efron B: Computer-intensive methods in statistics. Sci Am 248:116-130, 1983 4. Dworkin RH, Turk DC, Wyrwich KW, Beaton D, Cleeland CS, Farrar JT, Haythornthwaite JA, Jensen MP,
5. Eccleston C, Palermo TM, de C Williams A, Lewandowski A, Morley S, Fisher E, Law E: Psychological therapies for the management of chronic and recurrent pain in children and adolescents. Cochrane Database Syst Rev 12:CD003968, 2012
Hirschfeld et al 6. Farrar JT, Young JP Jr, LaMoreaux L, Werth JL, Poole RM: Clinical importance of changes in chronic pain intensity measured on an 11-point numerical pain rating scale. Pain 94:149-158, 2001 7. Hirschfeld G, Zernikow B: Cutpoints for mild, moderate, and severe pain on the VAS for children and adolescents— What can be learned from 10 million ANOVAs? Pain pii: S0304-3959(13)00290-X, 2013 8. Hirschfeld G, Zernikow B: Variability of ‘‘optimal’’ cut points for mild, moderate, and severe pain: Neglected problems when comparing groups. Pain 154:154-159, 2013 € bner B, Hechler T, Dobe M, Damschen U, Kosfelder J, 9. Hu Denecke H, Schroeder S, Zernikow B: [Pain-related disability in adolescents suffering from chronic pain. Preliminary examination of the Pediatric Pain Disability Index (P-PDI)]. Schmerz (Berlin, Germany) 23:20, 2009 10. Jensen MP, Smith DG, Ehde DM, Robinsin LR: Pain site and the effects of amputation pain: Further clarification of the meaning of mild, moderate, and severe pain. Pain 91: 317-322, 2001 11. McConahay T, Bryson M, Bulloch B: Clinically significant changes in acute pain in a pediatric ED using the Color Analog Scale. Am J Emerg Med 25:739-742, 2007 12. McGrath PJ, Walco GA, Turk DC, Dworkin RH, Brown MT, Davidson K, Eccleston C, Finley GA, Goldschneider K, Haverkos L: Core outcome domains and measures for pediatric acute and chronic/recurrent pain clinical trials: PedIMMPACT recommendations. J Pain 9:771-783, 2008 13. Neyman J, Pearson ES: On the use and interpretation of certain test criteria for purposes of statistical inference: Part I. Biometrika 20A:175-240, 1928 14. Palermo TM, Eccleston C, Lewandowski AS, Williams AC, Morley S: Randomized controlled trials of psychological thera-
The Journal of Pain
39
pies for management of chronic pain in children and adolescents: An updated meta-analytic review. Pain 148:387-397, 2010 15. Powell CV, Kelly AM, Williams A: Determining the minimum clinically significant difference in visual analog pain score for children. Ann Emerg Med 37:28-31, 2001 16. R Core Team: R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing Vienna Austria [Internet], 2012. Available at: http://www. R-project.org/ 17. Salaffi F, Stancati A, Silvestri CA, Ciapetti A, Grassi W: Minimal clinically important changes in chronic musculoskeletal pain intensity measured on a numerical rating scale. Eur J Pain 8:283-291, 2004 18. Serlin RC, Mendoza TR, Nakamura Y, Edwards KR, Cleeland CS: When is cancer pain mild, moderate or severe? Grading pain severity by its interference with function. Pain 61:277-284, 1995 19. Sing T, Sander O, Beerenwinkel N, Lengauer T: ROCR: Visualizing classifier performance in R. Bioinformatics 21: 3940-3941, 2005 20. Swets JA: Measuring the accuracy of diagnostic systems. Science 240:1285-1293, 1988 21. Tomlinson D, von Baeyer CL, Stinson JN, Sung L: A systematic review of faces scales for the self-report of pain intensity in children. Pediatrics 126:e1168-e1198, 2010 22. Voepel-Lewis T, Burke CN, Jeffreys N, Malviya S, Tait AR: Do 0–10 numeric rating scores translate into clinically meaningful pain measures for children? Anesth Analg 112:415-421, 2011 23. von Baeyer CL, Spagrud LJ, McCormick JC, Choo E, Neville K, Connelly MA: Three new datasets supporting use of the Numerical Rating Scale (NRS-11) for children’s selfreports of pain intensity. Pain 143:223-227, 2009
Minimally Clinically Significant Differences for Adolescents With Chronic Pain—Variability of ROC-Based Cut Points Gerrit Hirschfeld, Julia Wager, Pia Schmidt, and Boris Zernikow German Paediatric Pain Centre, Children’s and Adolescents’ Hospital, Datteln, Germany; and Department of Children’s Pain Therapy and Paediatric Palliative Care, School of Medicine, Faculty of Health, Witten/Herdecke University, Witten, North Rhine-Westphalia, Germany.
Abstract: Assessing if an individual patient has achieved clinically meaningful changes in pain intensity is a core aspect in the evaluation of pain treatments. The aim of the present study was to define minimally clinically significant differences (MCSDs) for the numerical rating scale (0–10 NRS) in adolescents with chronic pain. Data from 153 adolescents who completed an inpatient treatment were analyzed. MCSDs were defined as those cut points that yielded an optimal balance between sensitivity and specificity with regard to patients’ global impression of change. The variability of the empirically defined cut points was quantified using bootstrapping. Our results show that raw changes of 1 NRS point and percent changes of 12.5% can be considered MCSDs both within the full sample and within various subsamples of patients. Applying the MCSDs developed for adults to pediatric patients yielded extremely low sensitivities; for example, only 22% of the children who described global improvement met the 50% decrease in pain criterion. Studies evaluating chronic pain treatments for adolescents should use MCSDs that are specifically developed for this group of patients. Raw changes of 1 point and 12.5% on the 0 to 10 NRS should be considered clinically meaningful. On a methodological level, we call for more systematic studies aimed at defining MCSDs that also address the variability of cut point estimates so as to foster the integration of findings. Perspective: Many studies are aimed at empirically defining cut points for clinically relevant pain using receiver operating characteristic-based methods. For the first time, we apply these methods to children and show that even when taking into account the variability of the method, cut points specific for children are needed. ª 2014 by the American Pain Society Key words: Pain, optimal cut points, receiver operating characteristic, bootstrap.
A
core aspect in the evaluation of pain therapies is the assessment whether a change in pain intensity can be considered clinically meaningful.12 Such an assessment requires reliable measures for pain intensity and standards for their interpretation, that is, cut points that define minimally clinically significant differences
Received February 15, 2013; Revised August 1, 2013; Accepted September 13, 2013. No funding was received for this study. The authors declare that they have no conflict of interest, including specific financial interests, relationships, or affiliations relevant to the study. Address reprint requests to Gerrit Hirschfeld, PhD, German Paediatric Pain Centre and Department of Children’s Pain Therapy and Paediatric Palliative Care, Children’s and Adolescents’ Hospital, Datteln, Witten/ Herdecke University, Dr.-Friedrich-Steiner Str. 5, 45711 Datteln, Germany. E-mail: [email protected] 1526-5900/$36.00 ª 2014 by the American Pain Society http://dx.doi.org/10.1016/j.jpain.2013.09.006
32
(MCSDs). The reliability and validity of scales such as the 11-point numerical rating scale (0–10 NRS)23 or faces pain scale21 has been established in children and adolescents, but there are no agreed-upon MCSDs for use in pediatric chronic pain. Following the landmark study by Farrar and colleagues,6 patients’ global impressions of change (PGIC) are used as anchors and evaluated against possible cut points by means of receiver operating characteristic (ROC) methods. This allows defining those cut points as MCSD that best differentiate between patients who report that their pain intensity has improved and patients who report that their pain intensity is unchanged or worsened. In the present paper, an ROC method is utilized for the first time to establish MCDS for the 0 to 10 NRS for adolescents with chronic pain. Previous research has identified conflicting standards to interpret change on the 0 to 10 NRS for adolescents.
Hirschfeld et al
Methods to Define Minimally Clinically Significant Change Used in the Farrar et al Study on a Pooled Adult Sample and Studies in Children and Adolescents
Table 1.
STUDY FIRST AUTHOR, YEAR 6
SAMPLE
SCALES
SETTING
PAIN
METHODS
PGIC
TARGET PGIC CATEGORIES
STATISTIC
MCSD
‘‘Much improved’’ or ‘‘very much improved’’ vs other categories Comparison between ‘‘a bit better’’ and ‘‘a bit worse’’
ROC (balance between sensitivity and specificity) for raw and percent change Difference between changes in ‘‘bit better’’ and ‘‘bit worse’’ groups Median of pre – post differences Median (and mean) of pre–post differences Mean of pre–post differences
1.74/30%
Farrar, 2001
Adults; n = 2,724
Pregabalin in diabetic neuropathy
NRS (0–10)
7-point: ‘‘very much improved’’ to ‘‘very much worse’’
Powell, 200115
Adolescents (8–15 years); n = 73
Emergency department
VAS (100 mm)
5-point: ‘‘heaps better’’ to ‘‘heaps worse’’
Bulloch, 20022
Adolescents (9.8 6 3.15 years); n = 121
Emergency department
FPS (0–6) CAS (100 mm)
5-point: ‘‘much less’’ to ‘‘much worse’’
Adolescents (5–12 years); n = 126 Adolescents (7–16 years); n = 113
Emergency department
CAS (100 mm)
5-point: ‘‘much less’’ to ‘‘much worse’’
Mean for each category of PGIC
Postoperatively
NRS (0–10)
5-point: ‘‘a lot better’’ to ‘‘a lot worse’’
Comparison between ‘‘a little better’’ and ‘‘a little worse’’
McConahay, 200711 Voepel-Lewis, 201222
‘‘Little less’’ pain
Mean
10 mm
1 face 20 mm (17 mm) 24 mm
1
Abbreviations: VAS, visual analog scale; FPS, faces pain scale; CAS, color analog scale.
The Journal of Pain
33
34
MCSD in Pediatric Chronic Pain
The Journal of Pain
Table 2.
Core Demographic Characteristics VARIABLE
Age (mean [SD]) Gender (female n (%]) Main location (n [%]) Head Back/Extremities Abdomen Days missed school in past 3 month (mean [SD]) Pediatric pain disability index (range 12–60) (mean [SD])
VALUE 15.54 (1.83) 115 (75%) 85 (56%) 39 (25%) 29 (19%) 15 days (21) 37 (9)
Four previous studies aimed to empirically establish MCSDs for pain intensity in adolescents with acute pain in the emergency department or postoperatively.2,11,15,22 In addition to using a variety of methods to assess pain intensity, they also differed in their use and interpretation of the global impression of change and statistical procedure (see Table 1). A recent study suggested a change of 1 on the 0 to 10 NRS as an MCSD for adolescents with acute pain,22 which is considerably different from the 1.74 cut point suggested for adults with chronic pain.6 Two problems make it hard to apply existing ‘‘pediatric’’ MCSDs to pediatric chronic pain. First, the pediatric studies all dealt with acute pain and changes within several hours. Second, these studies did not access sensitivity or specificity of the changes in pain intensity with regard to PGIC. As a result, it is unclear how well these cut points are able to differentiate between improved and unimproved pediatric chronic pain patients. The main aim of the present study was to define MCSDs for adolescents with chronic pain using the same methodology that was used to define MCSDs in adults. In addition to reporting the MCSDs as raw changes, we also calculated MCSDs for percent change as these seem more stable.4 Our study also had 2 secondary aims. First, we wanted to test whether possible deviations between the MCSDs for pediatric22 and adult6 samples are due to chance fluctuations or reflect reliable differences. Second, we wanted to test whether MCSDs are also feasible in subgroups of patients with different temporal pain qualities (constant vs intermittent) and patients with the same pain location (head vs abdomen vs musculoskeletal).
local ethics committee of the university hospital. Of all 204 patients, 51 (25%) had to be excluded because they did not provide the pain intensity or PGIC ratings after completion of treatment necessary for the analysis. Core demographic information for the 153 adolescents is given in Table 2.
Measures The 2 variables central to this paper were pain intensity and PGIC. Pain intensity ratings were collected at admission and on discharge. At both time points, children were asked to indicate their worst pain in the preceding 7 days (‘‘When you were experiencing the main pain, how strong was this pain mostly during the past 7 days?’’ [‘‘Wenn du deine Hauptschmerzen hattest, wie stark waren diese Schmerzen dann meistens in den letzten 7 Tagen’’]) on the 11-point NRS. The reliability and validity of this scale is well established in adolescents.23 PGIC in pain intensity were assessed at discharge. Five possible response categories were given, ranging from ‘‘much more intense pain’’ to ‘‘much less intense pain.’’ In order to describe the sample and perform additional subgroup analyses, we also collected data on the main pain location, constant pain, days absent from school, and pain-related disability.9
Data Analysis The data analysis for raw changes (pre – post) and percent changes ([pre – post]/pre) proceeded in 4 steps: describing pain intensity and PGIC ratings, defining MCSDs, performing bootstrap, and analyzing subgroups. First, change from pre- to posttreatment 0 to 10 NRS was calculated for each patient. These differences were
Methods Subjects In this study, data from all adolescents who completed a 3-week multimodal inpatient pain therapy at the German Paediatric Pain Centre in 2012 were retrospectively analyzed. This is a secondary analysis of an ongoing project to assess the effectiveness of the treatment. Patients as well as their parents provided written informed consent for data collection, electronic data storage, and data analysis when they first visited the German Paediatric Pain Centre as approved by the
Figure 1. NRS change from baseline to endpoint for raw change (left) and percent change (right). Error bars indicate 95% CI.
Hirschfeld et al
The Journal of Pain
35
Figure 2. ROC curves for NRS changes for raw change (left) and percent change (right). Broken lines represent chance classification. averaged within the 5 PGIC categories and plotted together with their confidence interval. Second, the MCSDs were defined empirically using the ROC-based method suggested by Farrar and colleagues.6 Specifically, the 2 positive PGIC categories (ie, improved, much improved) were combined into a single ‘‘meaningful change category.’’ The area under the curve (AUC) was calculated to estimate the overall accuracy of the change scores with regard to this category. AUC values are interpreted according to the standards developed by Swets20; values between .5 and .7 are considered ‘‘rather low’’; values between .7 and .9 are considered ‘‘useful for some purposes’’; and values higher than .9 are considered ‘‘rather high.’’ The sensitivity and specificity for all possible cut points in 0 to 10 NRS changes were computed, and the cut point associated with the smallest absolute difference between sensitivity and specificity was defined as MCSD. Third, the variability of MCSDs was quantified in order to test whether these are systematically different from the MCSDs developed for adults. Specifically, a nonparametric bootstrap with 1,000 repetitions was used to yield sufficiently stable estimates for the vari-
ability of MCSD.3 For each of the 1,000 randomly chosen pseudo-samples, the MCSDs were determined using the ROC-based method described above. For each pseudo-sample, MCSDs were recorded. The resulting distributions of MCSDs were inspected for feasible values with the help of descriptive statistics using 5% as a cut point. Fourth, the bootstrapping analysis was replicated in homogeneous subgroups of patients, that is, patients with the same temporal pain quality (constant vs intermittent) and patients with the same pain location. This was motivated by research into optimal cut points for mild, moderate, or severe pain that found several alternative cut points for groups of patients with different pain locations.8 For example, Jensen and colleagues found different cut points for phantom limb vs back pain.10 Importantly, differences between groups may be found once the variability of the optimal cut points is taken into account.7 The software R16 and the ROCR package19 was used for data analysis.
Performance of Different Cut Points for Raw Change
CUT POINT
Table 3. CUT POINT 10 8 7 6 5 4 3 2 1* 0 1 2 3 5
SENSITIVITY
SPECIFICITY
.01 .02 .04 .06 .12 .21 .32 .45 .71 .88 .97 .98 .99 1
.98 .98 .98 .95 .95 .95 .89 .82 .63 .28 .07 .04 0 0
NOTE. Positive cut points indicate improvement. *Optimal cut point according to the criterion of equal sensitivity and specificity.
Table 4. Performance of Different Cut Points for Percent Change
50 44.44 42.86 40 37.5 33.33 28.57 25 22.22 20 16.67 14.29 12.5* 11.11 10 0
SENSITIVITY
SPECIFICITY
.22 .23 .28 .29 .31 .36 .41 .42 .44 .51 .55 .62 .68 .7 .71 .88
.95 .95 .95 .95 .89 .89 .89 .81 .81 .79 .74 .72 .65 .63 .63 .28
NOTE. Positive cut points indicate improvement. In order to save space, extreme cut points (cut points > 50% and cut points < 0%) were truncated. *Optimal cut point according to the criterion of equal sensitivity and specificity.
36
The Journal of Pain
MCSD in Pediatric Chronic Pain
Figure 3. Frequency at which the cut points were identified as optimal for raw change (left) and percent change (right). Broken lines represent 5% of the pseudo-samples.
Results
Definition of MCSDs
At discharge most patients reported that their pain was ‘‘improved’’ (n = 68; 44%) or ‘‘much improved’’ (n = 28; 18%), some indicated that their pain did not change (n = 50; 33%), and very few indicated that their pain became ‘‘worse’’ (n = 7; 5%). On average, the raw 0 to 10 NRS reduced by 1.29 6 2.35 raw points and 15.74 6 39.30%. The levels of change in the 0 to 10 NRS for each PGIC categories are presented in Fig 1 for the change in raw points (left) and percent (right).
Raw and percent changes in the 0 to 10 NRS were systematically related to PGIC categories (Fig 2). The overall accuracy of the change scores as indexed by the AUC was .70 (95% confidence interval [CI] = .61–.77) for raw change and .70 (95% CI = .61–.78) for percent change, indicating that changes in 0 to 10 NRS yield predictions that are ‘‘useful for some purposes.’’20 Inspecting the possible cut points for the raw change showed that a difference score of 1 yielded the best balance between sensitivity (.71) and specificity (.63; Table 3). That is, only raw changes that are 1 point or larger can be considered clinically meaningful. Using 2 as a cut point resulted in considerably lower sensitivity (.45) and higher specificity (.82).
Figure 4. Results of the bootstrap analysis for subgroups with
Figure 5. Results of the bootstrap analysis for subgroups with
constant vs intermittent pain. Frequency at which the cut points were identified as optimal for raw change (left) and percent change (right). Broken lines represent 5% of the pseudo-samples.
different pain locations (head, extremities, abdomen). Frequency at which the cut points were identified as optimal for raw change (left) and percent change (right). Broken lines represent 5% of the pseudo-samples.
Description of Pain Intensity and PGIC Ratings
Hirschfeld et al With regard to percent change, a difference of 12.5 yielded the best balance between sensitivity (.68) and specificity (.65; Table 4). That is, only changes that are 12.5% or larger can be considered clinically meaningful. Using 50% as a cut point resulted in very low sensitivity (.22) and high specificity (.95) compared to adolescents’ PGIC.
Variability of MCSDs The analysis of the results in the bootstrapped pseudosamples revealed that MCSDs for raw changes were identified more consistently than MCSDs for percent changes. The cut point of 1 on the 0 to 10 NRS that was identified as MCSD in the whole sample was also identified in 986 of the total 1,000 pseudo-samples, and the alternative cut point 2 was identified in the remaining 14 pseudosamples (Fig 3, left). The results for the percent change showed a larger variability (Fig 3 right); the cut point 12.5% was identified as MCSD in 432 of the pseudosamples. Several other cut points were also identified in more than 50 of the pseudo-samples; specifically, the 3 cut points of 10%, 11.1%, and 14.3% were identified as MCSDs in 116, 125, and 299 of the pseudo-samples.
Results in Homogeneous Subgroups We repeated these analysis in several subgroups, the first by frequency (constant vs intermittent pain) and the second by main pain location (head vs abdomen vs extremities). The analysis for patients with constant vs intermittent pain showed that the raw change of 1 point was most frequently identified as MCSD in both subgroups (Fig 4). However, for adolescents with intermittent pain, a cut point of 2 was also identified as MCSD in more than 5% of the pseudo-samples. For the percent change, the results also showed less stable MCSD estimations for adolescents with intermittent pain. For adolescents with constant pain, the MCSDs ranged between 10% and 14.3%, whereas for adolescents with intermittent pain, values between 0% and 25% were identified as MCSDs. The analysis of subgroups based on main pain location showed large agreement between adolescents with chronic headache and musculoskeletal pain (Fig 5). For these 2 groups, 1 was identified as MCSD in most pseudo-samples. In adolescents with abdominal pain, 2 was identified as MCSD in the majority (70%) of the pseudo-samples, whereas 1 was identified as MCSD in only 29% of the pseudo-samples. The MCSD for percent change also indicates that the most frequent MCSD for adolescents with headache or musculoskeletal pain was 12.5% whereas the most frequent MCSD for adolescents with abdominal pain was 20%. Again the 12.5% cut point was also identified as MCSD in more than 5% of the pseudo-samples.
Discussion The present study investigated empirically defined MCSDs for pain intensity measured with the 0 to 10 NRS in adolescents with chronic pain after inpatient
The Journal of Pain
37
treatment. Using a methodology similar to the one established in adults, we identified 1 and 12.5% as MCSDs for adolescents with chronic pain. Most importantly, these are lower than the cut points developed for adults with chronic pain6 but similar to those defined for adolescents with acute pain.22 This was also true when taking into account the variability of this procedure using bootstrapping. Furthermore, MCSDs that were developed for adults with chronic pain, that is, reduction by 1.74 raw points and 30%,6 result in low detection rates and underestimate the effectiveness of treatments in children. Comparison of homogeneous subgroups revealed that different cut points were identified as optimal, but the MCSDs developed for the whole sample were feasible in all different subgroups. Thus, we argue for using ‘‘pediatric’’ MCSDs that are different from the ‘‘adult’’ MCSDs but similar in diverse pediatric chronic pain conditions. In the following, we discuss the lack of standardization of procedures to establish cut points for meaningful change and the relevance of estimating the variability of cut points. Our results demonstrate that specialized cut points need to be utilized when interpreting changes in pain ratings by children with chronic pain. Use of the cut points for raw and percent change developed for adolescents yielded very low sensitivities. That is, the majority of patients who after therapy describe their pain as ‘‘less intense’’ or ‘‘much less intense’’ do not meet the criteria developed for adults. Furthermore, when taking into account the variability of the method to define MCSDs, we were able to show that these differences are not due to chance fluctuations but rather reflect real differences between the groups. Our results agree with a recent study on MCSDs for acute pain in children and adolescents.22 In contrast to this, most primary research papers and reviews rely on 50% pain reduction as a criterion for improvement.5 The IMMPACT guideline has defined 10–20% decrease as minimally important, 30% as moderately important, and 50% as substantial4 but offers no recommendation for children. The PedIMMPACT guideline12 does not recommend a specific cut point for pain intensity. In light of the present data on MCSD for chronic pain and earlier results of MCSDs for acute pain in pediatric samples,22 it seems prudent to adopt these guidelines. At present, it is unclear what exactly brings about these differences in comparison to the initial study by Farrar et al. In addition to the different age, the underlying disease and treatment differed. Most patients in the Farrar et al study had neuropathic pain, whereas the majority of patients in the present sample had headache or musculoskeletal pain.6 Looking closely at the reductions in pain intensity for different underlying diseases in the Farrar et al study (see Fig 2), it seems that, for example, patients with osteoarthritis who reported improvement reduced their pain ratings much less than patients with peripheral diabetic neuropathies. In line with this, a recent study that analyzed data from patients with osteoarthritis found that 1 point is a reasonable MCSD for this group.17 Second, the type of treatment (analgesic vs psychotherapy) may also impact the expected
38
The Journal of Pain
MCSD in Pediatric Chronic Pain
treatment effect, which serves as an internal standard against which changes are evaluated. It seems likely that patients who take an active role in their treatment are more likely to perceive small improvements in pain intensity as meaningful. Clearly more research is needed to untangle the different factors that contribute to differences in MCSDs. The inspection of different subgroups of adolescents with chronic pain revealed that the MCSDs emerged with slightly different frequencies. For example, patients with constant pain seem to exhibit more stable MCSDs compared to those with intermittent pain. However, in a study of mild, moderate, and severe pain, a group with constant pain also exhibited highly variable cut points.8 Thus, it is unlikely that systematic differences underlie these differences in MCSDs. If one were interested in the relation between PGIC and changes in pain intensity ratings, a correlational analysis would be more adept than an analysis of MCSDs. For us, the subgroup analysis indicated that bootstrapping supports the idea that the MCSDs defined for the whole group are also feasible in all subgroups. On a methodological level, we call for taking into account the variability of the methods to define optimal cut points. Research into mild, moderate, and severe pain has already demonstrated that differences in optimal cut points may be due to chance fluctuations.8,18 Even though those studies use analyses of variance to define optimal cut points, similar problems arise with the ROCbased methods that have been used to define MCSDs. We suggest a 5% criterion to judge whether an existing cut point is feasible for a subgroup of patients. That is, as long as a preexisting cut point is found in 5% of the bootstrapped pseudo-samples, the data should be interpreted as supporting the preexisting cut point. It is important to note that this is procedure is highly conservative in the sense that it makes it hard to establish specialized cut points. We believe that this is in line with the logic of null hypothesis significance testing13 with the associated alpha level of 5%, according to which the null hypothesis is rejected only if less than 5% of the samples would yield a result more extreme than the one found. This bias against novel results is an important safeguard against supporting novel cut points. We believe that such a strategy may be a core aspect toward the harmonization of the interpretation for meaningful change.
Several limitations need to be taken into account when interpreting the results of this study. First, the sample size was relatively small compared to studies in adults.6 This may affect the variability of the cut points and calls for a replication in larger samples. Second, even though we tried to replicate the methods used in adults as closely as possible, we did not collect daily pain ratings, which could have been averaged, but rather asked for a weekly average. Hence, only rounded difference scores are possible so that fewer alternative cut points were tested. This might explain the seemingly larger variability in percent change scores. Third, it is important to note that pain is a multimodal construct, and other aspects such as quality of life and functional interference should be taken into account as well. But although these are only reported in subsets of studies, pain intensity is a core marker found in all studies.14 Fourth, the overall AUC values as well as sensitivity and specificity are somewhat smaller than those reported for adults. This may be due to the fact that Farrar and colleagues averaged several pain ratings to calculate pretreatment and posttreatment differences. Such an averaging most likely increases the reliability of the pain ratings and in turn enhances the probability of finding meaningful relations to other constructs. As averaging pretreatment or posttreatment pain ratings is unusual in studies of pediatric chronic pain, and studies into MCSDs, we believe that our methods give an accurate estimate of AUC that can be expected in typical studies. The main conclusion to be drawn from the present study is that raw changes of 1 point or more than 12.5% on the 0 to 10 NRS should be used as a criterion for clinically meaningful reduction in pain intensity. MCSDs for the intensity of chronic pain developed for adults are of little utility in adolescents and underestimate the number of patients that attain a meaningful improvement. On a more general level, we have shown that MCSDs are easy to calculate for a given sample but hard to integrate across different studies. A major problem in integrating the existing results is the lack of explicit assessment of variability in most studies. Instead of arguing for a complete overhaul of the idea of cut points,1 we suggest to improve on the methods we use to arrive at agreed-upon cut points. Specifically, we suggest using bootstrapping and the 5% criterion for the reporting and integration of MCSDs.
References
Kerns RD, Ader DN, Brandenburg N, Burke LB, Cella D, Chandler J, Cowan P, Dimitrova R, Dionne R, Hertz S, Jadad AR, Katz NP, Kehlet H, Kramer LD, Manning DC, McCormick C, McDermott MP, McQuay HJ, Patel S, Porter L, Quessy S, Rappaport BA, Rauschkolb C, Revicki DA, Rothman M, Schmader KE, Stacey BR, Stauffer JW, von Stein T, White RE, Witter J, Zavisic S: Interpreting the clinical importance of treatment outcomes in chronic pain clinical trials: IMMPACT recommendations. J Pain 9:105-121, 2008
1. Birnie KA, McGrath PJ, Chambers CT: When does pain matter? Acknowledging the subjectivity of clinical significance. Pain 153:2311-2314, 2012 2. Bulloch B, Tenenbein M: Assessment of clinically significant changes in acute pain in children. Acad Emerg Med 9: 199-202, 2002 3. Diaconis P, Efron B: Computer-intensive methods in statistics. Sci Am 248:116-130, 1983 4. Dworkin RH, Turk DC, Wyrwich KW, Beaton D, Cleeland CS, Farrar JT, Haythornthwaite JA, Jensen MP,
5. Eccleston C, Palermo TM, de C Williams A, Lewandowski A, Morley S, Fisher E, Law E: Psychological therapies for the management of chronic and recurrent pain in children and adolescents. Cochrane Database Syst Rev 12:CD003968, 2012
Hirschfeld et al 6. Farrar JT, Young JP Jr, LaMoreaux L, Werth JL, Poole RM: Clinical importance of changes in chronic pain intensity measured on an 11-point numerical pain rating scale. Pain 94:149-158, 2001 7. Hirschfeld G, Zernikow B: Cutpoints for mild, moderate, and severe pain on the VAS for children and adolescents— What can be learned from 10 million ANOVAs? Pain pii: S0304-3959(13)00290-X, 2013 8. Hirschfeld G, Zernikow B: Variability of ‘‘optimal’’ cut points for mild, moderate, and severe pain: Neglected problems when comparing groups. Pain 154:154-159, 2013 € bner B, Hechler T, Dobe M, Damschen U, Kosfelder J, 9. Hu Denecke H, Schroeder S, Zernikow B: [Pain-related disability in adolescents suffering from chronic pain. Preliminary examination of the Pediatric Pain Disability Index (P-PDI)]. Schmerz (Berlin, Germany) 23:20, 2009 10. Jensen MP, Smith DG, Ehde DM, Robinsin LR: Pain site and the effects of amputation pain: Further clarification of the meaning of mild, moderate, and severe pain. Pain 91: 317-322, 2001 11. McConahay T, Bryson M, Bulloch B: Clinically significant changes in acute pain in a pediatric ED using the Color Analog Scale. Am J Emerg Med 25:739-742, 2007 12. McGrath PJ, Walco GA, Turk DC, Dworkin RH, Brown MT, Davidson K, Eccleston C, Finley GA, Goldschneider K, Haverkos L: Core outcome domains and measures for pediatric acute and chronic/recurrent pain clinical trials: PedIMMPACT recommendations. J Pain 9:771-783, 2008 13. Neyman J, Pearson ES: On the use and interpretation of certain test criteria for purposes of statistical inference: Part I. Biometrika 20A:175-240, 1928 14. Palermo TM, Eccleston C, Lewandowski AS, Williams AC, Morley S: Randomized controlled trials of psychological thera-
The Journal of Pain
39
pies for management of chronic pain in children and adolescents: An updated meta-analytic review. Pain 148:387-397, 2010 15. Powell CV, Kelly AM, Williams A: Determining the minimum clinically significant difference in visual analog pain score for children. Ann Emerg Med 37:28-31, 2001 16. R Core Team: R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing Vienna Austria [Internet], 2012. Available at: http://www. R-project.org/ 17. Salaffi F, Stancati A, Silvestri CA, Ciapetti A, Grassi W: Minimal clinically important changes in chronic musculoskeletal pain intensity measured on a numerical rating scale. Eur J Pain 8:283-291, 2004 18. Serlin RC, Mendoza TR, Nakamura Y, Edwards KR, Cleeland CS: When is cancer pain mild, moderate or severe? Grading pain severity by its interference with function. Pain 61:277-284, 1995 19. Sing T, Sander O, Beerenwinkel N, Lengauer T: ROCR: Visualizing classifier performance in R. Bioinformatics 21: 3940-3941, 2005 20. Swets JA: Measuring the accuracy of diagnostic systems. Science 240:1285-1293, 1988 21. Tomlinson D, von Baeyer CL, Stinson JN, Sung L: A systematic review of faces scales for the self-report of pain intensity in children. Pediatrics 126:e1168-e1198, 2010 22. Voepel-Lewis T, Burke CN, Jeffreys N, Malviya S, Tait AR: Do 0–10 numeric rating scores translate into clinically meaningful pain measures for children? Anesth Analg 112:415-421, 2011 23. von Baeyer CL, Spagrud LJ, McCormick JC, Choo E, Neville K, Connelly MA: Three new datasets supporting use of the Numerical Rating Scale (NRS-11) for children’s selfreports of pain intensity. Pain 143:223-227, 2009
