Downloaded from ard.bmj.com on September 27, 2014 - Published by group.bmj.com
Clinical and epidemiological research
CONCISE REPORT
Defining criteria for high disease activity in juvenile idiopathic arthritis based on the Juvenile Arthritis Disease Activity Score Alessandro Consolaro,1 Nicolino Ruperto,1 Giulia Bracciolini,1 Alessia Frisina,1 Maria Chiara Gallo,1 Angela Pistorio,1 Sara Verazza,1 Giorgia Negro,1 Valeria Gerloni,2 Claudia Goldenstein-Schainberg,3 Flavio Sztajnbok,4 Nico M Wulffraat,5 Alberto Martini,1,6 Angelo Ravelli,1,6 for the Paediatric Rheumatology International Trials Organization (PRINTO) Handling editor Tore K Kvien ▸ Additional material is published online only. To view please visit the journal online (http://dx.doi.org/10.1136/ annrheumdis-2013-204186). 1
Istituto Giannina Gaslini, Genova, Italy Istituto Gaetano Pini, Milano, Italy 3 Faculdade de Medicina da Universidade de São Paulo, São Paulo/SP, Brazil 4 Universidade do Estado do Rio de Janeiro, Brazil 5 Wilhelmina Children’s Hospital, University Medical Centre Utrecht, Utrecht, The Netherlands 6 Università degli Studi di Genova, Genova, Italy 2
Correspondence to Dr Alessandro Consolaro, Pediatria II, Istituto G Gaslini, Largo G, Gaslini 5, Genova 16147, Italy; alessandroconsolaro@ ospedale-gaslini.ge.it Received 27 June 2013 Revised 8 November 2013 Accepted 3 December 2013 Published Online First 17 December 2013
ABSTRACT Objective To determine cutoff values for defining the state of high disease activity (HDA) in juvenile idiopathic arthritis (JIA) using the Juvenile Arthritis Disease Activity Score (JADAS). Methods For the selection of cutoff values, data from a clinical database including 609 patients were used. Optimal cutoff values were determined against external criteria by calculating the 25th and 10th centile of cumulative score distribution and through receiver operating characteristic curve analysis. External criteria were based on the therapeutic decision made by the attending doctor. Cross-validation was performed using five patient samples that included 1421 patients. Results The optimal cutoff values were those obtained through the 90% fixed sensitivity method. The selected JADAS cutoff values were the following: 4.2 and 8.5 for JADAS27 in oligoarthritis and polyarthritis, respectively; 4.2 and 10.5 for both JADAS10 and JADAS71 in oligoarthritis and polyarthritis, respectively. In crossvalidation analyses, the cutoff values showed strong ability to discriminate between different levels of American College of Rheumatology paediatric response in two clinical trials and could predict worse functional and radiographic outcome. Conclusions Cutoff values for classifying HDA in JIA using the JADAS were developed. In cross-validation analyses, they proved to have good construct and discriminant validity and ability to predict disease outcome.
PATIENTS AND METHODS JADAS calculation The JADAS is calculated as the arithmetic sum of the scores of the following variables: physician global rating of disease activity; parent/child rating of well-being; active joint count, assessed in 71, 27, or 10 joints (JADAS71, JADAS27, and JADAS10, respectively); and erythrocyte sedimentation rate, normalised to a 0–10 scale.2
Study population used for selection of cutoff values For the selection of cutoff values, we used data from 609 children with JIA3 who had undergone 1814 visits between 2007 and 2009.4 5 Children with systemic arthritis (n=46), rheumatoid factorpositive (n=9) and -negative (n=138) polyarthritis or extended oligoarthritis (n=96) were included in the polyarthritis group. The oligoarthritis group included children with persistent oligoarthritis (n=267). Children classified in other JIA categories (n=53) were assigned to the polyarthritis or oligoarthritis group based on the number of affected joints (≤4 or >4, respectively). Children with systemic JIA and active systemic manifestations were excluded. The study protocol was approved by the Istituto G Gaslini institutional review board.
Definition of HDA state INTRODUCTION
To cite: Consolaro A, Ruperto N, Bracciolini G, et al. Ann Rheum Dis 2014;73:1380–1383. 1380
Assessment of disease activity is a fundamental component of the clinical evaluation of children with juvenile idiopathic arthritis (JIA) because persistently active disease plays a major role in causing joint damage and physical functional disability.1 Recently, the first composite disease activity score for JIA, the Juvenile Arthritis Disease Activity Score (JADAS), was reported.2 To facilitate interpretation of JADAS scores, criteria (ie, cutoff values) are needed for identifying high and low levels of JIA activity. The purpose of this study was to determine and validate cutoff values in the JADAS that correspond to the state of high disease activity (HDA).
All patients’ visits were examined to identify those patients who met the criteria for HDA. To ensure face validity of the criteria, it was decided that they had to be based on the therapeutic decision made by the attending paediatric rheumatologist. The criteria for HDA in oligoarthritis were (a) intra-articular corticosteroid (IAC) administration in ≥1 joint; (b) start of a disease-modifying antirheumatic drug or (c) start of a biological agent. The criteria for HDA in polyarthritis were: (a) IAC administration in ≥3 joints; (b) start of a diseasemodifying antirheumatic drug; (c) start of a biological agent or (d) start of systemic corticosteroid therapy. All visits which did not meet the criteria for HDA were considered as reflecting low/moderate disease activity.
Consolaro A, et al. Ann Rheum Dis 2014;73:1380–1383. doi:10.1136/annrheumdis-2013-204186
Downloaded from ard.bmj.com on September 27, 2014 - Published by group.bmj.com
Clinical and epidemiological research Study populations used for cross-validation of cutoff values Five patient samples (not used in calculation of cutoff values) were used to cross-validate the cutoff values. The first and second samples comprised patients enrolled in controlled trials of methotrexate (MTX) (n=595)6 and abatacept (n=190).7 The third sample included 175 patients seen between 2009 and 2012 and followed up for at least 6 months.8 The fourth sample included 358 patients who underwent two or more visits between 1997 and 2002.9 The fifth sample consisted of 60 of 103 patients included in a study on the paediatric adaptation of the Sharp/van der Heijde score.10
RESULTS JADAS value by treatment decision The JADAS10 values by therapeutic intervention are presented in online supplementary table S1.
Selection of the optimal cutoff values for classification of HDA
Optimal cutoff values were determined against external criteria (ie, the states of HDA, defined as above) by calculating the 25th and 10th centiles of score distribution and through receiver operating characteristic (ROC) analysis. In ROC analysis, three methods were applied: (1) the closest point to (0,1)—that is, the point where the shoulder of the ROC curve is closest to the left upper corner of the graphic; (2) the fixed 90% specificity and (3) the fixed 90% sensitivity. Only one observation for each person was used in selection and validation of cutoff values.
The cutoff values obtained through the various statistical approaches are presented in table 1. In selecting the final cutoff values, we reasoned that since the ‘gold standard’ was HDA, in order to reduce the risk of misclassifying patients whose disease was active, more importance should be given to sensitivity—that is, to the proportion of patients with active disease who are correctly classified. However, a minimum specificity of 75% was required to minimise the rate of misclassification of patients with low/moderate disease activity as having HDA. Based on these considerations, it was felt that the optimal cutoff values were those provided by the 90% fixed sensitivity method. Notably, all area under the curves were >0.90, which reflects, according to a rule of thumb, ‘outstanding discrimination’.16 The graphs of ROC curves are illustrated in online supplementary figure S1.
Cross-validation study
Results of cross-validation analyses
Cross-validation of cutoff values was based on assessment of construct, discriminant and predictive validity through the following approaches. (1) Calculation of the percentage of patients who had a JADAS above the HDA cutoff values in clinical trials in relation to level of improvement according to the American College of Rheumatology paediatric criteria11 in the MTX6 and abatacept7 trials. (2) Calculation of the percentage of patients who had a JADAS above the HDA cutoff values in relation to subjective assessment of disease states5 12 by doctors and parents. (3) Assessment of the ability of cutoff values to predict lack of attainment of inactive disease13 and achievement of worse functional status on the Childhood Health Assessment Questionnaire (CHAQ).14 15 (4) Assessment of the ability of cutoff values to predict progression of radiographic joint damage.10 Quantitative and percentage data were compared by Mann– Whitney U test and χ2 test, respectively. The statistical packages used were Statistica (StatSoft) and Stata, V.7 (StataCorp).
In the MTX and abatacept trial samples, the proportion of patients with JADAS above the HDA cutoff value was greatest among non-responders and lowest among those with 70% improvement (figure 1 and online supplementary figure S2). In the third sample, the percentage of patients with JADAS above the cutoff value was greater among patients judged as having active disease and flare or as not being in an acceptable symptom state, and lower among patients judged as being in remission or in an acceptable symptom state (figure 2 and online supplementary figure S3). In the fourth sample, the percentage of patients with inactive disease or with a CHAQ score of 0 at the final visit was lower among patients who had a JADAS above the cutoff value at the first visit than among those who did not (results not shown). In the fifth sample, the adapted Sharp/van der Heijde score at 3 years was higher among patients who had JADAS above the cutoff values at least twice during the 3 years of observation than in those who did not (online supplementary figure S4).
Statistical analyses used for cutoff selection
Table 1 JADAS cutoff values for classification of patients into high disease activity state according to five different methods for determining optimal cut-off points Methods for determination of optimal cutoff values* 25th Centile Oligoarthritis† JADAS10 JADAS27 JADAS71 Polyarthritis† JADAS10 JADAS27 JADAS71
10th Centile
(0,1)
90% Specificity
90% Sensitivity
AUC (95% CI)
7.9 (75.2/95.1) 7.2 (76.1/94.6) 7.9 (75.2/95.1)
4.5 (89.4/88.8) 4.5 (89.4/89.8) 4.5 (89.4/88.8)
4.9 (89.4/89.3) 4.9 (89.4/90.2) 4.9 (89.4/89.3)
5.5 (85.8/90.2) 4.9 (89.4/90.2) 5.5 (85.8/90.2)
4.2 (90.3/87.8) 4.2 (90.3/89.3) 4.2 (90.3/87.8)
0.95 (0.92 to 0.97) 0.95 (0.92 to 0.97) 0.95 (0.92 to 0.97)
12 (76.7/94.3) 11 (75.2/93.5) 12 (76.7/94.3)
10.5 (89.1/92.7) 9 (88.4/91) 10.5 (89.1/92.7)
7.8 (96.1/89.8) 6.8 (96.1/88.2) 7.8 (96.1/89.8)
8 (94.6/90.2) 7.9 (92.2/90.2) 8 (94.6/90.2)
10.5 (89.1/92.7) 8.5 (91.5/91) 10.5 (89.1/92.7)
0.96 (0.94 to 0.98) 0.96 (0.93 to 0.98) 0.96 (0.94 to 0.98)
The final cutoff values are highlighted in bold. *Values are the cutoff (sensitivity/specificity) value; 25th and 10th centiles are the cutoff values according to the 25th and 10th centiles of the cumulative score distribution; (0,1) is the cutoff value according to the closest point to (0,1); 90% specificity is the cutoff value according to fixed 90% specificity; 90% sensitivity is the cutoff value according to fixed 90% specificity (see ‘Patients and methods’ for details). †The number of positive results according to external criteria for high disease activity state/total number of patients was 113/205 for oligoarthritis and 129/245 for polyarthritis. AUC, area under the curve; JADAS, Juvenile Arthritis Disease Activity Score.
Consolaro A, et al. Ann Rheum Dis 2014;73:1380–1383. doi:10.1136/annrheumdis-2013-204186
1381
Downloaded from ard.bmj.com on September 27, 2014 - Published by group.bmj.com
Clinical and epidemiological research
Figure 1 Percentage of patients with a Juvenile Arthritis Disease Activity Score 10 (JADAS10) above the cutoff values for high disease activity state according to response at 6 months in the methotrexate trial (n=595) as assessed by American College of Rheumatology (ACR) paediatric criteria (not improved, or improved by 30%, 50%, or 70% (ACR 30, ACR 50 and ACR 70, respectively)). p
Clinical and epidemiological research
CONCISE REPORT
Defining criteria for high disease activity in juvenile idiopathic arthritis based on the Juvenile Arthritis Disease Activity Score Alessandro Consolaro,1 Nicolino Ruperto,1 Giulia Bracciolini,1 Alessia Frisina,1 Maria Chiara Gallo,1 Angela Pistorio,1 Sara Verazza,1 Giorgia Negro,1 Valeria Gerloni,2 Claudia Goldenstein-Schainberg,3 Flavio Sztajnbok,4 Nico M Wulffraat,5 Alberto Martini,1,6 Angelo Ravelli,1,6 for the Paediatric Rheumatology International Trials Organization (PRINTO) Handling editor Tore K Kvien ▸ Additional material is published online only. To view please visit the journal online (http://dx.doi.org/10.1136/ annrheumdis-2013-204186). 1
Istituto Giannina Gaslini, Genova, Italy Istituto Gaetano Pini, Milano, Italy 3 Faculdade de Medicina da Universidade de São Paulo, São Paulo/SP, Brazil 4 Universidade do Estado do Rio de Janeiro, Brazil 5 Wilhelmina Children’s Hospital, University Medical Centre Utrecht, Utrecht, The Netherlands 6 Università degli Studi di Genova, Genova, Italy 2
Correspondence to Dr Alessandro Consolaro, Pediatria II, Istituto G Gaslini, Largo G, Gaslini 5, Genova 16147, Italy; alessandroconsolaro@ ospedale-gaslini.ge.it Received 27 June 2013 Revised 8 November 2013 Accepted 3 December 2013 Published Online First 17 December 2013
ABSTRACT Objective To determine cutoff values for defining the state of high disease activity (HDA) in juvenile idiopathic arthritis (JIA) using the Juvenile Arthritis Disease Activity Score (JADAS). Methods For the selection of cutoff values, data from a clinical database including 609 patients were used. Optimal cutoff values were determined against external criteria by calculating the 25th and 10th centile of cumulative score distribution and through receiver operating characteristic curve analysis. External criteria were based on the therapeutic decision made by the attending doctor. Cross-validation was performed using five patient samples that included 1421 patients. Results The optimal cutoff values were those obtained through the 90% fixed sensitivity method. The selected JADAS cutoff values were the following: 4.2 and 8.5 for JADAS27 in oligoarthritis and polyarthritis, respectively; 4.2 and 10.5 for both JADAS10 and JADAS71 in oligoarthritis and polyarthritis, respectively. In crossvalidation analyses, the cutoff values showed strong ability to discriminate between different levels of American College of Rheumatology paediatric response in two clinical trials and could predict worse functional and radiographic outcome. Conclusions Cutoff values for classifying HDA in JIA using the JADAS were developed. In cross-validation analyses, they proved to have good construct and discriminant validity and ability to predict disease outcome.
PATIENTS AND METHODS JADAS calculation The JADAS is calculated as the arithmetic sum of the scores of the following variables: physician global rating of disease activity; parent/child rating of well-being; active joint count, assessed in 71, 27, or 10 joints (JADAS71, JADAS27, and JADAS10, respectively); and erythrocyte sedimentation rate, normalised to a 0–10 scale.2
Study population used for selection of cutoff values For the selection of cutoff values, we used data from 609 children with JIA3 who had undergone 1814 visits between 2007 and 2009.4 5 Children with systemic arthritis (n=46), rheumatoid factorpositive (n=9) and -negative (n=138) polyarthritis or extended oligoarthritis (n=96) were included in the polyarthritis group. The oligoarthritis group included children with persistent oligoarthritis (n=267). Children classified in other JIA categories (n=53) were assigned to the polyarthritis or oligoarthritis group based on the number of affected joints (≤4 or >4, respectively). Children with systemic JIA and active systemic manifestations were excluded. The study protocol was approved by the Istituto G Gaslini institutional review board.
Definition of HDA state INTRODUCTION
To cite: Consolaro A, Ruperto N, Bracciolini G, et al. Ann Rheum Dis 2014;73:1380–1383. 1380
Assessment of disease activity is a fundamental component of the clinical evaluation of children with juvenile idiopathic arthritis (JIA) because persistently active disease plays a major role in causing joint damage and physical functional disability.1 Recently, the first composite disease activity score for JIA, the Juvenile Arthritis Disease Activity Score (JADAS), was reported.2 To facilitate interpretation of JADAS scores, criteria (ie, cutoff values) are needed for identifying high and low levels of JIA activity. The purpose of this study was to determine and validate cutoff values in the JADAS that correspond to the state of high disease activity (HDA).
All patients’ visits were examined to identify those patients who met the criteria for HDA. To ensure face validity of the criteria, it was decided that they had to be based on the therapeutic decision made by the attending paediatric rheumatologist. The criteria for HDA in oligoarthritis were (a) intra-articular corticosteroid (IAC) administration in ≥1 joint; (b) start of a disease-modifying antirheumatic drug or (c) start of a biological agent. The criteria for HDA in polyarthritis were: (a) IAC administration in ≥3 joints; (b) start of a diseasemodifying antirheumatic drug; (c) start of a biological agent or (d) start of systemic corticosteroid therapy. All visits which did not meet the criteria for HDA were considered as reflecting low/moderate disease activity.
Consolaro A, et al. Ann Rheum Dis 2014;73:1380–1383. doi:10.1136/annrheumdis-2013-204186
Downloaded from ard.bmj.com on September 27, 2014 - Published by group.bmj.com
Clinical and epidemiological research Study populations used for cross-validation of cutoff values Five patient samples (not used in calculation of cutoff values) were used to cross-validate the cutoff values. The first and second samples comprised patients enrolled in controlled trials of methotrexate (MTX) (n=595)6 and abatacept (n=190).7 The third sample included 175 patients seen between 2009 and 2012 and followed up for at least 6 months.8 The fourth sample included 358 patients who underwent two or more visits between 1997 and 2002.9 The fifth sample consisted of 60 of 103 patients included in a study on the paediatric adaptation of the Sharp/van der Heijde score.10
RESULTS JADAS value by treatment decision The JADAS10 values by therapeutic intervention are presented in online supplementary table S1.
Selection of the optimal cutoff values for classification of HDA
Optimal cutoff values were determined against external criteria (ie, the states of HDA, defined as above) by calculating the 25th and 10th centiles of score distribution and through receiver operating characteristic (ROC) analysis. In ROC analysis, three methods were applied: (1) the closest point to (0,1)—that is, the point where the shoulder of the ROC curve is closest to the left upper corner of the graphic; (2) the fixed 90% specificity and (3) the fixed 90% sensitivity. Only one observation for each person was used in selection and validation of cutoff values.
The cutoff values obtained through the various statistical approaches are presented in table 1. In selecting the final cutoff values, we reasoned that since the ‘gold standard’ was HDA, in order to reduce the risk of misclassifying patients whose disease was active, more importance should be given to sensitivity—that is, to the proportion of patients with active disease who are correctly classified. However, a minimum specificity of 75% was required to minimise the rate of misclassification of patients with low/moderate disease activity as having HDA. Based on these considerations, it was felt that the optimal cutoff values were those provided by the 90% fixed sensitivity method. Notably, all area under the curves were >0.90, which reflects, according to a rule of thumb, ‘outstanding discrimination’.16 The graphs of ROC curves are illustrated in online supplementary figure S1.
Cross-validation study
Results of cross-validation analyses
Cross-validation of cutoff values was based on assessment of construct, discriminant and predictive validity through the following approaches. (1) Calculation of the percentage of patients who had a JADAS above the HDA cutoff values in clinical trials in relation to level of improvement according to the American College of Rheumatology paediatric criteria11 in the MTX6 and abatacept7 trials. (2) Calculation of the percentage of patients who had a JADAS above the HDA cutoff values in relation to subjective assessment of disease states5 12 by doctors and parents. (3) Assessment of the ability of cutoff values to predict lack of attainment of inactive disease13 and achievement of worse functional status on the Childhood Health Assessment Questionnaire (CHAQ).14 15 (4) Assessment of the ability of cutoff values to predict progression of radiographic joint damage.10 Quantitative and percentage data were compared by Mann– Whitney U test and χ2 test, respectively. The statistical packages used were Statistica (StatSoft) and Stata, V.7 (StataCorp).
In the MTX and abatacept trial samples, the proportion of patients with JADAS above the HDA cutoff value was greatest among non-responders and lowest among those with 70% improvement (figure 1 and online supplementary figure S2). In the third sample, the percentage of patients with JADAS above the cutoff value was greater among patients judged as having active disease and flare or as not being in an acceptable symptom state, and lower among patients judged as being in remission or in an acceptable symptom state (figure 2 and online supplementary figure S3). In the fourth sample, the percentage of patients with inactive disease or with a CHAQ score of 0 at the final visit was lower among patients who had a JADAS above the cutoff value at the first visit than among those who did not (results not shown). In the fifth sample, the adapted Sharp/van der Heijde score at 3 years was higher among patients who had JADAS above the cutoff values at least twice during the 3 years of observation than in those who did not (online supplementary figure S4).
Statistical analyses used for cutoff selection
Table 1 JADAS cutoff values for classification of patients into high disease activity state according to five different methods for determining optimal cut-off points Methods for determination of optimal cutoff values* 25th Centile Oligoarthritis† JADAS10 JADAS27 JADAS71 Polyarthritis† JADAS10 JADAS27 JADAS71
10th Centile
(0,1)
90% Specificity
90% Sensitivity
AUC (95% CI)
7.9 (75.2/95.1) 7.2 (76.1/94.6) 7.9 (75.2/95.1)
4.5 (89.4/88.8) 4.5 (89.4/89.8) 4.5 (89.4/88.8)
4.9 (89.4/89.3) 4.9 (89.4/90.2) 4.9 (89.4/89.3)
5.5 (85.8/90.2) 4.9 (89.4/90.2) 5.5 (85.8/90.2)
4.2 (90.3/87.8) 4.2 (90.3/89.3) 4.2 (90.3/87.8)
0.95 (0.92 to 0.97) 0.95 (0.92 to 0.97) 0.95 (0.92 to 0.97)
12 (76.7/94.3) 11 (75.2/93.5) 12 (76.7/94.3)
10.5 (89.1/92.7) 9 (88.4/91) 10.5 (89.1/92.7)
7.8 (96.1/89.8) 6.8 (96.1/88.2) 7.8 (96.1/89.8)
8 (94.6/90.2) 7.9 (92.2/90.2) 8 (94.6/90.2)
10.5 (89.1/92.7) 8.5 (91.5/91) 10.5 (89.1/92.7)
0.96 (0.94 to 0.98) 0.96 (0.93 to 0.98) 0.96 (0.94 to 0.98)
The final cutoff values are highlighted in bold. *Values are the cutoff (sensitivity/specificity) value; 25th and 10th centiles are the cutoff values according to the 25th and 10th centiles of the cumulative score distribution; (0,1) is the cutoff value according to the closest point to (0,1); 90% specificity is the cutoff value according to fixed 90% specificity; 90% sensitivity is the cutoff value according to fixed 90% specificity (see ‘Patients and methods’ for details). †The number of positive results according to external criteria for high disease activity state/total number of patients was 113/205 for oligoarthritis and 129/245 for polyarthritis. AUC, area under the curve; JADAS, Juvenile Arthritis Disease Activity Score.
Consolaro A, et al. Ann Rheum Dis 2014;73:1380–1383. doi:10.1136/annrheumdis-2013-204186
1381
Downloaded from ard.bmj.com on September 27, 2014 - Published by group.bmj.com
Clinical and epidemiological research
Figure 1 Percentage of patients with a Juvenile Arthritis Disease Activity Score 10 (JADAS10) above the cutoff values for high disease activity state according to response at 6 months in the methotrexate trial (n=595) as assessed by American College of Rheumatology (ACR) paediatric criteria (not improved, or improved by 30%, 50%, or 70% (ACR 30, ACR 50 and ACR 70, respectively)). p
