BMJ 2015;350:h2464 doi: 10.1136/bmj.h2464 (Published 8 May 2015)
Page 1 of 3
Endgames
ENDGAMES STATISTICAL QUESTION
How to read a receiver operating characteristic curve Philip Sedgwick reader in medical statistics and medical education Institute for Medical and Biomedical Education, St George’s, University of London, London, UK
Researchers investigated the use of vital signs as a screening test to identify brain lesions in patients with impaired consciousness. The setting was an emergency department in Japan. In total, 529 consecutive patients presenting with impaired consciousness, as assessed by a score of less than 15 on the Glasgow coma scale, were studied. Patients were followed until discharge. The vital signs of systolic and diastolic blood pressure plus pulse rate were recorded on arrival. The diagnosis of a brain lesion was determined on the basis of brain imaging and neurological examination. In total, 312 patients (59%) were diagnosed with a brain lesion.1 The performance of each vital sign as a screening test for diagnosed brain lesions was evaluated separately. The measurement scale for each vital sign was categorised using equal sized strata. Each stratum for a vital sign was taken successively as the threshold between a “negative” and “positive” screening test result; all measurements with values greater than the categorised strata were considered a “positive” result and all others were considered “negative.” If the result was positive, the patient was deemed at “high risk” of a brain lesion; otherwise the patient was deemed at “low risk” of a brain lesion. For each stratum of a vital sign the sensitivity and specificity were derived and used to plot a receiver operating characteristic curve for the vital sign (figure). The area beneath the curve was 0.90 for systolic blood pressure, 0.82 for diastolic pressure, and 0.63 for pulse rate.
Receiver operating characteristic curves for each of the three vital signs as screening tests for diagnosed brain lesions. For each vital sign, selected cut-off points between a positive and negative screening test result are shown
Which of the following statements, if any, are true? a) The value of (1 minus specificity) represents the proportion of patients without a diagnosed brain lesion identified as positive on screening b) For successive cut-off scores of a vital sign, as the sensitivity of the screening test increases the specificity decreases c) The vital sign that best distinguished between patients with diagnosed brain lesions and those without was systolic blood pressure
Answers
Statements a, b, and c are all true.
The study aimed to establish which vital sign best predicted brain lesions in patients with impaired consciousness and then
[email protected] For personal use only: See rights and reprints http://www.bmj.com/permissions
Subscribe: http://www.bmj.com/subscribe
BMJ 2015;350:h2464 doi: 10.1136/bmj.h2464 (Published 8 May 2015)
Page 2 of 3
ENDGAMES
select the value of that vital sign that gave the best threshold between a “positive” (high risk) and “negative” (low risk) result as a screening test for brain lesions. The measurement scales for each of the three vital signs were categorised using equal sized strata. For each category of each vital sign, the numbers of patients with a diagnosed brain lesion were obtained (table⇓). For each vital sign, the strata were taken successively as the cut-off point between a positive and negative screening test result; all patients with a vital sign measurement greater than the threshold were deemed to have a positive screening test result and the others a negative result. For each stratum of each vital sign, sensitivity and specificity were derived and used to plot a receiver operating characteristic curve for the vital sign. For each vital sign, the receiver operating characteristic curve was obtained by plotting the sensitivity against (1 minus specificity) for each stratum. The curve therefore allows the association between sensitivity and specificity to be explored as the threshold between a positive and negative screening test result changes. The sensitivity and specificity of a screening test has been described in a previous question.2 Sensitivity was the proportion of patients with a diagnosed brain lesion who were correctly identified as positive by the vital sign. Specificity was the proportion of patients without a diagnosed brain lesion who were correctly identified as negative by the vital sign. Therefore, the value of (1 minus specificity) is the proportion of patients without a diagnosed brain lesion who were identified incorrectly as positive by the vital sign (a is true). The value of (1 minus specificity) is referred to as the “false positive rate.”
There is a trade-off between sensitivity and the false positive rate when choosing the threshold between a positive and negative result for a vital sign as a screening test for brain lesions. For example, as the cut-off stratum for the vital sign pulse rate decreases from more than 130 beats/min to less than 50 beats/min, a higher proportion of patients with diagnosed brain lesions would be correctly identified as positive (high risk), resulting in a greater sensitivity. However, as the cut-off stratum decreases, the proportion of patients without a diagnosed brain lesion correctly identified as negative (low risk) becomes smaller, resulting in a lower specificity (b is true) and therefore a higher false positive rate. The receiver operating characteristic curve therefore provides a graphical representation for each threshold of the proportion of patients with diagnosed brain lesions correctly identified as positive against the proportion of patients without diagnosed brain lesions incorrectly identified as positive. Ideally, sensitivity should be as high as possible and the false positive rate as low as possible; a low false positive rate equates to a high specificity. Therefore, the choice of the optimal cut-off stratum for a vital sign depends on the implications of correctly identifying patients with a diagnosed brain lesion (true positive) and incorrectly identifying patients without a diagnosed lesion as positive (false positives). If for a vital sign the cut-off stratum between a positive and negative screening test result predicted those patients with diagnosed brain lesions and those without with 100% accuracy, both sensitivity and specificity would be equal to 1, and the false positive rate would equal zero. Hence the receiver operating characteristic curve would have to pass through the top left hand corner of the plot. The curve would start at the origin, go vertically up the y axis to a sensitivity of 1.0, and
For personal use only: See rights and reprints http://www.bmj.com/permissions
then horizontally across to a false positive rate of 1.0. However, a screening test will rarely be 100% accurate. A vital sign will generally have a good ability to distinguish between patients with diagnosed brain lesions and those without if the sensitivity and specificity are high and close to 1.0 (100%) for a range of cut-off strata. Therefore, the closer a curve for a vital sign comes to the top left hand corner of the plot the better the vital sign will be at distinguishing between patients with diagnosed brain lesions and those without. However, the closer the curve is to the diagonal on the receiver operating characteristic curve the worse the vital sign will be at distinguishing between patients with diagnosed brain lesions and those without. The diagonal can be seen on the figure as the line from (0,0), where sensitivity and the false positive rate both equal zero, to (1,1), where sensitivity and the false positive rate both equal one. Points on this line represent a sensitivity and false positive rate of equal value at each cut-off stratum. This line would indicate a screening test that had no ability to distinguish between patients with diagnosed brain lesions and those without; it identifies exactly the same proportion of each patient group as high risk at a particular cut-off point. As described, the closer a curve for a vital sign comes to the top left hand corner of the plot the better the vital sign is at distinguishing between patients with diagnosed brain lesions and those without. Therefore, the overall accuracy of the three vital signs as screening tests for the diagnosis of brain lesions can be compared by the area beneath each curve, which is typically expressed as a proportion. The vital sign with the curve nearest to the top left corner, which was therefore best at distinguishing between patients with a diagnosed brain lesion and those without, was systolic blood pressure (0.90), compared with 0.82 for diastolic pressure and 0.63 for pulse rate (c is true).
For each vital sign, the cut-off stratum closest to the top left hand corner of the plot would typically be chosen as the optimal threshold between a positive and negative screening test result. For systolic blood pressure, this value is 120-129 mm Hg, which gives a sensitivity of 88% and specificity of 78%; for diastolic blood pressure this value is 70-79 mm Hg (sensitivity 77%, specificity 72%), and for pulse rate it is 80-89 beats/min (sensitivity 57%, specificity 61%). However, choosing the cut-off stratum closest to the top left hand corner of the plot assumes that the implications of a false negative result (a patient with a diagnosed brain lesion being incorrectly identified as negative on screening) and a false positive result are similar. The receiver operating characteristic curve is probably most useful when comparing the performance of two or more screening tests. Although a receiver operating characteristic curve for a single screening test may have limited value, it is still a useful graphical representation if there are many possible cut-off points. Competing interests: None declared. 1 2
Ikeda M, Matsunaga T, Irabu N, et al. Using vital signs to diagnose impaired consciousness: cross sectional observational study. BMJ 2002;325:800. Sedgwick P, Joekes K. Evaluating the performance of a screening test for depression in primary care. BMJ 2015;350:h1801.
Cite this as: BMJ 2015;350:h2464 © BMJ Publishing Group Ltd 2015
Subscribe: http://www.bmj.com/subscribe
BMJ 2015;350:h2464 doi: 10.1136/bmj.h2464 (Published 8 May 2015)
Page 3 of 3
ENDGAMES
Table Table 1| Number of patients in each stratum with and without a diagnosed brain lesion, together with stratum specific sensitivity, specificity,
and (1 minus specificity) for each vital sign Stratum
Patients (n)
Sensitivity
Specificity
1 minus specificity
With brain lesion Without brain lesion Systolic blood pressure (mm Hg)
Page 1 of 3
Endgames
ENDGAMES STATISTICAL QUESTION
How to read a receiver operating characteristic curve Philip Sedgwick reader in medical statistics and medical education Institute for Medical and Biomedical Education, St George’s, University of London, London, UK
Researchers investigated the use of vital signs as a screening test to identify brain lesions in patients with impaired consciousness. The setting was an emergency department in Japan. In total, 529 consecutive patients presenting with impaired consciousness, as assessed by a score of less than 15 on the Glasgow coma scale, were studied. Patients were followed until discharge. The vital signs of systolic and diastolic blood pressure plus pulse rate were recorded on arrival. The diagnosis of a brain lesion was determined on the basis of brain imaging and neurological examination. In total, 312 patients (59%) were diagnosed with a brain lesion.1 The performance of each vital sign as a screening test for diagnosed brain lesions was evaluated separately. The measurement scale for each vital sign was categorised using equal sized strata. Each stratum for a vital sign was taken successively as the threshold between a “negative” and “positive” screening test result; all measurements with values greater than the categorised strata were considered a “positive” result and all others were considered “negative.” If the result was positive, the patient was deemed at “high risk” of a brain lesion; otherwise the patient was deemed at “low risk” of a brain lesion. For each stratum of a vital sign the sensitivity and specificity were derived and used to plot a receiver operating characteristic curve for the vital sign (figure). The area beneath the curve was 0.90 for systolic blood pressure, 0.82 for diastolic pressure, and 0.63 for pulse rate.
Receiver operating characteristic curves for each of the three vital signs as screening tests for diagnosed brain lesions. For each vital sign, selected cut-off points between a positive and negative screening test result are shown
Which of the following statements, if any, are true? a) The value of (1 minus specificity) represents the proportion of patients without a diagnosed brain lesion identified as positive on screening b) For successive cut-off scores of a vital sign, as the sensitivity of the screening test increases the specificity decreases c) The vital sign that best distinguished between patients with diagnosed brain lesions and those without was systolic blood pressure
Answers
Statements a, b, and c are all true.
The study aimed to establish which vital sign best predicted brain lesions in patients with impaired consciousness and then
[email protected] For personal use only: See rights and reprints http://www.bmj.com/permissions
Subscribe: http://www.bmj.com/subscribe
BMJ 2015;350:h2464 doi: 10.1136/bmj.h2464 (Published 8 May 2015)
Page 2 of 3
ENDGAMES
select the value of that vital sign that gave the best threshold between a “positive” (high risk) and “negative” (low risk) result as a screening test for brain lesions. The measurement scales for each of the three vital signs were categorised using equal sized strata. For each category of each vital sign, the numbers of patients with a diagnosed brain lesion were obtained (table⇓). For each vital sign, the strata were taken successively as the cut-off point between a positive and negative screening test result; all patients with a vital sign measurement greater than the threshold were deemed to have a positive screening test result and the others a negative result. For each stratum of each vital sign, sensitivity and specificity were derived and used to plot a receiver operating characteristic curve for the vital sign. For each vital sign, the receiver operating characteristic curve was obtained by plotting the sensitivity against (1 minus specificity) for each stratum. The curve therefore allows the association between sensitivity and specificity to be explored as the threshold between a positive and negative screening test result changes. The sensitivity and specificity of a screening test has been described in a previous question.2 Sensitivity was the proportion of patients with a diagnosed brain lesion who were correctly identified as positive by the vital sign. Specificity was the proportion of patients without a diagnosed brain lesion who were correctly identified as negative by the vital sign. Therefore, the value of (1 minus specificity) is the proportion of patients without a diagnosed brain lesion who were identified incorrectly as positive by the vital sign (a is true). The value of (1 minus specificity) is referred to as the “false positive rate.”
There is a trade-off between sensitivity and the false positive rate when choosing the threshold between a positive and negative result for a vital sign as a screening test for brain lesions. For example, as the cut-off stratum for the vital sign pulse rate decreases from more than 130 beats/min to less than 50 beats/min, a higher proportion of patients with diagnosed brain lesions would be correctly identified as positive (high risk), resulting in a greater sensitivity. However, as the cut-off stratum decreases, the proportion of patients without a diagnosed brain lesion correctly identified as negative (low risk) becomes smaller, resulting in a lower specificity (b is true) and therefore a higher false positive rate. The receiver operating characteristic curve therefore provides a graphical representation for each threshold of the proportion of patients with diagnosed brain lesions correctly identified as positive against the proportion of patients without diagnosed brain lesions incorrectly identified as positive. Ideally, sensitivity should be as high as possible and the false positive rate as low as possible; a low false positive rate equates to a high specificity. Therefore, the choice of the optimal cut-off stratum for a vital sign depends on the implications of correctly identifying patients with a diagnosed brain lesion (true positive) and incorrectly identifying patients without a diagnosed lesion as positive (false positives). If for a vital sign the cut-off stratum between a positive and negative screening test result predicted those patients with diagnosed brain lesions and those without with 100% accuracy, both sensitivity and specificity would be equal to 1, and the false positive rate would equal zero. Hence the receiver operating characteristic curve would have to pass through the top left hand corner of the plot. The curve would start at the origin, go vertically up the y axis to a sensitivity of 1.0, and
For personal use only: See rights and reprints http://www.bmj.com/permissions
then horizontally across to a false positive rate of 1.0. However, a screening test will rarely be 100% accurate. A vital sign will generally have a good ability to distinguish between patients with diagnosed brain lesions and those without if the sensitivity and specificity are high and close to 1.0 (100%) for a range of cut-off strata. Therefore, the closer a curve for a vital sign comes to the top left hand corner of the plot the better the vital sign will be at distinguishing between patients with diagnosed brain lesions and those without. However, the closer the curve is to the diagonal on the receiver operating characteristic curve the worse the vital sign will be at distinguishing between patients with diagnosed brain lesions and those without. The diagonal can be seen on the figure as the line from (0,0), where sensitivity and the false positive rate both equal zero, to (1,1), where sensitivity and the false positive rate both equal one. Points on this line represent a sensitivity and false positive rate of equal value at each cut-off stratum. This line would indicate a screening test that had no ability to distinguish between patients with diagnosed brain lesions and those without; it identifies exactly the same proportion of each patient group as high risk at a particular cut-off point. As described, the closer a curve for a vital sign comes to the top left hand corner of the plot the better the vital sign is at distinguishing between patients with diagnosed brain lesions and those without. Therefore, the overall accuracy of the three vital signs as screening tests for the diagnosis of brain lesions can be compared by the area beneath each curve, which is typically expressed as a proportion. The vital sign with the curve nearest to the top left corner, which was therefore best at distinguishing between patients with a diagnosed brain lesion and those without, was systolic blood pressure (0.90), compared with 0.82 for diastolic pressure and 0.63 for pulse rate (c is true).
For each vital sign, the cut-off stratum closest to the top left hand corner of the plot would typically be chosen as the optimal threshold between a positive and negative screening test result. For systolic blood pressure, this value is 120-129 mm Hg, which gives a sensitivity of 88% and specificity of 78%; for diastolic blood pressure this value is 70-79 mm Hg (sensitivity 77%, specificity 72%), and for pulse rate it is 80-89 beats/min (sensitivity 57%, specificity 61%). However, choosing the cut-off stratum closest to the top left hand corner of the plot assumes that the implications of a false negative result (a patient with a diagnosed brain lesion being incorrectly identified as negative on screening) and a false positive result are similar. The receiver operating characteristic curve is probably most useful when comparing the performance of two or more screening tests. Although a receiver operating characteristic curve for a single screening test may have limited value, it is still a useful graphical representation if there are many possible cut-off points. Competing interests: None declared. 1 2
Ikeda M, Matsunaga T, Irabu N, et al. Using vital signs to diagnose impaired consciousness: cross sectional observational study. BMJ 2002;325:800. Sedgwick P, Joekes K. Evaluating the performance of a screening test for depression in primary care. BMJ 2015;350:h1801.
Cite this as: BMJ 2015;350:h2464 © BMJ Publishing Group Ltd 2015
Subscribe: http://www.bmj.com/subscribe
BMJ 2015;350:h2464 doi: 10.1136/bmj.h2464 (Published 8 May 2015)
Page 3 of 3
ENDGAMES
Table Table 1| Number of patients in each stratum with and without a diagnosed brain lesion, together with stratum specific sensitivity, specificity,
and (1 minus specificity) for each vital sign Stratum
Patients (n)
Sensitivity
Specificity
1 minus specificity
With brain lesion Without brain lesion Systolic blood pressure (mm Hg)
