ORIGINAL CLINICAL STUDY
Specialist Nurse Screening for Retinopathy of PrematurityVA Pilot Study Shaheen P. Shah, FRCOphth, MD(Res),*Þ Zhen Wu, BHSc (Nursing),* Sharron Iverson, BHSc (Nursing),* and Shuan Dai, MBBS, MS, FRANZCO*
Purpose: To compare the accuracy of retinopathy of prematurity (ROP) screening between nurse specialists and an expert pediatric ophthalmologist. Design: A comparative case series of ROP screening in a level 2 neonatal intensive care unit setting where there is a higher threshold for accepting very premature or unwell neonates. Methods: Trained specialist ROP nurses acquired wide-field digital images, graded ROP, and proposed a follow-up plan. This was compared with the findings by an ROP expert ophthalmologist. Outcomes include sensitivity, specificity, positive predictive value, and negative predictive value of ROP grading by trained ROP nurses. Results: Mean gestational age was 28.6 weeks, and mean birth weight was 1184 g of the 64 consecutive neonates included. A total 316 eye screens was performed. Image acquisition, grading, and a management plan by the ROP nurse were possible in all screens. In right eyes, the presence of any ROP (stage 90) was 15%. Sensitivity, specificity, positive predictive value, and negative predictive value of ROP grading were 91.7% (95% CI, 73%Y99%), 80.6% (95% CI, 72.9%Y86.9%), 45.8% (95% CI, 31.4%Y60.80%), and 98.2% (95% CI, 93.6%Y99.8%), respectively. Agreement on the management plan occurred in 84.8% of cases. In virtually all circumstances of disagreement, ROP nurses exaggerated the ROP grading present and/or recommended a repeat screen when discharge from service was more appropriate. Conclusions: Our preliminary findings demonstrated good agreement between ROP nurses and the ROP expert ophthalmologist. Further research in expanding the role of utilizing nonphysician health workers in ROP screening is suggested. Key Words: Retinopathy of prematurity, Screening, Wide-field digital imaging, Telemedicine, Agreement (Asia-Pac J Ophthalmol 2013;2: 300Y304)
R
etinopathy of prematurity (ROP) is a retinal ischemic disorder that affects premature infants of low birth weight (BW). Retinopathy of prematurity is one of the few causes of childhood visual loss that is preventable, and many evidence-based guidelines exist for the screening of ROP. The goal of an effective screening program is to identify the relatively few neonates who require treatment from among the much larger number of at-risk infants while minimizing the number of stressful examinations required for these sick infants.1
From the *Department of Ophthalmology, Auckland, New Zealand, Auckland; and †International Centre for Eye Health, London School of Hygiene and Tropical Medicine, London, UK. Received for publication May 1, 2013; accepted May 28, 2013. The authors have no funding or conflicts of interest to declare. Reprints: Shuan Dai, MBBS, MS, FRANZCO, Department of Ophthalmology, Auckland District Health Board, Private Bag 92-189, Auckland, New Zealand. E-mail: [email protected]. Copyright * 2013 by Asia Pacific Academy of Ophthalmology ISSN: 2162-0989 DOI: 10.1097/APO.0b013e31829dc72b
300
www.apjo.org
Screening is traditionally conducted by the ophthalmologist using a binocular indirect ophthalmoscope. More recently, a number of studies have demonstrated excellent agreement with the traditional method when ophthalmologists screen for ROP using images captured with wide-field digital retinal imaging (WFDRI) cameras.2 Image acquisition by a trained nonphysician is widely accepted; however, a recent review3 highlighted the need for further research into the possibility of ROP grading by trained nonphysician health professionals as is currently the accepted standard in diabetic retinopathy screening.4 To our knowledge, no studies have been published on a screening program undertaken by trained ROP nurses under the supervision of ophthalmologists using WFDRI.
MATERIALS AND METHODS This study was an observational, blinded agreement study of ROP screening using images of ROP gathered by wide-field retinal imaging. Currently, ROP screening in 2 peripheral level 2 neonatal intensive care units (NICUs) approximately 20 km away from the tertiary level 3 NICU is conducted solely on ‘‘store-and-forward’’ telemedicine technology. This is a technology where medical data are captured for subsequent interpretation remotely by an expert. These peripheral units have a higher threshold for accepting very premature or unwell neonates from the city’s level 3 unit, and typically only babies older than 30 weeks’ corrected gestational age (cGA) or gestational age (GA) and weighing more than 1250 g who are medically improving are allowed onto the unit. Two specialist-trained ROP nurses (Z.W., S.I.), each with more than 5 years of experience in performing Retcam WFDRI, travel weekly with a medical photographer to the 2 peripheral NICUs and perform the ROP screening examinations. As per a standard protocol, pupils are dilated with cyclopentolate 0.5% and phenylephrine 2.5%, instilled 30 to 45 minutes before screening. The trained nurse operates the handpiece of Retcam-Shuttle (Clarity Medical Systems, Pleasanton, Calif ), which is equipped with 130-degree ROP lens while the photographer focuses and captures the image on the Retcam. A second NICU nurse swaddles the infant and monitors vital signs. A minimum of 3 images, (i) posterior pole, (ii) nasal to disc periphery, and (iii) temporal to disc periphery), are taken from each eye. At the end of the session, all the images are uploaded to the secure hospital server. Either the same or the next working day the images are read by the ROP specialist pediatric ophthalmologist (S.D.) at a review station in the tertiary ophthalmology unit. The decision on further management (for example, the date of the next screen) is taken and then relayed back to the peripheral units electronically. For the purposes of this study, all neonates consecutively undergoing ROP screening in both peripheral NICUs over a 13-month period were recruited. In addition to the normal screening process described above, the specialist nurses, after completing each screen, documented the ROP stage, zone, and presence of plus disease on each eye as per the international
Asia-Pacific Journal of Ophthalmology
&
Volume 2, Number 5, September/October 2013
Copyright © 2013 Asia Pacific Academy of Ophthalmology. Unauthorized reproduction of this article is prohibited.
Asia-Pacific Journal of Ophthalmology
&
Volume 2, Number 5, September/October 2013
Screening for Retinopathy of Prematurity
TABLE 1. Gestational Age, cGA, and BW of the 68 Neonates Included in This Study Corrected Gestational Age* G34.5 wk
Q34.5 and Q37.5 wk
937.5 wk
Total
Birth Weight (g)
Gestational Age, wk
n
%
n
%
n
%
n
%
Mean
SD
G27.5 Q27.5 and G29 Q29 Total
4 9 4 17
23.5 52.9 23.5 100
12 9 17 38
31.6 23.7 44.7 100
4 4 4 12
33.3 33.3 33.3 100
20 22 25 67
29.9 32.8 37.3 100
932.3 1273.9 1312.2 1186.2
177.3 279.6 310.5 311.6
Median 897.5 1300 1240 1210
IQR 820Y990 1230Y1380 1100Y1500 915Y1346
*At first screen. IQR, interquartile range.
Early Treatment for Retinopathy of Prematurity guidelines5 and proposed a plan of management for the neonate. Data were entered onto a study-specific database and exported to STATA (Statistical Software: Release 11.0; StataCorp, College Station, TX, 2001) for management and analysis. The unique National Health Index number and the date of examination were used to link the 2 assessments. Data entry errors were examined by cross-checking data points with the original data collection forms on a randomly selected 10% of the sample (no data entry errors were found in this subset). Birth weight, GA, (calculated from first day of last menstrual period), and cGA were categorized into approximate tertiles. Referral warranted ROP was defined as any ROP in zone 1, presence of plus disease, or presence of any stage 3 ROP detected by the specialist ROP nurse at any time during infant’s hospital course.6 Sensitivity, specificity, and positive and negative predictive values6 for digital imaging by a trained ROP nurse were determined for the presence of ‘‘any ROP’’ (defined as stage 90, any zone in the eye). Interobserver agreement using the J statistic was used to measure chance-adjusted agreement for the presence of disease based on an accepted scale.7 The findings of each specialist nurse were compared with those of the expert. As disease in the right and left eyes of an individual are highly correlated, analysis was conducted on right and left eyes separately. This project was approved by the Auckland District Health Board Research Review Committee.
RESULTS A total of 67 neonates were screened and included in the study. Nineteen neonates had only 1 screening performed before discharge, 18 had 2 separate screening examinations, 21 had 3 examinations, and 10 neonates had 4 examinations. Therefore, a total of 158 neonatal screens were performed. In no circumstances were ROP screens not able to be performed on the scheduled date. There were no reported adverse events. Neonates with GAs of less than 27.5 weeks, 27.5 weeks or greater but less than 29 weeks, and 29 weeks or greater accounted for 29.9%, 32.8%, and 37.3% of the population, respectively (Table 1). Mean GA was 28.6 (SD, 2.0) weeks, and mean BW was 1186.2 (SD, 311.6) g. The cGA dates of the first screen performed by the specialist nurse were as follows: 17 children (25.4%) with cGA of less than 34.5 weeks; 38 children (56.7%) were 34.5 weeks or greater but less than 37.5 weeks; and 12 children (17.9%) were 37.5 weeks or greater. The results for each nurse specialist when compared individually to the expert were similar, and therefore the combined data are presented. Diseases between eyes of the same * 2013 Asia-Pacific Academy of Ophthalmology
individual were similar; the presence of ‘‘any ROP’’ (by expert) in the right and left eye was 15% and 16%, respectively (Table 2). No child was noted to have plus disease by either assessor. One ‘‘referral warranted’’ mismatch was found (nurse: left eye stage 3 zone 2; physician: left eye stage 1 zone 2). Overall mismatch in assessment of ‘‘any ROP’’ existed in 55 (17.4%) of 316 screens. In the vast majority, these were false positives (nurses graded ‘‘any ROP’’ present when none was). Importantly, there were 5 false-negative screens (1.6%, all zone 3 or peripheral zone 2 with stage 1 disease). Sensitivity of grading of ‘‘any’’ ROP was 91.7% (95% CI, 73%Y99%) and 88% (95% CI, 68.8%Y97.5%), and specificity was 80.6% (95% CI, 72.9%Y86.9%) and 82% (95% CI, 74.4%Y88.1%) in right and left eyes, respectively. Results were very similar in right and left eyes, and thus only right-eye data were tabulated (Table 2). Incorrect level of staging occurred in 69 screens (21.8%). Generally, nurse staging was more likely to report a worse staging than that actually present, but in 7 screens (2.2%) nurses reported a lower stage than that graded by the ROP expert. Agreement of the management plan between the 2 screeners was 84.8%. J Values of agreement of presence or absence of ‘‘any ROP,’’ stage/zone of disease, and management are shown in Table 3.
DISCUSSION Retinopathy of prematurity is a leading cause of childhood blindness,8 and worldwide we are currently experiencing the ‘‘third ROP epidemic’’8 through higher premature birth rates and improved neonatal survival. A rapid expansion of neonatal services provision in economically emerging economies in addition to a shortage of trained ophthalmologists conducting ROP screening and less willingness to undertake ROP management by experienced ophthalmologists due to increasingly hazardous nature of litigation has also negatively affected screening programs worldwide. These problems of capacity and resource have led to intense operational research to determine alternative safe screening models from the traditional method. In particular, they have focused on the possibility of utilizing the skills of allied nonphysician health workers. Our ROP nurses have extensive experience in ROP screening, and this provided an ideal and safe setting in which to evaluate the accuracy of this alternative model of screening. The study findings led us to believe that an alternative model of screening, one where WFDRI acquisition, grading, and management by a trained nurse (supported by an off-site, highly accessible ROP expert) may be an answer to some of the current problems in ROP screening outlined above. The www.apjo.org
Copyright © 2013 Asia Pacific Academy of Ophthalmology. Unauthorized reproduction of this article is prohibited.
301
Asia-Pacific Journal of Ophthalmology
Shah et al
&
Volume 2, Number 5, September/October 2013
TABLE 2. Assessment of Nurse Grading Compared With the Criterion-Standard Expert ROP Screening Any ROP Nurse Present
Absent
Present % (Row) Absent % (Row) Total
22 91.7 26 19.4 48
2 8.3 108 80.6 110
% (Row)
30.4
69.6
Present % (Row) Absent % (Row) Total
22 88 24 18.0 46
3 12 109 82.0 112
% (Row)
29.1
70.9
0 % (Row) 1 % (Row) 2 % (Row) Total % (Row)
0 108 80.6 2 9.1 0 0 110 69.6
2 % (Row) 3 % (Row) Total % (Row)
2 41 87.2 43 38.7 84 53.2
95% Confidence Interval
Total
Right eye
Physician
24 100 134 100 158
Prevalence Sensitivity Specificity Likelihood ratio (+) Positive predictive value Negative predictive value
100.0
15% 91.70% 80.60% 4.72 45.80%
10%Y21.80% 73%Y99% 72.90%Y86.90% 3.28Y6.81 31.40%Y60.80%
98.20%
93.60%Y99.80%
16% 88% 82% 4.88 47.80%
11%Y22.50% 68.80%Y97.50% 74.40%Y88.10% 3.3Y7.2 32.90%Y63.10%
97.30%
92.40%Y99.40%
Left eye
Physician
25 100 133 100 158
Prevalence Sensitivity Specificity Likelihood ratio (+) Positive predictive value Negative predictive value
100.0
Stage of ROP Nurse Right eye
Physician
1 25 18.7 14 63.6 1 50 40 25.3
2 1 0.7 6 27.3 1 50 8 5.1
134 100 22 100 2 100 158 100
Zone of ROP Nurse Right eye
Physician
3 6 12.8 68 61.3 74 46.8
Total 47 100 111 100 158 100
Management
Physician
Follow-up in NICU % (Row) OPD % (Row) Total % (Row)
Follow-up in NICU 96 93 17 30.9 113 71.5
Nurse Follow-up in OPD 7 7 38 69.1 45 28.5
Total 103 100 55 100 158 100
*Numbers in bold and italics indicate mismatches between nurse and expert ROP assessments.
evidence to support this model of screening is generated from a number of broad perspectives: (i) An internationally accepted ROP classification system has led to a standardization of
302
www.apjo.org
diagnosis and a standardization of treatment guidelines. Robust evidence-based studies have allowed a simplification of the ROP decision-making algorithm, and ultimately, it is the * 2013 Asia-Pacific Academy of Ophthalmology
Copyright © 2013 Asia Pacific Academy of Ophthalmology. Unauthorized reproduction of this article is prohibited.
Asia-Pacific Journal of Ophthalmology
&
Volume 2, Number 5, September/October 2013
Screening for Retinopathy of Prematurity
TABLE 3. J Agreement for ‘‘Any ROP,’’ Management, Stage, and Zone of ROP
Any ROP
Agreement
Expected Agreement
J
SE
P
82.3% 82.9% 84.8% 77.9% 78.5% 69.0% 69.6%
63.7% 64.3% 56.5% 62.6% 63.2% 48.7% 49.0%
51.2% 52.2% 65.1% 40.7% 41.5% 39.5% 40.5%
0.0723 0.0738 0.0787 0.0649 0.0661 0.0708 0.0712
G0.001 G0.001 G0.001 G0.001 G0.001 G0.001 G0.001
Right eye Left eye
Follow-up plan Stage of ROP Zone of ROP
Right eye Left eye Right eye Left eye
presence of significant ROP in zone I or plus disease that is the critical finding that determines treatment.9,10 This simplification, by creating a focus on visualizing the pathology at the posterior pole, has made ROP easier to manage. (ii) The advent of new technologies that allow relatively easy capture of excellent quality wide-field images of the retina, particularly of the posterior pole, has revolutionized the traditional screening method using the indirect ophthalmoscope. Many studies have shown that nonphysician health workers can successfully use these systems to capture quality WFDRIs,11 and as a result, many units around the world now rely solely on these professionals for image acquisition. This can be reiterated in our study where image acquisition was possible in all cases. (iii) Screening by analysis of WFDRI alone is shown to be safe, and there is good level 1 evidence that retinal photography has high accuracy for the detection of clinically significant ROP.10 Sensitivities of 100% and specificities of 97.9% in detecting infants with treatment-requiring ROP by ophthalmologists reading Retcam 2 or Retcam-Shuttle images when compared with binocular indirect ophthalmoscope have been achieved,12 and the American Academy of Ophthalmology recently concluded that WFDRI provided unique advantages in ROP management through objective documentation of findings and improved recognition of disease progression.10 (iv) Management decisions (eg, treatment, next screening date) by expert ophthalmologists reading WFDRIs remotely using store-and-forward telemedicine have also been found to have very high sensitivity and specificity,6,11,13 and studies using this method in Canada and Germany reported no adverse outcomes over a 4.5and 6-year period, respectively.14,15 Dedicated trained readers of retinal images for the purposes of screening for diabetic retinopathy are now considered standard, yet one may argue that the retinal changes that determine the management decisions in ROP screening are not as difficult to detect as those in diabetic retinopathy. Richter and colleagues’3 comprehensive ROP review concluded that additional research on the accuracy of trained nurses for image grading was required, and Williams et al,16 in the first study of this nature, compared grading of WFDRI images by nonexperts who were trained by attending only 2 hours of training on image-based ROP diagnosis. Even with this short duration of training, several of their nonexpert graders performed comparably to the experts.16 The investigators were concerned about a high number of ‘‘unknown’’ responses by the nonexperts. This was found not to be a problem in our study. A number of studies used trained nonspecialists to screen for ROP, and the results have been encouraging. Zepeda-Romero et al17 trained a neonatologist and a pediatrician to use the indirect ophthalmoscope; Saunders et al18 trained a neonatologist to use the direct ophthalmoscope, and Azad et al19 trained residents and nurses with the same. In all studies, examination was directed only at observing vascular * 2013 Asia-Pacific Academy of Ophthalmology
changes at the posterior pole. Without objective documentation of the retinal appearance, these methods are limited (if a second ‘‘expert opinion’’ was required, this would necessitate a repeat onsite examination). The telemedicine approach to screening of ROP is, however, limited by several factors, the most relevant of which is the fact that dynamic ‘‘real-time’’ visualization of the retina is not yet widely feasible (ie, areas of concern or uncertainty cannot be revisited by the remote reader who is limited to reading only captured still images). Indeed, this may explain some of the false-positive results in the present study. Other factors such as the technical aspects, network maintenance and data security, and the high set-up and maintenance costs involved as well as the acceptability of telemedicine to patients and medical providers are barriers limiting this approach. Results of an ongoing multicenter study in the United States, which has established a grading center for WFDRI grading by nonphysician graders (eROP study, http://clinicaltrials.gov/ct2/show/NCT01264276), will be useful in directing future ROP screening models. The high sensitivity and specificity values found in this comparative study suggest safety in methodology. Clinically, ROP findings between right and left eyes are generally quite similar, but most published studies base statistics on eye-level screens, thus exaggerating sample size and statistical power. In this study, we attempted to limit the analysis to the person level. Most mismatching was conservative in nature (false positives), and false negatives were rare. The high number of follow-up screens recommended when discharge was deemed appropriate by the expert was of particular concern as these examinations are not without risk to the neonate. To our knowledge, this is the first study to examine nonphysician clinical decision making in ROP, and the results were encouraging. Although all NICUs are stratified to admit neonates according to levels of complexity appropriate for their personnel and resources, they each must provide an effective ROP service, and it is in the ‘‘low-risk’’ NICU setting that the results of this study apply. Thus, the applicability of our findings is limited by (i) the low prevalence of ‘‘any ROP’’ and (ii) the lack of clinically significant ROP pathology in the sample (eg, plus disease or zone 1 disease), and thus further research is warranted in a tertiary level 3 NICU, where more disease is anticipated. Our results reflect current ROP screening in level 2 units in Auckland; it is our impression that, with high standard of modern neonatal care, the large levels of ‘‘referral warranted’’ ROP detected only a decade ago6 are now simply not observed. Overall agreement of ‘‘any ROP’’ was greater than 80%, but the J coefficient found in this study would be classed as ‘‘moderate.’’ The value of the J coefficient, however, is limited as these statistics are affected by prevalence of the condition, and low J values in conditions of low prevalence do not necessarily reflect low rates of overall agreement.20 www.apjo.org
Copyright © 2013 Asia Pacific Academy of Ophthalmology. Unauthorized reproduction of this article is prohibited.
303
Asia-Pacific Journal of Ophthalmology
Shah et al
A number of operational rules and responsibilities would need to be clearly defined for this screening model. Unlike diabetic retinopathy, ROP progression is aggressive, and therefore robust time-defined protocols are critical (eg, clear channels of access to expert second opinion as well as clearly defined clinical and medicolegal liabilities). The nurses in this study gained their training from ‘‘on the job’’ observational experience; however, training programs with certification of nonspecialist screeners would need to be agreed and formalized. REFERENCES 1. Section on Ophthalmology, American Academy of Pediatrics, American Academy of Ophthalmology, American Association for Pediatric Ophthalmology and Strabismus. Screening examination of premature infants for retinopathy of prematurity. Pediatrics. 2006;117:572Y576. 2. Salcone EM, Johnston S, VanderVeen D. Review of the use of digital imaging in retinopathy of prematurity screening. Semin Ophthalmol. 2010;25:214Y217. 3. Richter GM, Williams SL, Starren J, et al. Telemedicine for retinopathy of prematurity diagnosis: evaluation and challenges. Surv Ophthalmol. 2009;54:671Y685. 4. Taylor CR, Merin LM, Salunga AM, et al. Improving diabetic retinopathy screening ratios using telemedicine-based digital retinal imaging technology: the Vine Hill study. Diabetes Care. 2007;30: 574Y578. 5. Good WV, Hardy RJ, Dobson V, et al. Final visual acuity results in the early treatment for retinopathy of prematurity study. Arch Ophthalmol. 2010;128:663Y671. 6. Ells AL, Holmes JM, Astle WF, et al. Telemedicine approach to screening for severe retinopathy of prematurity: a pilot study. Ophthalmology. 2003;110:2113Y2117. 7. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33:159Y174. 8. Gilbert C. Retinopathy of prematurity: a global perspective of the epidemics, population of babies at risk and implications for control. Early Hum Dev. 2008;84:77Y82.
304
www.apjo.org
&
Volume 2, Number 5, September/October 2013
9. Early Treatment for Retinopathy of Prematurity Cooperative Group. Revised indications for the treatment of retinopathy of prematurity: results of the early treatment for retinopathy of prematurity randomized trial. Arch Ophthalmol. 2003;121:1684Y1694. 10. Chiang MF, Melia M, Buffenn AN, et al. Detection of clinically significant retinopathy of prematurity using wide-angle digital retinal photography: a report by the American Academy of Ophthalmology. Ophthalmology. 2012;119:1272Y1280. 11. Chiang MF, Wang L, Busuioc M, et al. Telemedical retinopathy of prematurity diagnosis: accuracy, reliability, and image quality. Arch Ophthalmol. 2007;125:1531Y1538. 12. Dai S, Chow K, Vincent A. Efficacy of wide-field digital retinal imaging for retinopathy of prematurity screening. Clin Exp Ophthalmol. 2011;39:23Y29. 13. Silva RA, Murakami Y, Lad EM, et al. Stanford University network for diagnosis of retinopathy of prematurity (SUNDROP): 36-month experience with telemedicine screening. Ophthalmic Surg Lasers Imaging. 2011;42:12Y19. 14. Weaver DT, Murdock TJ. Telemedicine detection of type 1 ROP in a distant neonatal intensive care unit. J AAPOS. 2012;16:229Y233. 15. Lorenz B, Spasovska K, Elflein H, et al. Wide-field digital imaging based telemedicine for screening for acute retinopathy of prematurity (ROP). Six-year results of a multicentre field study. Graefes Arch Clin Exp Ophthalmol. 2009;247:1251Y1262. 16. Williams SL, Wang L, Kane SA, et al. Telemedical diagnosis of retinopathy of prematurity: accuracy of expert versus non-expert graders. Br J Ophthalmol. 2010;94:351Y356. 17. Zepeda-Romero LC, Barrera-de-Leon JC, Gonzalez-Bernal C, et al. The utility of non-ophthalmologist examination of eyes at risk for serious retinopathy of prematurity. Ophthalmic Epidemiol. 2011;18:264Y268. 18. Saunders RA, Bluestein EC, Berland JE, et al. Can non-ophthalmologists screen for retinopathy of prematurity? J Pediatr Ophthalmol Strabismus. 1995;32:302Y304; discussion 5. 19. Azad RV, Manjunatha NP, Pal N, et al. Retinopathy of prematurity screening by non-retinologists. Indian J Pediatr. 2006;73:515Y518. 20. Viera AJ, Garrett JM. Understanding interobserver agreement: the kappa statistic. Fam Med. 2005;37:360Y363.
* 2013 Asia-Pacific Academy of Ophthalmology
Copyright © 2013 Asia Pacific Academy of Ophthalmology. Unauthorized reproduction of this article is prohibited.
Specialist Nurse Screening for Retinopathy of PrematurityVA Pilot Study Shaheen P. Shah, FRCOphth, MD(Res),*Þ Zhen Wu, BHSc (Nursing),* Sharron Iverson, BHSc (Nursing),* and Shuan Dai, MBBS, MS, FRANZCO*
Purpose: To compare the accuracy of retinopathy of prematurity (ROP) screening between nurse specialists and an expert pediatric ophthalmologist. Design: A comparative case series of ROP screening in a level 2 neonatal intensive care unit setting where there is a higher threshold for accepting very premature or unwell neonates. Methods: Trained specialist ROP nurses acquired wide-field digital images, graded ROP, and proposed a follow-up plan. This was compared with the findings by an ROP expert ophthalmologist. Outcomes include sensitivity, specificity, positive predictive value, and negative predictive value of ROP grading by trained ROP nurses. Results: Mean gestational age was 28.6 weeks, and mean birth weight was 1184 g of the 64 consecutive neonates included. A total 316 eye screens was performed. Image acquisition, grading, and a management plan by the ROP nurse were possible in all screens. In right eyes, the presence of any ROP (stage 90) was 15%. Sensitivity, specificity, positive predictive value, and negative predictive value of ROP grading were 91.7% (95% CI, 73%Y99%), 80.6% (95% CI, 72.9%Y86.9%), 45.8% (95% CI, 31.4%Y60.80%), and 98.2% (95% CI, 93.6%Y99.8%), respectively. Agreement on the management plan occurred in 84.8% of cases. In virtually all circumstances of disagreement, ROP nurses exaggerated the ROP grading present and/or recommended a repeat screen when discharge from service was more appropriate. Conclusions: Our preliminary findings demonstrated good agreement between ROP nurses and the ROP expert ophthalmologist. Further research in expanding the role of utilizing nonphysician health workers in ROP screening is suggested. Key Words: Retinopathy of prematurity, Screening, Wide-field digital imaging, Telemedicine, Agreement (Asia-Pac J Ophthalmol 2013;2: 300Y304)
R
etinopathy of prematurity (ROP) is a retinal ischemic disorder that affects premature infants of low birth weight (BW). Retinopathy of prematurity is one of the few causes of childhood visual loss that is preventable, and many evidence-based guidelines exist for the screening of ROP. The goal of an effective screening program is to identify the relatively few neonates who require treatment from among the much larger number of at-risk infants while minimizing the number of stressful examinations required for these sick infants.1
From the *Department of Ophthalmology, Auckland, New Zealand, Auckland; and †International Centre for Eye Health, London School of Hygiene and Tropical Medicine, London, UK. Received for publication May 1, 2013; accepted May 28, 2013. The authors have no funding or conflicts of interest to declare. Reprints: Shuan Dai, MBBS, MS, FRANZCO, Department of Ophthalmology, Auckland District Health Board, Private Bag 92-189, Auckland, New Zealand. E-mail: [email protected]. Copyright * 2013 by Asia Pacific Academy of Ophthalmology ISSN: 2162-0989 DOI: 10.1097/APO.0b013e31829dc72b
300
www.apjo.org
Screening is traditionally conducted by the ophthalmologist using a binocular indirect ophthalmoscope. More recently, a number of studies have demonstrated excellent agreement with the traditional method when ophthalmologists screen for ROP using images captured with wide-field digital retinal imaging (WFDRI) cameras.2 Image acquisition by a trained nonphysician is widely accepted; however, a recent review3 highlighted the need for further research into the possibility of ROP grading by trained nonphysician health professionals as is currently the accepted standard in diabetic retinopathy screening.4 To our knowledge, no studies have been published on a screening program undertaken by trained ROP nurses under the supervision of ophthalmologists using WFDRI.
MATERIALS AND METHODS This study was an observational, blinded agreement study of ROP screening using images of ROP gathered by wide-field retinal imaging. Currently, ROP screening in 2 peripheral level 2 neonatal intensive care units (NICUs) approximately 20 km away from the tertiary level 3 NICU is conducted solely on ‘‘store-and-forward’’ telemedicine technology. This is a technology where medical data are captured for subsequent interpretation remotely by an expert. These peripheral units have a higher threshold for accepting very premature or unwell neonates from the city’s level 3 unit, and typically only babies older than 30 weeks’ corrected gestational age (cGA) or gestational age (GA) and weighing more than 1250 g who are medically improving are allowed onto the unit. Two specialist-trained ROP nurses (Z.W., S.I.), each with more than 5 years of experience in performing Retcam WFDRI, travel weekly with a medical photographer to the 2 peripheral NICUs and perform the ROP screening examinations. As per a standard protocol, pupils are dilated with cyclopentolate 0.5% and phenylephrine 2.5%, instilled 30 to 45 minutes before screening. The trained nurse operates the handpiece of Retcam-Shuttle (Clarity Medical Systems, Pleasanton, Calif ), which is equipped with 130-degree ROP lens while the photographer focuses and captures the image on the Retcam. A second NICU nurse swaddles the infant and monitors vital signs. A minimum of 3 images, (i) posterior pole, (ii) nasal to disc periphery, and (iii) temporal to disc periphery), are taken from each eye. At the end of the session, all the images are uploaded to the secure hospital server. Either the same or the next working day the images are read by the ROP specialist pediatric ophthalmologist (S.D.) at a review station in the tertiary ophthalmology unit. The decision on further management (for example, the date of the next screen) is taken and then relayed back to the peripheral units electronically. For the purposes of this study, all neonates consecutively undergoing ROP screening in both peripheral NICUs over a 13-month period were recruited. In addition to the normal screening process described above, the specialist nurses, after completing each screen, documented the ROP stage, zone, and presence of plus disease on each eye as per the international
Asia-Pacific Journal of Ophthalmology
&
Volume 2, Number 5, September/October 2013
Copyright © 2013 Asia Pacific Academy of Ophthalmology. Unauthorized reproduction of this article is prohibited.
Asia-Pacific Journal of Ophthalmology
&
Volume 2, Number 5, September/October 2013
Screening for Retinopathy of Prematurity
TABLE 1. Gestational Age, cGA, and BW of the 68 Neonates Included in This Study Corrected Gestational Age* G34.5 wk
Q34.5 and Q37.5 wk
937.5 wk
Total
Birth Weight (g)
Gestational Age, wk
n
%
n
%
n
%
n
%
Mean
SD
G27.5 Q27.5 and G29 Q29 Total
4 9 4 17
23.5 52.9 23.5 100
12 9 17 38
31.6 23.7 44.7 100
4 4 4 12
33.3 33.3 33.3 100
20 22 25 67
29.9 32.8 37.3 100
932.3 1273.9 1312.2 1186.2
177.3 279.6 310.5 311.6
Median 897.5 1300 1240 1210
IQR 820Y990 1230Y1380 1100Y1500 915Y1346
*At first screen. IQR, interquartile range.
Early Treatment for Retinopathy of Prematurity guidelines5 and proposed a plan of management for the neonate. Data were entered onto a study-specific database and exported to STATA (Statistical Software: Release 11.0; StataCorp, College Station, TX, 2001) for management and analysis. The unique National Health Index number and the date of examination were used to link the 2 assessments. Data entry errors were examined by cross-checking data points with the original data collection forms on a randomly selected 10% of the sample (no data entry errors were found in this subset). Birth weight, GA, (calculated from first day of last menstrual period), and cGA were categorized into approximate tertiles. Referral warranted ROP was defined as any ROP in zone 1, presence of plus disease, or presence of any stage 3 ROP detected by the specialist ROP nurse at any time during infant’s hospital course.6 Sensitivity, specificity, and positive and negative predictive values6 for digital imaging by a trained ROP nurse were determined for the presence of ‘‘any ROP’’ (defined as stage 90, any zone in the eye). Interobserver agreement using the J statistic was used to measure chance-adjusted agreement for the presence of disease based on an accepted scale.7 The findings of each specialist nurse were compared with those of the expert. As disease in the right and left eyes of an individual are highly correlated, analysis was conducted on right and left eyes separately. This project was approved by the Auckland District Health Board Research Review Committee.
RESULTS A total of 67 neonates were screened and included in the study. Nineteen neonates had only 1 screening performed before discharge, 18 had 2 separate screening examinations, 21 had 3 examinations, and 10 neonates had 4 examinations. Therefore, a total of 158 neonatal screens were performed. In no circumstances were ROP screens not able to be performed on the scheduled date. There were no reported adverse events. Neonates with GAs of less than 27.5 weeks, 27.5 weeks or greater but less than 29 weeks, and 29 weeks or greater accounted for 29.9%, 32.8%, and 37.3% of the population, respectively (Table 1). Mean GA was 28.6 (SD, 2.0) weeks, and mean BW was 1186.2 (SD, 311.6) g. The cGA dates of the first screen performed by the specialist nurse were as follows: 17 children (25.4%) with cGA of less than 34.5 weeks; 38 children (56.7%) were 34.5 weeks or greater but less than 37.5 weeks; and 12 children (17.9%) were 37.5 weeks or greater. The results for each nurse specialist when compared individually to the expert were similar, and therefore the combined data are presented. Diseases between eyes of the same * 2013 Asia-Pacific Academy of Ophthalmology
individual were similar; the presence of ‘‘any ROP’’ (by expert) in the right and left eye was 15% and 16%, respectively (Table 2). No child was noted to have plus disease by either assessor. One ‘‘referral warranted’’ mismatch was found (nurse: left eye stage 3 zone 2; physician: left eye stage 1 zone 2). Overall mismatch in assessment of ‘‘any ROP’’ existed in 55 (17.4%) of 316 screens. In the vast majority, these were false positives (nurses graded ‘‘any ROP’’ present when none was). Importantly, there were 5 false-negative screens (1.6%, all zone 3 or peripheral zone 2 with stage 1 disease). Sensitivity of grading of ‘‘any’’ ROP was 91.7% (95% CI, 73%Y99%) and 88% (95% CI, 68.8%Y97.5%), and specificity was 80.6% (95% CI, 72.9%Y86.9%) and 82% (95% CI, 74.4%Y88.1%) in right and left eyes, respectively. Results were very similar in right and left eyes, and thus only right-eye data were tabulated (Table 2). Incorrect level of staging occurred in 69 screens (21.8%). Generally, nurse staging was more likely to report a worse staging than that actually present, but in 7 screens (2.2%) nurses reported a lower stage than that graded by the ROP expert. Agreement of the management plan between the 2 screeners was 84.8%. J Values of agreement of presence or absence of ‘‘any ROP,’’ stage/zone of disease, and management are shown in Table 3.
DISCUSSION Retinopathy of prematurity is a leading cause of childhood blindness,8 and worldwide we are currently experiencing the ‘‘third ROP epidemic’’8 through higher premature birth rates and improved neonatal survival. A rapid expansion of neonatal services provision in economically emerging economies in addition to a shortage of trained ophthalmologists conducting ROP screening and less willingness to undertake ROP management by experienced ophthalmologists due to increasingly hazardous nature of litigation has also negatively affected screening programs worldwide. These problems of capacity and resource have led to intense operational research to determine alternative safe screening models from the traditional method. In particular, they have focused on the possibility of utilizing the skills of allied nonphysician health workers. Our ROP nurses have extensive experience in ROP screening, and this provided an ideal and safe setting in which to evaluate the accuracy of this alternative model of screening. The study findings led us to believe that an alternative model of screening, one where WFDRI acquisition, grading, and management by a trained nurse (supported by an off-site, highly accessible ROP expert) may be an answer to some of the current problems in ROP screening outlined above. The www.apjo.org
Copyright © 2013 Asia Pacific Academy of Ophthalmology. Unauthorized reproduction of this article is prohibited.
301
Asia-Pacific Journal of Ophthalmology
Shah et al
&
Volume 2, Number 5, September/October 2013
TABLE 2. Assessment of Nurse Grading Compared With the Criterion-Standard Expert ROP Screening Any ROP Nurse Present
Absent
Present % (Row) Absent % (Row) Total
22 91.7 26 19.4 48
2 8.3 108 80.6 110
% (Row)
30.4
69.6
Present % (Row) Absent % (Row) Total
22 88 24 18.0 46
3 12 109 82.0 112
% (Row)
29.1
70.9
0 % (Row) 1 % (Row) 2 % (Row) Total % (Row)
0 108 80.6 2 9.1 0 0 110 69.6
2 % (Row) 3 % (Row) Total % (Row)
2 41 87.2 43 38.7 84 53.2
95% Confidence Interval
Total
Right eye
Physician
24 100 134 100 158
Prevalence Sensitivity Specificity Likelihood ratio (+) Positive predictive value Negative predictive value
100.0
15% 91.70% 80.60% 4.72 45.80%
10%Y21.80% 73%Y99% 72.90%Y86.90% 3.28Y6.81 31.40%Y60.80%
98.20%
93.60%Y99.80%
16% 88% 82% 4.88 47.80%
11%Y22.50% 68.80%Y97.50% 74.40%Y88.10% 3.3Y7.2 32.90%Y63.10%
97.30%
92.40%Y99.40%
Left eye
Physician
25 100 133 100 158
Prevalence Sensitivity Specificity Likelihood ratio (+) Positive predictive value Negative predictive value
100.0
Stage of ROP Nurse Right eye
Physician
1 25 18.7 14 63.6 1 50 40 25.3
2 1 0.7 6 27.3 1 50 8 5.1
134 100 22 100 2 100 158 100
Zone of ROP Nurse Right eye
Physician
3 6 12.8 68 61.3 74 46.8
Total 47 100 111 100 158 100
Management
Physician
Follow-up in NICU % (Row) OPD % (Row) Total % (Row)
Follow-up in NICU 96 93 17 30.9 113 71.5
Nurse Follow-up in OPD 7 7 38 69.1 45 28.5
Total 103 100 55 100 158 100
*Numbers in bold and italics indicate mismatches between nurse and expert ROP assessments.
evidence to support this model of screening is generated from a number of broad perspectives: (i) An internationally accepted ROP classification system has led to a standardization of
302
www.apjo.org
diagnosis and a standardization of treatment guidelines. Robust evidence-based studies have allowed a simplification of the ROP decision-making algorithm, and ultimately, it is the * 2013 Asia-Pacific Academy of Ophthalmology
Copyright © 2013 Asia Pacific Academy of Ophthalmology. Unauthorized reproduction of this article is prohibited.
Asia-Pacific Journal of Ophthalmology
&
Volume 2, Number 5, September/October 2013
Screening for Retinopathy of Prematurity
TABLE 3. J Agreement for ‘‘Any ROP,’’ Management, Stage, and Zone of ROP
Any ROP
Agreement
Expected Agreement
J
SE
P
82.3% 82.9% 84.8% 77.9% 78.5% 69.0% 69.6%
63.7% 64.3% 56.5% 62.6% 63.2% 48.7% 49.0%
51.2% 52.2% 65.1% 40.7% 41.5% 39.5% 40.5%
0.0723 0.0738 0.0787 0.0649 0.0661 0.0708 0.0712
G0.001 G0.001 G0.001 G0.001 G0.001 G0.001 G0.001
Right eye Left eye
Follow-up plan Stage of ROP Zone of ROP
Right eye Left eye Right eye Left eye
presence of significant ROP in zone I or plus disease that is the critical finding that determines treatment.9,10 This simplification, by creating a focus on visualizing the pathology at the posterior pole, has made ROP easier to manage. (ii) The advent of new technologies that allow relatively easy capture of excellent quality wide-field images of the retina, particularly of the posterior pole, has revolutionized the traditional screening method using the indirect ophthalmoscope. Many studies have shown that nonphysician health workers can successfully use these systems to capture quality WFDRIs,11 and as a result, many units around the world now rely solely on these professionals for image acquisition. This can be reiterated in our study where image acquisition was possible in all cases. (iii) Screening by analysis of WFDRI alone is shown to be safe, and there is good level 1 evidence that retinal photography has high accuracy for the detection of clinically significant ROP.10 Sensitivities of 100% and specificities of 97.9% in detecting infants with treatment-requiring ROP by ophthalmologists reading Retcam 2 or Retcam-Shuttle images when compared with binocular indirect ophthalmoscope have been achieved,12 and the American Academy of Ophthalmology recently concluded that WFDRI provided unique advantages in ROP management through objective documentation of findings and improved recognition of disease progression.10 (iv) Management decisions (eg, treatment, next screening date) by expert ophthalmologists reading WFDRIs remotely using store-and-forward telemedicine have also been found to have very high sensitivity and specificity,6,11,13 and studies using this method in Canada and Germany reported no adverse outcomes over a 4.5and 6-year period, respectively.14,15 Dedicated trained readers of retinal images for the purposes of screening for diabetic retinopathy are now considered standard, yet one may argue that the retinal changes that determine the management decisions in ROP screening are not as difficult to detect as those in diabetic retinopathy. Richter and colleagues’3 comprehensive ROP review concluded that additional research on the accuracy of trained nurses for image grading was required, and Williams et al,16 in the first study of this nature, compared grading of WFDRI images by nonexperts who were trained by attending only 2 hours of training on image-based ROP diagnosis. Even with this short duration of training, several of their nonexpert graders performed comparably to the experts.16 The investigators were concerned about a high number of ‘‘unknown’’ responses by the nonexperts. This was found not to be a problem in our study. A number of studies used trained nonspecialists to screen for ROP, and the results have been encouraging. Zepeda-Romero et al17 trained a neonatologist and a pediatrician to use the indirect ophthalmoscope; Saunders et al18 trained a neonatologist to use the direct ophthalmoscope, and Azad et al19 trained residents and nurses with the same. In all studies, examination was directed only at observing vascular * 2013 Asia-Pacific Academy of Ophthalmology
changes at the posterior pole. Without objective documentation of the retinal appearance, these methods are limited (if a second ‘‘expert opinion’’ was required, this would necessitate a repeat onsite examination). The telemedicine approach to screening of ROP is, however, limited by several factors, the most relevant of which is the fact that dynamic ‘‘real-time’’ visualization of the retina is not yet widely feasible (ie, areas of concern or uncertainty cannot be revisited by the remote reader who is limited to reading only captured still images). Indeed, this may explain some of the false-positive results in the present study. Other factors such as the technical aspects, network maintenance and data security, and the high set-up and maintenance costs involved as well as the acceptability of telemedicine to patients and medical providers are barriers limiting this approach. Results of an ongoing multicenter study in the United States, which has established a grading center for WFDRI grading by nonphysician graders (eROP study, http://clinicaltrials.gov/ct2/show/NCT01264276), will be useful in directing future ROP screening models. The high sensitivity and specificity values found in this comparative study suggest safety in methodology. Clinically, ROP findings between right and left eyes are generally quite similar, but most published studies base statistics on eye-level screens, thus exaggerating sample size and statistical power. In this study, we attempted to limit the analysis to the person level. Most mismatching was conservative in nature (false positives), and false negatives were rare. The high number of follow-up screens recommended when discharge was deemed appropriate by the expert was of particular concern as these examinations are not without risk to the neonate. To our knowledge, this is the first study to examine nonphysician clinical decision making in ROP, and the results were encouraging. Although all NICUs are stratified to admit neonates according to levels of complexity appropriate for their personnel and resources, they each must provide an effective ROP service, and it is in the ‘‘low-risk’’ NICU setting that the results of this study apply. Thus, the applicability of our findings is limited by (i) the low prevalence of ‘‘any ROP’’ and (ii) the lack of clinically significant ROP pathology in the sample (eg, plus disease or zone 1 disease), and thus further research is warranted in a tertiary level 3 NICU, where more disease is anticipated. Our results reflect current ROP screening in level 2 units in Auckland; it is our impression that, with high standard of modern neonatal care, the large levels of ‘‘referral warranted’’ ROP detected only a decade ago6 are now simply not observed. Overall agreement of ‘‘any ROP’’ was greater than 80%, but the J coefficient found in this study would be classed as ‘‘moderate.’’ The value of the J coefficient, however, is limited as these statistics are affected by prevalence of the condition, and low J values in conditions of low prevalence do not necessarily reflect low rates of overall agreement.20 www.apjo.org
Copyright © 2013 Asia Pacific Academy of Ophthalmology. Unauthorized reproduction of this article is prohibited.
303
Asia-Pacific Journal of Ophthalmology
Shah et al
A number of operational rules and responsibilities would need to be clearly defined for this screening model. Unlike diabetic retinopathy, ROP progression is aggressive, and therefore robust time-defined protocols are critical (eg, clear channels of access to expert second opinion as well as clearly defined clinical and medicolegal liabilities). The nurses in this study gained their training from ‘‘on the job’’ observational experience; however, training programs with certification of nonspecialist screeners would need to be agreed and formalized. REFERENCES 1. Section on Ophthalmology, American Academy of Pediatrics, American Academy of Ophthalmology, American Association for Pediatric Ophthalmology and Strabismus. Screening examination of premature infants for retinopathy of prematurity. Pediatrics. 2006;117:572Y576. 2. Salcone EM, Johnston S, VanderVeen D. Review of the use of digital imaging in retinopathy of prematurity screening. Semin Ophthalmol. 2010;25:214Y217. 3. Richter GM, Williams SL, Starren J, et al. Telemedicine for retinopathy of prematurity diagnosis: evaluation and challenges. Surv Ophthalmol. 2009;54:671Y685. 4. Taylor CR, Merin LM, Salunga AM, et al. Improving diabetic retinopathy screening ratios using telemedicine-based digital retinal imaging technology: the Vine Hill study. Diabetes Care. 2007;30: 574Y578. 5. Good WV, Hardy RJ, Dobson V, et al. Final visual acuity results in the early treatment for retinopathy of prematurity study. Arch Ophthalmol. 2010;128:663Y671. 6. Ells AL, Holmes JM, Astle WF, et al. Telemedicine approach to screening for severe retinopathy of prematurity: a pilot study. Ophthalmology. 2003;110:2113Y2117. 7. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33:159Y174. 8. Gilbert C. Retinopathy of prematurity: a global perspective of the epidemics, population of babies at risk and implications for control. Early Hum Dev. 2008;84:77Y82.
304
www.apjo.org
&
Volume 2, Number 5, September/October 2013
9. Early Treatment for Retinopathy of Prematurity Cooperative Group. Revised indications for the treatment of retinopathy of prematurity: results of the early treatment for retinopathy of prematurity randomized trial. Arch Ophthalmol. 2003;121:1684Y1694. 10. Chiang MF, Melia M, Buffenn AN, et al. Detection of clinically significant retinopathy of prematurity using wide-angle digital retinal photography: a report by the American Academy of Ophthalmology. Ophthalmology. 2012;119:1272Y1280. 11. Chiang MF, Wang L, Busuioc M, et al. Telemedical retinopathy of prematurity diagnosis: accuracy, reliability, and image quality. Arch Ophthalmol. 2007;125:1531Y1538. 12. Dai S, Chow K, Vincent A. Efficacy of wide-field digital retinal imaging for retinopathy of prematurity screening. Clin Exp Ophthalmol. 2011;39:23Y29. 13. Silva RA, Murakami Y, Lad EM, et al. Stanford University network for diagnosis of retinopathy of prematurity (SUNDROP): 36-month experience with telemedicine screening. Ophthalmic Surg Lasers Imaging. 2011;42:12Y19. 14. Weaver DT, Murdock TJ. Telemedicine detection of type 1 ROP in a distant neonatal intensive care unit. J AAPOS. 2012;16:229Y233. 15. Lorenz B, Spasovska K, Elflein H, et al. Wide-field digital imaging based telemedicine for screening for acute retinopathy of prematurity (ROP). Six-year results of a multicentre field study. Graefes Arch Clin Exp Ophthalmol. 2009;247:1251Y1262. 16. Williams SL, Wang L, Kane SA, et al. Telemedical diagnosis of retinopathy of prematurity: accuracy of expert versus non-expert graders. Br J Ophthalmol. 2010;94:351Y356. 17. Zepeda-Romero LC, Barrera-de-Leon JC, Gonzalez-Bernal C, et al. The utility of non-ophthalmologist examination of eyes at risk for serious retinopathy of prematurity. Ophthalmic Epidemiol. 2011;18:264Y268. 18. Saunders RA, Bluestein EC, Berland JE, et al. Can non-ophthalmologists screen for retinopathy of prematurity? J Pediatr Ophthalmol Strabismus. 1995;32:302Y304; discussion 5. 19. Azad RV, Manjunatha NP, Pal N, et al. Retinopathy of prematurity screening by non-retinologists. Indian J Pediatr. 2006;73:515Y518. 20. Viera AJ, Garrett JM. Understanding interobserver agreement: the kappa statistic. Fam Med. 2005;37:360Y363.
* 2013 Asia-Pacific Academy of Ophthalmology
Copyright © 2013 Asia Pacific Academy of Ophthalmology. Unauthorized reproduction of this article is prohibited.
