The Laryngoscope C 2014 The American Laryngological, V
Rhinological and Otological Society, Inc.
Interrater Reliability of Sleep Videofluoroscopy for Airway Obstruction in Obstructive Sleep Apnea Dong-Kyu Kim, MD; Woo-Hyun Lee, MD; Chul Hee Lee, MD; Chae Seo Rhee, MD; Jeong-Whun Kim, MD Objectives/Hypothesis: Sleep videofluoroscopy (SVF) has been introduced to identify upper airway obstruction. This study was aimed to determine the interrater reliability of SVF in patients with obstructive sleep apnea (OSA). Study Design: A retrospective analysis. Methods: On the basis of apnea-hypopnea index in full-night attended polysomnography, 374 consecutive OSA patients who underwent SVF were enrolled in this study. The SVF was evaluated by three independent reviewers. Interrater reliabilities were assessed by evaluating agreement of the obstructive anatomic structures (soft palate, tongue base, tonsils, and epiglottis) and airway levels (velopharynx, oropharynx, and hypopharynx) between the reviewers. Results: In a comparison between an unblinded and a blinded well-experienced sleep surgeons, the interrater reliability for the presence of obstruction was the highest for the soft palate at the level of the velopharynx (Cohen’s kappa value, 0.919) and the lowest for the soft palate at the level of the oropharynx (Cohen’s kappa value, 0.757). In a blind comparison between a well-experienced and less-experienced sleep surgeons, the interrater reliability for the presence of obstruction was also the highest for the soft palate at the level of the velopharynx (Cohen’s kappa value, 0.938) and the lowest for the palatine tonsils at the level of the oropharynx (Cohen’s kappa value, 0.635). Conclusion: This study showed that SVF was a diagnostic modality that can be used to evaluate upper airway obstruction without significant interrater disagreements. Key Words: Obstructive sleep apnea, sleep videofluoroscopy, fluoroscopy, interrater, reliability, validity. Level of Evidence: 4. Laryngoscope, 124:1267–1271, 2014
INTRODUCTION Obstructive sleep apnea (OSA) is a multilevel, dynamic, and complex disease caused by repetitive upper airway collapses during sleep. Although the disease severity of OSA can be diagnosed by polysomnography, the information on the anatomical problems cannot be obtained from polysomnography. When surgery is considered for the management of OSA, preoperative identification of the obstruction sites and levels in the upper airway is very important. A precise anatomical investigation of the upper airway may lead to successful individualized treatment, including single-level or multilevel surgeries. Variable methods have been attempted to achieve the most precise localization of the obstructive uller sites, including Friedman staging system,1 M€
From the Department of Otorhinolaryngology–Head and Neck Surgery, Chuncheon Sacred Heart Hospital, Hallym University College of Medicine (D-K.K.), Chuncheon; and the Departments of Otorhinolaryngology, Seoul National University College of Medicine, Seoul National University Bundang Hospital (W-H.L., C.H.L., C.S.R., J-W.K.), Seongnam, Republic of Korea. Editor’s Note: This Manuscript was accepted for publication November 4, 2013. The authors have no funding, financial relationships, or conflicts of interest to disclose. Send correspondence to Jeong-Whun Kim, MD, PhD, Associate Professor, Department of Otorhinolaryngology, Seoul National University College of Medicine, Seoul National University Bundang Hospital, 166 Goomi-ro, Bundang-gu, Seongnam, 463–707, South Korea. E-mail: [email protected] DOI: 10.1002/lary.24509
Laryngoscope 124: May 2014
maneuver,2,3 cephalometry,4,5 dynamic computed tomography (CT), and magnetic resonance imaging (MRI) scans.6,7 However, these diagnostic tools have clinical limitations in that they are usually performed during wakefulness and produce only static information on the upper airway. Recently, some dynamic airway evaluations were developed and introduced, including drug-induced sleep endoscopy (DISE),8–11 sleep MRI,12,13 and sleep videofluoroscopy (SVF)14–17 during sleep. DISE has been performed popularly and also known to be a safe, feasible, and valid assessment in OSA patients. Sleep MRI could also localize the obstructive sites in the upper airway during sleep. In addition, a Barrera’s study12 showed a higher reliability of sleep MRI. In the previous studies, SVF could provide high resolution dynamic images of the upper airway during induced sleep.14–17 Thus, SVF may be helpful to establish surgical plans for patients based on the dynamic characteristics of their upper airway. There were studies showing that the SVF may also be helpful in the selection of candidates for uvulopalatopharyngoplasty.14,18,19 Another recent study demonstrated that the SVF could show mouth breathing in addition to abnormal movements of the upper airway and that the severity of mouth opening was a significant factor to predict surgical outcomes.20 Although several previous studies showed that the SVF is a useful diagnostic modality for evaluating obstruction sites of the upper airway in OSA,14–20,19–22 Kim et al.: Interrater Reliability of Sleep Videofluoroscopy
1267
the interrater reliability of SVF has not been studied. Thus, the objective of this study was to analyze the agreement of the SVF findings evaluated by different examiners.
MATERIALS AND METHODS Subjects We retrospectively studied 374 consecutive individuals who were diagnosed as having OSA. All patients underwent full-night attended polysomnography and SVF from September 2010 through June 2012. Patients with apnea-hypopnea index (AHI) 5 and body mass index (BMI) < 40 were included. Exclusion criteria were as follows: 1) younger than 18 years of age; 2) severe medical diseases and allergy to midazolam or sedatives; 3) previous sleep surgery; and 4) craniofacial syndromes, neuromuscular diseases, or psychiatric disorders. This study was approved by the internal review board of Seoul National University Bundang Hospital.
Sleep Videofluoroscopy Protocol All OSA patients underwent SVF, as previously described, for about 2 minutes.14–17 All of the procedures were conducted after an informed consent was documented. Briefly, they were placed in the supine position on a C-arm table (Allura Xper FD20; Philips, Amsterdam, Netherlands) with their head on a comfortable pillow. An intravenous fluid line was placed on the patient arm for injection of midazolam (Bukwang Pharmacy, Seoul, Korea) and to prepare emergency situations that could accidentally occur during SVF. All images were real-time twodimensional lateral views acquired with a rotational angiographic C-arm system. Patients were instructed to breathe in and out naturally. Oxygen saturation was monitored throughout the examination using a finger pulse oximeter (Care Vision; Medical supply, Seoul, Korea). An otorhinolaryngologic doctor monitored breathing and oxygen saturation of the patients throughout the study. After the soft palate, palatine tonsils, tongue base, and epiglottis could clearly be delineated by SVF, image recording was started. During normal respiration before sedation, an awake event was recorded for 15 seconds. Thereafter, all lights in the examination room were turned off to help the patients fall asleep; sleep was also induced by intravenously administering midazolam (initial dose: 0.05 mg/kg). If patients did not fall asleep after the initial injection, an additional 0.02 mg/kg of midazolam was injected to a maximum dose of 0.1 mg/ kg. The drug-induced sleep was indirectly assessed by development of snoring and oxygen desaturation. In most of the patients with OSA, patients began to snore soon after administration of midazolam, and oxygen saturation decreased with respiratory disturbances. Only when oxygen saturation dropped by 4% or more were videofluoroscopic images for oxygen desaturation recorded. Two 15-second desaturation sleep events were recorded as desaturation sleep events. We could observe that upper airway obstruction was associated with oxygen desaturation during SVF in most of the patients. After the study was completed, flumazenil (0.02 mg/kg, Bukwang pharmacy, Seoul, Korea), the antidote of midazolam, was administrated intravenously to reverse the sedative effect of midazolam and to help the patients resume their normal life as soon as possible.
Evaluation of Sleep Videofluoroscopic Findings and Interrater Agreement The patterns of upper airway obstruction were investigated according to the previous studies in terms of anatomic
Laryngoscope 124: May 2014
1268
Fig. 1. This is a sample chart used during analysis. Each white cell is filled with 0 or 1 as a degree of obstruction. Zero denotes no obstruction and 1 denotes collapse of the upper airway. Gray cells mean that the specific structure is not involved in the specific obstruction level. The velopharynx and oropharynx are divided by the lower margin of the soft palate, and the oropharynx and hypopharynx are divided by the tip of the epiglottis. structures and airway levels involved in the obstruction.14–16 The analysis was performed for the desaturation sleep events. The obstructive anatomic structures involved the soft palate, tongue base, tonsils, and epiglottis. The obstructive airway level was divided into the velopharynx, oropharynx, and hypopharynx. While the velopharynx and oropharynx were divided by the lower margin of the soft palate, the oropharynx and hypopharynx were divided by the tip of the epiglottis. The patterns of obstruction were categorized as follows: obstruction by the soft palate at the velopharynx; obstruction by the soft palate, palatine tonsils, or tongue base at the oropharynx; and obstruction by the tongue base or epiglottis at the hypopharynx (Figure 1). To assess the interrater agreements, the video images of SVF were reviewed by three different sleep surgeons: two well-experienced sleep surgeons who had reviewed more than 200 SVF images and a less-experienced sleep surgeon who had reviewed less than 20 SVF images. One well-experienced sleep surgeon (J.-W.K.) unblindly analyzed all the findings of SVF at the first time. Thereafter, the digitally recorded video images were blindly reviewed independently by the other wellexperienced sleep surgeon (C.S.R.) and a less-experienced sleep surgeon (D.-K.K.) without being aware of their Friedman stage and polysomnographic results.
Statistical Analysis All descriptive results were expressed as a mean 6 standard deviation. To analyze interrater reliabilities, the percentage of agreement was calculated and the Cohen’s kappa test was used. The degree of agreement was considered moderate if the kappa values were between 0.41 and 0.60, substantial if between 0.61 and 0.80, and almost perfect or excellent if between 0.81 and 1.00.17 If the Cohen’s kappa values were greater than 0.6, the test was considered acceptable as a recommended method for clinical decision.23 All statistical analyses were performed using SPSS statistics version 19.0 (SPSS, Chicago, IL).
RESULTS Characteristics of Patients A total of 374 consecutive subjects (72 women and 302 men) were included in this study. Out of them, 44 Kim et al.: Interrater Reliability of Sleep Videofluoroscopy
Fig. 2. Prevalence of obstructive anatomical structures and levels evaluated by different sleep surgeons (unblinded well-experienced, blinded well-experienced, and blinded less-experienced). hypo 5 hypopharynx; oro 5 oropharynx; velo 5 velopharynx.
patients had mild OSA (5 AHI < 15), 102 patients had moderate OSA (15 (AHI < 30), and 228 patients had severe OSA (AHI 30). Their mean age was 48.4 years (6 13), mean AHI 28.2/h (6 24.4), and mean BMI 25.6 kg/m2 (6 2.9). The mean Epworth sleepiness scale and Pittsburgh sleep quality index was 10.6 and 8.5, respectively.
tion by the soft palate at the level of the oropharynx, and the highest kappa value was 0.919 for the obstruction by the soft palate at the level of the velopharynx. In the comparison between the blinded well-experienced and the less-experienced sleep surgeons, the lowest kappa value was 0.635 for the obstruction by the palatine tonsils at the level of the oropharynx, and the highest kappa value was 0.938 for the obstruction by the soft palate at the level of the velopharynx.
Prevalence of Obstructive Structures and Levels The prevalence of obstructive structures and levels evaluated by each examiner is shown in Figure 2. The most prevalent obstructive structures and levels were the soft palate at the level of the velopharynx for the unblinded and blinded well-experienced sleep surgeons (57% and 58%, respectively), whereas it was the soft palate at the level of the oropharynx (64%) for the blinded less-experienced sleep surgeon. The least prevalent obstructive structures and levels were the palatine tonsils at the level of the oropharynx for all observers.
Interrater Reliability of Obstructive Structures and Levels The interrater reliabilities analyzed by percentages of agreement and kappa values are presented in Table I. In the comparison between the well-experienced sleep surgeons, the lowest rate of agreement was 90.8% for the obstruction by the soft palate at the level of the oropharynx, and the highest rate of agreement was 99.1% for the obstruction by the palatine tonsils at the level of the oropharynx. In the blinded comparison between the well-experienced and less-experienced sleep surgeons, the lowest rate of agreement was 89.6% for the obstruction by the tongue base at the level of the oropharynx, and the highest rate of agreement was 97.2% for the obstruction by the soft palate at the level of the velopharynx. In the analysis of kappa values, the comparison between the two well-experienced sleep surgeons showed that the lowest kappa value was 0.757 for the obstrucLaryngoscope 124: May 2014
DISCUSSION Among the treatment modalities, surgery and mandibular advancement devices should be administered to the patients after thorough evaluation of the upper airway. Although the overall success rate of uvulopalatopharyngoplasty for OSA has been reported to be less than optimal (40.7%) in the literature,24 some studies showed higher success rates in preevaluated patients.25–27 Anatomical factors are also involved in successful outcomes in the patients using mandibular TABLE I. Interrater Reliability of Sleep Video Fluoroscopy Results. Comparison Between Unblinded and Blinded Well-Experienced Sleep Surgeons % Agreement
Cohen’s Kappa
Comparison Between Blinded Well- and Less-Experienced Sleep Surgeons % Agreement
Cohen’s Kappa
97.2
0.938
Obstruction at the Velopharynx Soft palate
96.4
0.919
Obstruction at the Oropharynx Soft palate 90.8 0.757
91.5
0.794
Palatine tonsils
0.918
94.2
0.635
Tongue base 93.7 0.840 Obstruction at the Hypopharynx
89.6
0.782
99.1
Tongue base
92.1
0.775
92.8
0.783
Epiglottis
94.0
0.826
92.6
0.776
Kim et al.: Interrater Reliability of Sleep Videofluoroscopy
1269
advancement devices.16,17,28 Even if a precise evaluation of the obstructive sites by reliable methods is indispensable as a preoperative evaluation, there is no standard method. The SVF is one of the new methods to evaluate dynamic abnormalities of the upper airway. This study was designed to evaluate the interrater reliability of SVF examination. Although several studies have already reported the efficacy of SVF for the anatomical evaluation of OSA,14–17 to our knowledge the present study is the first for determining the rater-torater reliability of SVF. In our study, interrater agreement was studied including both well-experienced and less-experienced sleep surgeons to identify whether experiences for SVF analysis can influence the accuracy of analyses. According to our results, the obstructive level-bylevel agreements were more than 90% in the comparison between the unblinded and blinded well-experienced sleep surgeons. In the oropharyngeal level, the agreement for the soft palate was slightly lower than in the other subsites. This might be attributed to a frequent overlapping of a backward and downward movement of the soft palate, with a backward movement of the tongue base and a forward movement of the posterior pharyngeal wall. In the comparison between the blinded welland less-experienced sleep surgeons, the agreement was also high, suggesting that SVF analysis may require a short learning curve. In the present study, the kappa values for the interrater reliability were all higher than 0.6, meaning substantial-to-perfect agreements. (Table II). However, there were some differences between the two comparison arms. For the obstruction at the velopharyngeal and hypopharyngeal levels, different testers showed almost similar analysis results. On the other hand, at the oropharyngeal levels the agreements were higher in the comparison between the two wellexperienced surgeons than in the comparison between the well- and less-experienced surgeons: The agreements between the two experts for the soft palate, palatine tonsils, and tongue base at the level of the oropharynx were 0.757, 0.918, and 0.840, respectively; those between the expert and the beginner for the soft palate, palatine tonsils, and tongue base were 0.794, 0.635, and 0.782, respectively. This implies that the analysis of the oropharyngeal level may be relatively more difficult for beginners, and those less-experienced surgeons should pay more attention to the evaluation of the oropharyngeal level during SVF analysis. The complex anatomy of the oropharynx in SVF video clips may also be associated with the higher difficulty. Although the SVF is thought to be suited for dynamic evaluation of upper airway, it also has some limitations in OSA patients. First, a selection bias can be present because the SVF was performed during induced sleep by midazolam; therefore, only a part of total sleep events can be recorded. However, this selection bias is also present in other dynamic airway examination such as DISE8–11 and dynamic sleep MRI scan.12,13 Second, it is still unclear whether midazolaminduced sleep is physiologic. However, an article showed that polysomnography performed during midazolamLaryngoscope 124: May 2014
1270
induced sleep was highly correlated with polysomnography during natural sleep.29 Third, we did not monitor the stage of sleep or depth of sleep using concurrent polysomnography or bispectral index. However, because we could not observe any upper airway obstructions before sedation—and we analyzed the images only when upper airway obstructions were observed in association with oxygen desaturation—the obstructive findings are likely to be assessed as sleep-related breathing events. Useful diagnostic methods should demonstrate important features, such as safety, validity, and reliability. The SVF showed safety and validity in the previous studies.14–17,20 The results of the present study may provide an evidence for the interrater reliability of SVF findings. We found that the concordance rate and kappa value of SVF were higher than DISE.10 The previous study of interrater reliability in DISE showed relatively lower kappa values (0.42, 0.47, and 0.53 in tonsils, lateral wall of the hypopharynx and oropharynx, respectively).10 From these results, it is conceivable that SVF may be easier to evaluate for upper airway obstruction than DISE. The reason for the SVF to show higher interrater reliabilities may be as follows: The study population was different between SVF and DISE in the studies; the DISE was performed by sleep surgeons; and the analysis results may be affected by their endoscopic experience as well as anatomical knowledge, whereas SVF was conducted according to a preset standard protocol by a videofluoroscope without involvement of surgeon’s technical experiences. This study also has some limitations: 1) The number of video reviewers was limited. 2) Three video reviewers have worked in the same hospital; thus, further studies need to be conducted with a larger number of samples and reviewers working in various sleep centers. 3) Because an anatomical evaluation is not always likely to guarantee the surgical results, a prospective cohort study is needed for surgery or mandibular advance devices to be applied to the patients according to the anatomical diagnosis using SVF. 4) Because a lateral movement of the pharyngeal walls cannot be exactly shown in lateral video clips of SVF, it was excluded from the present study.
CONCLUSION The interrater reliability of SVF analysis for OSA was excellent and its analysis results performed by a less-experienced sleep surgeon were quite comparable to those by experienced sleep surgeons, suggesting its short learning curve. Therefore, SVF may provide enough information on the obstructive sites and levels, which would help to plan appropriate treatment for OSA based on the dynamic anatomy of the upper airway during sleep.
BIBLIOGRAPHY 1. Friedman M, Ibrahim H, Joseph NJ. Staging of obstructive sleep apnea/ hypopnea syndrome: a guide to appropriate treatment. Laryngoscope 2004;114:454–459. 2. Sher AE, Thorpy MJ, Shprintzen RJ, Spielman AJ, Burack B, McGregor PA. Predictive value of Muller maneuver in selection of patients for uvulopalatopharyngoplasty. Laryngoscope 1985;95:1483–1487.
Kim et al.: Interrater Reliability of Sleep Videofluoroscopy
3. Terris DJ, Hanasono MM, Liu YC. Reliability of the Muller maneuver and its association with sleep-disordered breathing. Laryngoscope 2000;110: 1819–1823. 4. Poirrier AL, Pire S, Raskin S, Limme M, Poirrier R. Contribution of postero-anterior cephalometry in obstructive sleep apnea. Laryngoscope 2012;122:2350–2354. 5. Julia-Serda G, Perez-Penate G, Saavedra-Santana P, et al. Usefulness of cephalometry in sparing polysomnography of patients with suspected obstructive sleep apnea. Sleep Breath 2006;10:181–187. 6. Bhattacharyya N, Blake SP, Fried MP. Assessment of the airway in obstructive sleep apnea syndrome with 3-dimensional airway computed tomography. Otolaryngol Head Neck Surg 2000;123:444–449. 7. Donnelly LF, Casper KA, Chen B, Koch BL. Defining normal upper airway motion in asymptomatic children during sleep by means of cine MR techniques. Radiology 2002;223:176–180. 8. Ravesloot MJ, de Vries N. One hundred consecutive patients undergoing drug-induced sleep endoscopy: results and evaluation. Laryngoscope 2011;121:2710–2716. 9. De Vito A, Agnoletti V, Berrettini S, et al. Drug-induced sleep endoscopy: conventional versus target controlled infusion techniques-a randomized controlled study. Eur Arch Otorhinolaryngol 2011;268:457–462. 10. Kezirian EJ, White DP, Malhotra A, Ma W, McCulloch CE, Goldberg AN. Interrater reliability of drug-induced sleep endoscopy. Arch Otolaryngol Head Neck Surg 2010;136:393–397. 11. Rodriguez-Bruno K, Goldberg AN, McCulloch CE, Kezirian EJ. Test-retest reliability of drug-induced sleep endoscopy. Otolaryngol Head Neck Surg 2009;140:646–651.12. 12. Barrera JE. Sleep magnetic resonance imaging: dynamic characteristics of the airway during sleep in obstructive sleep apnea syndrome. Laryngoscope 2011;121:1327–1335. 13. Moon IJ, Han DH, Kim JW, et al. Sleep magnetic resonance imaging as a new diagnostic method in obstructive sleep apnea syndrome. Laryngoscope 2010;120:2546–2554. 14. Lee CH, Mo JH, Kim BJ, et al. Evaluation of soft palate changes using sleep videofluoroscopy in patients with obstructive sleep apnea. Arch Otolaryngol Head Neck Surg 2009;135:168–172. 15. Lee CH, Hong SL, Rhee CS, Kim SW, Kim JW. Analysis of upper airway obstruction by sleep videofluoroscopy in obstructive sleep apnea: a large population-based study Laryngoscope 2012;122:237–241. 16. Lee CH, Kim JW, Lee HJ, et al. An investigation of upper airway changes associated with mandibular advancement device using sleep videofluoro-
Laryngoscope 124: May 2014
17.
18. 19.
20.
21.
22.
23. 24.
25. 26.
27.
28.
29.
scopy in patients with obstructive sleep apnea. Arch Otolaryngol Head Neck Surg 2009;135:910–914. Lee CH, Kim JW, Lee HJ, et al. Determinants of treatment outcome after use of the mandibular advancement device in patients with obstructive sleep apnea. Arch Otolaryngol Head Neck Surg 2010;136: 677–681. Rama AN, Tekwani SH, Kushida CA. Sites of obstruction in obstructive sleep apnea. Chest 2002;122:1139–1147. Walsh JK, Katsantonis GP, Schweitzer PK, Verde JN, Muehlbach M. Somnofluoroscopy: cineradiographic observation of obstructive sleep apnea. Sleep 1985;8:294–297. Lee CH, Mo JH, Seo BS, Kim DY, Yoon IY, Kim JW. Mouth opening during sleep may be a critical predictor of surgical outcome after uvulopalatopharyngoplasty for obstructive sleep apnea. J Clin Sleep Med 2010;6: 157–162. Suratt PM, Dee P, Atkinson RL, Armstrong P, Wilhoit SC. Fluoroscopic and computed tomographic features of the pharyngeal airway in obstructive sleep apnea. Am Rev Respir Dis 1983;127:487–492. Pepin JL, Ferretti G, Veale D, et al. Somnofluoroscopy, computed tomography, and cephalometry in the assessment of the airway in obstructive sleep apnoea. Thorax 1992;47:150–156. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977;33:159–174. Sher AE, Schechtman KB, Piccirillo JF. The efficacy of surgical modifications of the upper airway in adults with obstructive sleep apnea syndrome. Sleep 1996;19:156–177. Li HY, Lee LA. Relocation pharyngoplasty for obstructive sleep apnea. Laryngoscope 2009;119:2472–2477. Pang KP, Woodson BT. Expansion sphincter pharyngoplasty: a new technique for the treatment of obstructive sleep apnea. Otolaryngol Head Neck Surg 2007;137:110–114. Vicini C, Dallan I, Canzi P, et al. Transoral robotic surgery of the tongue base in obstructive sleep apnea-hypopnea syndrome: anatomic considerations and clinical experience. Head Neck 2012;34:15–22. Lee CH, Mo JH, Choi IJ, et al. The mandibular advancement device and patient selection in the treatment of obstructive sleep apnea. Arch Otolaryngol Head Neck Surg 2009;135:439–444. Gregorio MG, Jacomelli M, Inoue D, Genta PR, de Figueiredo AC, Lorenzi-Filho G. Comparison of full versus short induced-sleep polysomnography for the diagnosis of sleep apnea. Laryngoscope 2011;121:1098– 1103.
Kim et al.: Interrater Reliability of Sleep Videofluoroscopy
1271
Rhinological and Otological Society, Inc.
Interrater Reliability of Sleep Videofluoroscopy for Airway Obstruction in Obstructive Sleep Apnea Dong-Kyu Kim, MD; Woo-Hyun Lee, MD; Chul Hee Lee, MD; Chae Seo Rhee, MD; Jeong-Whun Kim, MD Objectives/Hypothesis: Sleep videofluoroscopy (SVF) has been introduced to identify upper airway obstruction. This study was aimed to determine the interrater reliability of SVF in patients with obstructive sleep apnea (OSA). Study Design: A retrospective analysis. Methods: On the basis of apnea-hypopnea index in full-night attended polysomnography, 374 consecutive OSA patients who underwent SVF were enrolled in this study. The SVF was evaluated by three independent reviewers. Interrater reliabilities were assessed by evaluating agreement of the obstructive anatomic structures (soft palate, tongue base, tonsils, and epiglottis) and airway levels (velopharynx, oropharynx, and hypopharynx) between the reviewers. Results: In a comparison between an unblinded and a blinded well-experienced sleep surgeons, the interrater reliability for the presence of obstruction was the highest for the soft palate at the level of the velopharynx (Cohen’s kappa value, 0.919) and the lowest for the soft palate at the level of the oropharynx (Cohen’s kappa value, 0.757). In a blind comparison between a well-experienced and less-experienced sleep surgeons, the interrater reliability for the presence of obstruction was also the highest for the soft palate at the level of the velopharynx (Cohen’s kappa value, 0.938) and the lowest for the palatine tonsils at the level of the oropharynx (Cohen’s kappa value, 0.635). Conclusion: This study showed that SVF was a diagnostic modality that can be used to evaluate upper airway obstruction without significant interrater disagreements. Key Words: Obstructive sleep apnea, sleep videofluoroscopy, fluoroscopy, interrater, reliability, validity. Level of Evidence: 4. Laryngoscope, 124:1267–1271, 2014
INTRODUCTION Obstructive sleep apnea (OSA) is a multilevel, dynamic, and complex disease caused by repetitive upper airway collapses during sleep. Although the disease severity of OSA can be diagnosed by polysomnography, the information on the anatomical problems cannot be obtained from polysomnography. When surgery is considered for the management of OSA, preoperative identification of the obstruction sites and levels in the upper airway is very important. A precise anatomical investigation of the upper airway may lead to successful individualized treatment, including single-level or multilevel surgeries. Variable methods have been attempted to achieve the most precise localization of the obstructive uller sites, including Friedman staging system,1 M€
From the Department of Otorhinolaryngology–Head and Neck Surgery, Chuncheon Sacred Heart Hospital, Hallym University College of Medicine (D-K.K.), Chuncheon; and the Departments of Otorhinolaryngology, Seoul National University College of Medicine, Seoul National University Bundang Hospital (W-H.L., C.H.L., C.S.R., J-W.K.), Seongnam, Republic of Korea. Editor’s Note: This Manuscript was accepted for publication November 4, 2013. The authors have no funding, financial relationships, or conflicts of interest to disclose. Send correspondence to Jeong-Whun Kim, MD, PhD, Associate Professor, Department of Otorhinolaryngology, Seoul National University College of Medicine, Seoul National University Bundang Hospital, 166 Goomi-ro, Bundang-gu, Seongnam, 463–707, South Korea. E-mail: [email protected] DOI: 10.1002/lary.24509
Laryngoscope 124: May 2014
maneuver,2,3 cephalometry,4,5 dynamic computed tomography (CT), and magnetic resonance imaging (MRI) scans.6,7 However, these diagnostic tools have clinical limitations in that they are usually performed during wakefulness and produce only static information on the upper airway. Recently, some dynamic airway evaluations were developed and introduced, including drug-induced sleep endoscopy (DISE),8–11 sleep MRI,12,13 and sleep videofluoroscopy (SVF)14–17 during sleep. DISE has been performed popularly and also known to be a safe, feasible, and valid assessment in OSA patients. Sleep MRI could also localize the obstructive sites in the upper airway during sleep. In addition, a Barrera’s study12 showed a higher reliability of sleep MRI. In the previous studies, SVF could provide high resolution dynamic images of the upper airway during induced sleep.14–17 Thus, SVF may be helpful to establish surgical plans for patients based on the dynamic characteristics of their upper airway. There were studies showing that the SVF may also be helpful in the selection of candidates for uvulopalatopharyngoplasty.14,18,19 Another recent study demonstrated that the SVF could show mouth breathing in addition to abnormal movements of the upper airway and that the severity of mouth opening was a significant factor to predict surgical outcomes.20 Although several previous studies showed that the SVF is a useful diagnostic modality for evaluating obstruction sites of the upper airway in OSA,14–20,19–22 Kim et al.: Interrater Reliability of Sleep Videofluoroscopy
1267
the interrater reliability of SVF has not been studied. Thus, the objective of this study was to analyze the agreement of the SVF findings evaluated by different examiners.
MATERIALS AND METHODS Subjects We retrospectively studied 374 consecutive individuals who were diagnosed as having OSA. All patients underwent full-night attended polysomnography and SVF from September 2010 through June 2012. Patients with apnea-hypopnea index (AHI) 5 and body mass index (BMI) < 40 were included. Exclusion criteria were as follows: 1) younger than 18 years of age; 2) severe medical diseases and allergy to midazolam or sedatives; 3) previous sleep surgery; and 4) craniofacial syndromes, neuromuscular diseases, or psychiatric disorders. This study was approved by the internal review board of Seoul National University Bundang Hospital.
Sleep Videofluoroscopy Protocol All OSA patients underwent SVF, as previously described, for about 2 minutes.14–17 All of the procedures were conducted after an informed consent was documented. Briefly, they were placed in the supine position on a C-arm table (Allura Xper FD20; Philips, Amsterdam, Netherlands) with their head on a comfortable pillow. An intravenous fluid line was placed on the patient arm for injection of midazolam (Bukwang Pharmacy, Seoul, Korea) and to prepare emergency situations that could accidentally occur during SVF. All images were real-time twodimensional lateral views acquired with a rotational angiographic C-arm system. Patients were instructed to breathe in and out naturally. Oxygen saturation was monitored throughout the examination using a finger pulse oximeter (Care Vision; Medical supply, Seoul, Korea). An otorhinolaryngologic doctor monitored breathing and oxygen saturation of the patients throughout the study. After the soft palate, palatine tonsils, tongue base, and epiglottis could clearly be delineated by SVF, image recording was started. During normal respiration before sedation, an awake event was recorded for 15 seconds. Thereafter, all lights in the examination room were turned off to help the patients fall asleep; sleep was also induced by intravenously administering midazolam (initial dose: 0.05 mg/kg). If patients did not fall asleep after the initial injection, an additional 0.02 mg/kg of midazolam was injected to a maximum dose of 0.1 mg/ kg. The drug-induced sleep was indirectly assessed by development of snoring and oxygen desaturation. In most of the patients with OSA, patients began to snore soon after administration of midazolam, and oxygen saturation decreased with respiratory disturbances. Only when oxygen saturation dropped by 4% or more were videofluoroscopic images for oxygen desaturation recorded. Two 15-second desaturation sleep events were recorded as desaturation sleep events. We could observe that upper airway obstruction was associated with oxygen desaturation during SVF in most of the patients. After the study was completed, flumazenil (0.02 mg/kg, Bukwang pharmacy, Seoul, Korea), the antidote of midazolam, was administrated intravenously to reverse the sedative effect of midazolam and to help the patients resume their normal life as soon as possible.
Evaluation of Sleep Videofluoroscopic Findings and Interrater Agreement The patterns of upper airway obstruction were investigated according to the previous studies in terms of anatomic
Laryngoscope 124: May 2014
1268
Fig. 1. This is a sample chart used during analysis. Each white cell is filled with 0 or 1 as a degree of obstruction. Zero denotes no obstruction and 1 denotes collapse of the upper airway. Gray cells mean that the specific structure is not involved in the specific obstruction level. The velopharynx and oropharynx are divided by the lower margin of the soft palate, and the oropharynx and hypopharynx are divided by the tip of the epiglottis. structures and airway levels involved in the obstruction.14–16 The analysis was performed for the desaturation sleep events. The obstructive anatomic structures involved the soft palate, tongue base, tonsils, and epiglottis. The obstructive airway level was divided into the velopharynx, oropharynx, and hypopharynx. While the velopharynx and oropharynx were divided by the lower margin of the soft palate, the oropharynx and hypopharynx were divided by the tip of the epiglottis. The patterns of obstruction were categorized as follows: obstruction by the soft palate at the velopharynx; obstruction by the soft palate, palatine tonsils, or tongue base at the oropharynx; and obstruction by the tongue base or epiglottis at the hypopharynx (Figure 1). To assess the interrater agreements, the video images of SVF were reviewed by three different sleep surgeons: two well-experienced sleep surgeons who had reviewed more than 200 SVF images and a less-experienced sleep surgeon who had reviewed less than 20 SVF images. One well-experienced sleep surgeon (J.-W.K.) unblindly analyzed all the findings of SVF at the first time. Thereafter, the digitally recorded video images were blindly reviewed independently by the other wellexperienced sleep surgeon (C.S.R.) and a less-experienced sleep surgeon (D.-K.K.) without being aware of their Friedman stage and polysomnographic results.
Statistical Analysis All descriptive results were expressed as a mean 6 standard deviation. To analyze interrater reliabilities, the percentage of agreement was calculated and the Cohen’s kappa test was used. The degree of agreement was considered moderate if the kappa values were between 0.41 and 0.60, substantial if between 0.61 and 0.80, and almost perfect or excellent if between 0.81 and 1.00.17 If the Cohen’s kappa values were greater than 0.6, the test was considered acceptable as a recommended method for clinical decision.23 All statistical analyses were performed using SPSS statistics version 19.0 (SPSS, Chicago, IL).
RESULTS Characteristics of Patients A total of 374 consecutive subjects (72 women and 302 men) were included in this study. Out of them, 44 Kim et al.: Interrater Reliability of Sleep Videofluoroscopy
Fig. 2. Prevalence of obstructive anatomical structures and levels evaluated by different sleep surgeons (unblinded well-experienced, blinded well-experienced, and blinded less-experienced). hypo 5 hypopharynx; oro 5 oropharynx; velo 5 velopharynx.
patients had mild OSA (5 AHI < 15), 102 patients had moderate OSA (15 (AHI < 30), and 228 patients had severe OSA (AHI 30). Their mean age was 48.4 years (6 13), mean AHI 28.2/h (6 24.4), and mean BMI 25.6 kg/m2 (6 2.9). The mean Epworth sleepiness scale and Pittsburgh sleep quality index was 10.6 and 8.5, respectively.
tion by the soft palate at the level of the oropharynx, and the highest kappa value was 0.919 for the obstruction by the soft palate at the level of the velopharynx. In the comparison between the blinded well-experienced and the less-experienced sleep surgeons, the lowest kappa value was 0.635 for the obstruction by the palatine tonsils at the level of the oropharynx, and the highest kappa value was 0.938 for the obstruction by the soft palate at the level of the velopharynx.
Prevalence of Obstructive Structures and Levels The prevalence of obstructive structures and levels evaluated by each examiner is shown in Figure 2. The most prevalent obstructive structures and levels were the soft palate at the level of the velopharynx for the unblinded and blinded well-experienced sleep surgeons (57% and 58%, respectively), whereas it was the soft palate at the level of the oropharynx (64%) for the blinded less-experienced sleep surgeon. The least prevalent obstructive structures and levels were the palatine tonsils at the level of the oropharynx for all observers.
Interrater Reliability of Obstructive Structures and Levels The interrater reliabilities analyzed by percentages of agreement and kappa values are presented in Table I. In the comparison between the well-experienced sleep surgeons, the lowest rate of agreement was 90.8% for the obstruction by the soft palate at the level of the oropharynx, and the highest rate of agreement was 99.1% for the obstruction by the palatine tonsils at the level of the oropharynx. In the blinded comparison between the well-experienced and less-experienced sleep surgeons, the lowest rate of agreement was 89.6% for the obstruction by the tongue base at the level of the oropharynx, and the highest rate of agreement was 97.2% for the obstruction by the soft palate at the level of the velopharynx. In the analysis of kappa values, the comparison between the two well-experienced sleep surgeons showed that the lowest kappa value was 0.757 for the obstrucLaryngoscope 124: May 2014
DISCUSSION Among the treatment modalities, surgery and mandibular advancement devices should be administered to the patients after thorough evaluation of the upper airway. Although the overall success rate of uvulopalatopharyngoplasty for OSA has been reported to be less than optimal (40.7%) in the literature,24 some studies showed higher success rates in preevaluated patients.25–27 Anatomical factors are also involved in successful outcomes in the patients using mandibular TABLE I. Interrater Reliability of Sleep Video Fluoroscopy Results. Comparison Between Unblinded and Blinded Well-Experienced Sleep Surgeons % Agreement
Cohen’s Kappa
Comparison Between Blinded Well- and Less-Experienced Sleep Surgeons % Agreement
Cohen’s Kappa
97.2
0.938
Obstruction at the Velopharynx Soft palate
96.4
0.919
Obstruction at the Oropharynx Soft palate 90.8 0.757
91.5
0.794
Palatine tonsils
0.918
94.2
0.635
Tongue base 93.7 0.840 Obstruction at the Hypopharynx
89.6
0.782
99.1
Tongue base
92.1
0.775
92.8
0.783
Epiglottis
94.0
0.826
92.6
0.776
Kim et al.: Interrater Reliability of Sleep Videofluoroscopy
1269
advancement devices.16,17,28 Even if a precise evaluation of the obstructive sites by reliable methods is indispensable as a preoperative evaluation, there is no standard method. The SVF is one of the new methods to evaluate dynamic abnormalities of the upper airway. This study was designed to evaluate the interrater reliability of SVF examination. Although several studies have already reported the efficacy of SVF for the anatomical evaluation of OSA,14–17 to our knowledge the present study is the first for determining the rater-torater reliability of SVF. In our study, interrater agreement was studied including both well-experienced and less-experienced sleep surgeons to identify whether experiences for SVF analysis can influence the accuracy of analyses. According to our results, the obstructive level-bylevel agreements were more than 90% in the comparison between the unblinded and blinded well-experienced sleep surgeons. In the oropharyngeal level, the agreement for the soft palate was slightly lower than in the other subsites. This might be attributed to a frequent overlapping of a backward and downward movement of the soft palate, with a backward movement of the tongue base and a forward movement of the posterior pharyngeal wall. In the comparison between the blinded welland less-experienced sleep surgeons, the agreement was also high, suggesting that SVF analysis may require a short learning curve. In the present study, the kappa values for the interrater reliability were all higher than 0.6, meaning substantial-to-perfect agreements. (Table II). However, there were some differences between the two comparison arms. For the obstruction at the velopharyngeal and hypopharyngeal levels, different testers showed almost similar analysis results. On the other hand, at the oropharyngeal levels the agreements were higher in the comparison between the two wellexperienced surgeons than in the comparison between the well- and less-experienced surgeons: The agreements between the two experts for the soft palate, palatine tonsils, and tongue base at the level of the oropharynx were 0.757, 0.918, and 0.840, respectively; those between the expert and the beginner for the soft palate, palatine tonsils, and tongue base were 0.794, 0.635, and 0.782, respectively. This implies that the analysis of the oropharyngeal level may be relatively more difficult for beginners, and those less-experienced surgeons should pay more attention to the evaluation of the oropharyngeal level during SVF analysis. The complex anatomy of the oropharynx in SVF video clips may also be associated with the higher difficulty. Although the SVF is thought to be suited for dynamic evaluation of upper airway, it also has some limitations in OSA patients. First, a selection bias can be present because the SVF was performed during induced sleep by midazolam; therefore, only a part of total sleep events can be recorded. However, this selection bias is also present in other dynamic airway examination such as DISE8–11 and dynamic sleep MRI scan.12,13 Second, it is still unclear whether midazolaminduced sleep is physiologic. However, an article showed that polysomnography performed during midazolamLaryngoscope 124: May 2014
1270
induced sleep was highly correlated with polysomnography during natural sleep.29 Third, we did not monitor the stage of sleep or depth of sleep using concurrent polysomnography or bispectral index. However, because we could not observe any upper airway obstructions before sedation—and we analyzed the images only when upper airway obstructions were observed in association with oxygen desaturation—the obstructive findings are likely to be assessed as sleep-related breathing events. Useful diagnostic methods should demonstrate important features, such as safety, validity, and reliability. The SVF showed safety and validity in the previous studies.14–17,20 The results of the present study may provide an evidence for the interrater reliability of SVF findings. We found that the concordance rate and kappa value of SVF were higher than DISE.10 The previous study of interrater reliability in DISE showed relatively lower kappa values (0.42, 0.47, and 0.53 in tonsils, lateral wall of the hypopharynx and oropharynx, respectively).10 From these results, it is conceivable that SVF may be easier to evaluate for upper airway obstruction than DISE. The reason for the SVF to show higher interrater reliabilities may be as follows: The study population was different between SVF and DISE in the studies; the DISE was performed by sleep surgeons; and the analysis results may be affected by their endoscopic experience as well as anatomical knowledge, whereas SVF was conducted according to a preset standard protocol by a videofluoroscope without involvement of surgeon’s technical experiences. This study also has some limitations: 1) The number of video reviewers was limited. 2) Three video reviewers have worked in the same hospital; thus, further studies need to be conducted with a larger number of samples and reviewers working in various sleep centers. 3) Because an anatomical evaluation is not always likely to guarantee the surgical results, a prospective cohort study is needed for surgery or mandibular advance devices to be applied to the patients according to the anatomical diagnosis using SVF. 4) Because a lateral movement of the pharyngeal walls cannot be exactly shown in lateral video clips of SVF, it was excluded from the present study.
CONCLUSION The interrater reliability of SVF analysis for OSA was excellent and its analysis results performed by a less-experienced sleep surgeon were quite comparable to those by experienced sleep surgeons, suggesting its short learning curve. Therefore, SVF may provide enough information on the obstructive sites and levels, which would help to plan appropriate treatment for OSA based on the dynamic anatomy of the upper airway during sleep.
BIBLIOGRAPHY 1. Friedman M, Ibrahim H, Joseph NJ. Staging of obstructive sleep apnea/ hypopnea syndrome: a guide to appropriate treatment. Laryngoscope 2004;114:454–459. 2. Sher AE, Thorpy MJ, Shprintzen RJ, Spielman AJ, Burack B, McGregor PA. Predictive value of Muller maneuver in selection of patients for uvulopalatopharyngoplasty. Laryngoscope 1985;95:1483–1487.
Kim et al.: Interrater Reliability of Sleep Videofluoroscopy
3. Terris DJ, Hanasono MM, Liu YC. Reliability of the Muller maneuver and its association with sleep-disordered breathing. Laryngoscope 2000;110: 1819–1823. 4. Poirrier AL, Pire S, Raskin S, Limme M, Poirrier R. Contribution of postero-anterior cephalometry in obstructive sleep apnea. Laryngoscope 2012;122:2350–2354. 5. Julia-Serda G, Perez-Penate G, Saavedra-Santana P, et al. Usefulness of cephalometry in sparing polysomnography of patients with suspected obstructive sleep apnea. Sleep Breath 2006;10:181–187. 6. Bhattacharyya N, Blake SP, Fried MP. Assessment of the airway in obstructive sleep apnea syndrome with 3-dimensional airway computed tomography. Otolaryngol Head Neck Surg 2000;123:444–449. 7. Donnelly LF, Casper KA, Chen B, Koch BL. Defining normal upper airway motion in asymptomatic children during sleep by means of cine MR techniques. Radiology 2002;223:176–180. 8. Ravesloot MJ, de Vries N. One hundred consecutive patients undergoing drug-induced sleep endoscopy: results and evaluation. Laryngoscope 2011;121:2710–2716. 9. De Vito A, Agnoletti V, Berrettini S, et al. Drug-induced sleep endoscopy: conventional versus target controlled infusion techniques-a randomized controlled study. Eur Arch Otorhinolaryngol 2011;268:457–462. 10. Kezirian EJ, White DP, Malhotra A, Ma W, McCulloch CE, Goldberg AN. Interrater reliability of drug-induced sleep endoscopy. Arch Otolaryngol Head Neck Surg 2010;136:393–397. 11. Rodriguez-Bruno K, Goldberg AN, McCulloch CE, Kezirian EJ. Test-retest reliability of drug-induced sleep endoscopy. Otolaryngol Head Neck Surg 2009;140:646–651.12. 12. Barrera JE. Sleep magnetic resonance imaging: dynamic characteristics of the airway during sleep in obstructive sleep apnea syndrome. Laryngoscope 2011;121:1327–1335. 13. Moon IJ, Han DH, Kim JW, et al. Sleep magnetic resonance imaging as a new diagnostic method in obstructive sleep apnea syndrome. Laryngoscope 2010;120:2546–2554. 14. Lee CH, Mo JH, Kim BJ, et al. Evaluation of soft palate changes using sleep videofluoroscopy in patients with obstructive sleep apnea. Arch Otolaryngol Head Neck Surg 2009;135:168–172. 15. Lee CH, Hong SL, Rhee CS, Kim SW, Kim JW. Analysis of upper airway obstruction by sleep videofluoroscopy in obstructive sleep apnea: a large population-based study Laryngoscope 2012;122:237–241. 16. Lee CH, Kim JW, Lee HJ, et al. An investigation of upper airway changes associated with mandibular advancement device using sleep videofluoro-
Laryngoscope 124: May 2014
17.
18. 19.
20.
21.
22.
23. 24.
25. 26.
27.
28.
29.
scopy in patients with obstructive sleep apnea. Arch Otolaryngol Head Neck Surg 2009;135:910–914. Lee CH, Kim JW, Lee HJ, et al. Determinants of treatment outcome after use of the mandibular advancement device in patients with obstructive sleep apnea. Arch Otolaryngol Head Neck Surg 2010;136: 677–681. Rama AN, Tekwani SH, Kushida CA. Sites of obstruction in obstructive sleep apnea. Chest 2002;122:1139–1147. Walsh JK, Katsantonis GP, Schweitzer PK, Verde JN, Muehlbach M. Somnofluoroscopy: cineradiographic observation of obstructive sleep apnea. Sleep 1985;8:294–297. Lee CH, Mo JH, Seo BS, Kim DY, Yoon IY, Kim JW. Mouth opening during sleep may be a critical predictor of surgical outcome after uvulopalatopharyngoplasty for obstructive sleep apnea. J Clin Sleep Med 2010;6: 157–162. Suratt PM, Dee P, Atkinson RL, Armstrong P, Wilhoit SC. Fluoroscopic and computed tomographic features of the pharyngeal airway in obstructive sleep apnea. Am Rev Respir Dis 1983;127:487–492. Pepin JL, Ferretti G, Veale D, et al. Somnofluoroscopy, computed tomography, and cephalometry in the assessment of the airway in obstructive sleep apnoea. Thorax 1992;47:150–156. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977;33:159–174. Sher AE, Schechtman KB, Piccirillo JF. The efficacy of surgical modifications of the upper airway in adults with obstructive sleep apnea syndrome. Sleep 1996;19:156–177. Li HY, Lee LA. Relocation pharyngoplasty for obstructive sleep apnea. Laryngoscope 2009;119:2472–2477. Pang KP, Woodson BT. Expansion sphincter pharyngoplasty: a new technique for the treatment of obstructive sleep apnea. Otolaryngol Head Neck Surg 2007;137:110–114. Vicini C, Dallan I, Canzi P, et al. Transoral robotic surgery of the tongue base in obstructive sleep apnea-hypopnea syndrome: anatomic considerations and clinical experience. Head Neck 2012;34:15–22. Lee CH, Mo JH, Choi IJ, et al. The mandibular advancement device and patient selection in the treatment of obstructive sleep apnea. Arch Otolaryngol Head Neck Surg 2009;135:439–444. Gregorio MG, Jacomelli M, Inoue D, Genta PR, de Figueiredo AC, Lorenzi-Filho G. Comparison of full versus short induced-sleep polysomnography for the diagnosis of sleep apnea. Laryngoscope 2011;121:1098– 1103.
Kim et al.: Interrater Reliability of Sleep Videofluoroscopy
1271
