Cardiopulmonar y Imaging • Original Research Kligerman et al. Computer-Aided Detection of Pulmonary Embolism on CT Angiography

Downloaded from www.ajronline.org by Univeristy of Brighton on 07/04/14 from IP address 194.81.203.94. Copyright ARRS. For personal use only; all rights reserved

Cardiopulmonary Imaging Original Research

Missed Pulmonary Emboli on CT Angiography: Assessment With Pulmonary Embolism–ComputerAided Detection Seth J. Kligerman1 Kian Lahiji Jeffrey R. Galvin Carly Stokum Charles S. White Kligerman SJ, Lahiji K, Galvin JR, Stokum C, White CS

OBJECTIVE. The purpose of this study is to assess the use of a pulmonary embolism (PE)– computer-aided detection (CADx) program in the detection of PE missed in clinical practice. MATERIALS AND METHODS. Pulmonary CT angiography (CTA) studies (n = 6769) performed between January 2009 and July 2012 were retrospectively assessed by a thoracic radiologist. In studies that were positive for PE, all prior contrast-enhanced pulmonary CTA studies were reviewed. Missed PE was deemed to have occurred if PE was not described in the final interpretation. The presence, proximal extent, and number of PEs were agreed on by three thoracic radiologists. Studies with missed acute PE and available slice thickness of 2 mm or less were assessed with a prototype PE-CADx program. False-positive PE-CADx marks were analyzed. Outcomes of missed acute PEs were assessed in patients with both follow-up imaging and clinical data. RESULTS. Fifty-three studies with overlooked acute PE met our inclusion criteria for PECADx assessment. The PE-CADx program identified at least one PE in 77.4% of instances (41/53). PE-CADx correctly marked at least one PE in 23 of 23 cases (100%) with multiple PEs and 18 of 30 (60%) cases with a solitary PE (p < 0.001). PE-CADx per-study sensitivity was significantly higher for segmental (65.5%) than for subsegmental (91.7%) PEs (p = 0.002). PE-CADx averaged 3.8 false-positive marks per case (range, 0–23 marks). Fourteen patients with missed PE who were not receiving anticoagulation therapy developed new PEs, including nine with an isolated subsegmental PE on the initial CT scan. CONCLUSION. PE-CADx correctly identified 77.4% of cases of acute PE that were previously missed in clinical practice.

P

Keywords: computer-aided detection, missed, pulmonary embolism DOI:10.2214/AJR.13.11049 Received April 4, 2013; accepted after revision May 29, 2013. 1

All authors: Department of Diagnostic Radiology and Nuclear Medicine, University of Maryland School of Medicine, 22 S Greene St, Baltimore, MD 21201. Address correspondence to S. J. Kligerman ([email protected]).

AJR 2014; 202:65–73 0361–803X/14/2021–65 © American Roentgen Ray Society

ulmonary thromboembolic disease is the third most common cause of cardiovascular-related death after myocardial infarction and stroke [1]. Although mortality rates from untreated pulmonary embolism (PE) vary, a rate as high as 30% has been reported [2–4]. With the advent of MDCT, CT has largely replaced ventilation-perfusion scanning as the most commonly ordered study to evaluate for PE, and its sensitivity is comparable to that of invasive pulmonary angiography for the detection of PE to the segmental level [5, 6]. Although CT allows optimized noninvasive assessment of the distal pulmonary vasculature, a vast amount of information is produced because hundreds of images are routinely reconstructed for each scan. The detailed evaluation of each distal subsegmental and subsubsegmental vessel in this large dataset represents a daunting challenge in the context of the high CT volume in many clinical practices.

Over the past several years, computer-aided detection (CADx) algorithms have been developed as a second reader for multiple thoracic CT applications, including lung nodule and PE detection [7]. However, PE-CADx has yet to gain widespread acceptance. In part, this is due to a lack of reimbursement but it is also likely related to concerns about false-positive (FP) markings, the time required to run the program, and a perception that overlooked PEs are uncommon. The rate of observer error in the detection of PE is unknown, but it is recognized that PEs are sometimes overlooked [8, 9]. The purpose of this article is to assess the efficacy of a latest-generation PE-CADx program in the detection of PEs overlooked in clinical practice by attending radiologists. Materials and Methods Patient Selection This study was approved by our institutional review board and is HIPAA compliant. Between

AJR:202, January 2014 65

Downloaded from www.ajronline.org by Univeristy of Brighton on 07/04/14 from IP address 194.81.203.94. Copyright ARRS. For personal use only; all rights reserved

Kligerman et al. January 1, 2009, and July 31, 2012, all pulmonary CT angiography (CTA) studies were retrospectively assessed by a fellowship-trained thoracic radiologist with 5 years of experience. Studies were identified by searching our PACS (IMPAX, AGFA Healthcare) between the aforementioned dates using the keywords “CTA chest” and “CT angio chest.” If an acute PE was judged to be present and the attending radiologist failed to mention the presence of a PE in the report, it was recorded as a “potentially missed” PE study. If a PE was prospectively mentioned in the report or was regarded as equivocal by the initial interpreting radiologist, it was not included in the study. Two additional thoracic radiologists with 22 and 33 years of experience, respectively, reviewed each “potentially missed” PE study, and if both concurred with the initial radiologist, it was categorized as a missed PE. If either of the additional thoracic radiologists was not certain of the diagnosis of a PE, then the study was excluded. In addition, for all PE-positive pulmonary CTA studies (either detected or missed PE) performed between January 2009 and July 2012, prior pulmonary CTA studies available on the PACS from 2004 to 2008 were also reviewed. These prior studies were assessed for the presence of undiagnosed PE using the inclusion criteria described already.

CT Technique Multiple scanner types with a range of 16–256 slices were used for the pulmonary CTA studies. Because of the long time over which the studies were completed, only current protocols are de-

scribed. At present, a collimation of 0.625 mm and a pitch of 0.758 are used for all studies. Tube voltage of 100 kV is used for patients with a body mass index (BMI) of 30 or less, which is increased to 120 kV in patients with a BMI greater than 30. Although the standard pulmonary CTA protocol uses a tube current–time product of 250 mAs, this parameter was adjusted by the technologist on the basis of a departmental weight-based adjustment. At our institution, a tube current–time product of 200, 250, 300, and 350 mAs is used in patients with a BMI 25 or less, greater than 25 but 30 or less, greater than 30 but 35 or less, and greater than 35 but 40 or less, respectively. For morbidly obese patients (BMI > 40), the technologist may increase tube current–time product up to 500 mAs according to the size of the patient and the distribution of weight. Tube current modulation is used for all examinations. Contrast agent bolus timing is obtained after a test injection using 20 mL of contrast agent. The diagnostic study is performed with 65 mL of contrast agent. Both the test bolus and diagnostic study are injected through a power injector at 5 mL/s. For Philips Healthcare scanners, a B filter is used for reconstruction, whereas a B-30 filter is used for scans performed on a Siemens Healthcare system. All pulmonary CTA studies performed from 2006 to 2012 were reconstructed using a slice thickness of 0.9 or 1 mm. Coronal and sagittal 5-mm-thick maximum-intensityprojection reconstructions were provided with each study. In addition, since 2007, all pulmonary CTA studies could be reviewed using a third-party vendor (AquariusNET version 4.4, TeraRecon) that is opened directly from the PACS.

Pulmonary Embolism Characterization Studies with missed PEs were individually assessed by three thoracic radiologists and were categorized according to the number of PEs (single vs multiple), most proximal location of PE (lobar, segmental, or subsegmental), and chronicity (acute or chronic). Chronic PE was defined by eccentric location in the arterial lumen, webbing, or narrowing or obliteration of the lumen. In the absence of these findings, the PE was classified as acute. If both acute and chronic PEs were present in the same study, only acute PEs were considered for evaluation. If a PE was present that extended into multiple smaller segmental or subsegmental branches without discontinuity, it was counted as a single PE. Discontinuous PEs were classified as being separate PEs only if the discontinuous filling defects were located in a distinct segment or subsegment. The three radiologists provided a consensus opinion of the characteristics of all PEs. If there was a disagreement in the classification that could not be resolved after consensus review, a fourth thoracic radiologist with 37 years of experience served as a tiebreaker.

Pulmonary Embolism Conspicuity For each study, the three primary radiologists independently graded the conspicuity of the PE in the study on a scale of 1 (most subtle) to 5 (very obvious) (Table 1 and Fig. 1). This score was determined for the overall subtlety of PEs for the entire study and not for each individual PE. If there was disagreement regarding the conspicuity after consensus review, the fourth radiologist was consulted.

TABLE 1: Parameters Used For Qualitative Scoring by Radiologists Qualitative Score

PE Conspicuity

PA Enhancement

Motion

Noise

1

Extremely subtle: PE easily missed and presence subtle even in retrospect

Very poor: degree of PA enhancement renders most segments and subsegments as nondiagnostic; main and lobar vessels diagnostic

Very poor: degree of motion renders large portions of the study nondiagnostic

Very poor: degree of noise renders large portions of the study nondiagnostic

2

Subtle: PE visible in retrospect, but missed during prospective evaluation is frequently expected to occur

Poor: degree of PA enhancement severely limits CT evaluation in some segments and many subsegments but much of study is still diagnostic

Poor: degree of motion severely limits CT evaluation in many areas but majority of study still diagnostic

Poor: degree of noise severely limits CT evaluation in many areas but majority of study still diagnostic

3

Intermediate: PE evident in retrospect and, although subtle, would often be detected during prospective evaluation

Suboptimal: PA enhancement is diagnostic in all segments but in some subsegments is nondiagnostic

Suboptimal: CT evaluation limited due to motion but diagnostic in most regions

Suboptimal: CT evaluation limited because of noise but diagnostic in most regions

4

Obvious: PE not subtle and would be expected to be detected in most instances

Mild limitation: Mildly suboptimal enhancement but diagnosis is possible at all levels

Mild limitation: CT evaluation mildly suboptimal because of motion but diagnosis is not substantially impaired

Mild limitation: Evaluation mildly suboptimal because of noise but diagnosis is not substantially impaired

5

Very obvious: PE not expected to be overlooked

Good: Ideal PA enhancement; all levels well visualized

Good: No perceived motion

Good: Image is free of any perceived noise

Note—PA = pulmonary artery, PE = pulmonary embolism.

66

AJR:202, January 2014

Computer-Aided Detection of Pulmonary Embolism on CT Angiography

Downloaded from www.ajronline.org by Univeristy of Brighton on 07/04/14 from IP address 194.81.203.94. Copyright ARRS. For personal use only; all rights reserved

6769 pulmonary CTA studies performed from January 1, 2009, through July 31, 2012, retrospectively reviewed

703 studies with acute or chronic PE

82 prior pulmonary CTA studies before 2009 from 749 cases with acute or chronic PE reviewed

638 studies with prospectively detected PE excluded

70 studies with prospectively detected PE excluded 65 potentially missed PE studies

12 potentially missed PE studies

20 studies with missed chronic PE excluded

1 study with missed chronic PE excluded 45 potentially missed acute PE studies

11 potentially missed acute PE studies

1 study not confirmed with missed PE after consensus review excluded

2 studies with missed acute PE and slice thickness > 2 mm excluded 44 missed acute PE studies confirmed by consensus review

9 missed acute PE studies confirmed by consensus review

53 missed acute PE studies assessed with PE-CADx

Fig. 1—Flow diagram of scans with missed pulmonary embolism (PE) that met inclusion criteria for assessment with PE computer-aided detection (CADx) prototype. From 6769 pulmonary CT angiography (CTA) cases retrospectively reviewed between January 1, 2009, and July 31, 2012, 703 cases had either acute or chronic pulmonary emboli. Of those, 44 pulmonary CTA examinations were independently confirmed to be missed PE examinations by three thoracic radiologists and had slice thickness of ≤ 2 mm, which was required for PE-CADx assessment. In patients with 703 positive PE examinations, all prior contrast-enhanced pulmonary CTA studies obtained from January 1, 2004, through December 31, 2008, were also reviewed. From these studies, 11 additional studies with missed acute PE were discovered. Only nine of those examinations had slice thickness ≤ 2 mm required for PE-CADx evaluation and were independently confirmed to be positive for PE by same three thoracic radiologists.

Image Quality Several aspects of image quality were also subjectively graded using the same method as described already, including the quality of the contrast enhancement of the pulmonary arteries, respiratory motion, and image noise. Each parameter was scored on a 5-point scale as 1 (poor) to 5 (ideal) (Table 1). Studies with one category (enhancement, noise, or motion) ranked as suboptimal (score of 3) or poor (score of 2) were classified as “mildly limited,” whereas studies with two or more categories ranked as suboptimal or poor or those with a single category rated as very poor (score of 1) were classified as “moderately limited.” Studies in which all categories were ranked mildly limited or good were classified as “good” (Fig. 2). Filtered back projection reconstruction series were used for PE-CADx analysis and for grading of PE conspicuity and image quality because iterative reconstruction was not available at our institution until 2010.

Pulmonary Embolism Computer-Aided Detection Analysis The missed PE studies were evaluated by a noncommercially available PE-CADx prototype developed by Philips Healthcare that was located on a separate workstation. The prototype requires a slice thickness of 2 mm or less to analyze DICOM data; thus, studies not meeting this criterion were excluded. Before PE-CADx analysis, all missed PEs, as agreed on by consensus, were digitally marked (Easyscil, Philips Healthcare) by the primary thoracic radiologist and were designated as the reference standard. A digital marking was performed by placing a crosshair on the proximal and distal portions of each PE. Subsequently, the PE-CADx program was run using an algorithm that has been described in detail elsewhere [10]. The time required to run the PE-CADx program was recorded for each study.

PE-CADx performance was evaluated for its ability to correctly identify individual missed PE. PE-CADx marks corresponding to the reference standard digital mark were counted as a true-positive (TP) or “hit” (Figs. 1 and 2). To be classified as a TP, more than 50% of the ovoid PE-CADx mark was required to be placed between the crosshairs of the reference-standard marking. If CADx failed to identify a mark or most of the ovoid CADx mark fell outside of the reference-standard crosshairs, it was judged a false-negative (FN) result or “miss” (Figs. 1 and 2). The location of each TP and FN mark was recorded on the basis of standard anatomic lobar, segmental, and subsegmental anatomy. If the PE-CADx marked two portions of the same PE, it was classified as a single correct mark. Only the most proximal mark was counted as a TP. In addition, PE-CADx performance was assessed for its ability to correctly stratify the entire study

AJR:202, January 2014 67

Downloaded from www.ajronline.org by Univeristy of Brighton on 07/04/14 from IP address 194.81.203.94. Copyright ARRS. For personal use only; all rights reserved

Kligerman et al.

A

B

C

D

Fig. 2—Conspicuity scoring of missed pulmonary emboli (PE) in four different patients. A, 51-year-old man with PE. Subtlety score 1. Coronal image from pulmonary CT angiography (CTA) study shows isolated right upper lobe anterior subsegmental PE (arrow). Circle indicates true-positive (TP) mark by PE–computer-aided detection (CADx) prototype. B, 67-year-old man with PE. Subtlety score 2. Axial image from pulmonary CTA study shows subsegmental CADx marks (circles). Two TP marks are in right middle lobe (arrow) and right lower lobe (arrowhead). False-positive (FP) mark in left lower lobe (asterisk) was due to bronchial wall thickening adjacent to area of aspiration pneumonia. C, 28-year-old woman with PE. Subtlety score 3. Axial reformat (multiplanar reformation) from pulmonary CTA study shows isolated subsegmental PE (circle and arrow) in lateral segment of right lower lobe. D, 68-year-old man with PE. Subtlety score 4. Axial image from pulmonary CTA study shows segmental PE (arrow and circle) in medial segment of right middle lobe. Additional segmental PE was present in left upper lobe (not shown).

as positive for PE. A TP study was designated by a marking of at least one PE, whereas an FN study was indicated if the CADx did not indicate a PE. The sensitivity of PE-CADx was assessed in relation to the PE subtlety score, the presence of single or multiple PEs, and most proximal PE location (lobar, segmental, or subsegmental). If a mark made by the PE-CADx program did not correspond to the reference standard digital mark of a PE, the CADx mark was classified as an FP (Figs. 1 and 2). Each FP mark was reviewed by the primary thoracic radiologist to determine its underlying cause. If no identifiable cause for the FP mark was identified, it was classified as “unknown.” Moreover, if a suspected FP CADx mark appeared to correspond to a PE that was missed by all three review-

68

ing radiologists during the consensus review, the CADx mark was then assessed by all three radiologists. If all three agreed that the mark corresponded to a PE, it was recategorized as a TP.

Clinical Outcomes In each patient with a missed acute PE study, all subsequent contrast-enhanced CT studies were evaluated by all three thoracic radiologists to assess for changes in clot burden. Follow-up studies were classified as revealing no new acute PEs (including complete resolution, decrease, stability of PE, or evolution to chronic disease) or as showing the development of a new PE. New PEs were categorized by location and were confirmed by all three reviewing radiologists. The interval

between the two studies was documented. The electronic medical records were reviewed to determine whether patients were receiving anticoagulation therapy and the reason for anticoagulation therapy between the initial and repeat scans, and the records were also reviewed for deaths. Imaging outcomes were assessed according to whether the PE-CADx stratified a study as a TP study (PECADx hit) or an FN study (PE-CADx miss).

Statistical Analysis Proportional relationships were tested using the Pearson chi-square test and Fisher exact test where appropriate. A significance level of 0.05 was used for determination. Investigated associations included the location of the PE (segmental vs subsegmental) and

AJR:202, January 2014

Computer-Aided Detection of Pulmonary Embolism on CT Angiography TABLE 2: Per-Study Pulmonary Embolism (PE) Computer-Aided Detection (CADx) Performance in Relation to Number, Location, and Conspicuity 1 Downloaded from www.ajronline.org by Univeristy of Brighton on 07/04/14 from IP address 194.81.203.94. Copyright ARRS. For personal use only; all rights reserved

Conspicuity

2

3

4 Hit

Miss

Total

Hit

Miss

Hit (%)

Hit

Miss

Hit (%)

Hit

Miss

Hit (%)

Hit (%)

Hit

Miss

Hit (%)

2

3

40

4

5

44.4

6

2

75

12

10

54.5

0

1

0

5

1

83.3

1

0

100

6

2

75

2

3

40

4

6

40

11

3

78.6

1

0

100

18

12

60

2

0

100

3

0

2

0

7

0

100

Single PE Subsegmental Segmental Total Multiple PEs Subsegmental

100

Segmental

4

0

100

3

0

100

9

0

100

16

0

100

Total

6

0

100

6

0

100

11

0

100

23

0

100

10

6

17

3

85

12

0

100

41

12

Combined total

2

3

40

62.5

77.4

Note—Data are no. of PEs. Hit means that the PE-CADx correctly marked at least one PE in the study. Miss means that the PE-CADx failed to mark any PE.

the number of PEs detected (single vs multiple). To compare the subtlety score of the PEs and the image quality, the conspicuity rating and image quality levels were tested for linear associations using a MantelHaenszel chi-square test with level investigation by Fisher exact test. Comparisons between the number of FPs in relationship to the study quality were calculated using the Mann-Whitney U Test. Proportional relationships were tested using SAS software (version 9.2, SAS Institute).

Results Missed Pulmonary Embolism Studies Six thousand seven hundred sixty-nine pulmonary CTA studies performed between January 1, 2009, and July 31, 2012, were retrospectively assessed. Of these, 703 studies found either acute or chronic PE. From these 703 studies, 44 were confirmed to have a missed acute PE by consensus review (Fig. 1). Among the 703 studies that were positive for PE, an additional 82 prior pulmonary CTA studies obtained from January 1, 2004, to December 31, 2008, were available on the PACS

for review. Of these 82 studies, an additional 11 with missed acute PE were confirmed by consensus. Two of these studies were excluded because only a slice thickness of greater than 2 mm was available and thus could not be assessed by the PE-CADx (Fig. 1). In total, 53 pulmonary CTA studies in 52 patients with missed acute PE were identified that met the inclusion criteria for PE-CADx analysis. The 52 patients consisted of 28 men and 24 women (average age, 52 years; age range, 22–82 years). Twenty different radiologists interpreted the 53 missed PE studies, including four fellowship-trained thoracic radiologists. The average and median length of practice since residency of these 20 radiologists was 14.2 years and 11 years, respectively, with a range of 1–35 years. Pulmonary Embolism Computer-Aided Detection Performance Overall, there were a total of 146 missed PEs identified in the 53 cases. CADx correctly marked 103 of 146 of these missed PEs. In

addition, CADx made two additional marks that, after consensus agreement, corresponded to PEs that had not been previously identified by the three reviewing radiologists during their initial evaluation. The two reclassified TP marks were additional subsegmental PEs in patients with multiple PEs. With inclusion of these reclassified TP marks, the overall sensitivity of PE-CADx on a per-clot basis was 71.9% (105/146). PE-CADx identified 27 of 38 segmental (71.1%) and 78 of 108 (72.2%) subsegmental PEs, respectively. The PE-CADx processed 17.6 images per second with an average (± SD) reconstruction time per study of 37.5 ± 8.9 seconds. Among the 53 missed PE studies, 29 (54.7%) had a single PE and 24 (45.3%) had multiple PEs. The most proximal PE was in a subsegmental vessel in 29 cases and in a segmental vessel in 24 cases. With respect to conspicuity, five (9.4%), 16 (30.2%), 20 (37.8%), and 12 (22.6%) cases were categorized as very subtle (score of 1), subtle (score of 2), intermediate (score of 3), and obvi-

TABLE 3: Per-Study Pulmonary Embolism (PE) Computer-Aided Detection (CADx) Performance in Relation to Image Quality Factor

Good (n = 35)

Mildly Limited (n = 4)

Moderate Limited (n = 14)

Total (n = 53)

Bolus

4.9

4.5

3.2

4.4

Motion

4.4

3.3

3.9

4.2

Noise

4.4

4

2.6

3.9

Conspicuity

2.7

2.8

2.8

2.7

No. of single PEs/no. of multiple PEs

23/12

2/2

4/10

29/24

Subsegmental (%)

62.5

25

14.3

54.7

PE-CADx sensitivity (%)

68.6

100

92.9

77.4

3.3 ± 2.3

1.3 ± 1.9

8.4 ± 7.5

3.8 ± 5.3

No. of false-positive PE-CADx marks per case (mean ± SD) Note—Data are image quality score, except where noted otherwise.

AJR:202, January 2014 69

Downloaded from www.ajronline.org by Univeristy of Brighton on 07/04/14 from IP address 194.81.203.94. Copyright ARRS. For personal use only; all rights reserved

Kligerman et al. Fig. 3—64-year-old man with pulmonary embolism. Axial multiplanar reformation image from pulmonary CT angiography study shows numerous false-positive (FP) marks (circles) due to pulmonary veins (PVs). Over one third of FP marks in study were due to PVs (arrows) and nearly one third of FP marks were due to respiratory motion. Truepositive mark is seen in inferior segment of lingua (arrowhead), and falsenegative mark is seen in posterior segment of left lower lobe (asterisk).

study sensitivity increased as the PE conspicuity score improved (Table 2).

ous (score of 4), respectively. No study was scored as very obvious. On a per-study basis, the PE-CADx algorithm made at least one TP mark in 41 of 53 cases with a missed PE, a sensitivity of 77.4%. Of the 30 cases with isolated PEs, PE-CADx had a per-study sensitivity of 60% (18/30), as compared with 100% (23/23) in those with multiple PEs (p = 0.0006). PECADx per-patient sensitivity was 91.7% (22/24) and 65.5% (19/29) when the most proximal PE was in a segmental vessel versus subsegmental vessel, respectively. This difference was statistically significant (p = 0.02). For lesion subtlety, PE-CADx per-

Analysis Based on Study Quality Overall, 35 scans were graded as “good” quality, four as “mildly limited,” and 14 as “moderately limited.” There was no linear association detected in relation to image quality and PE-CADx sensitivity (p = 0.1) (Table 3). In the 35 studies graded as good, the most proximal embolus was in a subsegmental vessel in 62.5%, as compared with 25% and 14.3% of those studies graded as “mildly limited” or “moderately limited,” respectively.

A

B

False-Positive Pulmonary Embolism ComputerAided Detection Marks Among the 53 missed PE cases, the PECADx algorithm made 202 FP marks, an average of 3.8 ± 5.3 FP marks per case (range, 0–23 marks per case). Compared with the studies with “good” image quality, “moderately limited” studies had significantly more FP marks (p < 0.0001), whereas “mildly limited” studies did not show a significant difference. Cardiac or respiratory motion accounted for 72 FP marks (35.6%), whereas the pulmonary vein was responsible for 64 FP marks (31.7%) (Fig. 3). Poor vascular opacification, image noise, and bronchial wall thickening or consolidation adjacent to a pulmonary artery led to 19 (9.4%), 18 (8.9%), and 18 (8.9%) FP marks, respectively (Figs. 2 and 4). In eight instances (4%), the cause of the FP mark could not be determined. A surgical suture, an area of parenchymal scarring, and a pulmonary artery branch point each led to a single FP mark (0.5% each). As noted already, two FP marks were reclassified as TP after consensus review. In total, the PE-CADx algorithm made 307 marks (105 TP and 202 FP) for an average of 5.8 ± 6.7 marks per case (range, 0–30 marks per case). Clinical Outcome Of the 53 cases with missed PE, follow-up imaging and clinical data were available in 26

C

Fig. 4—Subjective image quality of contrast-enhanced studies in three different patients. A, “Good” image quality in 32-year-old woman with pulmonary embolism (PE). Coronal multiplanar reformation (MPR) shows good contrast bolus, no respiratory motion, and low image noise and was classified as having “good” image quality. True-positive mark (circle) of isolated subsegmental pulmonary embolism (PE) (arrow) with subtlety score of 2 is shown. B, “Mildly limited” image quality in 45-year-old man with multifocal pneumonia and PE. Coronal MPR shows good pulmonary arterial enhancement and low image noise. However, there is extensive respiratory motion, which led to four false-positive (FP) marks (circles), two of which are shown in right upper lobe (white arrows). Isolated right lower lobe posterior segmental PE (black arrow) with subtlety score of 3 was missed by PE–computer-aided detection (CADx) prototype. C, “Moderately limited” image quality in 58-year-old woman with metastatic breast cancer and PE. Coronal MPR shows poor contrast bolus, moderate image noise, and mild respiratory motion. Although PE-CADx correctly detected two subsegmental PE (white arrows), there were numerous FP marks (circles) due to respiratory motion (asterisk), pulmonary vein (black arrow), and noise within small subsegmental vessels (arrowheads). A total of 12 FP marks were made in this case.

70

AJR:202, January 2014

Computer-Aided Detection of Pulmonary Embolism on CT Angiography TABLE 4: Imaging Outcomes of Patients with Missed Pulmonary Embolism (PE) Stratified by Embolic Burden on Initial Scan

Downloaded from www.ajronline.org by Univeristy of Brighton on 07/04/14 from IP address 194.81.203.94. Copyright ARRS. For personal use only; all rights reserved

No New PE

Embolic Burden

New PE

Improveda Prior PEs

Unchanged Prior PEs

Hit

Miss

Hit

2

1

Miss

New Subsegmental PE

Total

New Segmental PE

New Lobar PE

New Main PE

Hit

Hit

Hit

Miss

Hit

Miss

Hit

Miss

1

2

3

1

2

2

2

1

Miss

Miss

Total Hit

Miss

6

3

Single PE on initial study Subsegmental Segmental

2

1

1

Multiple PEs on initial study Subsegmental

2

Segmental

2

Total

6b

2 3b

2

4

3c

9

1 1 3

4

1

1 1

2

4

2

2

3 1

11

3

Note—Data are no. of PEs. Hit means that PE–computer-aided detection (CADx) correctly marked at least one PE on initial study. Miss means that PE-CADx failed to mark any PE on initial study. Embolic burden refers to the number and location of PE on initial CT with missed PE. aImproved by either decreased size or resolution of prior PE. bAll four patients with multiple PEs and three of five patients with a single PE were receiving anticoagulation therapy between initial and follow-up CT scan. cAll three patients without change in PE were rescanned within 2 days.

studies (26 patients), of which PE-CADx correctly identified PE in 20 (76.9%). Details are provided in Table 4. Average time between the initial (missed PE) and follow-up study was 96.7 days (range, 1–516 days) with a median of 35 days. Fourteen patients (53.8%) developed new PE in the interim, including nine patients with isolated subsegmental PEs (Figs. 5 and 6). Seven of the nine (77.8%) patients whose PEs resolved or decreased on follow-up imaging were receiving anticoagulation therapy between the two scans. Each of the three patients with no change in PE burden between scans was rescanned within a very short interval (≤ 2 days) because of continued hypoxia, chest pain, or shortness of breath. Discussion Although there have been multiple studies looking at the impact of PE-CADx on diagnosis and workflow, to our knowledge, the current study is the first to assess its capability in patients with missed PEs, whose overlooked PEs would be expected to be subtle and in whom identification of such PEs would likely have greater clinical importance. In our study, we retrospectively evaluated over 6800 pulmonary CTA scans and identified 53 pulmonary CTA studies with a slice thickness of 2 mm or less in which acute PE was overlooked on the final interpretation. A PE-CADx prototype algorithm correctly marked at least one PE in 77.4% of these studies, permitting them to be correctly characterized as showing embolic disease in retrospect. In many instances, this might have led to initiation of

anticoagulation therapy. In support of this hypothesis, the available follow-up data showed that many of our untreated patients with overlooked PE had increased clot burden when ultimately diagnosed. Since 2002, when Masutani et al. [11] showed that PE-CADx could detect PE with a high degree of sensitivity, multiple studies have found a wide range of effectiveness for this technique, with overall sensitivity between 30.7% and 92% [7–9, 12–17]. Our overall and per-study sensitivities are slightly lower than those in recent studies [8, 9, 12, 13,

14, 16, 17], likely because we restricted our study cohort to PEs missed in clinical practice, which were therefore more likely to be subtle. All 146 PEs were either segmental (n = 38) or subsegmental (n = 108) in location. Moreover, we did not exclude contrast-enhanced cases according to the type of study, coexistent lung disease, vascular attenuation, image noise, or respiratory motion, as has been done in many prior investigations [7]. In our study, the sensitivity of PE-CADx correlated with the proximal extent and number (single vs multiple) of PEs in addition to

A

B

Fig. 5—43-year-old man who presented to emergency department with shortness of breath. A, Axial image from pulmonary CT angiography (CTA) study shows subtle left lower lobe pulmonary embolism (PE) (arrow) that was not seen during initial interpretation but was detected by PE–computer-aided detection, as indicated by circle. Circle in right lower lobe is false-positive mark due to adjacent bronchial wall thickening (asterisk). Patient was discharged without diagnosis of PE. B, Fifty-six days later, patient returned to emergency department with chest pain. Image from repeat pulmonary CTA shows resolution of left lower lobe PE (asterisk) but new subsegmental PE in right lower lobe (arrow).

AJR:202, January 2014 71

Downloaded from www.ajronline.org by Univeristy of Brighton on 07/04/14 from IP address 194.81.203.94. Copyright ARRS. For personal use only; all rights reserved

Kligerman et al. the conspicuity of the PEs. The prototype algorithm had a per-study sensitivity of 91.7% in instances where the most proximal PE was in a segmental vessel, compared with 65.5% of cases if the PE was localized to the subsegmental vessels. This decreasing sensitivity for smaller more-peripheral vessels is concordant with the results of previous studies [9, 14–16, 18]. In the current study, PE-CADx per-study sensitivity improved as our subjective conspicuity score increased, and, in general, the conspicuity score was higher in cases with multiple PEs and more proximal extent. The computer algorithm detects PEs by segmenting and tracking vessels from a central to a peripheral location while assessing for abrupt differences in vascular attenuation, similar to the method used by a trained radiologist to map out the pulmonary vasculature. Therefore, it is not surprising that PEs perceived as more subtle by radiologists were more likely to be missed by the PE-CADx prototype. However, the prototype did detect at least one PE in all six cases (100%) with multiple PEs in the very subtle or subtle categories, compared with only six of 15 cases (40%) with only a single PE in these same categories. Overall, the CADx program had higher perstudy sensitivity for cases with multiple PEs, presumably because these studies provide the PE-CADx algorithm multiple opportunities to detect at least one PE. Subjective components of image quality, including vascular attenuation, motion, and image noise, were not related to the sensitivity of the PE-CADx software, a finding that has been reported previously [10, 19]. However, it is noteworthy that, in our cases classified as moderately limited because of these parameters, PE-CADx per-study sensitivity was 92.9%, compared with 68.6% for cases with good image quality. This unexpected finding is presumably related to the higher percentage of cases with moderately limited image quality that had multiple PEs located in more proximal vessels. The number of FP studies per case was directly related to the subjective assessment of image quality. Our average of 3.8 FP marks per study using the PE-CADx prototype is somewhat lower than reported in prior studies, which showed 3.9–11.4 FP studies per case, despite our use of a missed PE cohort [8– 10, 13, 15–21]. In particular, in studies with “good” image quality, there were 3.3 FP marks per case, which is lower than the 4.2 FP marks reported by Dewailly et al. [19] in their studies with “good” image quality but greater than

72

A

B

Fig. 6—60-year-old woman with vomiting and shortness of breath due to large-bowel obstruction. A, Axial oblique image from pulmonary CT angiography (CTA) shows very subtle pulmonary embolism (PE) (arrow) in left upper lobe, which was not seen on initial interpretation but was detected by PE–computer-aided detection, as indicated by circle. This was only PE seen on study. B, Forty-nine days later, patient returned to emergency department with severe dyspnea and chest pain. Repeat pulmonary CTA showed massive saddle embolism (arrows).

the 1.6 FP marks reported by Wittenberg et al. [10] in studies with “excellent” image quality. This number was more than double (8.4 FP marks per case) in our “moderately limited” studies. Although most FP marks would likely be recognized and dismissed by skilled interpreters, there is the possibility that negative PE studies could be incorrectly interpreted as positive, leading to unnecessary treatment and a potentially adverse outcome [8]. With an average processing time of 37.5 seconds, this PE-CADx software, if it is approved by the U.S. Food and Drug Administration, could be incorporated into the daily workflow of many practices, particularly if it is done in the background before or during review of the study. Given its performance in this study, we would recommend using the software as a second reader in pulmonary CTA cases initially interpreted as negative by a radiologist. Although the program’s sensitivity is not affected by image quality, we also suggest that the algorithm is most useful in cases with good or only mildly limited image quality because PE-CADx use may be more time consuming in studies with poor image quality because of the increasing number of FP results. In cases interpreted as positive, the advantage of using PE-CADx as a second reader is less clear. One potential indication in positive pulmonary CTA studies may be in those initially interpreted as having an isolated subsegmental PE because the discovery of additional PEs would provide the treating physician with a stronger impetus to begin anticoagulation therapy, as noted in the next paragraph.

Debate exists regarding the clinical importance of isolated PEs, but patients with isolated lobar or segmental PEs typically receive anticoagulation therapy [22]. The uncertainty regarding isolated subsegmental PEs is greater. One concern with the use of PE-CADx is that it may lead to an increase in the diagnosis and treatment of patients with small subsegmental PEs whose clots might resolve without treatment. However, on the basis of available follow-up data, we documented new PEs in nine of 12 cases with missed isolated subsegmental PEs, including one patient who developed a massive saddle PE. PE-CADx identified a miss in most of these initial studies. Our results suggest that an isolated subsegmental PE, although sometimes innocuous, may progress, occasionally substantially. The finding of a single isolated subsegmental PE by either a radiologist or PECADx software should be considered seriously, and clinicians should incorporate the risks of treatment in the context of other patient-specific factors, such as underlying malignancy, the presence of coexistent deep venous thrombosis, history of PE, and extent of symptoms [22– 26]. The current study suggests that PE-CADx may lead to a decrease in time to diagnosis and treatment, a decrease in unnecessary repeat imaging, and, presumably, decreased morbidity in some patients. Not surprisingly, patients whose studies initially showed multiple missed PE tended to return with new embolic disease if they were not already receiving anticoagulation therapy, and PE-CADx correctly identified missed PEs in all four such patients who later presented with new PEs.

AJR:202, January 2014

Downloaded from www.ajronline.org by Univeristy of Brighton on 07/04/14 from IP address 194.81.203.94. Copyright ARRS. For personal use only; all rights reserved

Computer-Aided Detection of Pulmonary Embolism on CT Angiography One of the limitations of our study is the possibility that some of our CT studies designated as overlooked PE were in fact negative. To minimize this possibility, three experienced thoracic radiologists independently reviewed each study and were required to agree that PE was present. We did not process negative control cases with the PE-CADx system to assess specificity and negative predictive value because this was not part of the study objective and has been reported in numerous prior studies. We did not assess the accuracy of radiologists in the detection of PE using modern CT scanners because this was not a part of our study objectives. Similarly, we did not evaluate specific imaging factors or PE characteristics that may have led to missed PE. We have shown that a high percentage of patients with missed PEs, even those with missed single subsegmental PEs, present later with new PEs if not treated with anticoagulation therapy. These data are subject to selection bias because we had followup imaging and clinical data for only 26 of our 53 cases. It remains possible that all embolic disease spontaneously resolved in the other 27 cases for which we do not have follow-up information. However, even if such selection bias occurred, our results show that PE-CADx could have improved the time to diagnosis, obviated further imaging, and decreased morbidity in many of our patients. In summary, our study shows the potential value of PE-CADx in a highly relevant group of patients, those with overlooked PE at the time of initial clinical assessment, a cohort that has not been investigated previously. References 1. Goldhaber SZ, Bounameaux H. Pulmonary embolism and deep vein thrombosis. Lancet 2012; 379:1835–1846 2. Anderson FA Jr, Wheeler HB, Goldberg RJ, et al. A population-based perspective of the hospital incidence and case-fatality rates of deep vein thrombosis and pulmonary embolism: the Worcester DVT Study. Arch Intern Med 1991; 151:933–938 3. Calder KK, Herbert M, Henderson SO. The mortality of untreated pulmonary embolism in emergency department patients. Ann Emerg Med 2005; 45:302–310

4. Tapson VF. Acute pulmonary embolism. N Engl J Med 2008; 358:1037–1052 5. Baile EM, King GG, Muller NL, et al. Spiral computed tomography is comparable to angiography for the diagnosis of pulmonary embolism. Am J Respir Crit Care Med 2000; 161:1010–1015 6. Sostman HD, Stein PD, Gottschalk A, Matta F, Hull R, Goodman L. Acute pulmonary embolism: sensitivity and specificity of ventilationperfusion scintigraphy in PIOPED II study. Radiology 2008; 246:941–946 7. Chan HP, Hadjiiski L, Zhou C, Sahiner B. Computer-aided diagnosis of lung cancer and pulmonary embolism in computed tomography: a review. Acad Radiol 2008; 15:535–555 8. Wittenberg R, Berger FH, Peters JF, et al. Acute pulmonary embolism: effect of a computer-assisted detection prototype on diagnosis—an observer study. Radiology 2012; 262:305–313 9. Wittenberg R, Peters JF, Sonnemans JJ, Prokop M, Schaefer-Prokop CM. Computer-assisted detection of pulmonary embolism: evaluation of pulmonary CT angiograms performed in an oncall setting. Eur Radiol 2010; 20:801–806 10. Wittenberg R, Peters JF, Sonnemans JJ, Bipat S, Prokop M, Schaefer-Prokop CM. Impact of image quality on the performance of computer-aided detection of pulmonary embolism. AJR 2011; 196:95–101 11. Masutani Y, MacMahon H, Doi K. Computerized detection of pulmonary embolism in spiral CT angiography based on volumetric image analysis. IEEE Trans Med Imaging 2002; 21:1517–1523 12. Blackmon KN, Florin C, Bogoni L, et al. Computer-aided detection of pulmonary embolism at CT pulmonary angiography: can it improve performance of inexperienced readers? Eur Radiol 2011; 21:1214–1223 13. Buhmann S, Herzog P, Liang J, et al. Clinical evaluation of a computer-aided diagnosis (CAD) prototype for the detection of pulmonary embolism. Acad Radiol 2007; 14:651–658 14. Das M, Muhlenbruch G, Helm A, et al. Computer-aided detection of pulmonary embolism: influence on radiologists’ detection performance with respect to vessel segments. Eur Radiol 2008; 18:1350–1355 15. Engelke C, Schmidt S, Bakai A, Auer F, Marten K. Computer-assisted detection of pulmonary embolism: performance evaluation in consensus with experienced and inexperienced chest radiologists. Eur Radiol 2008; 18:298–307

16. Schoepf UJ, Schneider AC, Das M, Wood SA, Cheema JI, Costello P. Pulmonary embolism: computer-aided detection at multidetector row spiral computed tomography. J Thorac Imaging 2007; 22:319–323 17. Wittenberg R, Peters JF, Weber M, et al. Standalone performance of a computer-assisted detection prototype for detection of acute pulmonary embolism: a multi-institutional comparison. Br J Radiol 2012; 85:758–764 18. Zhou C, Chan H-P, Patel S, et al. Preliminary investigation of computer-aided detection of pulmonary embolism in three-dimensional computed tomography pulmonary angiography images. Acad Radiol 2005; 12:782–792 19. Dewailly M, Remy-Jardin M, Duhamel A, et al. Computer-aided detection of acute pulmonary embolism with 64-slice multi-detector row computed tomography: impact of the scanning conditions and overall image quality in the detection of peripheral clots. J Comput Assist Tomogr 2010; 34:23–30 20. Engelke C, Schmidt S, Auer F, Rummeny EJ, Marten K. Does computer-assisted detection of pulmonary emboli enhance severity assessment and risk stratification in acute pulmonary embolism? Clin Radiol 2010; 65:137–144 21. Maizlin ZV, Vos PM, Godoy MC, Cooperberg PL. Computer-aided detection of pulmonary embolism on CT angiography: initial experience. J Thorac Imaging 2007; 22:324–329 22. Carrier M, Kimpton M, Le Gal G, et al. The management of a sub-segmental pulmonary embolism: a cross-sectional survey of Canadian thrombosis physicians. J Thromb Haemost 2011; 9:1412–1415 23. Eyer BA, Goodman LR, Washington L. Clinicians’ response to radiologists’ reports of isolated subsegmental pulmonary embolism or inconclusive interpretation of pulmonary embolism using MDCT. AJR 2005; 184:623–628 24. Stein PD, Goodman LR, Hull RD, Dalen JE, Matta F. Diagnosis and management of isolated subsegmental pulmonary embolism: review and assessment of the options. Clin Appl Thromb Hemost 2012; 18:20–26 25. Carrier M, Righini M, Le Gal G. Symptomatic subsegmental pulmonary embolism: what is the next step? J Thromb Haemost 2012; 10:1486–1490 26. Le Gal G, Righini M, Parent F, van Strijen M, Couturaud F. Diagnosis and management of subsegmental pulmonary embolism. J Thromb Haemost 2006; 4:724–731

AJR:202, January 2014 73

Missed pulmonary emboli on CT angiography: assessment with pulmonary embolism-computer-aided detection.

The purpose of this study is to assess the use of a pulmonary embolism (PE)- computer-aided detection (CADx) program in the detection of PE missed in ...
775KB Sizes 0 Downloads 0 Views