Note: This copy is for your personal non-commercial use only. To order presentation-ready copies for distribution to your colleagues or clients, contact us at www.rsna.org/rsnarights.
Original Research n Ultrasonography
Hepatocellular Carcinoma: Stiffness Value and Ratio to Discriminate Malignant from Benign Focal Liver Lesions1 Qiang Lu, MD, PhD Wenwu Ling, MD Changli Lu, MD Jiawu Li, MD Lin Ma, MD Jierong Quan, MD Du He, MD Jianping Liu, MD Jiaying Yang, MD, PhD Tianfu Wen, MD Hong Wu, MD, PhD Hongguang Zhu, MD, PhD Yan Luo, MD
Purpose:
To investigate the use of stiffness value and stiffness ratio (ratio of lesion to background liver parenchyma values) to discriminate malignant from benign focal liver lesions by using histologic results as the reference standard.
Materials and Methods:
This study was approved by the institutional review board, and written informed consent was obtained. Three hundred seventy-three patients with focal liver lesions proven at histologic examination underwent measurement of liver stiffness with elastography point quantification. First, stiffness values in two regions of the background liver parenchyma (at 0.5–2 cm and .2 cm from the lesion periphery) near 163 hepatocellular carcinomas were analyzed to determine a reference background liver for calculating the stiffness ratio. Second, the use of the lesion stiffness value and the stiffness ratio for prediction of liver malignancy was investigated in a cohort of patients with 58 benign and 201 malignant lesions. Results were validated in another independent cohort of patients with 25 benign and 89 malignant lesions by using analysis of the area under the receiver operating characteristic (AUC) curve.
Results:
The coefficient of variation for the background liver at 0.5–2 cm from the lesion was higher (196%) than that at greater than 2 cm from the lesion (66%). In the development phase, diagnostic accuracy with use of the stiffness value was significantly higher than that with use of the stiffness ratio for discrimination of malignant from benign lesions (AUC, 0.86 vs 0.66, respectively; P , .001). Diagnostic performance with the stiffness value was lower than that with the stiffness ratio (AUC, 0.53 vs 0.86, respectively; P , .001) for discrimination of cirrhotic nodules from other benign lesions. Diagnostic performance with the stiffness value was significantly lower than that with the stiffness ratio (AUC, 0.58 vs 0.71 respectively; P = .007) for discrimination of metastasis from primary liver cancers. In the validation phase, similar findings were revealed for the discrimination of malignant from benign lesions (AUC, 0.87 vs 0.67; P , .001) and discrimination between metastasis and primary liver cancers (AUC, 0.49 vs 0.73; P , .001).
Conclusion:
Use of stiffness values measured in the liver parenchyma at more than 2 cm from the lesion allowed better diagnostic performance than did values measured in a region closer to the tumor. Stiffness value was more accurate than stiffness ratio for differentiation of malignant from benign focal liver lesions, but the stiffness ratio might be useful for subclassification of benign and malignant lesions.
From the Departments of Ultrasound (Q.L., W.L., J.L., L.M., J.Q., Y.L.), Pathology (C.L., D.H., J.L.), and Hepatobiliary Surgery (J.Y., T.W., H.W.), West China Hospital of Sichuan University, 37th Guoxue Xiang, Jiang Xi Street, Chengdu, Sichuan 610041, China; and Department of Pathology, Shanghai Medical College, Fudan University, Shanghai, China (H.Z.). Received May 29, 2013; revision requested July 13; revision received April 4; accepted April 22; final version accepted November 17. Supported by the National Natural Science Foundation of China (grants 30870715 and 81101060). Address correspondence to Y.L. (e-mail: [email protected]). RSNA, 2015
q
Radiology: Volume 000: Number 0— 2015 n radiology.rsna.org
RSNA, 2015
q
Online supplemental material is available for this article. 1
ULTRASONOGRAPHY: Stiffness Value and Ratio to Discriminate Malignant from Benign Focal Liver Lesions
L
iver cancer is the second leading cause of death in men and the sixth leading cause of death in women worldwide (1). Therefore, it is crucial to differentiate between benign and malignant liver lesions. Ultrasonographically (US) based elastographic technologies such as strain imaging (2–5) and shear-wave elastographic methods (6–10) have been developed to estimate the elastic modulus of tissues noninvasively. In strain imaging, the relative stiffness properties of tissues in the region of interest can be measured and quantified as the stiffness ratio, which is calculated from the strain in the lesion compared with that in the background parenchyma. In shear-wave elastography, including supersonic shear-wave imaging (6), shear-wave dispersion US vibrometry (7), acoustic radiation force impulse imaging (8), and elastography point quantification (9,10), the tissue stiffness value and/or the stiffnessrelated shear-wave velocity can be quantified. Both strain imaging and shear-wave elastographic technology have been demonstrated to be useful in the assessment of breast, thyroid, and prostate cancers (3–5,11–13). In liver applications, the lesions generally occur in a broad spectrum of background parenchyma including normal, fatty, fibrotic, and cirrhotic livers. The
Advance in Knowledge nn Use of the stiffness value was better than use of the stiffness ratio (ratio of lesion value to background liver value; area under the receiver operating characteristic curve [AUC], .871 vs .670, respectively; P , .001) for differentiation between malignant and benign focal liver lesions; whereas the stiffness ratio showed better accuracy than the stiffness value in the classification of cirrhotic nodules and other benign lesions (AUC, .858) and in the classification of metastasis, hepatocellular carcinoma, and intrahepatic cholangiocarcinoma (AUC, .706). 2
differences in the stiffness properties of the background liver make the differentiation between malignant and benign focal liver lesions by means of US elastography more complex (14– 20). The purpose of this study was to investigate use of the lesion stiffness value and stiffness ratio (lesion values to background liver values) for discrimination of malignant from benign focal liver lesions by using histologic results as the reference standard.
Materials and Methods Patients This study was approved by our ethics committee with a requirement for written informed consent. Between March 2011 and October 2013, 403 consecutive patients who met the inclusion criteria (Appendix E1 [online]) before their planned surgical liver resection underwent measurement of liver stiffness at West China Hospital. Thirty patients were excluded from the evaluation for the following reasons: (a) A total of 17 patients had received radiation therapy (n = 2), transarterial chemoembolization (n = 12), or radiofrequency ablation (n = 3); (b) five patients were unable to hold their breath during stiffness measurement; and (c) 18 patients did not have histologic results from the area in the liver at 0.5–2 cm from the lesion periphery. Thus, 373 patients who met the inclusion criteria and did not meet the exclusion criteria were included. The 373 patients were allocated to two cohorts in chronological order. Cohort 1 patients, who were recruited Implications for Patient Care nn The stiffness value of focal liver lesions may be used to distinguish between malignant and benign liver lesions. nn Values for the liver parenchyma at greater than 2 cm from the lesion periphery may be considered reference values for the background liver for calculating the stiffness ratio.
Lu et al
between March 2011 and April 2013, underwent studies to determine the reference background liver for calculation of the stiffness ratio and thresholds for the classifications of malignant (163 patients with hepatocellular carcinoma [HCC], 18 with intrahepatic cholangiocarcinoma [ICC], and 20 with hepatic metastasis) and benign (11 patients with cirrhotic nodules, 12 with focal nodular hyperplasia and 35 with hemangioma) focal liver lesions. Patients in cohort 2, who were recruited between May 2013 and October 2013, included 64 patients with HCC, 18 with ICC, seven with metastasis, one with a cirrhotic nodule, six with focal nodular hyperplasia, and 18 with hemangioma, underwent studies to validate the developed thresholds. The characteristics of the two cohorts are presented in Table E1 (online).
B-Mode US Examination Before liver stiffness measurement, all participants underwent B-mode liver US. The examinations were conducted by one of two radiologists from the Ultrasonography Department (L.Y. and L.Q., with more than 5 years of experience in hepatic applications). The US scan was performed with a US system (iU22; Royal Philips, Amsterdam, the
Published online before print 10.1148/radiol.14131164 Content codes: Radiology 2015; 000:1–9 Abbreviations: AUC = area under receiver operating characteristic curve FNH = focal nodular hyperplasia HCC = hepatocellular carcinoma ICC = intrahepatic cholangiocarcinoma Author contributions: Guarantors of integrity of entire study, Q.L., C.L., J.L., D.H., Y.L.; study concepts/study design or data acquisition or data analysis/interpretation, all authors; manuscript drafting or manuscript revision for important intellectual content, all authors; approval of final version of submitted manuscript, all authors; literature research, Q.L., C.L., J.L., D.H., J.L., H.W., Y.L.; clinical studies, Q.L., W.L., C.L., J.L., L.M., J.Q., D.H., J.Y., T.W., H.W., H.Z., Y.L.; experimental studies, C.L., D.H., J.Y., H.W., H.Z., Y.L.; statistical analysis, W.L., C.L., L.M., J.Q., D.H., H.W., Y.L.; and manuscript editing, Q.L., C.L., D.H., H.W., Y.L. Conflicts of interest are listed at the end of this article.
radiology.rsna.org n Radiology: Volume 000: Number 0— 2015
ULTRASONOGRAPHY: Stiffness Value and Ratio to Discriminate Malignant from Benign Focal Liver Lesions
Lu et al
Figure 1
Figure 1: Elastography point quantification superimposed on B-mode US images of HCC (left) and two areas of background liver (background I and II, center and right) show examples of liver stiffness measurements. Clinical images were acquired in a 69-year-old man who was proven to have HCC and cirrhosis of the liver at pathologic examination. Liver stiffness value was expressed in kilopascals by using the Young modulus.
Netherlands) equipped with an elastography point quantification feature and a 1–5-MHz transducer (C5–1, Royal Philips). The settings, including the gain, time-gain compensation, dynamic range, focal length, and mechanical index were optimized for each examination.
Liver Stiffness Measurement Elastography point quantification (Royal Philips), which was integrated in the same system as that used in the B-mode US examination, was used to assess the liver stiffness in the lesion and background liver parenchyma. Elastography point quantification technology uses focused ultrasound to induce shear-wave propagation inside tissue. Conventional US is applied to track the shear wave, and the Voigt model is used to estimate the elasticity of tissue (9,10). The quantified elastic measurement in the region of interest is expressed in kilopascals and is superimposed on a B-mode US image (Fig 1a). The size of the region of interest is dependent on depth, approximately 1.0 cm by 1.5 cm at a depth of 4 cm. The maximum penetration depth is 7 cm. The liver stiffness measurement was performed in areas without blood vessels during a 5-second breath hold at inspiration. The measurements were performed by one of two radiologists (L.W. and L.J., with more than 1 year of experience performing elastography point quantification). For patients with
multiple lesions, the largest lesion was chosen as the index lesion. Liver stiffness was measured in three locations: in the peripheral area of the largest lesion, in the liver at 0.5–2 cm from the lesion periphery (background 1), and greater than 2 cm from the lesion periphery (background 2). All the stiffness measurements in background 2 were accessed in liver segments 4, 5, or 6 (15). The measurement was performed five times for each group. The mean value of five measurements for each individual was used in the statistical analysis.
Histologic Examination Surgical specimens of the lesions and background areas were fixed with 10% formalin and routinely embedded in paraffin. The tissue sections were stained with hematoxylin and eosin (Fig E1 [online]). The hematoxylin and eosin sections were independently evaluated by two pathologists (C.L. and D.H., with 7 and 8 years of experience, respectively) and were reviewed by a gastroenterologic pathologist (Z.H., with 30 years of experience). When disagreements occurred, the three pathologists reviewed the sections independently and repeatedly until at least two pathologists reached consensus. The fibrosis severity in background 1 was determined according to the Scheuer classification system (21). Stages 1, 2, or 3 were classified as fibrotic liver, while stage 4 was classified as cirrhotic liver.
Radiology: Volume 000: Number 0— 2015 n radiology.rsna.org
Statistical Analysis To determine the intraobserver variability of the stiffness measurements, coefficients of variation were calculated and analyzed in different types of lesions. The patients were assigned to two independent cohorts, cohort 1 (n = 259) and cohort 2 (n = 114). In cohort 1, stiffness values in two regions of the background liver near the 163 HCCs were first analyzed to determine a reference background liver value for calculating stiffness ratio. Then, thresholds for stiffness value and stiffness ratio for the differentiation of malignant versus benign lesions, early HCC (diameter 3 cm) versus cirrhotic nodules, cirrhotic nodules versus other benign lesions, and metastasis versus primary liver cancers were investigated. In cohort 2, stiffness value and stiffness ratio for differentiation of malignant versus benign lesions and metastasis versus primary liver cancers were validated. A Mann-Whitney unpaired test was performed with the log 2-transformed data to determine the differences between each two categories. All of the P values were two sided. A difference was considered significant if a P value was less than .05. Receiver operating characteristic curve analysis (22) was performed on stiffness measurements. Diagnostic performance was evaluated by using area under the receiver operating characteristic curve (AUC) analysis (23). Sensitivity, specificity, positive prediction value, and negative prediction 3
ULTRASONOGRAPHY: Stiffness Value and Ratio to Discriminate Malignant from Benign Focal Liver Lesions
value were calculated with an optimal cutoff value that maximized the sum of sensitivity and specificity. In addition, comparison of receiver operating characteristic curve analysis was performed to determine the difference in diagnostic performance with stiffness value and with stiffness ratio. The difference was considered significant if the P value was less than .05. Software (10.4.7.0; MedCalc, Ostend, Belgium) was used to perform the Mann-Whitney unpaired test, boxand-whisker plot, and receiver operating characteristic curve analysis. All measurements were used in the statistical analysis.
Results Intraobserver Variability of Liver Stiffness Measurements Intraobserver variability of liver stiffness measurements in different lesions and background 2 are shown in Figure E2 and Table E2 (online). Malignant tumors showed significantly higher variability than did benign lesions (P , .0001). In comparison, there were no significant differences in variability of background 2 measurements between the benign and malignant focal liver lesions (P = .05). Stiffness Values in Background Liver Parenchyma near the HCC The stiffness values in the lesions and background parenchyma of 163 patients with HCC in cohort 1 are illustrated in Figure 2a. In these patients, all the stiffness values in background 2 were measured in liver segments 4, 5, or 6. Stiffness values in background 1 in 15 patients (9.2%) were obtained in the left lobe of the liver. The coefficient of variation for background 1 (196%) was higher than that for background 2 (66%) and even higher than that for the HCC lesions (113%). The diagnostic accuracy (measured as the AUC) for discrimination of HCC from background 1 was significantly lower than that for HCC versus background 2 (AUC, 0.75 vs 0.84; P , .001, Fig 2b). 4
Lu et al
Figure 2
Figure 2: A, Box-and-whisker plot for liver stiffness values in two areas of background liver and in HCC. B, Graph shows AUC estimations for HCC versus background 1 (blue line; AUC, 0.75) and HCC versus background 2 (red dashed line; AUC, 0.84). Liver stiffness value was expressed in kilopascals by using the Young modulus. Red squares in plot indicate measurements larger than upper quartile plus three times the interquartile range.
Stiffness Value and Stiffness Ratio Profiles in Focal Liver Lesions Stiffness values for six different types of focal liver lesions in 373 patients are shown in Figure 3a. The mean stiffness values and ranges expressed in kilopascals for patients with hemangioma, focal nodular hyperplasia (FNH), cirrhotic nodules, HCC, ICC, and metastasis were 9.3 (range, 3.1–41), 10 (range, 2.9–26), 11 (range, 4.4–49), 34 (range, 4.4–188), 25 (range, 5.5–79), and 30 (range, 4.7–64), respectively (Table 1). No significant difference in values was found between cirrhotic nodules and other benign lesions (P = .91). Similarly, no significant difference in values was observed for metastasis versus HCC or ICC (P = .24). The distribution of stiffness values in background 2 for patients with different focal liver lesions is shown in Figure 3b. The mean stiffness values
and ranges expressed in kilopascals in background 2 for patients with hemangioma, FNHs, cirrhotic nodules, HCCs, ICCs, and metastases were 4.6 (range, 2.6–12), 4.3 (range, 2.7–7.0), 15 (range, 5.0–42), 11 (range, 1.8–43), 7.2 (range, 3.1–20), and 5.9 (range, 3.1–12), respectively (Table 1). Significantly higher stiffness values were observed in background 2 near cirrhotic nodules compared with those near benign lesions (P , .0001). In comparison, background 2 near metastases showed significantly lower stiffness values than did those near HCCs and ICCs (P , .0001). The stiffness ratios (lesion to background 2) are illustrated in Figure 3c. The mean stiffness ratios and ranges for patients with hemangiomas, FNHs, cirrhotic nodules, HCCs, ICCs, and metastases were 2.2 (range, 0.4–6.7), 2.7 (range, 0.5–5.1), 0.9 (range, 0.3–4.0), 4.1 (range, 0.5–37),
radiology.rsna.org n Radiology: Volume 000: Number 0— 2015
ULTRASONOGRAPHY: Stiffness Value and Ratio to Discriminate Malignant from Benign Focal Liver Lesions
Figure 3
Figure 3: Stiffness values and stiffness ratios for 53 hemangiomas, 18 FNHs, 12 cirrhotic nodules, 227 HCCs, 36 ICCs, and 27 metastases. A, Box-and-whisker plot shows distribution of liver stiffness values in patients with hemangioma, FNH, cirrhotic nodule, HCC, ICC, and metastasis (malignant vs benign lesions [P , .0001], cirrhotic nodule vs other benign lesions [P = .9072] and metastasis vs HCC and ICC [P = .2449]). B, Box-and-whisker plot shows distribution of liver stiffness values in background 2 of patients with hemangioma, FNH, cirrhotic nodule, HCC, ICC, and metastasis (malignant vs benign lesions [P , .0001], cirrhotic nodules vs other benign lesions [P , .0001] and metastasis vs HCC and ICC [P , .0001]). C, Box-andwhisker plot shows liver stiffness ratios (lesion to background 2 values) in patients with hemangioma, FNH, cirrhotic nodules, HCC, ICC, and metastasis (malignant vs benign lesions [P = .0001], cirrhotic nodules vs other benign lesions [P = .0001], and metastasis vs HCC and ICC [P = .0002]). Liver stiffness values were expressed in kilopascals by using the Young modulus. Red squares in plots indicate measurements larger than upper quartile plus three times the interquartile range. Radiology: Volume 000: Number 0— 2015 n radiology.rsna.org
Lu et al
4.4 (range, 0.9–18), and 5.6 (range, 1.0–13), respectively (Table 1). Cirrhotic nodules had significantly lower stiffness ratios than did other benign lesions (P = .0001). In comparison, metastases had significantly higher stiffness ratios than did HCCs and ICCs (P = .0002).
Development Phase: Stiffness Value and Stiffness Ratio in Liver Malignancy Prediction In cohort 1 (n = 259), malignant tumors had significantly higher stiffness values and stiffness ratios than did benign lesions (P , .0001 and P = .0003, respectively; Table 2). For discrimination of malignant from benign lesions, diagnostic accuracy with use of the stiffness value was significantly higher than that with use of the stiffness ratio (AUC, 0.86 vs 0.69; P , .001; Fig 4a). With a cutoff value of 13 kPa for the stiffness value, the sensitivity, specificity, positive predictive value, and negative predictive value were 78%, 83%, 94%, and 52%, respectively (Table 2). With a cutoff value of 1.3 for the stiffness ratio, the sensitivity, specificity, positive predictive value, and negative predictive value were 79%, 45%, 83%, and 38%, respectively. For discrimination of early HCCs from cirrhotic nodules (Table 2, Fig 4b), diagnostic accuracy with the stiffness value was lower than that with the stiffness ratio (AUC, 0.77 vs 0.83; P = .32). For discrimination of cirrhotic nodules from other benign lesions (Table 2, Fig 4c), diagnostic performance with the stiffness value was significantly lower than that with the stiffness ratio (AUC, 0.53 vs 0.86; P , .001). For discrimination of metastasis from HCC and ICC (Table 2, Fig 4d), the diagnostic performance with the stiffness value was significantly lower than that with the stiffness ratio (AUC, 0.58 vs 0.71; P = .007). Validation Phase: Stiffness Value and Stiffness Ratio for Prediction of Liver Malignancy The same thresholds developed in cohort 1 for the classification of malignant and benign focal liver lesions were applied to predict liver malignancy at 5
ULTRASONOGRAPHY: Stiffness Value and Ratio to Discriminate Malignant from Benign Focal Liver Lesions
Lu et al
Table 1 Stiffness Value Profiles in Focal Liver Lesions and Background 2 Lesion (kPa) Variable Benign (n = 83) Hemangioma (n = 53) FNH (n = 18) Cirrhotic nodule (n = 12) Malignant (n = 290) HCC (n = 227) ICC (n = 36) Metastasis (n = 27)
Background 2 (kPa)
Stiffness Ratio (Lesion vs Background 2)
Median
Mean
Median
Mean
P Value*
Median
Mean
8.4 (2.9–49) 8.3 (3.1–41) 8.6 (2.9–26) 8.0 (4.4–49) 21 (4.4–188) 16 (4.4–188) 16 (5.5–79) 26 (4.7–64)
9.8 9.3 10 11 32 34 25 30
4.3 (2.6–42) 4.1 (2.6–12) 3.8 (2.7–7.0) 13 (5.0–42) 8.1 (18–43) 8.9 (1.8–43) 6.2 (3.1–20) 5.1 (3.1–12)
6 4.6 4.3 15 10 11 7.2 5.9
, .0001 , .0001 .0001 .0496 , .0001 , .0001 , .0001 , .0001
1.6 (0.3–6.7) 2.0 (0.4–6.7) 2.9 (0.5–5.1) 0.7 (0.3–4.0) 2.5 (0.5–37) 2.2 (0.5–37) 2.7 (0.9–18) 4.9 (1.0–13)
2.1 2.2 2.7 0.9 4.3 4.1 4.4 5.6
Note.—Data in parentheses are the range. Background 2 = background liver at greater than 2 cm from the lesion periphery.
Table 2 Diagnostic Performance with Use of Stiffness Value and Stiffness Ratio in Liver Malignancy Prediction Cohort, Diagnoses, and Stiffness Value and Ratio Development phase (cohort 1) Malignant (n = 201) vs benign lesions (n = 58) Stiffness value Stiffness ratio Early HCC (n = 33) vs cirrhotic nodule (n = 11) Stiffness value Stiffness ratio Cirrhotic nodule (n = 11) vs other benign lesions (n = 47) Stiffness value Stiffness ratio Metastasis (n = 20) vs ICC and HCC (n = 181) Stiffness value Stiffness ratio Validation phase (cohort 2) Malignant (n = 89) vs benign lesions (n = 25) Stiffness value Stiffness ratio Metastasis (n = 7) vs ICC and HCC (n = 82) Stiffness value Stiffness ratio
P Value
Mean Ratio*
AUC
Criterion (kPa)
,.0001 .0003
3.5 2.3
0.86 (0.81, 0.90) 0.66 (0.60, 0.72)
. 13 . 1.3
.0089 .0013
2.5 4.0
0.77 (0.61, 0.88) 0.83 (0.68, 0.93)
.7435 .0002
1.2 0.4
.2166 .0025
Sensitivity (%)
Specificity (%)
PPV (%)
NPV (%)
78 79
83 45
94 83
52 38
. 13 . 0.76
67 82
91 82
96 93
48 60
0.53 (0.40, 0.66) 0.86 (0.74, 0.94)
. 4.3 0.76
100 73
24 89
22 62
100 93
0.9 1.3
0.58 (0.51, 0.65) 0.71 (0.64, 0.77)
. 22 . 4.0
65 70
82 75
13 23
93 96
,.0001 ,.0001
2.8 1.5
0.87 (0.82, 0.91) 0.67 (0.61, 0.73)
. 13 , 1.3
74 82
84 28
94 80
48 30
.9211 .0419
1.0 1.7
0.49 (0.38, 0.60) 0.73 (0.63, 0.82)
. 22 . 4.0
29 57
54 73
5 15
90 95
Note.—Data in parentheses are 95% confidence intervals. Early HCC = Barcelona Clinic Liver Cancer stage 0–A (diameter # 3 cm), PPV = positive predictive value, NPV = negative predictive value * Mean ratio of the two categories.
independent validation (cohort 2; n = 114). Similarly, malignant tumors had significantly higher stiffness values and 6
stiffness ratios than did the benign lesions (P , .0001 and P , .0001, respectively; Table 2). For discrimination
of malignant from benign lesions, diagnostic accuracy with the stiffness value was significantly higher than that with
radiology.rsna.org n Radiology: Volume 000: Number 0— 2015
ULTRASONOGRAPHY: Stiffness Value and Ratio to Discriminate Malignant from Benign Focal Liver Lesions
Figure 4
Figure 4: Graphs show comparison of liver stiffness values (blue dashed line) and stiffness ratio (lesion to background 2, red dashed line) in patient cohort 1. A, Graph shows AUC estimates for stiffness values (blue dashed line; AUC, 0.86) and stiffness ratios (red dashed line; AUC, 0.66) in malignant versus benign liver lesions (P , .001). B, Graph shows AUC estimates for stiffness values (AUC, 0.77) and stiffness ratios (AUC, 0.83) for early HCCs (diameter 3 cm) versus cirrhotic nodules (P = .319). C, Graph shows AUC estimates for stiffness values (AUC, 0.53) and stiffness ratios (AUC, 0.86) for cirrhotic nodules versus those benign lesions (P , .001). D, Graph shows AUC estimates for stiffness values (AUC, 0.58) and stiffness ratios (AUC, 0.71) in metastasis versus HCC and ICC (P = .007).
the stiffness ratio (AUC, 0.87 vs 0.67; P , .001; Fig E3a [online]). With the cutoff stiffness value developed in cohort 1 of 13 kPa, sensitivity, specificity, positive predictive value, and negative predictive value were 74%, 84%, 94%, and 48%, respectively (Table 2). With the cutoff values developed in cohort 1 of 1.3 for the stiffness ratio, the sensitivity, specificity, positive predictive value, and negative predictive value were 82%, 28%, 80%, and 30%, respectively. For discrimination of metastasis from HCC and ICC, diagnostic performance with the stiffness value was significantly lower than that with
the stiffness ratio (AUC, 0.49 vs 0.73; P , .001, Table 2; Fig E3b [online]).
Discussion In this study, we investigated clinical use of the lesion stiffness value and stiffness ratio for prediction of malignancy in the liver in a large cohort of 373 patients by using elastography point quantification technology. Our study results revealed that hemangioma, FNH, HCC, ICC, and metastasis showed significantly higher stiffness values than did background liver parenchyma (all, P , .05) when the area measured was greater than 2
Radiology: Volume 000: Number 0— 2015 n radiology.rsna.org
Lu et al
cm from the lesion. This finding is different from that of Heide et al (16), who reported that the HCC was softer than the background liver, whereas Gallotti et al (20) reported higher stiffness values in the lesions than in the background liver for patients with hemangioma, FNH, HCC, and metastasis. The discrepancy might be due to the regions of background liver used in the different studies. The determination of reference background liver is crucial to understanding the role of the stiffness ratio in the characterization of focal liver lesions. We investigated two regions of the background liver on the basis of routine sampling practice for liver surgical specimens for histologic examination. In clinical conditions, shear-wave boundary effects (6–10) and biologic characteristics of tissue near tumors such as satellite nodules and stromal reactions (24) contribute to measurement variation. This is most likely why the measurements in the background tissue near the HCCs showed even greater variation than did the measurements in the HCCs. In our previous study (15), we did not observe significantly different measurements in the same liver segment between intercostal and subcostal scans. However, we found significant effects of liver location on the stiffness measurements. Liver segment 5 showed the lowest variation, while comparable stiffness values were revealed in liver segments 4, 5, and 6. Thus, for the stiffness measurements in the background liver, we located the measurement region of interest in liver segments 4–6 when they were available. We were unable to obtain stiffness measurements in the background liver at 0.5–2 cm from the lesion in liver segments 4–6 in 15 patients. This also might have contributed to the great variation in stiffness measurements in the tissue near the HCCs. Therefore, we recommend calculating stiffness ratio by using the background liver at greater than 2 cm from the lesion periphery as reference. Most of the HCCs and cirrhotic nodules in the 373 patients were found in fibrotic or cirrhotic livers, whereas 7
ULTRASONOGRAPHY: Stiffness Value and Ratio to Discriminate Malignant from Benign Focal Liver Lesions
most of the benign lesions and hepatic metastases occurred in normal livers. By comparing the stiffness value and stiffness ratio in the characterization of focal liver lesions, we found that the stiffness value was superior to the stiffness ratio for differentiation between malignant and benign liver lesions. The difference in the stiffness values of the background livers may have compromised malignancy prediction with use of the stiffness ratio. On the other hand, the low stiffness ratio in cirrhotic nodules has the potential to allow differentiation between cirrhotic nodules and other benign lesions. Similarly, the high stiffness ratio in hepatic metastasis might be useful in the subclassification of metastasis and primary liver cancers (HCC and ICC). By using quantitative acoustic radiation force impulse imaging technology, Heide et al (16) showed statistically comparable stiffness values between 38 benign lesions and 24 malignant tumors (P = .28). In comparison, Yu et al (17) showed a significant difference in the stiffness values between 41 benign and 64 malignant lesions (P , .001), with sensitivity and specificity of 68% (28 of 41) and 69% (44 of 64), respectively. Cho et al (18) investigated 51 patients with 17 hepatic hemangiomas and 43 malignant liver lesions and determined that the positive predictive value and specificity of the stiffness quantification for malignancy were 89% and 81%, respectively. Park et al (19) demonstrated an AUC of 0.74 for discrimination of 39 malignant from eight benign liver lesions. Our study results confirmed that the stiffness value could be considered a noninvasive predictor for malignancy in focal liver lesions. For the differentiation of malignant and benign liver lesions with stiffness value, sensitivity and specificity results in the development phase were 78% (156 of 159) and 83% (48 of 58), respectively. By using the same thresholds developed in cohort 1, similar sensitivity (74%, 66 of 89) and specificity (84%, 21 of 25) were obtained at independent validation in cohort 2. 8
The differentiation of early HCCs from cirrhotic nodules is important in patients with a cirrhotic liver. Gheorghe et al (14) reported high diagnostic accuracy (AUC, 0.94) of strain imaging for the differentiation of early HCCs (1–3 cm) and cirrhotic nodules in 42 patients with cirrhosis. In the development phase, our study results showed fair accuracy (sensitivity, 67%; specificity, 91%; AUC, 0.77) with use of the stiffness value and improved accuracy with use of the stiffness ratio (sensitivity, 82%; specificity, 82%; AUC, 0.83) in 44 patients with cirrhosis. Unfortunately, we could not validate this finding, because only one patient with a cirrhotic nodule was included in the validation phase. Taken together, these results indicate that the liver stiffness measurement may be a useful tool for prediction of early HCCs and for follow-up of cirrhotic nodules. Unfortunately, only eight of the patients included had very early HCCs (diameter , 2 cm), which limited our ability to evaluate the diagnostic performance of the stiffness value and stiffness ratio for discrimination of very early HCCs from cirrhotic nodules. There were several limitations in our study. First, only patients with lesions for which resections were planned were included in the cohorts. This limited the types and numbers of benign lesions investigated in this study. Further large-scale confirmation studies with rare types of lesions are needed to validate our findings. Moreover, the effect of distance between the transducer and region of interest measured on stiffness assessment was not directly evaluated. In addition, interobserver variability was not documented in the patient groups. In conclusion, we identified that the liver parenchyma at greater than 2 cm from the lesion periphery provides useful reference values for calculating the stiffness ratio. Our findings revealed that the stiffness value allowed better diagnostic performance than did stiffness ratio for differentiation between malignant and benign focal liver lesions, whereas stiffness ratio might be useful in the subclassifications of benign and malignant liver lesions.
Lu et al
Acknowledgments: We would like to thank Xiaomin Li, PhD, and Ying Wu, PhD, MD, for editorial assistance in the preparation of the manuscript. Disclosures of Conflicts of Interest: Q.L. disclosed no relevant relationships. W.L. disclosed no relevant relationships. C.L. disclosed no relevant relationships. J.L. disclosed no relevant relationships. L.M. disclosed no relevant relationships. J.Q. disclosed no relevant relationships. D.H. disclosed no relevant relationships. J.L. disclosed no relevant relationships. J.Y. disclosed no relevant relationships. T.W. disclosed no relevant relationships. H.W. disclosed no relevant relationships. H.Z. disclosed no relevant relationships. Y.L. disclosed no relevant relationships.
References 1. Bosch FX, Ribes J, Cléries R, Díaz M. Epidemiology of hepatocellular carcinoma. Clin Liver Dis 2005;9(2):191–211, v. 2. Wells PN, Liang HD. Medical ultrasound: imaging of soft tissue strain and elasticity. J R Soc Interface 2011;8(64):1521–1549. 3. Zhi H, Xiao XY, Yang HY, Ou B, Wen YL, Luo BM. Ultrasonic elastography in breast cancer diagnosis: strain ratio vs 5-point scale. Acad Radiol 2010;17(10):1227–1233. 4. Xing P, Wu L, Zhang C, Li S, Liu C, Wu C. Differentiation of benign from malignant thyroid lesions: calculation of the strain ratio on thyroid sonoelastography. J Ultrasound Med 2011;30(5):663–669. 5. Zhang Y, Tang J, Li YM, et al. Differentiation of prostate cancer from benign lesions using strain index of transrectal real-time tissue elastography. Eur J Radiol 2012;81(5):857– 862. 6. Bercoff J, Tanter M, Fink M. Supersonic shear imaging: a new technique for soft tissue elasticity mapping. IEEE Trans Ultrason Ferroelectr Freq Control 2004;51(4):396– 409. 7. Chen S, Urban MW, Pislaru C, et al. Shearwave dispersion ultrasound vibrometry (SDUV) for measuring tissue elasticity and viscosity. IEEE Trans Ultrason Ferroelectr Freq Control 2009;56(1):55–62. 8. Fahey BJ, Nelson RC, Bradway DP, Hsu SJ, Dumont DM, Trahey GE. In vivo visualization of abdominal malignancies with acoustic radiation force elastography. Phys Med Biol 2008;53(1):279–293. 9. Xie H, Shamdasani V, Fernandez AT, et al. Shear wave Dispersion Ultrasound Vibrometry (SDUV) on an ultrasound system: In vivo measurement of liver viscoelasticity in healthy animals. IEEE Ultrasonics Symposium 2011; 912–915.
radiology.rsna.org n Radiology: Volume 000: Number 0— 2015
ULTRASONOGRAPHY: Stiffness Value and Ratio to Discriminate Malignant from Benign Focal Liver Lesions
10. Chen S, Sanchez W, Callstrom MR, et al. Assessment of liver viscoelasticity by using shear waves induced by ultrasound radiation force. Radiology 2013;266(3):964–970.
15. Ling W, Lu Q, Quan J, Ma L, Luo Y. Assessment of impact factors on shear wave based liver stiffness measurement. Eur J Radiol 2013;82(2):335–341.
11. Berg WA, Cosgrove DO, Doré CJ, et al. Shear-wave elastography improves the specificity of breast US: the BE1 multinational study of 939 masses. Radiology 2012;262(2):435–449.
16. Heide R, Strobel D, Bernatik T, Goertz RS. Characterization of focal liver lesions (FLL) with acoustic radiation force impulse (ARFI) elastometry. Ultraschall Med 2010; 31(4):405–409.
12. Sebag F, Vaillant-Lombard J, Berbis J, et al. Shear wave elastography for thyroid nodules. Clin Thyroidol 2012;23(1):7–8.
17. Yu H, Wilson SR. Differentiation of benign from malignant liver masses with Acoustic Radiation Force Impulse technique. Ultrasound Q 2011;27(4):217–223.
13. Correas JM, Tissier AM, Khairoune A, Khoury G, Eiss D, Hélénon O. Ultrasound elastography of the prostate: state of the art. Diagn Interv Imaging 2013;94(5):551–560. 14. Gheorghe L, Iacob S, Iacob R, et al. Real time elastography - a non-invasive diagnostic method of small hepatocellular carcinoma in cirrhosis. J Gastrointestin Liver Dis 2009;18(4):439– 446.
18. Cho SH, Lee JY, Han JK, Choi BI. Acoustic radiation force impulse elastography for the evaluation of focal solid hepatic lesions: preliminary findings. Ultrasound Med Biol 2010;36(2):202–208. 19. Park H, Park JY, Kim Y, et al. Characterization of focal liver masses using acoustic radiation force impulse elastography. World J Gastroenterol 2013;19(2):219–226.
Radiology: Volume 000: Number 0— 2015 n radiology.rsna.org
Lu et al
20. Gallotti A, D’Onofrio M, Romanini L, Cantisani V, Pozzi Mucelli R. Acoustic Radiation Force Impulse (ARFI) ultrasound imaging of solid focal liver lesions. Eur J Radiol 2012; 81(3):451–455. 21. Desmet Manns chronic staging.
VJ, Gerber M, Hoofnagle JH, M, Scheuer PJ. Classification of hepatitis: diagnosis, grading and Hepatology 1994;19(6):1513–1520.
22. DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics 1988;44(3):837–845. 23. Hanley JA, McNeil BJ. A method of com paring the areas under receiver operating characteristic curves derived from the same cases. Radiology 1983;148(3):839–843. 24. Hoshida Y, Villanueva A, Kobayashi M, et al. Gene expression in fixed tissues and outcome in hepatocellular carcinoma. N Engl J Med 2008;359(19):1995–2004.
9
Original Research n Ultrasonography
Hepatocellular Carcinoma: Stiffness Value and Ratio to Discriminate Malignant from Benign Focal Liver Lesions1 Qiang Lu, MD, PhD Wenwu Ling, MD Changli Lu, MD Jiawu Li, MD Lin Ma, MD Jierong Quan, MD Du He, MD Jianping Liu, MD Jiaying Yang, MD, PhD Tianfu Wen, MD Hong Wu, MD, PhD Hongguang Zhu, MD, PhD Yan Luo, MD
Purpose:
To investigate the use of stiffness value and stiffness ratio (ratio of lesion to background liver parenchyma values) to discriminate malignant from benign focal liver lesions by using histologic results as the reference standard.
Materials and Methods:
This study was approved by the institutional review board, and written informed consent was obtained. Three hundred seventy-three patients with focal liver lesions proven at histologic examination underwent measurement of liver stiffness with elastography point quantification. First, stiffness values in two regions of the background liver parenchyma (at 0.5–2 cm and .2 cm from the lesion periphery) near 163 hepatocellular carcinomas were analyzed to determine a reference background liver for calculating the stiffness ratio. Second, the use of the lesion stiffness value and the stiffness ratio for prediction of liver malignancy was investigated in a cohort of patients with 58 benign and 201 malignant lesions. Results were validated in another independent cohort of patients with 25 benign and 89 malignant lesions by using analysis of the area under the receiver operating characteristic (AUC) curve.
Results:
The coefficient of variation for the background liver at 0.5–2 cm from the lesion was higher (196%) than that at greater than 2 cm from the lesion (66%). In the development phase, diagnostic accuracy with use of the stiffness value was significantly higher than that with use of the stiffness ratio for discrimination of malignant from benign lesions (AUC, 0.86 vs 0.66, respectively; P , .001). Diagnostic performance with the stiffness value was lower than that with the stiffness ratio (AUC, 0.53 vs 0.86, respectively; P , .001) for discrimination of cirrhotic nodules from other benign lesions. Diagnostic performance with the stiffness value was significantly lower than that with the stiffness ratio (AUC, 0.58 vs 0.71 respectively; P = .007) for discrimination of metastasis from primary liver cancers. In the validation phase, similar findings were revealed for the discrimination of malignant from benign lesions (AUC, 0.87 vs 0.67; P , .001) and discrimination between metastasis and primary liver cancers (AUC, 0.49 vs 0.73; P , .001).
Conclusion:
Use of stiffness values measured in the liver parenchyma at more than 2 cm from the lesion allowed better diagnostic performance than did values measured in a region closer to the tumor. Stiffness value was more accurate than stiffness ratio for differentiation of malignant from benign focal liver lesions, but the stiffness ratio might be useful for subclassification of benign and malignant lesions.
From the Departments of Ultrasound (Q.L., W.L., J.L., L.M., J.Q., Y.L.), Pathology (C.L., D.H., J.L.), and Hepatobiliary Surgery (J.Y., T.W., H.W.), West China Hospital of Sichuan University, 37th Guoxue Xiang, Jiang Xi Street, Chengdu, Sichuan 610041, China; and Department of Pathology, Shanghai Medical College, Fudan University, Shanghai, China (H.Z.). Received May 29, 2013; revision requested July 13; revision received April 4; accepted April 22; final version accepted November 17. Supported by the National Natural Science Foundation of China (grants 30870715 and 81101060). Address correspondence to Y.L. (e-mail: [email protected]). RSNA, 2015
q
Radiology: Volume 000: Number 0— 2015 n radiology.rsna.org
RSNA, 2015
q
Online supplemental material is available for this article. 1
ULTRASONOGRAPHY: Stiffness Value and Ratio to Discriminate Malignant from Benign Focal Liver Lesions
L
iver cancer is the second leading cause of death in men and the sixth leading cause of death in women worldwide (1). Therefore, it is crucial to differentiate between benign and malignant liver lesions. Ultrasonographically (US) based elastographic technologies such as strain imaging (2–5) and shear-wave elastographic methods (6–10) have been developed to estimate the elastic modulus of tissues noninvasively. In strain imaging, the relative stiffness properties of tissues in the region of interest can be measured and quantified as the stiffness ratio, which is calculated from the strain in the lesion compared with that in the background parenchyma. In shear-wave elastography, including supersonic shear-wave imaging (6), shear-wave dispersion US vibrometry (7), acoustic radiation force impulse imaging (8), and elastography point quantification (9,10), the tissue stiffness value and/or the stiffnessrelated shear-wave velocity can be quantified. Both strain imaging and shear-wave elastographic technology have been demonstrated to be useful in the assessment of breast, thyroid, and prostate cancers (3–5,11–13). In liver applications, the lesions generally occur in a broad spectrum of background parenchyma including normal, fatty, fibrotic, and cirrhotic livers. The
Advance in Knowledge nn Use of the stiffness value was better than use of the stiffness ratio (ratio of lesion value to background liver value; area under the receiver operating characteristic curve [AUC], .871 vs .670, respectively; P , .001) for differentiation between malignant and benign focal liver lesions; whereas the stiffness ratio showed better accuracy than the stiffness value in the classification of cirrhotic nodules and other benign lesions (AUC, .858) and in the classification of metastasis, hepatocellular carcinoma, and intrahepatic cholangiocarcinoma (AUC, .706). 2
differences in the stiffness properties of the background liver make the differentiation between malignant and benign focal liver lesions by means of US elastography more complex (14– 20). The purpose of this study was to investigate use of the lesion stiffness value and stiffness ratio (lesion values to background liver values) for discrimination of malignant from benign focal liver lesions by using histologic results as the reference standard.
Materials and Methods Patients This study was approved by our ethics committee with a requirement for written informed consent. Between March 2011 and October 2013, 403 consecutive patients who met the inclusion criteria (Appendix E1 [online]) before their planned surgical liver resection underwent measurement of liver stiffness at West China Hospital. Thirty patients were excluded from the evaluation for the following reasons: (a) A total of 17 patients had received radiation therapy (n = 2), transarterial chemoembolization (n = 12), or radiofrequency ablation (n = 3); (b) five patients were unable to hold their breath during stiffness measurement; and (c) 18 patients did not have histologic results from the area in the liver at 0.5–2 cm from the lesion periphery. Thus, 373 patients who met the inclusion criteria and did not meet the exclusion criteria were included. The 373 patients were allocated to two cohorts in chronological order. Cohort 1 patients, who were recruited Implications for Patient Care nn The stiffness value of focal liver lesions may be used to distinguish between malignant and benign liver lesions. nn Values for the liver parenchyma at greater than 2 cm from the lesion periphery may be considered reference values for the background liver for calculating the stiffness ratio.
Lu et al
between March 2011 and April 2013, underwent studies to determine the reference background liver for calculation of the stiffness ratio and thresholds for the classifications of malignant (163 patients with hepatocellular carcinoma [HCC], 18 with intrahepatic cholangiocarcinoma [ICC], and 20 with hepatic metastasis) and benign (11 patients with cirrhotic nodules, 12 with focal nodular hyperplasia and 35 with hemangioma) focal liver lesions. Patients in cohort 2, who were recruited between May 2013 and October 2013, included 64 patients with HCC, 18 with ICC, seven with metastasis, one with a cirrhotic nodule, six with focal nodular hyperplasia, and 18 with hemangioma, underwent studies to validate the developed thresholds. The characteristics of the two cohorts are presented in Table E1 (online).
B-Mode US Examination Before liver stiffness measurement, all participants underwent B-mode liver US. The examinations were conducted by one of two radiologists from the Ultrasonography Department (L.Y. and L.Q., with more than 5 years of experience in hepatic applications). The US scan was performed with a US system (iU22; Royal Philips, Amsterdam, the
Published online before print 10.1148/radiol.14131164 Content codes: Radiology 2015; 000:1–9 Abbreviations: AUC = area under receiver operating characteristic curve FNH = focal nodular hyperplasia HCC = hepatocellular carcinoma ICC = intrahepatic cholangiocarcinoma Author contributions: Guarantors of integrity of entire study, Q.L., C.L., J.L., D.H., Y.L.; study concepts/study design or data acquisition or data analysis/interpretation, all authors; manuscript drafting or manuscript revision for important intellectual content, all authors; approval of final version of submitted manuscript, all authors; literature research, Q.L., C.L., J.L., D.H., J.L., H.W., Y.L.; clinical studies, Q.L., W.L., C.L., J.L., L.M., J.Q., D.H., J.Y., T.W., H.W., H.Z., Y.L.; experimental studies, C.L., D.H., J.Y., H.W., H.Z., Y.L.; statistical analysis, W.L., C.L., L.M., J.Q., D.H., H.W., Y.L.; and manuscript editing, Q.L., C.L., D.H., H.W., Y.L. Conflicts of interest are listed at the end of this article.
radiology.rsna.org n Radiology: Volume 000: Number 0— 2015
ULTRASONOGRAPHY: Stiffness Value and Ratio to Discriminate Malignant from Benign Focal Liver Lesions
Lu et al
Figure 1
Figure 1: Elastography point quantification superimposed on B-mode US images of HCC (left) and two areas of background liver (background I and II, center and right) show examples of liver stiffness measurements. Clinical images were acquired in a 69-year-old man who was proven to have HCC and cirrhosis of the liver at pathologic examination. Liver stiffness value was expressed in kilopascals by using the Young modulus.
Netherlands) equipped with an elastography point quantification feature and a 1–5-MHz transducer (C5–1, Royal Philips). The settings, including the gain, time-gain compensation, dynamic range, focal length, and mechanical index were optimized for each examination.
Liver Stiffness Measurement Elastography point quantification (Royal Philips), which was integrated in the same system as that used in the B-mode US examination, was used to assess the liver stiffness in the lesion and background liver parenchyma. Elastography point quantification technology uses focused ultrasound to induce shear-wave propagation inside tissue. Conventional US is applied to track the shear wave, and the Voigt model is used to estimate the elasticity of tissue (9,10). The quantified elastic measurement in the region of interest is expressed in kilopascals and is superimposed on a B-mode US image (Fig 1a). The size of the region of interest is dependent on depth, approximately 1.0 cm by 1.5 cm at a depth of 4 cm. The maximum penetration depth is 7 cm. The liver stiffness measurement was performed in areas without blood vessels during a 5-second breath hold at inspiration. The measurements were performed by one of two radiologists (L.W. and L.J., with more than 1 year of experience performing elastography point quantification). For patients with
multiple lesions, the largest lesion was chosen as the index lesion. Liver stiffness was measured in three locations: in the peripheral area of the largest lesion, in the liver at 0.5–2 cm from the lesion periphery (background 1), and greater than 2 cm from the lesion periphery (background 2). All the stiffness measurements in background 2 were accessed in liver segments 4, 5, or 6 (15). The measurement was performed five times for each group. The mean value of five measurements for each individual was used in the statistical analysis.
Histologic Examination Surgical specimens of the lesions and background areas were fixed with 10% formalin and routinely embedded in paraffin. The tissue sections were stained with hematoxylin and eosin (Fig E1 [online]). The hematoxylin and eosin sections were independently evaluated by two pathologists (C.L. and D.H., with 7 and 8 years of experience, respectively) and were reviewed by a gastroenterologic pathologist (Z.H., with 30 years of experience). When disagreements occurred, the three pathologists reviewed the sections independently and repeatedly until at least two pathologists reached consensus. The fibrosis severity in background 1 was determined according to the Scheuer classification system (21). Stages 1, 2, or 3 were classified as fibrotic liver, while stage 4 was classified as cirrhotic liver.
Radiology: Volume 000: Number 0— 2015 n radiology.rsna.org
Statistical Analysis To determine the intraobserver variability of the stiffness measurements, coefficients of variation were calculated and analyzed in different types of lesions. The patients were assigned to two independent cohorts, cohort 1 (n = 259) and cohort 2 (n = 114). In cohort 1, stiffness values in two regions of the background liver near the 163 HCCs were first analyzed to determine a reference background liver value for calculating stiffness ratio. Then, thresholds for stiffness value and stiffness ratio for the differentiation of malignant versus benign lesions, early HCC (diameter 3 cm) versus cirrhotic nodules, cirrhotic nodules versus other benign lesions, and metastasis versus primary liver cancers were investigated. In cohort 2, stiffness value and stiffness ratio for differentiation of malignant versus benign lesions and metastasis versus primary liver cancers were validated. A Mann-Whitney unpaired test was performed with the log 2-transformed data to determine the differences between each two categories. All of the P values were two sided. A difference was considered significant if a P value was less than .05. Receiver operating characteristic curve analysis (22) was performed on stiffness measurements. Diagnostic performance was evaluated by using area under the receiver operating characteristic curve (AUC) analysis (23). Sensitivity, specificity, positive prediction value, and negative prediction 3
ULTRASONOGRAPHY: Stiffness Value and Ratio to Discriminate Malignant from Benign Focal Liver Lesions
value were calculated with an optimal cutoff value that maximized the sum of sensitivity and specificity. In addition, comparison of receiver operating characteristic curve analysis was performed to determine the difference in diagnostic performance with stiffness value and with stiffness ratio. The difference was considered significant if the P value was less than .05. Software (10.4.7.0; MedCalc, Ostend, Belgium) was used to perform the Mann-Whitney unpaired test, boxand-whisker plot, and receiver operating characteristic curve analysis. All measurements were used in the statistical analysis.
Results Intraobserver Variability of Liver Stiffness Measurements Intraobserver variability of liver stiffness measurements in different lesions and background 2 are shown in Figure E2 and Table E2 (online). Malignant tumors showed significantly higher variability than did benign lesions (P , .0001). In comparison, there were no significant differences in variability of background 2 measurements between the benign and malignant focal liver lesions (P = .05). Stiffness Values in Background Liver Parenchyma near the HCC The stiffness values in the lesions and background parenchyma of 163 patients with HCC in cohort 1 are illustrated in Figure 2a. In these patients, all the stiffness values in background 2 were measured in liver segments 4, 5, or 6. Stiffness values in background 1 in 15 patients (9.2%) were obtained in the left lobe of the liver. The coefficient of variation for background 1 (196%) was higher than that for background 2 (66%) and even higher than that for the HCC lesions (113%). The diagnostic accuracy (measured as the AUC) for discrimination of HCC from background 1 was significantly lower than that for HCC versus background 2 (AUC, 0.75 vs 0.84; P , .001, Fig 2b). 4
Lu et al
Figure 2
Figure 2: A, Box-and-whisker plot for liver stiffness values in two areas of background liver and in HCC. B, Graph shows AUC estimations for HCC versus background 1 (blue line; AUC, 0.75) and HCC versus background 2 (red dashed line; AUC, 0.84). Liver stiffness value was expressed in kilopascals by using the Young modulus. Red squares in plot indicate measurements larger than upper quartile plus three times the interquartile range.
Stiffness Value and Stiffness Ratio Profiles in Focal Liver Lesions Stiffness values for six different types of focal liver lesions in 373 patients are shown in Figure 3a. The mean stiffness values and ranges expressed in kilopascals for patients with hemangioma, focal nodular hyperplasia (FNH), cirrhotic nodules, HCC, ICC, and metastasis were 9.3 (range, 3.1–41), 10 (range, 2.9–26), 11 (range, 4.4–49), 34 (range, 4.4–188), 25 (range, 5.5–79), and 30 (range, 4.7–64), respectively (Table 1). No significant difference in values was found between cirrhotic nodules and other benign lesions (P = .91). Similarly, no significant difference in values was observed for metastasis versus HCC or ICC (P = .24). The distribution of stiffness values in background 2 for patients with different focal liver lesions is shown in Figure 3b. The mean stiffness values
and ranges expressed in kilopascals in background 2 for patients with hemangioma, FNHs, cirrhotic nodules, HCCs, ICCs, and metastases were 4.6 (range, 2.6–12), 4.3 (range, 2.7–7.0), 15 (range, 5.0–42), 11 (range, 1.8–43), 7.2 (range, 3.1–20), and 5.9 (range, 3.1–12), respectively (Table 1). Significantly higher stiffness values were observed in background 2 near cirrhotic nodules compared with those near benign lesions (P , .0001). In comparison, background 2 near metastases showed significantly lower stiffness values than did those near HCCs and ICCs (P , .0001). The stiffness ratios (lesion to background 2) are illustrated in Figure 3c. The mean stiffness ratios and ranges for patients with hemangiomas, FNHs, cirrhotic nodules, HCCs, ICCs, and metastases were 2.2 (range, 0.4–6.7), 2.7 (range, 0.5–5.1), 0.9 (range, 0.3–4.0), 4.1 (range, 0.5–37),
radiology.rsna.org n Radiology: Volume 000: Number 0— 2015
ULTRASONOGRAPHY: Stiffness Value and Ratio to Discriminate Malignant from Benign Focal Liver Lesions
Figure 3
Figure 3: Stiffness values and stiffness ratios for 53 hemangiomas, 18 FNHs, 12 cirrhotic nodules, 227 HCCs, 36 ICCs, and 27 metastases. A, Box-and-whisker plot shows distribution of liver stiffness values in patients with hemangioma, FNH, cirrhotic nodule, HCC, ICC, and metastasis (malignant vs benign lesions [P , .0001], cirrhotic nodule vs other benign lesions [P = .9072] and metastasis vs HCC and ICC [P = .2449]). B, Box-and-whisker plot shows distribution of liver stiffness values in background 2 of patients with hemangioma, FNH, cirrhotic nodule, HCC, ICC, and metastasis (malignant vs benign lesions [P , .0001], cirrhotic nodules vs other benign lesions [P , .0001] and metastasis vs HCC and ICC [P , .0001]). C, Box-andwhisker plot shows liver stiffness ratios (lesion to background 2 values) in patients with hemangioma, FNH, cirrhotic nodules, HCC, ICC, and metastasis (malignant vs benign lesions [P = .0001], cirrhotic nodules vs other benign lesions [P = .0001], and metastasis vs HCC and ICC [P = .0002]). Liver stiffness values were expressed in kilopascals by using the Young modulus. Red squares in plots indicate measurements larger than upper quartile plus three times the interquartile range. Radiology: Volume 000: Number 0— 2015 n radiology.rsna.org
Lu et al
4.4 (range, 0.9–18), and 5.6 (range, 1.0–13), respectively (Table 1). Cirrhotic nodules had significantly lower stiffness ratios than did other benign lesions (P = .0001). In comparison, metastases had significantly higher stiffness ratios than did HCCs and ICCs (P = .0002).
Development Phase: Stiffness Value and Stiffness Ratio in Liver Malignancy Prediction In cohort 1 (n = 259), malignant tumors had significantly higher stiffness values and stiffness ratios than did benign lesions (P , .0001 and P = .0003, respectively; Table 2). For discrimination of malignant from benign lesions, diagnostic accuracy with use of the stiffness value was significantly higher than that with use of the stiffness ratio (AUC, 0.86 vs 0.69; P , .001; Fig 4a). With a cutoff value of 13 kPa for the stiffness value, the sensitivity, specificity, positive predictive value, and negative predictive value were 78%, 83%, 94%, and 52%, respectively (Table 2). With a cutoff value of 1.3 for the stiffness ratio, the sensitivity, specificity, positive predictive value, and negative predictive value were 79%, 45%, 83%, and 38%, respectively. For discrimination of early HCCs from cirrhotic nodules (Table 2, Fig 4b), diagnostic accuracy with the stiffness value was lower than that with the stiffness ratio (AUC, 0.77 vs 0.83; P = .32). For discrimination of cirrhotic nodules from other benign lesions (Table 2, Fig 4c), diagnostic performance with the stiffness value was significantly lower than that with the stiffness ratio (AUC, 0.53 vs 0.86; P , .001). For discrimination of metastasis from HCC and ICC (Table 2, Fig 4d), the diagnostic performance with the stiffness value was significantly lower than that with the stiffness ratio (AUC, 0.58 vs 0.71; P = .007). Validation Phase: Stiffness Value and Stiffness Ratio for Prediction of Liver Malignancy The same thresholds developed in cohort 1 for the classification of malignant and benign focal liver lesions were applied to predict liver malignancy at 5
ULTRASONOGRAPHY: Stiffness Value and Ratio to Discriminate Malignant from Benign Focal Liver Lesions
Lu et al
Table 1 Stiffness Value Profiles in Focal Liver Lesions and Background 2 Lesion (kPa) Variable Benign (n = 83) Hemangioma (n = 53) FNH (n = 18) Cirrhotic nodule (n = 12) Malignant (n = 290) HCC (n = 227) ICC (n = 36) Metastasis (n = 27)
Background 2 (kPa)
Stiffness Ratio (Lesion vs Background 2)
Median
Mean
Median
Mean
P Value*
Median
Mean
8.4 (2.9–49) 8.3 (3.1–41) 8.6 (2.9–26) 8.0 (4.4–49) 21 (4.4–188) 16 (4.4–188) 16 (5.5–79) 26 (4.7–64)
9.8 9.3 10 11 32 34 25 30
4.3 (2.6–42) 4.1 (2.6–12) 3.8 (2.7–7.0) 13 (5.0–42) 8.1 (18–43) 8.9 (1.8–43) 6.2 (3.1–20) 5.1 (3.1–12)
6 4.6 4.3 15 10 11 7.2 5.9
, .0001 , .0001 .0001 .0496 , .0001 , .0001 , .0001 , .0001
1.6 (0.3–6.7) 2.0 (0.4–6.7) 2.9 (0.5–5.1) 0.7 (0.3–4.0) 2.5 (0.5–37) 2.2 (0.5–37) 2.7 (0.9–18) 4.9 (1.0–13)
2.1 2.2 2.7 0.9 4.3 4.1 4.4 5.6
Note.—Data in parentheses are the range. Background 2 = background liver at greater than 2 cm from the lesion periphery.
Table 2 Diagnostic Performance with Use of Stiffness Value and Stiffness Ratio in Liver Malignancy Prediction Cohort, Diagnoses, and Stiffness Value and Ratio Development phase (cohort 1) Malignant (n = 201) vs benign lesions (n = 58) Stiffness value Stiffness ratio Early HCC (n = 33) vs cirrhotic nodule (n = 11) Stiffness value Stiffness ratio Cirrhotic nodule (n = 11) vs other benign lesions (n = 47) Stiffness value Stiffness ratio Metastasis (n = 20) vs ICC and HCC (n = 181) Stiffness value Stiffness ratio Validation phase (cohort 2) Malignant (n = 89) vs benign lesions (n = 25) Stiffness value Stiffness ratio Metastasis (n = 7) vs ICC and HCC (n = 82) Stiffness value Stiffness ratio
P Value
Mean Ratio*
AUC
Criterion (kPa)
,.0001 .0003
3.5 2.3
0.86 (0.81, 0.90) 0.66 (0.60, 0.72)
. 13 . 1.3
.0089 .0013
2.5 4.0
0.77 (0.61, 0.88) 0.83 (0.68, 0.93)
.7435 .0002
1.2 0.4
.2166 .0025
Sensitivity (%)
Specificity (%)
PPV (%)
NPV (%)
78 79
83 45
94 83
52 38
. 13 . 0.76
67 82
91 82
96 93
48 60
0.53 (0.40, 0.66) 0.86 (0.74, 0.94)
. 4.3 0.76
100 73
24 89
22 62
100 93
0.9 1.3
0.58 (0.51, 0.65) 0.71 (0.64, 0.77)
. 22 . 4.0
65 70
82 75
13 23
93 96
,.0001 ,.0001
2.8 1.5
0.87 (0.82, 0.91) 0.67 (0.61, 0.73)
. 13 , 1.3
74 82
84 28
94 80
48 30
.9211 .0419
1.0 1.7
0.49 (0.38, 0.60) 0.73 (0.63, 0.82)
. 22 . 4.0
29 57
54 73
5 15
90 95
Note.—Data in parentheses are 95% confidence intervals. Early HCC = Barcelona Clinic Liver Cancer stage 0–A (diameter # 3 cm), PPV = positive predictive value, NPV = negative predictive value * Mean ratio of the two categories.
independent validation (cohort 2; n = 114). Similarly, malignant tumors had significantly higher stiffness values and 6
stiffness ratios than did the benign lesions (P , .0001 and P , .0001, respectively; Table 2). For discrimination
of malignant from benign lesions, diagnostic accuracy with the stiffness value was significantly higher than that with
radiology.rsna.org n Radiology: Volume 000: Number 0— 2015
ULTRASONOGRAPHY: Stiffness Value and Ratio to Discriminate Malignant from Benign Focal Liver Lesions
Figure 4
Figure 4: Graphs show comparison of liver stiffness values (blue dashed line) and stiffness ratio (lesion to background 2, red dashed line) in patient cohort 1. A, Graph shows AUC estimates for stiffness values (blue dashed line; AUC, 0.86) and stiffness ratios (red dashed line; AUC, 0.66) in malignant versus benign liver lesions (P , .001). B, Graph shows AUC estimates for stiffness values (AUC, 0.77) and stiffness ratios (AUC, 0.83) for early HCCs (diameter 3 cm) versus cirrhotic nodules (P = .319). C, Graph shows AUC estimates for stiffness values (AUC, 0.53) and stiffness ratios (AUC, 0.86) for cirrhotic nodules versus those benign lesions (P , .001). D, Graph shows AUC estimates for stiffness values (AUC, 0.58) and stiffness ratios (AUC, 0.71) in metastasis versus HCC and ICC (P = .007).
the stiffness ratio (AUC, 0.87 vs 0.67; P , .001; Fig E3a [online]). With the cutoff stiffness value developed in cohort 1 of 13 kPa, sensitivity, specificity, positive predictive value, and negative predictive value were 74%, 84%, 94%, and 48%, respectively (Table 2). With the cutoff values developed in cohort 1 of 1.3 for the stiffness ratio, the sensitivity, specificity, positive predictive value, and negative predictive value were 82%, 28%, 80%, and 30%, respectively. For discrimination of metastasis from HCC and ICC, diagnostic performance with the stiffness value was significantly lower than that with
the stiffness ratio (AUC, 0.49 vs 0.73; P , .001, Table 2; Fig E3b [online]).
Discussion In this study, we investigated clinical use of the lesion stiffness value and stiffness ratio for prediction of malignancy in the liver in a large cohort of 373 patients by using elastography point quantification technology. Our study results revealed that hemangioma, FNH, HCC, ICC, and metastasis showed significantly higher stiffness values than did background liver parenchyma (all, P , .05) when the area measured was greater than 2
Radiology: Volume 000: Number 0— 2015 n radiology.rsna.org
Lu et al
cm from the lesion. This finding is different from that of Heide et al (16), who reported that the HCC was softer than the background liver, whereas Gallotti et al (20) reported higher stiffness values in the lesions than in the background liver for patients with hemangioma, FNH, HCC, and metastasis. The discrepancy might be due to the regions of background liver used in the different studies. The determination of reference background liver is crucial to understanding the role of the stiffness ratio in the characterization of focal liver lesions. We investigated two regions of the background liver on the basis of routine sampling practice for liver surgical specimens for histologic examination. In clinical conditions, shear-wave boundary effects (6–10) and biologic characteristics of tissue near tumors such as satellite nodules and stromal reactions (24) contribute to measurement variation. This is most likely why the measurements in the background tissue near the HCCs showed even greater variation than did the measurements in the HCCs. In our previous study (15), we did not observe significantly different measurements in the same liver segment between intercostal and subcostal scans. However, we found significant effects of liver location on the stiffness measurements. Liver segment 5 showed the lowest variation, while comparable stiffness values were revealed in liver segments 4, 5, and 6. Thus, for the stiffness measurements in the background liver, we located the measurement region of interest in liver segments 4–6 when they were available. We were unable to obtain stiffness measurements in the background liver at 0.5–2 cm from the lesion in liver segments 4–6 in 15 patients. This also might have contributed to the great variation in stiffness measurements in the tissue near the HCCs. Therefore, we recommend calculating stiffness ratio by using the background liver at greater than 2 cm from the lesion periphery as reference. Most of the HCCs and cirrhotic nodules in the 373 patients were found in fibrotic or cirrhotic livers, whereas 7
ULTRASONOGRAPHY: Stiffness Value and Ratio to Discriminate Malignant from Benign Focal Liver Lesions
most of the benign lesions and hepatic metastases occurred in normal livers. By comparing the stiffness value and stiffness ratio in the characterization of focal liver lesions, we found that the stiffness value was superior to the stiffness ratio for differentiation between malignant and benign liver lesions. The difference in the stiffness values of the background livers may have compromised malignancy prediction with use of the stiffness ratio. On the other hand, the low stiffness ratio in cirrhotic nodules has the potential to allow differentiation between cirrhotic nodules and other benign lesions. Similarly, the high stiffness ratio in hepatic metastasis might be useful in the subclassification of metastasis and primary liver cancers (HCC and ICC). By using quantitative acoustic radiation force impulse imaging technology, Heide et al (16) showed statistically comparable stiffness values between 38 benign lesions and 24 malignant tumors (P = .28). In comparison, Yu et al (17) showed a significant difference in the stiffness values between 41 benign and 64 malignant lesions (P , .001), with sensitivity and specificity of 68% (28 of 41) and 69% (44 of 64), respectively. Cho et al (18) investigated 51 patients with 17 hepatic hemangiomas and 43 malignant liver lesions and determined that the positive predictive value and specificity of the stiffness quantification for malignancy were 89% and 81%, respectively. Park et al (19) demonstrated an AUC of 0.74 for discrimination of 39 malignant from eight benign liver lesions. Our study results confirmed that the stiffness value could be considered a noninvasive predictor for malignancy in focal liver lesions. For the differentiation of malignant and benign liver lesions with stiffness value, sensitivity and specificity results in the development phase were 78% (156 of 159) and 83% (48 of 58), respectively. By using the same thresholds developed in cohort 1, similar sensitivity (74%, 66 of 89) and specificity (84%, 21 of 25) were obtained at independent validation in cohort 2. 8
The differentiation of early HCCs from cirrhotic nodules is important in patients with a cirrhotic liver. Gheorghe et al (14) reported high diagnostic accuracy (AUC, 0.94) of strain imaging for the differentiation of early HCCs (1–3 cm) and cirrhotic nodules in 42 patients with cirrhosis. In the development phase, our study results showed fair accuracy (sensitivity, 67%; specificity, 91%; AUC, 0.77) with use of the stiffness value and improved accuracy with use of the stiffness ratio (sensitivity, 82%; specificity, 82%; AUC, 0.83) in 44 patients with cirrhosis. Unfortunately, we could not validate this finding, because only one patient with a cirrhotic nodule was included in the validation phase. Taken together, these results indicate that the liver stiffness measurement may be a useful tool for prediction of early HCCs and for follow-up of cirrhotic nodules. Unfortunately, only eight of the patients included had very early HCCs (diameter , 2 cm), which limited our ability to evaluate the diagnostic performance of the stiffness value and stiffness ratio for discrimination of very early HCCs from cirrhotic nodules. There were several limitations in our study. First, only patients with lesions for which resections were planned were included in the cohorts. This limited the types and numbers of benign lesions investigated in this study. Further large-scale confirmation studies with rare types of lesions are needed to validate our findings. Moreover, the effect of distance between the transducer and region of interest measured on stiffness assessment was not directly evaluated. In addition, interobserver variability was not documented in the patient groups. In conclusion, we identified that the liver parenchyma at greater than 2 cm from the lesion periphery provides useful reference values for calculating the stiffness ratio. Our findings revealed that the stiffness value allowed better diagnostic performance than did stiffness ratio for differentiation between malignant and benign focal liver lesions, whereas stiffness ratio might be useful in the subclassifications of benign and malignant liver lesions.
Lu et al
Acknowledgments: We would like to thank Xiaomin Li, PhD, and Ying Wu, PhD, MD, for editorial assistance in the preparation of the manuscript. Disclosures of Conflicts of Interest: Q.L. disclosed no relevant relationships. W.L. disclosed no relevant relationships. C.L. disclosed no relevant relationships. J.L. disclosed no relevant relationships. L.M. disclosed no relevant relationships. J.Q. disclosed no relevant relationships. D.H. disclosed no relevant relationships. J.L. disclosed no relevant relationships. J.Y. disclosed no relevant relationships. T.W. disclosed no relevant relationships. H.W. disclosed no relevant relationships. H.Z. disclosed no relevant relationships. Y.L. disclosed no relevant relationships.
References 1. Bosch FX, Ribes J, Cléries R, Díaz M. Epidemiology of hepatocellular carcinoma. Clin Liver Dis 2005;9(2):191–211, v. 2. Wells PN, Liang HD. Medical ultrasound: imaging of soft tissue strain and elasticity. J R Soc Interface 2011;8(64):1521–1549. 3. Zhi H, Xiao XY, Yang HY, Ou B, Wen YL, Luo BM. Ultrasonic elastography in breast cancer diagnosis: strain ratio vs 5-point scale. Acad Radiol 2010;17(10):1227–1233. 4. Xing P, Wu L, Zhang C, Li S, Liu C, Wu C. Differentiation of benign from malignant thyroid lesions: calculation of the strain ratio on thyroid sonoelastography. J Ultrasound Med 2011;30(5):663–669. 5. Zhang Y, Tang J, Li YM, et al. Differentiation of prostate cancer from benign lesions using strain index of transrectal real-time tissue elastography. Eur J Radiol 2012;81(5):857– 862. 6. Bercoff J, Tanter M, Fink M. Supersonic shear imaging: a new technique for soft tissue elasticity mapping. IEEE Trans Ultrason Ferroelectr Freq Control 2004;51(4):396– 409. 7. Chen S, Urban MW, Pislaru C, et al. Shearwave dispersion ultrasound vibrometry (SDUV) for measuring tissue elasticity and viscosity. IEEE Trans Ultrason Ferroelectr Freq Control 2009;56(1):55–62. 8. Fahey BJ, Nelson RC, Bradway DP, Hsu SJ, Dumont DM, Trahey GE. In vivo visualization of abdominal malignancies with acoustic radiation force elastography. Phys Med Biol 2008;53(1):279–293. 9. Xie H, Shamdasani V, Fernandez AT, et al. Shear wave Dispersion Ultrasound Vibrometry (SDUV) on an ultrasound system: In vivo measurement of liver viscoelasticity in healthy animals. IEEE Ultrasonics Symposium 2011; 912–915.
radiology.rsna.org n Radiology: Volume 000: Number 0— 2015
ULTRASONOGRAPHY: Stiffness Value and Ratio to Discriminate Malignant from Benign Focal Liver Lesions
10. Chen S, Sanchez W, Callstrom MR, et al. Assessment of liver viscoelasticity by using shear waves induced by ultrasound radiation force. Radiology 2013;266(3):964–970.
15. Ling W, Lu Q, Quan J, Ma L, Luo Y. Assessment of impact factors on shear wave based liver stiffness measurement. Eur J Radiol 2013;82(2):335–341.
11. Berg WA, Cosgrove DO, Doré CJ, et al. Shear-wave elastography improves the specificity of breast US: the BE1 multinational study of 939 masses. Radiology 2012;262(2):435–449.
16. Heide R, Strobel D, Bernatik T, Goertz RS. Characterization of focal liver lesions (FLL) with acoustic radiation force impulse (ARFI) elastometry. Ultraschall Med 2010; 31(4):405–409.
12. Sebag F, Vaillant-Lombard J, Berbis J, et al. Shear wave elastography for thyroid nodules. Clin Thyroidol 2012;23(1):7–8.
17. Yu H, Wilson SR. Differentiation of benign from malignant liver masses with Acoustic Radiation Force Impulse technique. Ultrasound Q 2011;27(4):217–223.
13. Correas JM, Tissier AM, Khairoune A, Khoury G, Eiss D, Hélénon O. Ultrasound elastography of the prostate: state of the art. Diagn Interv Imaging 2013;94(5):551–560. 14. Gheorghe L, Iacob S, Iacob R, et al. Real time elastography - a non-invasive diagnostic method of small hepatocellular carcinoma in cirrhosis. J Gastrointestin Liver Dis 2009;18(4):439– 446.
18. Cho SH, Lee JY, Han JK, Choi BI. Acoustic radiation force impulse elastography for the evaluation of focal solid hepatic lesions: preliminary findings. Ultrasound Med Biol 2010;36(2):202–208. 19. Park H, Park JY, Kim Y, et al. Characterization of focal liver masses using acoustic radiation force impulse elastography. World J Gastroenterol 2013;19(2):219–226.
Radiology: Volume 000: Number 0— 2015 n radiology.rsna.org
Lu et al
20. Gallotti A, D’Onofrio M, Romanini L, Cantisani V, Pozzi Mucelli R. Acoustic Radiation Force Impulse (ARFI) ultrasound imaging of solid focal liver lesions. Eur J Radiol 2012; 81(3):451–455. 21. Desmet Manns chronic staging.
VJ, Gerber M, Hoofnagle JH, M, Scheuer PJ. Classification of hepatitis: diagnosis, grading and Hepatology 1994;19(6):1513–1520.
22. DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics 1988;44(3):837–845. 23. Hanley JA, McNeil BJ. A method of com paring the areas under receiver operating characteristic curves derived from the same cases. Radiology 1983;148(3):839–843. 24. Hoshida Y, Villanueva A, Kobayashi M, et al. Gene expression in fixed tissues and outcome in hepatocellular carcinoma. N Engl J Med 2008;359(19):1995–2004.
9
