Value of information methods for assessing a new diagnostic test.

Featured Article Received 23 September 2013,

Accepted 11 December 2013

Published online 9 January 2014 in Wiley Online Library

(wileyonlinelibrary.com) DOI: 10.1002/sim.6085

Value of information methods for assessing a new diagnostic test Maggie Hong Chena and Andrew R. Willanb*† Value-of-information methods are applied to assess the evidence in support of a new diagnostic test and, where the evidence is insufficient for decision making, to determine the optimal sample size for future studies. Net benefit formulations are derived under various diagnostic and treatment scenarios. The expressions for the expected opportunity loss of adopting strategies that include the new test are given. Expressions for the expected value of information from future studies are derived. One-sample and two-sample designs, with or without known prevalence, are considered. An example is given. Copyright © 2014 John Wiley & Sons, Ltd. Keywords:

value of information; diagnostic tests; full Bayesian approach, incremental net benefit

1. Introduction Diagnostic tests are an important tool in the provision of health care and contribute significantly to healthcare costs [1]. A good diagnostic test provides information about the patient’s condition and influences the healthcare provider’s plan for patient care [2]. Although the development of new diagnostic tests has accelerated recently, the associated methodology lags behind that for treatment evaluations [3–7], and improvements are needed to guide future research. The design and analysis of empirical studies of diagnostic tests are generally based on classical statistical analysis of performance measures, such as sensitivity, specificity and the area under the receiver operating characteristic curve [8]. This approach has been criticized because it fails to account for the trade-off between sensitivity and specificity by ignoring the relative utilities (value) of the diagnostic outcomes, namely true positive (tp), false positive (fp), true negative (tn) and false negative (fn). In this paper, we propose a full Bayesian decision theory approach for examining evidence in support of a new diagnostic test with the purpose of identifying whether or not the evidence is sufficient for adopting the ‘best’ diagnostic strategy and, where the evidence is in insufficient, identifying the ‘optimal’ design for future research. Because the evidence from clinical studies has associated uncertainty in the model parameters, such as sensitivity and specificity, decisions based on the evidence, such as implementing a new diagnostic strategy, can be wrong. In decision theory, the possibility of a wrong decision is associated with an expected opportunity loss (EOL), which can be expressed in monetary terms. The opportunity loss is a function of the model parameters, and by taking a Bayesian approach, the current evidence can be used to derive probability distributions for the model parameters. Taking the expectation of the opportunity loss with respect to the model parameters provides the EOL. Consider a potential future study. By applying Bayes theorem to update the current evidence with data from the future study, the uncertainty and therefore the EOL can be reduced. In this situation, the expectation of opportunity loss must be taken not only with respect to the ‘updated’ model parameters but also with respect to data from the future study. The reduction in the EOL is referred to as the expected value of sample information (EVSI) of the future study. If the EVSI of the future study exceeds the study’s total cost (TC), then it is considered worthwhile, and the future study that maximizes the difference between

a Dalla

Lana School of Public Health, University of Toronto, Toronto, ON, Canada in Child Health Evaluative Sciences, SickKids Research Institute, Toronto, ON, Canada *Correspondence to: Andrew R. Willan, Program in Child Health Evaluative Sciences, SickKids Research Institute, Toronto, ON, Canada. † E-mail: [email protected] b Program

Statist. Med. 2014, 33 1801–1815

1801

Copyright © 2014 John Wiley & Sons, Ltd.

M. H. CHEN AND A. R. WILLAN

EVSI and TC is considered optimal. The application of decision theory in this context is often referred to as value-of-information (VOI) methods and has been applied extensively in the assessment of evidence in support of therapeutic interventions and the consideration of performing future randomized clinical trials [9–31]. In the next section, we apply VOI methods to the assessment of evidence (i.e., sensitivity and specificity) in support of a new diagnostic test. Following many others [32–46], our approach requires the identification of the health outcomes and healthcare costs for the four diagnostic outcomes. We use net benefit as the measure of utility. For simplicity, we assume that there exists a gold standard that can be used to establish the definitive diagnosis (i.e., 100% accurate) and that no other diagnostic test is available. The assumption of no other diagnostic test can easily be relaxed as noted in the discussion. Both one-sample and two-sample designs for current evidence and future studies are considered. In a one-sample design, a cross section of subjects suspected of having the disease is sampled, and the results of the diagnostic test and the gold standard are recorded. In a two-sample design, subjects known to have the disease are sampled, and independently, subjects known not to have the disease are sampled. The results of the diagnostic test are recorded for the subjects in both samples. We illustrate the methods with an example.

2. Methods Consider a treatable disease for which the prevalence among those suspected of having it is denoted by g. Assume there exists a gold standard test that can accurately distinguish between those who have the disease and those who do not. Suppose further that there is a new diagnostic test for which the evidence regarding its sensitivity and specificity supports its use for identifying patients with the disease. The purpose of this paper is to examine the use of VOI methods to determine if current evidence regarding sensitivity and specificity of the new test is sufficient for its adoption. In VOI methodology, the current evidence is considered sufficient if for all potential future studies, the value of the additional evidence (EVSI) is less than the study costs (TC). If the current evidence is not sufficient, VOI methods can be used to determine the optimal design for a future study. In this section, we establish the formulations for the quantities required for applying VOI methods for the evaluation of a new diagnostic test. We start with the definition and formulations for net benefit (utility) for various strategies for diagnosing and treating patients. Based on current evidence, the expected net benefit for each strategy can be determined, and the net benefit-maximizing strategy identified. We then derive the per-patient EOL for the decision to adopt the net benefit-maximizing strategy (EOLpp0 /. Attention is then given to reducing the EOL by considering a future study. We derive the EOL with the evidence updated from a future study, which is denoted EOLpp1 . Following that, we derive the EVSI and the TC of the future study. The EVSI is the difference between EOLpp0 and EOLpp1 , multiplied by the number of potential patients for whom the test would be relevant [24]. The TC of the future study is exclusively financial and is usually assumed to include a fixed amount plus an amount per patient. The expected net gain (ENG) of the future study is the difference between EVSI and TC. If the ENG is negative for all designs and sample sizes, then the optimal decision is to adopt the net benefit-maximizing strategy based on current evidence because no further research is worthwhile (i.e., the evidence is considered sufficient for decision making). On the other hand, if the ENG is positive for some design and sample size, the optimal decision is to carry out a future study with the design and sample size that maximize the ENG. We assume that the net benefit-maximizing diagnostic strategy based on current evidence includes the use of the new test because otherwise there would be little interest in studying it further. Let etp , etn , efp and ef n be the health outcome (effectiveness) for a subject who is a true positive (tp), a true negative (tn), a false positive (fp) and a false negative (fn), respectively. The healthcare costs for such patients are similarly defined as ctp , ctn , cfp and cf n . Net benefit is defined in monetary terms as NBij D eij cij ; i 2 ft; f g; j 2 fp; ng, where is the threshold value of the willingness to pay for a unit of health outcome. The quantities eij and cij , and therefore NBij ; i D ft; f g; j D fp; ng, are assumed known. Assuming there is no other existing suitable diagnostic test other than the gold standard, there are five reasonable diagnostic/treatment strategies available for patients suspected of having the disease:

1802

Treat: No treat:

Treat all subjects suspected of having the disease. Treat no subjects suspected of having the disease.


Statist. Med. 2014, 33 1801–1815


Gold: Apply the gold standard test and treat those identified as having the disease. Test: Apply the new test and treat only those who test positive. Test-gold: Apply the new test, and then apply the gold standard to those who test positive and treat those identified as having the disease. The remainder of this section is organized as follows. In Section 2.1, we consider the case where current evidence comes from a two-sample study. The net benefit of the two strategies that include the new test can be expressed as a linear function of sensitivity and specificity. This provides expressions for the current mean and variance of incremental net benefit and leads to an expression for the prior per-patient EOL by taking the expectation with respect to sensitivity and specificity. VOI methods are based on a Bayesian approach where sensitivity and specificity are considered random variables, the distributions for which are based on current evidence. Then, considering a future two-sample study, an expression for posterior per-patient EOL is derived as a function of sample size. In Section 2.2, the solution for the case where current and future evidence comes from a one-sample study is given. Here, the net benefit of the two strategies that include the new test can be expressed as linear functions of the probabilities that a subject is a true positive, a false negative or a true negative. 2.1. Current evidence from a two-sample design study In this section, we assume that the disease prevalence (g/ is known. In a two-sample study, the diagnostic test result is determined in a sample of subjects known to have the disease and independently in a sample of subjects known to be disease free. The net benefit of each diagnostic/treatment strategy is given in Table I, where X, Y and Ctest are the sensitivity, specificity and cost of the new test, respectively, and Cgold is the cost of the gold standard test. Cgold and Ctest could include an amount reflecting the willingness to pay to avoid the respective tests if they are invasive or otherwise undesirable. Let NBrel D maxfNBtreat ; NBno treat ; NBgold g, then the strategy associated with NBrel is the net benefit-maximizing strategy without using the new test and will be considered the current standard. The incremental net benefit (INB) of adopting the Test strategy is defined as INBtest D NBtest NBrel and, using the entries from Table I, can be written as a linear function of the sensitivity and specificity, given by INBtest D a1;test X C a2;test Y C a3;test , where a1;test D g.NBtp NBfn / and a2;test D .1 g/.NBtn NBfp /. The term a3;test depends on which strategy is associated with NBrel and is given in the second column of Table II. Similarly, the incremental net benefit of adopting the Testgold strategy is defined as INBtest-gold D NBtest-gold NBrel and can be written as a linear function of the sensitivity and specificity, given by INBtest-gold D a1;test-goldX C a2;test-gold Y C a3;test-gold, where a1;test-gold D g.NBtp NBf n Cgold / and a2;test-gold D .1 g/Cgold . Again, the term a3;test-gold depends on which strategy is associated with NBrel and is given in the third column of Table II. Suppose the current evidence regarding the new test comes from a two-sample study in which ˛1 1 patients tested positive in a sample of ˛1 C ˛2 2 patients who were known to have the disease and ˛3 1 patients tested negative in a independent sample of ˛3 C˛4 2 patients who were known to be disease free. If we use uniform uninformative priors for sensitivity and specificity (i.e., Beta(1,1)), then the posterior distributions for sensitivity and specificity are given by Beta.˛1 ; ˛2 / and Beta.˛3 ; ˛4 /, respectively. This is simply an application of Bayes theorem where the prior beta distributions are combined with the likelihood for binomial sampling to yield the posterior beta distributions. The mean and vari2 ance for sensitivity and specificity are given by ˛i =˛i;iC1 and ˛i ˛iC1 =f˛i;iC1 .˛i;iC1 1/g, respectively,

Table I. Net benefit by diagnostic/treatment strategy with current evidence from a two-sample study. Strategy

Net benefit

Treat

NBtreat D gNBtp C .1 g/NBfp

No treat

NBno

Gold Test

NBgold D gNBtp C .1 g/NBt n Cgold ˚ ˚ NBtest D g X NBtp C .1 X/NBfn C .1 g/ Y NBtn C .1 Y /NBfp Ctest

Test-gold

NBtest-gold D gfX NBtp C .1 X/NBfn g C .1 g/NBtn Ctest fg.X C Y 1/ C .1 Y /gCgold

treat

D gNBfn C .1 g/NBtn


Statist. Med. 2014, 33 1801–1815

1803

X, Y and Ctest are the sensitivity, specificity and cost of the new test, respectively, and Cgold is the cost of the gold standard test.


Table II. The coefficient a3 by the strategy associated with NBrel with current evidence from two-sample study. Coefficient a3;test

a3;test-gold

g.NBfn NBtp / Ctest

g.NBfn NBtp / C .1 g/.NBtn NBfp /

Strategy associated with NBrel Treat

Ctest .1 g/Cgold No treat Gold

.1 g/.NBfp NBtn / Ctest

Ctest .1 g/Cgold

g.NBfn NBtp / C .1 g/.NBfp NBtn / CCgold Ctest

g.NBf n NBtp / Ctest C Cgold

where i = 1 for sensitivity and i = 3 for specificity, and where ˛j;j 0 D mean and variance for INBtest and INBtest-gold are given by EX;Y .INBk / D a1;k

2 VX;Y .INBk / D a1;k

Pj 0

iDj

˛i . Therefore, the current

˛1 ˛3 C a2;k C a3;k ˛1;2 ˛3;4

˛1 ˛2 2 ˛1;2 .˛1;2 C 1/

2 C a2;k

(1)

˛3 ˛4 2 ˛3;4 .˛3;4 C 1/

(2)

where k = test or test-gold. Let 0 D max fEX;Y .INBk / W k 2 ftest; test-goldgg and refer to strategyMax as that strategy (either test or test-gold) that has the larger expected INB (i.e., EX;Y .INBstrategyMax/ D 0 ) and let v0 D VX;Y .INBstrategyMax/. If 0 > 0, then strategyMax is the net benefit-maximizing strategy based on current evidence, and if no further evidence is forthcoming, adopting strategyMax is the optimal decision. However, because v0 > 0, adopting strategyMax is associated with a positive expected opportunity costs (EOL). If we assume that ˛1;2 and ˛3;4 are sufficiently large so that the distribution of INBstrategyMax is approximately normal, then from Willan and Pinto [24], the per-patient opportunity loss of adopting strategyMax is given by EOLpp0 D D.0 ; v0 /; where p ı ıp D.; v/ D v=.2/ expf2 .2v/g Œˆ. v/ I f 6 0g

(3)

ˆ./ is the cdf for a standard normal random variable and I fg is the indicator function. Thus, EOLpp0 depends on astrategyMax and ˛, where astrategyMax D .a1;strategyMax; a2;strategyMax; a3;strategyMax/ and ˛ D .˛1 ; ˛2 ; ˛3 ; ˛4 /. Consider a potential future two-sample study where among the patients with the disease, n1 test positive and n2 test negative and among the independently sampled disease-free patients, n3 test negative and n4 test positive. If the study is conducted, the posterior distributions for sensitivity and specificity would be Beta.˛1 C n1 ; ˛2 C n2 / and Beta.˛3 C n3 ; ˛4 C n4 /, respectively. Let n1 n3 ZO 1 D a1;strategyMax C a2;strategyMax C a3;strategyMax n1;2 n3;4

(4)

Pj 0 where nj;j 0 D iDj ni : ZO 1 is the estimate of incremental net benefit from the study data. n1 and n3 have Beta-binomial distributions, and the predictive distribution of ZO 1 has mean 0 and variance given by

1804

˛1 ˛2 .˛1;2 C n1;2 / ˛3 ˛4 .˛3;4 C n3;4 / 2 2 O D a1;strategyMax C a2;strategyMax vZO 1 D V .Z/ 2 2 ˛1;2 .˛1;2 C 1/n1;2 ˛3;4 .˛3;4 C 1/n3;4 Copyright © 2014 John Wiley & Sons, Ltd.

(5)

Statist. Med. 2014, 33 1801–1815


If we assume that n1;2 and n3;4 are sufficiently large so that the distribution of ´O is approximately normal, then from Willan and Pinto [24], the updated mean and variance of incremental net benefit are given by ( ) 0 ZO 1 1 1 1 1 D v1 and v1 D C C ; where v0 v1 v0 v1 (6) ˛1 ˛2 ˛3 ˛4 2 2 v1 D a1;strategyMax 2 C a2;strategyMax 2 ˛1;2 .n1;2 C 1/ ˛3;4 .n3;4 C 1/ Again, assuming that ZO 1 is approximately normal, the per-patient EOL after updating the evidence, which requires taking the expectation with respect to ZO 1 , is given by EOLpp1 .n1;2 ; n3;4 / D R1 EZO 1 D.1 ; v1 / D 1 D.1 ; v1 / fN .ZO 1 I 0 ; v´O 1 / d ZO 1 , where fN .I ; v/ is the pdf for a normal random variable with mean and variance v. R1 Willan and Pinto [24] show that 1 D.1 ; v1 / fN .ZO 1 I 0 ; vZO 1 / d ZO 1 D I1 C I2 C I3 , where . p ı I1 D v v0 =.2/exp.20 2v0 / vZO 1 . p ıp ı 3=2 I2 D 0 ˆ.0 v0 / C v0 exp.20 2v0 / .vZO 1 2/ (7) . . . q p I3 D 0 ˆ.0 vZO 1 v0 / v0 expf20 vZO 1 .2v02 /g 2vZO 1 Thus, EOLpp1 .n1;2 ; n3;4 / is a function of n1;2 and n3;4 and depends on astrategyMax and ˛. The expected VOI (EVSI) of the new study is given by EVSI.n1;2 ; n3;4 / D N fEOLpp0 EOLpp1 .n1;2 ; n3;4 /g where N is the number of potential patients for whom the test is relevant. Costs for the study are assumed to consist of a fixed cost .Cf / to set up and run the study and a variable cost per patient .Cv /. Therefore, the TC (TC) for the study is given by TC.n1;2 ; n3;4 / D Cf C n1;4 Cv

(8)

The ENG of the study is the EVSI minus the TC. Thus, ˚ ENG.n1;2 ; n3;4 / D N fEOLpp0 EOLpp1 .n1;2 ; n3;4 /g Cf C .n1;2 C n3;4 /Cv

(9)

Let .n1;2 ; n3;4 / be those values of .n1;2 ; n3;4 / that maximize ENG. If ENG.n1;2 ; n3;4 / 6 0, then the future study should not be performed. The current information is considered sufficient, and assuming 0 > 0, the optimal decision is to adopt strategyMax. On the other hand, if ENG.n1;2 ; n3;4 / > 0, then the future study is worthwhile. In this case, the optimal decision is to perform a study in which the test results are determined on n1;2 patients with the disease and n3;4 patients who are disease free. 2.2. Current evidence from a one-sample study


Statist. Med. 2014, 33 1801–1815

1805

Suppose now that the current evidence comes from a one-sample study where the disease status and test results were determined simultaneously on a sample of ˛1;4 4 patients, in which ˛1 1 were determined to have the disease and test positive, ˛2 1 were determined to have the disease and test negative, ˛3 1 were determined to be disease free and test negative and ˛4 1 were determined to be disease free and test positive. Let p D .p1 ; p2 ; p3 ; p4 /, where p1 is the probability that a patient is disease and test positive, p2 is the probability that a patient is disease positive and test negative, p3 is the probability that a patient is disease and test negative and p4 is the probability that a patient is disease negative and test positive. If we assume an uninformative Dirichlet prior (i.e., Dir(1,1,1,1)), then the distriı posterior 2 bution for p is Dir.˛/. Thus, the mean and variance of pi is ˛i =˛1;4 and ˛i .˛1;4 ˛i / f˛1;4 .˛1;4 C 1/g, ı 2 respectively, and the covariance of pi and pj is given by C.pi ; pj / D ˛i ˛j f˛1;4 .˛1;4 C 1/g. Assuming the prevalence (g/ is known, the expressions for net benefit for the diagnostic/treatment strategies are given in Table III, which are the same as those in Table I where p1 =p1;2 has been substituted for sensitivity (X/ and p3 =p3;4 has been substituted for specificity (Y /. Recalling that NBrel D maxfNBtreat ; NBno treat ; NBgold g and recognizing that p4 D 1 p1;3 , the incremental net benefit of adopting the test strategy can be expressed, using the entries in Table III, as a linear function of the first three


Table III. Net benefit by diagnostic/treatment strategy with current evidence from a one-sample study. Strategy

Net benefit

Treat

NBtreat D gNBtp C .1 g/NBfp

No treat

NBno

treat

D gNBfn C .1 g/NBtn

Gold

NBgold D gNBtp C .1 g/NBtn Cgold

Test

NBtest D p1 NBtp C p2 NBfn C p3 NBtn C p4 NBfp Ctest

Test-gold

NBtest- gold D p1 NBtp C p2 NBfn C p3;4 NBtn Ctest .p1 C p4 /Cgold

Table IV. The coefficient b4 by the strategy associated with NBrel , with current evidence from a one-sample study. Coefficient a4;test

a4;test-gold

g.NBfn NBtp / Ctest

.NBtn NBfp / C g.NBfp NBtp /

Strategy associated with NBrel Treat

Cgold Ctest No treat

NBfp NBtn C g.NBtn NBfn / Ctest

g.NBfn NBtn / Cgold Ctest

NBfp NBtn C g.NBtn NBtp / CCgold Ctest

g.NBtn NBtp / Ctest

Gold

Table V. Expected net benefit by diagnostic/treatment strategy with current evidence from a one-sample study with unknown prevalence. Strategy

Net benefit

Treat

NBtreat D p1;2 NBtp C p3;4 NBfp

No treat

NBno

Gold

NBgold D p1;2 NBtp C p3;4 NBtn Cgold

Test

NBtest D p1 NBtp C p2 NBfn C p3 NBtn C p4 NBfp Ctest

Test-gold

NBtest-gold D p1 NBtp C p2 NBfn C p3;4 NBtn Ctest .p1 C p4 /Cgold

treat

D p1;2 NBfn C p3;4 NBtn

elements of p, defined by INBtest D b1;test p1 C b2;test p2 C b3;test p3 C b4;test , where b1;test D NBtp NBfp , b2;test D NBfn NBfp and b3;test D NBtn NBfp . The term b4;test depends on which strategy is associated with NBrel and is given in the second column of Table IV. Similarly, the incremental net benefit of the test-gold strategy can be expressed as INBtest-gold D b1;test-gold p1 Cb2;test-gold p2 Cb3;test-gold p3 Cb4;test-gold , where b1;test-gold D NBfp NBtn , b2;test-gold D NBfn NBtn CCgold and b3;test-gold D Cgold . The term b4;test-gold depends on which strategy is associated with NBrel and is given in the third column of Table IV. If prevalence is unknown and the evidence regarding it comes solely from the current one-sample study, the net benefit for the strategies is given in Table V, which is the same as Table III with p1;2 substituted for prevalence (g/. For unknown prevalence, the coefficients b3;test and b3;test-gold remain the same, that is, b3;test D NBtn NBfp and b3;test-gold D Cgold . The other coefficients depend on which strategy is associated with NBrel and are given in Table VI. Recalling that p Dir.˛/, the current mean and variance for INBtest and INBtest-gold are given by Ep .INBk / D b1;k

Vp .INBk / D

1 2 ˛1;4 .˛1;4 C 1/

˛1 ˛2 ˛3 C b2;k C b3;k C b4;k ˛1;4 ˛1;4 ˛1;4

8 3 < X iD1

:

2 ˛i .˛1;4 ˛i / C 2 bi;k

X j >i

bi;k bj;k ˛i ˛j

(10) 9 = ;

(11)

1806

˚ where k = test or test-gold. We redefine 0 D max Ep .INBk / W k 2 ftest; test-goldg and again refer to strategyMax as that strategy (either test or test-gold) that has the larger expected INB (i.e., Copyright © 2014 John Wiley & Sons, Ltd.

Statist. Med. 2014, 33 1801–1815

NBtn NBfp

ab

NBfn NBtp CNBtn NBfp

NBtn NBfp

NBtp NBfn

CNBtn NBfp

Ctest

NBfn NBtp

0

NBfn NBtp CCgold Ctest

NBfp NBtn Ctest

b4;test

b2;test

b1;test

3;test D NBtn NBfp , regardless of strategy. bb 3;test-gold D Cgold , regardless of strategy.

Gold

No treat

Treat

Strategy associated with NBrel

1807


Coefficienta

0

NBtp NBfn

NBfp NBtn

b1;test-gold

Ctest Cgold

Cgold

Ctest

NBfn NBtp C Cgold

NBtn NBfp Cgold Ctest

NBfn NBtp C NBfp

b4;test-gold NBtn C Cgold

b2;test-gold

Coefficientb

Table VI. The coefficients b1 ; b2 and b4 by the strategy associated with NBrel with current evidence from a one-sample study with unknown prevalence.


Statist. Med. 2014, 33 1801–1815


Ep .INBstrategyMax/ D 0 ). Redefining v0 D Vp .INBstrategyMax/, the prior per-person opportunity cost of adopting strategyMax is given by EOLpp0 D D.0 ; v0 / and is a function of bstrategyMax and ˛, where bstrategyMax D .b1;strategyMax ; b2;strategyMax ; b3;strategyMax; b4;strategyMax /. Consider a possible future one-sample study in which the disease status and test results are determined simultaneously on a sample of n1;4 patients, in which n1 are determined to have the disease and test positive, n2 are determined to have the disease and test negative, n3 are determined to be disease free and test negative and n4 are determined to be disease free and test positive. The posterior distribution for p is Dir.˛ C n/, where n D .n1 ; n2 ; n3 ; n4 /. The estimate of incremental net benefit based on the data from the new study is given by n1 n2 n3 ZO 2 D b1;strategyMax C b2;strategyMax C b3;strategyMax C b4;strategyMax n1;4 n1;4 n1;4 n has a multivariate Polya compound distribution, and the predictive distribution of ZO 2 has mean 0 and variance given by vZO 2 D Vp .ZO 2 / D

.˛1;4 C n1;4 / A 2 ˛1;4 .˛1;4 C 1/n1;4

(12)

8 9 3 < 3 = X X 2 where A D ˛i .˛1;4 ˛i / C 2 bi;strategyMaxbj;strategyMax ˛i ˛j bi;strategyMax : ; iD1

(13)

j >i

If we assume that n1;4 is sufficiently large so that the distribution of ZO 2 is approximately normal, then the updated mean and variance of incremental net benefit are given by ( ) 0 ZO 2 1 1 1 1 2 D v2 C and v2 D C ; where v2 D 2 A (14) v0 v2 v0 v2 ˛1;4 .n1;4 C 1/ Again, assuming that ZO 2 is approximately normal, the per-patient EOL with the additional evidence from the new study, given by Z1 EOLpp2 .n1;4 / D EZO 2 D.2 ; v2 / D

D.2 ; v2 / fN .ZO 2 I 0 ; vZO 2 / dZO 2

(15)

1

is a function of n1;4 and depends on bstrategyMax and ˛ and can be evaluated using Equations 7 and 12. The expression for ENG becomes ENG.n1;4 / D N fEOLpp0 EOLpp2 .n1;4 /g .Cf C n1;4 Cv /

(16)

Again, if n1;4 maximizes ENG.n1;4 / and ENG.n1;4 / 6 0, the optimal decision, assuming 0 > 0, is to adopt strategyMax, and no additional research is required. On the other hand, if ENG.n1;4 / > 0, the optimal course of action is to conduct a cross-sectional study with n1;4 patients.

3. Example

1808

Pulmonary embolism is a life-threatening event. Treatment with anticoagulation is effective in reducing the risk of death. Anticoagulation, however, has a risk of major bleeding, which can be fatal, and therefore, avoiding unnecessary anticoagulation is desirable. A pulmonary angiogram (a catheterization and X-ray examination in which the vessels of the lungs are opacified) would provide accurate diagnostic information about whether or not the patient has pulmonary embolism. However, if the patient was pregnant, any examination associated with X-rays is generally avoided. MS Excel spreadsheets containing the analyses illustrated in the next three subsections are available at www.andywillan.com/downloads. Copyright © 2014 John Wiley & Sons, Ltd.

Statist. Med. 2014, 33 1801–1815


3.1. Current and future evidence from two-sample studies Suppose there is a diagnostic test for detecting a pulmonary embolism without the aid of X-ray that is safe for pregnant women. Assume further that the current knowledge regarding sensitivity and specificity of the test is characterized by two independent beta distributions Beta(40,10) and Beta(140,60), respectively. Assuming no prior information, this corresponds to having performed a two-sample study of 48 diseased patients, with 39 testing positive, and 198 non-diseased patients, with 139 testing negative. Among the suspected patients, assume that it is known that 19% have a pulmonary embolism (i.e., g D 0:19). There are three strategies available for pregnant women. The first strategy (test) is to perform the diagnostic test and treat with anticoagulation those that test positive. The second strategy (no treat) is not to perform the test and not treat. The third strategy (treat) is to treat all patients with anticoagulation without testing. The assumption here is that pregnant women cannot be given the gold standard because of the X-rays. The cost and effectiveness outcomes for different diagnostic outcomes are reported by Hunink et al. [47] and listed in Table VII. The net benefit for the three strategies is given in Table VIII. By inspection, one can see that, regardless of the threshold value for the willingness to pay for preventing a death, the net benefit for the no treat strategy will always exceed the net benefit for the treat strategy, and therefore,

Table VII. Effectiveness and cost by disease and treatment status for the pulmonary embolism example. Effectiveness (probability of survival)

Cost (Canadian $)

Net benefit (Canadian $)

Disease

Treated Not treated

.tp/ .fn/

0.99 0.75

3030 750

0:99 3030 0:75 750

No disease

Not treated Treated

.tn/ .fp/

1 0.90

0 3024

0:9 3024

Table VIII. Net benefit by diagnostic/treatment strategy with current evidence from a two-sample study for the pulmonary embolism example (formulae from Table I). Strategy

Net benefit

Treat

NBtreat D gNBtp C .1 g/NBfp D 0:9171 3025

No treat

NBno

Test

NBtest D gfX NBtp C .1 X/NBfn g C .1 g/fY NBtn C .1 Y /NBfp g Ctest

treat

D gNBfn C .1 g/NBtn D 0:9525 142:5

X, Y and Ctest are the sensitivity, specificity and cost of the new test, respectively.

Table IX. Data for the pulmonary embolism example with current and future evidence from two-sample studies. NBtp

etp ctp D 0:99 500;000 3030 D 491;970

NBfn

efn cfn D 0:75 500;000 750 D 374;250

NBtn

etn ctn D 1 500;000 0 D 500;000

NBfp

efp cfp D 0:9 500;000 3024 D 446;976

a1;test

g.NBtp NBfn / D 0:19.491;970 374;250/ D 22;367

a2;test

.1 g/.NBtn NBfp / D 0:81.500;000 446;976/ D 42;949

a3;test

.1 g/.NBfp NBtn / Ctest D 0:81.446;976 500;000/ 2500 D 45;449

E.INBtest /.D 0 /

a1;test E.X/ C a2;test E.Y / C a3;test D a1;test ˛˛1 C a2;test ˛˛3 C a3;test 1;2 3;4 140 D 22;367 40 50 C 42;949 200 45;449 D 2508:61

EOLpp0 .D D.0 ; v0 //

p ıp v0 =.2/ expf20 =.2v/ g 0 Œˆ.0 v0 / I f0 6 0g D 77:89


2 C a2;test

˛3 ˛3;4 2 ˛3;4 .˛3;4 C1/

2 2 2 a1;test V .X/ C a2;test V .Y / D a1;test

D 3;496;737

Statist. Med. 2014, 33 1801–1815

1809

˛1 ˛1;2 2 ˛1;2 .˛1;2 C1/

V .INBtest /.D v0 /

M. H. CHEN AND A. R. WILLAN 12,000,000

10,000,000

* 8,000,000 EVSI (n1,4 )

7, 478,742

6,000,000 * TC(n1,4 ) 3,595,000

4,000,000

2,000,000 * n1,4

0 0

500

1,000

1065 1,500

2,000

2,500

Total sample size (n1,4)

Figure 1. Expected value of sample information (EVSI) and total cost (TC) for the pulmonary embolism example with current and future evidence from case-referent studies.

no treat is the relevant strategy to compare to the test strategy. The relevant incremental net benefit is INBtest D NBtest NBno treat and can be written as a linear function of the sensitivity (X/ and specificity (Y / as INBtest D a1;test X Ca2;test Y Ca3;test , where a1;test D g.NBtp NBfn /, a2;test D .1g/.NBtn NBfp / and a3;test D .1 g/.NBfp NBtn / Ctest . If for the sake of illustration, we let the cost of the test equal $2500 and the threshold value for the willingness to pay to prevent a death equal $500,000, the values for NBij ; i 2 ft; f g; j 2 fp; ng; and aj;test ; j 2 f1; 2; 3g; are given in Table IX. The prior expected value .0 / and variance .v0 / of INBtest and the prior EOL per patient .D.0 ; v0 // are also given in Table IX. Consider a potential two-sample study where, in keeping with the prevalence of 0.19, the ratio of diseased to non-diseased patients is constrained to be 1:4 and where the fixed .Cf / and variable .Cv / costs are $400,000 and $3000, respectively. Again, for the sake of illustration, we assume that the relevant number of patients (N / is 150,000. The plot of the EVSI and TC is given in Figure 1 as a function of total sample size .n1;4 /. The ENG is maximized at n1;4 D 1065, where the number of diseased patients .n1;2 / is 213 and the number of non-diseased patients .n3;4 / is 852, and corresponds to a post-EOL per patient .EOLpp1 / of $28.03. The maximum ENG is $3,883,742 and corresponds to an EVSI of $7,478,742 and a TC of $3,595,000. Because the maximum ENG is positive, the optimal decision is to conduct a two-sample study with 213 diseased patients and 852 non-diseased patients.

3.2. Current and future evidence from one-sample studies with known prevalence

1810

Suppose the current evidence comes from a one-sample study where 39 patients were observed to be true positives, 9 were observed to be false negatives, 139 were observed to be true negatives and 59 were observed to be false positives. Assuming no prior information, the distribution of p is Dir.40; 10; 140; 60/. With known prevalence, the components of btest , the current prior .0 / and variance .v0 / of INBtest and the prior EOL per patient .EOLpp0 / are given in Table X. The plot of the EVSI and TC for a potential cross-sectional study is given in Figure 2 as a function of total sample size .n1;4 /. The ENG is maximized at n1;4 D 1245 and corresponds to a post-EOL per patient .EOLpp1 / of $30.91. The maximum ENG is $7,452,709 and corresponds to an EVSI of $11,587,709 and a TC of $4,135,000. Because the maximum ENG is positive, the optimal decision is to conduct a one-sample study with a total of 1245 patients. In this example, the prior variance of incremental net benefit is greater when the evidence comes from a one-sample study than when it comes from a two-sample study. As a result, the per-patient EOL is larger, and additional evidence has more value, leading to almost a twofold increase in the optimal ENG, albeit with a larger sample size. The optimal sample size is fairly robust in this example. The ENG, assuming prior evidence is from a one-sample study, at a sample size of 1065 (optimal when prior evidence is from a two-sample study) is $7,439,486, representing only a 0.18% reduction. Further, the ENG, assuming prior evidence is from a two-sample study, at a sample size of 1245 (optimal when prior evidence is from a one-sample study) is $3,822,449, representing only a 1.58% reduction. Copyright © 2014 John Wiley & Sons, Ltd.

Statist. Med. 2014, 33 1801–1815


Table X. Data for the pulmonary embolism example with current and future evidence from one-sample studies. b1;test

NBtp NBfp D 491;970 446;976 D 44;994

b2;test

NBfn NBfp D 374;250 446;976 D 72;726

b3;test

NBtn NBfp D 500;000 446;976 D 53;024

b4;test

NBfp NBtn C g.NBtn NBfn / Ctest D .446;976 500;000/ C 0:19.500;000 374;250/ 2500 D 31;632:5

E.INBtest /.D 0 /

b1;test E.p1 / C b2;test E.p2 / C b3;test E.p4 / C b4;test 1 2 3 = b1;t est ˛˛1;4 C b2;test ˛˛1;4 C b3;test ˛˛1;4 C b4;test 40 10 = 44;994 250 C .72;726/ 250 C 53;024 140 250 C .31;632:5/ D 2351:94

n o P3 2 b V .p / C 2 b b C.p ; p / i i;test j;test i j i D1 j >i i;test o P3 n 2 P3 2 .˛ 1 b ˛ .˛ ˛ / C 2 b b ˛ ˛ D f˛1;4 1;4 C 1/g 1;4 i i i;test j;test i j i D1 j >i i;test

P3

V .INBtest /.D v0 /

D 3;805;049 EOLpp0 .D D.0 ; v0 //

p ıp v0 =.2/ expf20 = .2v/g 0 Œˆ.0 v0 / I f0 6 0g D 108:16

16,000,000 14,000,000 * EVSI (n1,4 ) 11,587,709

12,000,000 10,000,000 8,000,000 6,000,000

* TC (n1,4 )

4,135,000

4,000,000 2,000,000 * n1,4

0 0

250

500

750

1,000

1,250

1245 1,500

1,750

2,000


Figure 2. Expected value of sample information (EVSI) and total cost (TC) for the pulmonary embolism example with current and future evidence from cross-sectional studies.

3.3. Current and future evidence from one-sample studies with unknown prevalence Consider the same prior evidence (i.e., prior p Dir.40; 10; 140; 60/), but assuming that the knowledge regarding prevalence comes solely from the 246-patient cross-sectional study, the components of btest , the prior mean .0 / and variance .v0 / of INBtest and the prior EOL per patient .EOLpp0 / are given in Table XI. The plot of the EVSI and TC for a potential cross-sectional study is given in Figure 3 as a function of total sample size .n1;4 /. The ENG is maximized at n1;4 D 1915 and corresponds to a postEOL per patient .EOLpp1 / of $43.86. The maximum ENG is $24,077,621 and corresponds to an EVSI of $30,222,621 and a TC of $6,145,000. Because the maximum ENG is positive, the optimal decision is to conduct a one-sample study with a total of 1915 patients. With unknown prevalence, the prior variance of incremental net benefit increases threefold, and the per-patient EOL is more than doubled (Tables X and XI). As a result, additional evidence has much more value, leading to a threefold increase in optimal ENG with a 54% increase in sample size. 3.4. The effect of the strength of evidence on the VOI solution


Statist. Med. 2014, 33 1801–1815

1811

The strength of the current evidence can have a profound effect on the VOI solution. In Section 3.1, we assumed that current evidence for sensitivity and specificity were characterized by two independent


Table XI. Data for the pulmonary embolism example with current and future evidence from one-sample studies with unknown prevalence. b1;test

NBtp NBfn C NBtn NBfp D 491;970 374;250 C 500;000 446;976 D 170;744

b2;test

NBtn NBfp D 500;000 446;976 D 53;024

b3;test

NBtn NBfp D 500;000 446;976 D 53;024

b4;test

NBfp NBtn Ctest D 446;976 500;000 2500 D 55;524 b1;test E.p1 / C b2;test E.p2 / C b3;test E.p4 / C b4;test = b1;test ˛˛1 C b2;test ˛˛2 C b3;test ˛˛3 C b4;test 1;4 1;4 1;4 40 10 = 170;744 250 C 53;024 250 C 53;024 140 250 C .55;524/ D 3609:44 o P3 P3 n 2 i D1 bi;test V .pi / C 2 j >i bi;test bj;test C.pi ; pj / n o P3 2 .˛ 1 P3 2 D f˛1;4 1;4 C 1/g i D1 bi;test ˛i .˛1;4 ˛i / C 2 j >i bi;test bj;test ˛i ˛j

E.INBtest /.D 0 /

V .INBtest /.D v0 /

EOLpp0 .D D.0 ; v0 //

D 11;373;407 p ıp v0 =.2/ expf20 =.2v/ g 0 Œˆ.0 v0 / I f0 6 0g D 245:35

40,000,000 35,000,000 * EVSI (n1,4 ) 30, 222,621

30,000,000 25,000,000 20,000,000 15,000,000 10,000,000 * TC (n1,4 )

6,145,000

5,000,000 * n1,4

0 0

500

1,000

1,500

2,000

1915 2,500

3,000

3,500

4,000


Figure 3. Expected value of sample information (EVSI) and total cost (TC) for the pulmonary embolism example with current and future evidence from cross-sectional studies with unknown prevalence. Table XII. Value-of-information solutions by strength of current evidence. Current knowledge

n

Expected value of sample information (n )

Total cost (n )

Current and future evidence from two-sample studies 7,478,742 3,595,000 Beta.40; 10/ and Beta.140; 60/a 1065 Beta.20; 5/ and Beta.70; 30/b 1345 31,410,213 4,435,000 Current and future evidence from one-sample studies with known prevalence 1245 11,587,709 4,135,000 Dir.40; 10; 140; 60/a Dir.20; 5; 70; 30/b 1435 40,394,869 4,705,000 Current and future evidence from one-sample studies with unknown prevalence 1915 30,222,621 6,145,000 Dir.40; 10; 140; 60/a Dir.20; 5; 70; 30/b 2015 85,942,466 6,445,000 a Base

Expected net gain (n ) 3,883,742 26,975,213 7,452,709 35,689,869 24,077,621 79,497,466

case. evidence.

b ‘Half’

1812

beta distributions Beta(40,10) and Beta(140,60), respectively. In Sections 3.2 and 3.3, we assumed the current evidence of p was characterized by Dir.40; 10; 140; 60/. In Table XII, the VOI solutions are given under the assumption that the prior distributions are based on ‘half’ as much evidence, that is, Beta.20; 5/ and Beta.70; 30/ and Dir.20; 5; 70; 30/, respectively. In comparison to the solutions given Copyright © 2014 John Wiley & Sons, Ltd.

Statist. Med. 2014, 33 1801–1815


in Sections 3.1–3.3, also provided in Table XII, the optimal sample size increases marginally, but the corresponding EVSI and ENG increase markedly. The explanation for this is that with much less prior evidence, the EOL is much higher, and therefore, additional evidence has more value. On the other hand, if we assume that the prior distributions are based on ‘twice’ as much evidence, that is, Beta.80; 20/ and Beta.280; 120/ and Dir.80; 20; 280; 120/, respectively, the optimal sample size is zero, and the current evidence would be considered sufficient for decision making; in this case, adopt the new diagnostic test.

4. Discussion A VOI approach for assessing the evidence in support of a new diagnostic test is introduced. The approach identifies situations where the current evidence is sufficient for adopting the new test and, in those situations where it is insufficient, identifies the optimal sample size for further study. The strengths of the VOI approach are that it considers the following: (i) the utility associated with the four possible diagnostic outcomes, thereby facilitating the trade-off between sensitivity and specificity; (ii) the strength and uncertainty of the current evidence; (iii) the value placed on health outcomes; (iv) the cost of additional evidence; and (v) the number of patients for whom the new test is relevant. Evidence from one-sample and two-sample studies are considered. Solutions are given for the situations where the prevalence is either known or estimated from a one-sample study. In our case, the formulations for net benefit were linear in the parameters of interest, for example, sensitivity and specificity. In situations where this is not the case, determining expressions for the mean and variance of net benefit would be more challenging and might require the use of numerical methods such as those facilitated by decision-analytic models [48]. While it is assumed that the prior evidence comes from a single study, the methods could be adapted if the prior evidence comes from multiple sources. Suppose, based on the evidence from the multiple sources, the current mean and variance for sensitivity are t and vt , respectively, then ˛1 and ˛2 can be derived from solving the equations t D ˛1 =˛1;2 and vt D ˛1 ˛2

ı˚ 2 ˛1;2 .˛1;2 C 1/

Similarly, ˛3 and ˛4 can be derived from the current mean and variance for specificity. Whether or not the prior evidence comes from a single study or multiple sources, if the prevalence is uncertain and comes from other sources, then because of the nonlinearity, decision-analytic models would need to be employed to determine the prior EOL and post-EOL [48]. Similarly, decision-analytic models would be needed if there is uncertainty regarding the net benefit of the diagnostic outcomes. This could be computationally intensive, especially for the post-EOL. Throughout the paper, it has been assumed that incremental net benefit is normally distributed. In general, this is considered a reasonable assumption if the parameters for the Beta distribution are equal to or greater than 10 and n1;2 and n3;4 are equal to or greater than 20 [49]. In small samples for which the assumption of normality is unlikely to hold, numerical methods, such as those facilitated by WinBUGS, would have to be applied to the Beta and Dirichlet distributions. The solutions provided in previous sections can be modified to allow for the existence of an existing diagnostic test, with known sensitivity and specificity, which is currently the standard of care. Because the current test is the standard of care, we assume that its net benefit exceeds that of the treat, no treat and gold strategies so that the relevant diagnostic strategy to compare test ˚ and test-gold to is the current test. The˚ net benefit of the current test is known and given by NBct D g Xct NBtp C .1 Xct /NBfn C .1 g/ Yct NBtn C .1 Yct /NBfp Cct , where Xct ; Yct and Cct are the sensitivity, specificity and cost of the current test, respectively.

Acknowledgements


Statist. Med. 2014, 33 1801–1815

1813

MHC was supported through a studentship by the Ontario Student Opportunity Trust Fund—Hospital for Sick Children Foundation Student Scholarship Program. ARW is funded through the Discovery Grant Program of the Natural Sciences and Engineering Research Council of Canada (grant number 44868-08). The authors wish to thank the reviewers and editors whose insightful comments improved the paper immeasurably.


References

1814

1. Epstein AM, Begg CB, McNeil BJ. The use of ambulatory testing in prepaid and fee-for-service group practices. New England Journal of Medicine 1986; 314(17):1089–1094. 2. Sox HC, Blatt MA, Higgins MC, Marton KI. Medical Decision Making. Butterworths-Heinemann: Boston, 1989. 3. Knottnerus A, Buntinx F. The Evidence Base of Clinical Diagnosis: Theory and Methods of Diagnostic Research. Wiley: New York, 2009. 4. Lijmer JG, Mol B, Heisterkamp S, et al. Empirical evidence of design-related bias in studies of diagnostic tests. Journal of the American Medical Association 1999; 282(11):1061–1066. 5. Reid MC, Lachs MS, Feinstein AR. Use of methodological standards in diagnostic test research: getting better but still not good. Journal of the American Medical Association 1995; 274(8):645–651. 6. Bossuyt PMM. The quality of reporting in diagnostic test research: getting better, still not optimal. Clinical Chemistry 2004; 50(3):465–466. 7. Mower WR. Evaluating bias and variability in diagnostic test reports. Annals of Emergency Medicine 1999; 33(1): 85–91. 8. Kazmierczak SC. Statistical techniques for evaluating the diagnostic utility of laboratory tests. Clinical Chemistry and Laboratory Medicine 1999; 37(11-12):1001–1009. 9. Spiegelhalter DJ. Statistical aids in clinical decision-making. Journal of the Royal Statistical Society. Series D (The Statistician) 1982; 31(1):19–36. 10. Claxton K, Posnett J. An economic approach to clinical trial design and research priority setting. Health Economics 1996; 5(6):513–524. 11. Claxton K, Thompson KM. A dynamic programming approach to the efficient design of clinical trials. Journal of Health Economics 2001; 20(5):797–822. 12. Eckermann S, Willan AR. Expected value of information and decision making in HTA. Health Economics 2007; 16(2):195–209. 13. Eckermann S, Willan AR. Time and expected value of sample information wait for no patient. Value in Health 2008; 11(3):522–526. 14. Eckermann S, Willan AR. The option value of delay in health technology assessment. Medical Decision Making 2008; 28(3):300–305. 15. Eckermann S, Willan AR. Globally optimal trial design for local decision making. Health Economics 2009; 18(2):203–216. 16. Gittins J, Pezeshk H. How large should a trial be? The Statistician 2000; 49(Part 2):177–197. 17. Gittins J, Pezeshk H. A behavioral Bayes method for determining the size of a clinical trial. Drug Information Journal 2000; 34(2):355–363. 18. Gittins J, Pezeshk H. (2002) A decision theoretic approach to sample size determination in clinical trials. Journal of Biopharmaceutical Statistics 2002; 12(4):535–551. 19. Halpern J, Brown Jr BW, Hornberger J. The sample size for a clinical trial: a Bayesian-decision theoretic approach. Statistics in Medicine 2001; 20(6):841–858. 20. Hornberger JC, Brown Jr BW, Halpern J. Designing a cost-effective clinical trial. Statistics in Medicine 1995; 14(20):2249–2259. 21. Hornberger J, Eghtesady P. The cost–benefit of a randomized trial to a health care organization. Controlled Clinical Trials 1998; 19(2):198–211. 22. Kikuchi T, Pezeshk H, Gittins J. A Bayesian cost–benefit approach to the determination of sample size in clinical trials. Statistics in Medicine 2008; 27(1):68–82. 23. Pezeshk H, Gittins J. A fully Bayesian approach to calculating sample sizes for clinical trials with binary response. Drug Information Journal 2002; 36(1):143–150. 24. Willan AR, Pinto EM. The expected value of information and optimal clinical trial design. Statistics in Medicine 2005; 24(12):1791–1806. Correction: Statistics in Medicine 2006; 25(4):720. 25. Willan AR. Clinical decision making and the expected value of information. Clinical Trials 2007; 4(3):279–285. 26. Willan AR. Optimal sample size determinations from an industry perspective based on the expected value of information. Clinical Trials 2008; 5(6):587–594. 27. Willan AR, Kowgier ME. Determining optimal sample sizes for multi-stage randomized clinical trials using value of information methods. Clinical Trials 2008; 5(4):289–300. 28. Willan AR, Eckermann S. Optimal clinical trial design using value of information methods with imperfect implementation. Health Economics 2010; 19(5):549–561. 29. Willan AR, Eckermann S. Accounting for between-study variation in incremental net benefit in value of information methodology. Health Economics 2012; 21(10):1183–1195. 30. Willan AR, Goeree R, Boutis K. Value of information methods for planning and analyzing clinical studies optimize decision making and research planning. Journal of Clinical Epidemiology 2012; 65(8):870–876. 31. Pezeshk H. Bayesian techniques for sample size determination in clinical trials: a short review. Statistical Methods in Medical Research 2003; 12(6):489–504. 32. Hilden J. The area under the ROC curve and its competitors. Medical Decision Making 1991; 11(2):95–101. 33. Hilden J, Glasziou P. Regret graphs, diagnostic uncertainty and Youden’s index. Statistics in Medicine 1998; 15(10):969–986. 34. Somoza E, Mossman D. “Biological markers” and psychiatric diagnosis: risk–benefit balancing using ROC analysis. Biological Psychiatry 1991; 29(8):811–826. 35. McFall RM, Treat TA. Quantifying the information value of clinical assessments with signal detection theory. Annual Review of Psychology 1999; 50(1):215–241.


Statist. Med. 2014, 33 1801–1815

M. H. CHEN AND A. R. WILLAN 36. Hilden J. Prevalence-free utility-respecting summary indices of diagnostic power do not exist. Statistics in Medicine 2000; 19(4):431–440. 37. Glasziou P, Hilden J. Test selection measures. Medical Decision Making 1989; 9(2):133–141. 38. Biggerstaff BJ. Comparing diagnostic tests: a simple graphic using likelihood ratios. Statistics in Medicine 2000; 19(5):649–663. 39. Metz CE. Basic principles of ROC analysis. Seminars in Nuclear Medicine 1978; 8(4):283–298. 40. Pauker SG, Kassirer JP. The threshold approach to clinical decision making. New England Journal of Medicine 1980; 302(20):1109–1117. 41. Cain KC, Greenes RA, Begg CB. Patient-oriented performance measures of diagnostic test: the U-factor. Medical Decision Making 1984; 4(1):33–46. 42. Sainfort F. Evaluation of medical technologies: a generalized ROC analysis. Medical Decision Making 1991; 11(3):208–221. 43. Djulbegovic B, Hozo I, Schwartz A, McMasters KM. Acceptable regret in medical decision making. Medical Hypotheses 1999; 53(3):253–259. 44. Tsalatsanis A, Hozo I, Vickers A, Djulbegovic BA. Regret theory approach to decision curve analysis: a novel method for eliciting decision makers’ preferences and decision-making. BMC Medical Informatics and Decision Making 2010; 10:51. 45. Hozo I, Djulbegovic B. When is diagnostic testing inappropriate or irrational? Acceptable regret approach. Medical Decision Making 2008; 28(4):540–553. 46. Hilden J, Glasziou P, Habbema JDF. A pitfall in utility assessment—patients’ undisclosed investment decisions. Medical Decision Making 1992; 12(1):39–43. 47. Hunink M, Glasziou P, Siegel J, Weeks J, Pliskin J, Elstein A, Weinstein M. Decision Making in Health and Medicine: Integrating Evidence and Values. Cambridge University Press: Cambridge, 2001. 48. Ades AE, Lu G, Claxton K. Expected value of sample information calculations in medical decision modeling. Medical Decision Making 2004; 24(2):207–227. 49. Box GEP, Hunter WG, Hunter SJ. Statistics for Experimenters. John Wiley & Sons: New York, 1978, p.130.

1815


Statist. Med. 2014, 33 1801–1815

A new condition for assessing the clinical efficiency of a diagnostic test.

Diagnostic value of a chemistry test profile.

Diagnostic value of secretin provocation test.

Determining the value of a periodontal diagnostic test.

A new practical diagnostic test for Creutzfeldt-Jakob disease.

The patient's role in assessing the value of diagnostic tests.

Assessing the value of cardiac biomarkers: going beyond diagnostic accuracy?

Limitations of diagnostic value of the sweat test.

New methods of imaging in diagnostic radiology.

The Hug-up Test: A New, Sensitive Diagnostic Test for Supraspinatus Tears.

Diagnostic value of free erythrocyte porphyrins and blood lead as a screening test for lead exposure.

Diagnostic methods for assessing maxillary skeletal and dental transverse deficiencies: A systematic review.

Acute myeloblastic leukaemia--a model for assessing value for money for new treatment programmes.

New diagnostic methods for laser plasma- and microwave-enhanced combustion.

Leukotriene D4 nasal provocation test: Rationale, methodology and diagnostic value.

A philosophy of methods development: The assimilation of new methods and information into aquatic microbial ecology.

Non-IgE-related diagnostic methods (LST, patch test).

The Script Concordance Test in anesthesiology: Validation of a new tool for assessing clinical reasoning.

Assessing interprofessional teamwork: pilot test of a new assessment module for practicing physicians.

Diagnostic test for dementias.

A new way of quantifying diagnostic information from multilead electrocardiogram for cardiac disease classification.

A Novel "Maximizing Kappa" Approach for Assessing the Ability of a Diagnostic Marker and Its Optimal Cutoff Value.

A Bayesian framework for estimating the incremental value of a diagnostic test in the absence of a gold standard.

A new integrated tool for assessing and monitoring test comparability and stability.