Intern Emerg Med DOI 10.1007/s11739-015-1230-8
CE-METHODOLOGICAL NOTES
When should we change our clinical practice based on the results of a clinical study? The hierarchy of evidence Giorgio Costantino1 • Nicola Montano1,2 • Giovanni Casazza2
Received: 11 March 2015 / Accepted: 23 March 2015 Ó SIMI 2015
Introduction In the first paper of this series [1], we described how to perform a bibliographic search aimed at retrieving the available evidence. Once we have collected the relevant literature, and before deciding if it could be helpful in our clinical practice, we have to judge its quality. For this purpose, a framework for ranking the quality of evidence can be of help in classifying clinical studies [2]. The hierarchy of evidence is usually represented with a pyramid: studies are ranked according to the quality of the evidence they provide: from those with the lowest (the base of the pyramid) to those with the highest level (the apex of the pyramid) (Fig. 1). To a first approximation, this schematic representation can be considered valid for all the types of studies. Nevertheless, in this brief note we will consider three different fields of clinical research (i.e. intervention, diagnosis and prognosis) and we will show that each field has its own hierarchy of evidence that is slightly different from the hierarchy of the others.
Intervention The efficacy of an intervention can be assessed by comparing an outcome in two groups of patients: patients receiving the intervention under evaluation, and patients
receiving placebo or another intervention (comparator). This aim can be achieved with different study designs. At a simple level, an observational study can be conducted by selecting a group of patients already receiving the new intervention and a group of patients already receiving the comparator. As an alternative, it is possible to prospectively enrol consecutive patients and let the clinical investigator choose which patients will receive the experimental intervention and which one the comparator. However, both these types of studies are at risk of providing biased results due to both intrinsic (i.e. voluntary selection by the investigator) or extrinsic (i.e. different patients characteristics between the two groups) confounding factors. To avoid such biases, a randomized clinical trial (RCT, where patients are randomly assigned to the intervention or to the comparator) can be used [3]. RCTs are generally considered the ‘‘gold standard’’ in clinical research. These studies are characterized by a high degree of internal validity (reduction of the effect of confounding factors and resulting low risk of bias), but the corresponding drawback is that their external validity (i.e. the generalizability of the findings of the study to patients other than those included in the trial) might be limited, due to the strict inclusion criteria adopted. However, the positive aspects of RCTs usually overcome their potential limitations. For this reason, the results of a well-designed and well-conducted RCT can contribute to change our clinical practice.
& Giovanni Casazza [email protected] 1
Fondazione IRCCS Ca’ Granda Ospedale Maggiore Policlinico di Milano, Milan, Italy
2
Dipartimento di Scienze Biomediche e Cliniche ‘‘L. Sacco’’, Universita` degli Studi di Milano, Via G.B. Grassi, 74, 20157 Milan, Italy
Diagnosis The clinical usefulness of a new diagnostic test is initially evaluated by assessing the ability of the test to distinguish people with a particular disease from people without that
123
Intern Emerg Med
disease. For this purpose, a case–control study can be used, although it cannot provide the diagnostic accuracy of the test. If subsequently we would like to assess the diagnostic accuracy of the new test (i.e. sensitivity and specificity), a cross-sectional study will be the optimal choice. In a crosssectional design, all patients suspected of having the disease will at the same time undergo the new test (index test) and the test considered as the reference standard.
Systemac reviews Randomized Clinical Trials Cohort studies Case-control studies Cross seconal surveys Case reports – Case series
Fig. 1 The pyramid of evidence
Fig. 2 The three-leaf clover of evidence
Thus, in the diagnostic field, the cross-sectional design has to be considered the ‘‘gold standard’’, and such a study can modify our practice. Finally, a diagnostic procedure could be considered as an intervention, and its clinical usefulness should be assessed through an RCT (for instance, patients admitted to Emergency Department for dyspnoea are randomized to undergo or not to BNP testing, and then survival is evaluated). Nevertheless, such studies are infrequent, and most of the time we are satisfied just to know the capacity of a test to diagnose a disease [4].
Prognosis There are different types of studies with different goals related to prognosis. Some of them are aimed at the identification of prognostic factors for a particular condition, whilst others are aimed at the construction of prognostic scores or clinical prediction rules. Prognostic studies can be performed using retrospective data provided by clinical databases. However, most of the typical biases of observational research may affect them (see above). To reduce the biases, prospective observational studies, based on the selection of a group of patients followed for INTERVENTION STUDIES
Systematic reviews of RCTs RCT Comparison studies without randomizaon Observaonal studies
Case reports Case series
123
Intern Emerg Med
an appropriate period of time, are the optimal choice. A first important distinction is between derivation and validation studies. Initially, the weighted variables independently associated with an outcome are merged to derive a prognostic score or a clinical prediction rule. Once a prognostic score has been derived, its clinical usefulness must be confirmed in a subsequent validation study. Such studies (internal validation, performed in the same centre, but including different patients; external validation, performed by different researchers in different centres) are needed to increase the validity of a score. In this context, derivation studies can only be considered as the preliminary step, while a well-designed and well-conducted prospectively designed external validation study can be considered the ‘‘gold standard’’ in prognostic research [5]. Only the results of an external validation study of a prognostic score (or clinical decision rule) should change our clinical practice.
Systematic reviews Systematic reviews with meta-analyses of methodologically correct clinical studies (in all the three types: interventions, diagnosis and prognosis) are considered to provide the highest level of evidence. A systematic review with meta-analysis of available ‘‘gold standard’’ studies (RCTs for intervention, crosssectional for diagnosis, prospective external validation cohorts for prognosis) provides the strongest evidence.
The three-leaf clover of evidence In conclusion, the hierarchy of evidence is not unique for all types of clinical questions. The same study design can be placed in a different position within the pyramid, depending on the type of clinical question (diagnostic, prognostic, intervention). For these reasons, the use of the pyramid to rate the evidence might be, in some circumstances, misleading and it would be more appropriate to talk about a three-leaf clover (Fig. 2).
The lowest level of evidence for the three areas of clinical research herein considered is provided by case reports and case series, which provide only very preliminary evidence on the efficacy of an intervention (or on the diagnostic or prognostic value of the factors under study). At the next levels, for each of the three areas there are three separate hierarchies. Finally, the highest level of evidence in all three areas is provided by systematic reviews with meta-analyses. When we read a high-quality systematic review, based upon high-quality studies, without significant heterogeneity, we can be confident that changing our clinical practice according to its results will bring more benefit than harm. Conflict of interest of interest.
The Authors declare that they have no conflict
Statement of human and animal rights All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards. This article does not contain any studies with human and animals performed by any of the authors.
References 1. Costantino G, Montano N, Casazza G (2015) When should we change our clinical practice based on the results of a clinical study? Searching for evidence: PICOS and PubMed. Intern Emerg Med. doi:10.1007/s11739-015-1225-5 2. Guyatt GH, Sackett DL, Sinclair JC, Hayward R, Cook DJ, Cook RJ (1995) Users’ guides to the medical literature. IX. A method for grading health care recommendations. JAMA 274:1800–1804 3. Pocock S (1985) Current issues in the design and interpretation of clinical trials. BMJ 290:39–42 4. Colli A, Fraquelli M, Casazza G, Conte D, Nikolova D, Duca P, Thorlund K, Gluud C (2014) The architecture of diagnostic research: from bench to bedside—research guidelines using liver stiffness as an example. Hepatology 60(1):408–418. doi:10.1002/ hep.26948 5. Hemingway H, Croft P, Perel P, Hayden JA, Abrams K, Timmis A, Briggs A, Udumyan R, Moons KG, Steyerberg EW, Roberts I, Schroter S, Altman DG, Riley RD, PROGRESS Group (2013) Prognosis research strategy (PROGRESS) 1: a framework for researching clinical outcomes. BMJ 346:e5595. doi:10.1136/bmj. e5595
123
CE-METHODOLOGICAL NOTES
When should we change our clinical practice based on the results of a clinical study? The hierarchy of evidence Giorgio Costantino1 • Nicola Montano1,2 • Giovanni Casazza2
Received: 11 March 2015 / Accepted: 23 March 2015 Ó SIMI 2015
Introduction In the first paper of this series [1], we described how to perform a bibliographic search aimed at retrieving the available evidence. Once we have collected the relevant literature, and before deciding if it could be helpful in our clinical practice, we have to judge its quality. For this purpose, a framework for ranking the quality of evidence can be of help in classifying clinical studies [2]. The hierarchy of evidence is usually represented with a pyramid: studies are ranked according to the quality of the evidence they provide: from those with the lowest (the base of the pyramid) to those with the highest level (the apex of the pyramid) (Fig. 1). To a first approximation, this schematic representation can be considered valid for all the types of studies. Nevertheless, in this brief note we will consider three different fields of clinical research (i.e. intervention, diagnosis and prognosis) and we will show that each field has its own hierarchy of evidence that is slightly different from the hierarchy of the others.
Intervention The efficacy of an intervention can be assessed by comparing an outcome in two groups of patients: patients receiving the intervention under evaluation, and patients
receiving placebo or another intervention (comparator). This aim can be achieved with different study designs. At a simple level, an observational study can be conducted by selecting a group of patients already receiving the new intervention and a group of patients already receiving the comparator. As an alternative, it is possible to prospectively enrol consecutive patients and let the clinical investigator choose which patients will receive the experimental intervention and which one the comparator. However, both these types of studies are at risk of providing biased results due to both intrinsic (i.e. voluntary selection by the investigator) or extrinsic (i.e. different patients characteristics between the two groups) confounding factors. To avoid such biases, a randomized clinical trial (RCT, where patients are randomly assigned to the intervention or to the comparator) can be used [3]. RCTs are generally considered the ‘‘gold standard’’ in clinical research. These studies are characterized by a high degree of internal validity (reduction of the effect of confounding factors and resulting low risk of bias), but the corresponding drawback is that their external validity (i.e. the generalizability of the findings of the study to patients other than those included in the trial) might be limited, due to the strict inclusion criteria adopted. However, the positive aspects of RCTs usually overcome their potential limitations. For this reason, the results of a well-designed and well-conducted RCT can contribute to change our clinical practice.
& Giovanni Casazza [email protected] 1
Fondazione IRCCS Ca’ Granda Ospedale Maggiore Policlinico di Milano, Milan, Italy
2
Dipartimento di Scienze Biomediche e Cliniche ‘‘L. Sacco’’, Universita` degli Studi di Milano, Via G.B. Grassi, 74, 20157 Milan, Italy
Diagnosis The clinical usefulness of a new diagnostic test is initially evaluated by assessing the ability of the test to distinguish people with a particular disease from people without that
123
Intern Emerg Med
disease. For this purpose, a case–control study can be used, although it cannot provide the diagnostic accuracy of the test. If subsequently we would like to assess the diagnostic accuracy of the new test (i.e. sensitivity and specificity), a cross-sectional study will be the optimal choice. In a crosssectional design, all patients suspected of having the disease will at the same time undergo the new test (index test) and the test considered as the reference standard.
Systemac reviews Randomized Clinical Trials Cohort studies Case-control studies Cross seconal surveys Case reports – Case series
Fig. 1 The pyramid of evidence
Fig. 2 The three-leaf clover of evidence
Thus, in the diagnostic field, the cross-sectional design has to be considered the ‘‘gold standard’’, and such a study can modify our practice. Finally, a diagnostic procedure could be considered as an intervention, and its clinical usefulness should be assessed through an RCT (for instance, patients admitted to Emergency Department for dyspnoea are randomized to undergo or not to BNP testing, and then survival is evaluated). Nevertheless, such studies are infrequent, and most of the time we are satisfied just to know the capacity of a test to diagnose a disease [4].
Prognosis There are different types of studies with different goals related to prognosis. Some of them are aimed at the identification of prognostic factors for a particular condition, whilst others are aimed at the construction of prognostic scores or clinical prediction rules. Prognostic studies can be performed using retrospective data provided by clinical databases. However, most of the typical biases of observational research may affect them (see above). To reduce the biases, prospective observational studies, based on the selection of a group of patients followed for INTERVENTION STUDIES
Systematic reviews of RCTs RCT Comparison studies without randomizaon Observaonal studies
Case reports Case series
123
Intern Emerg Med
an appropriate period of time, are the optimal choice. A first important distinction is between derivation and validation studies. Initially, the weighted variables independently associated with an outcome are merged to derive a prognostic score or a clinical prediction rule. Once a prognostic score has been derived, its clinical usefulness must be confirmed in a subsequent validation study. Such studies (internal validation, performed in the same centre, but including different patients; external validation, performed by different researchers in different centres) are needed to increase the validity of a score. In this context, derivation studies can only be considered as the preliminary step, while a well-designed and well-conducted prospectively designed external validation study can be considered the ‘‘gold standard’’ in prognostic research [5]. Only the results of an external validation study of a prognostic score (or clinical decision rule) should change our clinical practice.
Systematic reviews Systematic reviews with meta-analyses of methodologically correct clinical studies (in all the three types: interventions, diagnosis and prognosis) are considered to provide the highest level of evidence. A systematic review with meta-analysis of available ‘‘gold standard’’ studies (RCTs for intervention, crosssectional for diagnosis, prospective external validation cohorts for prognosis) provides the strongest evidence.
The three-leaf clover of evidence In conclusion, the hierarchy of evidence is not unique for all types of clinical questions. The same study design can be placed in a different position within the pyramid, depending on the type of clinical question (diagnostic, prognostic, intervention). For these reasons, the use of the pyramid to rate the evidence might be, in some circumstances, misleading and it would be more appropriate to talk about a three-leaf clover (Fig. 2).
The lowest level of evidence for the three areas of clinical research herein considered is provided by case reports and case series, which provide only very preliminary evidence on the efficacy of an intervention (or on the diagnostic or prognostic value of the factors under study). At the next levels, for each of the three areas there are three separate hierarchies. Finally, the highest level of evidence in all three areas is provided by systematic reviews with meta-analyses. When we read a high-quality systematic review, based upon high-quality studies, without significant heterogeneity, we can be confident that changing our clinical practice according to its results will bring more benefit than harm. Conflict of interest of interest.
The Authors declare that they have no conflict
Statement of human and animal rights All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards. This article does not contain any studies with human and animals performed by any of the authors.
References 1. Costantino G, Montano N, Casazza G (2015) When should we change our clinical practice based on the results of a clinical study? Searching for evidence: PICOS and PubMed. Intern Emerg Med. doi:10.1007/s11739-015-1225-5 2. Guyatt GH, Sackett DL, Sinclair JC, Hayward R, Cook DJ, Cook RJ (1995) Users’ guides to the medical literature. IX. A method for grading health care recommendations. JAMA 274:1800–1804 3. Pocock S (1985) Current issues in the design and interpretation of clinical trials. BMJ 290:39–42 4. Colli A, Fraquelli M, Casazza G, Conte D, Nikolova D, Duca P, Thorlund K, Gluud C (2014) The architecture of diagnostic research: from bench to bedside—research guidelines using liver stiffness as an example. Hepatology 60(1):408–418. doi:10.1002/ hep.26948 5. Hemingway H, Croft P, Perel P, Hayden JA, Abrams K, Timmis A, Briggs A, Udumyan R, Moons KG, Steyerberg EW, Roberts I, Schroter S, Altman DG, Riley RD, PROGRESS Group (2013) Prognosis research strategy (PROGRESS) 1: a framework for researching clinical outcomes. BMJ 346:e5595. doi:10.1136/bmj. e5595
123
