A critical appraisal of the systematic review process: Systematic reviews of zirconia single crowns Lisa A. Lang, DDS, MS, MBAa and Sorin T. Teich, DMD, MBAb Case Western Reserve University School of Dental Medicine, Cleveland, Ohio Statement of problem. Systematic reviews analyze the data of published research in an effort to assemble the scientific evidence to help clinicians apply evidence-based information in decision making. The quality of systematic reviews varies greatly. Purpose. This study critically appraised the current systematic review process by evaluating systematic reviews that pertain to zirconia-based single crowns. Materials and methods. The following PICO (patients, intervention, comparison, outcome) question was formulated: “In adults, how does the long-term prognosis of zirconia-based single crowns compare with conventional single crowns on natural teeth?” An electronic search was performed in PubMed and the Cochran Library for articles published in English between 1950 and October 2012. Additional manual searches were completed. To be included in the analysis, the study must have been a systematic review, published in an English-speaking peer-reviewed journal, and evaluated zirconia crowns on teeth. Two examiners qualitatively evaluated the publications with an Assessment of Multiple Systematic Reviews checklist and the Oxford Systematic Review Appraisal form. Results. Three systematic reviews were identified that met the search criteria. Two studies met 5 of the 11 Assessment of Multiple Systematic Reviews criteria, whereas the third met only 1 criterion. The same 2 studies met 3 of the 5 Oxford Systematic Review Appraisal criteria and the third met only 2 criteria. Conclusion. Because of the variation in methodologies, systematic reviews should be interpreted cautiously. The Assessment of Multiple Systematic Reviews Checklist and the Oxford Systematic Review Appraisal Sheet are practical tools for appraising and determining the quality of systematic reviews. (J Prosthet Dent 2014;-:---)

Clinical Implications Clinicians should critically appraise the quality of a systematic review before accepting its recommendations. During the 1990s, a process emerged in medicine of critically reviewing the scientific literature for evidence. A hierarchy of evidence was established to review the literature for the best available evidence. This pyramid of evidence developed by Sackett et al1 characterized the strength of the evidence, with systematic reviews that analyzed the data of published articles in a metaanalysis being at the apex of the pyramid2 and representing the strongest level. The hierarchy of evidence pyramid a

developed by Sackett et al1,2 was further refined by other researchers.3,4 Glover et al,3 differentiates between “filtered” and “unfiltered” information. Filtered information generates 3 categories of studies; systematic reviews, critically appraised topics, and critically appraised individual articles. Systematic reviews gather data from the literature, preferably randomized clinical trials (RCT), and, when possible, perform statistical analyses such as a metaanalysis. Critically appraised topics are

reviews in which the best available evidence, oftentimes the investigations included are clinical studies other than RCT, is evaluated. This information is synthesized, and recommendations are made based on the best available evidence. Critically appraised individual articles are publications in which the article is merely a synopsis of the literature. These publication types have been described in the hierarchy of evidence from highest to lowest.

Associate Professor, Chair, Department of Comprehensive Care; Assistant Dean of Clinical Education. Associate Professor, Department of Comprehensive Care; Assistant Dean of Clinical Operations.

b

Lang and Teich

2

Volume The next level of evidence is “unfiltered information.”3 RCTs, cohort studies, case-control studies, or case series or reports, and background information and/or expert opinion are all considered unfiltered information. Within this category, RCTs are regarded as the highest level of evidence, whereas expert opinion is deemed to be the lowest level. Cohort studies, casecontrol studies, and case reports are listed in their respective hierarchy of evidence. In vitro studies are considered to be a lower level of evidence than in vivo studies and serve as background information along with expert opinions. Rosner4 reexamined the principles of evidence-based medicine and cited the exclusion of numerous sources of research information (basic research, epidemiology, and health services research) as problematic. He further refined the hierarchy pyramid to include animal research and in vitro studies. In previous pyramids, the lowest level was labeled “background information” and “expert opinion.” Although background information may have been interpreted to include animal research and in vitro studies, this new hierarchy pyramid clearly defined animal research followed by in vitro research to be the lower level of the pyramid. In 1990, the leaders of the prosthodontic specialty convened at the Mayo Clinic in Minnesota to assess the current status of prosthodontics as a discipline and specialty of dentistry with regard to clinical practice, education, and research. The proceedings were published as an issue in The Journal of Prosthetic Dentistry. Laney5 authored the summary section and identified areas that are still pertinent to dental education today. With regard to dental education, recommendations were made to improve the integration and transfer of knowledge and technology. It was recognized that “properly dividing didactic and clinical instruction time between new and established treatment procedure would be one of the greatest challenges for dental educators.”5 With regard to research, concern was raised regarding the need for and capabilities

to carefully assess new treatment benefits for the patient. Specifically, recommendations were made that the clinician should review data on which manufacturers based their marketing and scrutinize whether products had undergone adequate testing to ensure successful patient outcomes.5 The consensus was that fundamental improvements were needed in the quality of prosthodontic literature. Therefore, a national prosthodontic research training symposium was recommended to upgrade research orientation and the capabilities of faculty, clinicians, and students.5,6 As a result, The Journal of Prosthetic Dentistry published a series of articles on the critical appraisal of the literature with regard to evidence-based dentistry (EBD) written by 10 prosthodontists who had completed training in the evidence-based process.6 EBD is defined by the American Dental Association “as an approach to oral health care that requires the judicious integration of systematic assessments of clinically relevant scientific evidence relating to the patient’s oral and medical conditions and history with the dentist’s clinical expertise and the patient’s treatment needs and preferences.”7 This is a 3-prong approach: critical review of the literature, the clinician’s expertise, and the patient’s needs and values. Organizations such as the Oxford Center for Evidence-based Medicine8 and US Department of Health and Human Services Agency for Health Care Policy and Research9,10 developed systems to further classify the levels of evidence. These classification systems take into account the limited number of available clinical research studies that may be systematically reviewed in certain cases to provide a strength index of the studies included in the review. In the practice of EBD, both the level of the evidence and the scientific validity of the review article must be examined. Systematic reviews vary greatly in the way that they are conducted and how they report research. In an effort to develop high-quality, reliable systematic

The Journal of Prosthetic Dentistry

-

Issue

-

reviews, the Cochran Collaboration publishes guidelines and protocols; proposed protocols in various fields are published in the Cochran Library before the initiation of the review.11 However, even with published criteria of how to plan, organize, and evaluate the literature, these variations still exist. These differences resulted in a varied quality of systematic reviews, which potentially influenced the article’s final recommendations. In 2007, Shea et al12 reported the development of an assessment tool for the quality of systematic reviews. The researchers presented a new streamlined instrument to assess the methodologic quality of systematic reviews through an 11-item questionnaire named the Assessment of Multiple Systematic Reviews (AMSTAR). The instrument was validated for agreement reliability, construct validity, and feasibility.13 The Centre for Evidence-based Medicine at Oxford University pursued the same direction and developed a Web site with tools to aid clinicians in the assessment of medical literature. Their systematic review critical appraisal sheet14 is formatted in a clear concise manner that allows the reader to evaluate the publication methodically to determine its quality. The principles of EBD have spread to dental education and are now part of the Commission on Dental Accreditation (CODA) standards for dental education15 in the United States. Several standards focus on critical thinking, integration of knowledge, and evidencebased practice. Some of the pertinent CODA standards related to these areas are depicted in Table I. As preparation for the integration of an evidence-based protocol into the clinical curriculum at the Case Western Reserve University School of Dental Medicine, a search for systematic reviews that pertain to ceramic zirconia-based single crowns (ZBSC) was undertaken by 2 faculty members to validate the AMSTAR and the Oxford critical appraisal tools. This study critically appraised the current systematic review process by evaluating systematic reviews that pertain to ZBSCs.

Lang and Teich

-

2014

3

Commission on Dental Accreditation Standards related to evidence-based dentistry

Table I.

Standard No.

Description

Intent

2-9

Graduates must be competent in the use of critical thinking and problem solving, including their use in the comprehensive care of patients, scientific inquiry, and research methodology.

2-14

Graduates must be competent in the application of biomedical science knowledge in the delivery of patient care.

2-21

Graduates must be competent to access, critically appraise, apply, and communicate scientific, and lay literature as it relates to providing evidence-based care.

5-2

Patient care must be evidenced based, integrating the best research evidence and patient values.

MATERIAL AND METHODS An electronic search was performed in PubMed and the Cochran Library by using Boolean operators for articles published in English between 1950 and October 2012. The search was designed to answer the PICO (patients, intervention, comparison, outcome) question, “In adults, how does the long-term prognosis of zirconia-based single crowns compare with conventional single crowns on natural teeth?” The following key words were used for the search: zirconia, all ceramic, crown(s), fixed dental prosthesis, and restorations.

Table II.

Demonstration of critical appraisal of scientific evidence in combination with clinical application and patient factors.

Apply advances in modern biology to clinical practice and to integrate new medical knowledge and therapies relevant to oral health care.

The dental school should use evidence to evaluate new technology and products and to guide diagnosis and treatment decisions.

The various searches revealed 453 titles, which were reviewed for duplication. The electronic search was supplemented by searches for selected articles obtained from the references of the articles produced by the electronic search and manual searches of articles published in 2012 in The Journal of Prosthetic Dentistry, the Journal of Prosthodontics, and the International Journal of Prosthodontics. The titles and abstracts of all the articles were reviewed for possible inclusion. The inclusion and exclusion criteria are cited in Table II. To be included in the analysis, the study must have been a systematic review, published in an English speaking

Inclusion and exclusion criteria

Inclusion Systematic review Study published in peer-reviewed journal

Exclusion Implant-supported crowns Multiunit fixed dental prosthesis

Study published in English

Studies that did not provide data that could be reviewed

Single-unit crowns on teeth

Cohort studies without clinical follow-up

Lang and Teich

peer-reviewed journal, and evaluated zirconia crowns on teeth. The initial determination of a study as being a systematic review was done with the “Clinical Queries” tool in PubMed,16 which separates search results indexed as systematic reviews. Implant-supported crowns, multiunit fixed dental prostheses, studies that did not provide data that could be reviewed and evaluated, and studies that were not systematic reviews were excluded. Studies that included crowns on natural teeth and implant crowns could be used if the article clearly separated the data. In these instances, the data from crowns on natural teeth were included and the implant crown data were excluded. Upon identification for inclusion, the full text of each article was reviewed. To compare the quality of the identified systematic reviews, 2 examiners (LAL and STT) evaluated the publications by using the AMSTAR12,13 and Oxford14 Systematic Review checklists. In addition, the protocol, established by the Cochrane Collaborations17 for the systematic review of metal-free restorations compared with metal-ceramic or conventional restorations was analyzed to determine if it followed the recommendations established by AMSTAR.

RESULTS After inclusion and exclusion criteria had been applied to the search results, 3 systematic reviews18-20 were included (Table III). Wang et al18 and Al-Amleh et al20 clearly defined a question to be addressed in their search, whereas Triwatana et al19 did not formulate a research question. All 3 systematic reviews18-20 established inclusion criteria for the conduct of their reviews. Wang et al18 had 2 independent data extractors, identified a consensus procedure for disagreements, and used multiple database resources to identify publications. Triwatana et al19 only searched PubMed to identify publications. AlAmleh et al20 mentioned searching Medline and PubMed only. None of the authors identified MeSH terms as part of their search strategy.

4

Volume

Table III. Study Wang et al18

Journal Int J Prosthodont

Year of Publication Type

Zirconia Included Studies Related

Randomized clinical All trials, prospective ceramic cohort studies, crowns retrospective studies with follow-up

2012

Lava Zirconia and Procera Zirconia

Sample Size

-

J Adv Prosthodont

2012

Zirconia crowns and FDP

RCTs, longitudinal prospective and retrospective cohort studies

216 Procera Zirconiay

IPS e.max ZirCAD Cercon and InCeram Procera

Ortorp et al21 (retrospective with clinical follow-up) J Oral Rehabil

2010

RCTs, prospective cohort studies, case series

Zirconia crowns and FDP

Cehreli et al24 (RCT) Ortorp et al21 (retrospective with clinical follow-up)

Cercon Zirconia and Procera Zirconia

Success Rate

39.2

94%

36

NA (0% core fracture, (2% veneering porcelain fracture)

CSR (%)

50 IPS e.max ZirCAD

36

100%

15 Cercon, 15 InCeram

24

93%

168 Procera Zirconiay

36

Total, 284

Beuer et al23 (prospective longitudinal cohort study) Cehreli et al24 (RCT)

Follow-up (mo)

Total, 233

17 Lava Zirconia

Ortorp et al21 (retrospective with clinical follow-up)

Al-Amleh et al20

Issue

Systematic review data

Schmitt et al22 (prospective clinical trial)

Triwatana et al19

-

100

92.7

Total, 219

15 Cercon, 15 InCeram

24

93%

204 Procera Zirconiay

36

94%

93

RCT, randomized clinical trial; FDP, fixed dental prosthesis.

The studies21-24 included in each of the systematic reviews are listed in Table III. A study by Wang et al18 included 2 studies21,22 that reported data on ZBSCs. For their analysis, the researchers pooled data from studies that investigated various types of ceramic crowns, but, for the purpose of this report, the data from the zirconia crowns only is included in Table III. Triwatana

et al19 and Al-Amleh et al20 investigated single crowns and FPDs.21-24 They reported the data separately, so their study data on single crowns could be included in this evaluation. These studies reported up to 3-year success rates that ranged from 93% to 100%. Because of the heterogeneity of studies included in the reviews, no meta-analyses were performed in any of the systematic reviews.

The Journal of Prosthetic Dentistry

The AMSTAR evaluations for the 3 systematic reviews are presented in Table IV. If either of the 2 evaluators disagreed on any given criterion, then the evaluation was discussed until a consensus was reached. The systematic reviews were then rank ordered, based on the number of affirmative answers given each publication. Wang et al18 and Al-Amleh et al20 received 5

Lang and Teich

-

2014

Table IV.

5 Assessment of systematic reviews by using Assessment of Multiple Systematic Reviews (AMSTAR) checklista,b

Answer No. (%) Wang et al18

Triwatana et al19

1. Was an “a priori” design provided? The research question and inclusion criteria should be established before the conduct of the review. (Note. Need to refer to a protocol, ethics approval, or predetermined and/or a priori published research objectives to score a “yes.”)

Y

N

Y

2. Was there duplicate study selection and data extraction? There should be at least 2 independent data extractors and a consensus procedure for disagreements should be in place. (Note. Two people do study selection, 2 people do data extraction, consensus process or 1 person checks the other’s work.)

Y

N

N

3. Was a comprehensive literature search performed? At least 2 electronic sources should be searched; the report must include years and databases used (eg, PubMed Central, EMBASE, and MEDLINE); key words, MeSH or both must be stated, and, where feasible, the search strategy should be provided. (Note. If at least 2 sources þ one supplementary strategy used, select “yes” [Cochrane register/Central counts as 2 sources; a gray literature search counts as supplementary]).

Y

N

Y

4. Was the status of publication (such as gray literature) used as an inclusion criterion? The researchers should state that they searched for reports regardless of their publication type; the researchers should state whether they excluded any reports from the systematic review on the basis of their publication status, language, or other factors. (Note. If review indicates that there was a search for “gray literature” or “unpublished literature,” indicate “yes.” Single database, dissertations, conference proceedings, and trial registries are all considered gray for this purpose; if searching a source that contains both gray and nongray, must specify that they were searching for gray or unpublished literature.)

N

N

N

5. Was a list of studies (included and excluded) provided? A list of included and excluded studies should be provided. (Note. Acceptable if the excluded studies are referenced; if there is an electronic link to the list but the link is dead, select “no.”)c

Y

N

Y

6. Were the characteristics of the included studies provided? In an aggregated form such as a table, data from the original studies should be provided on the participants, interventions, and outcomes. (Note. Acceptable if not in table format as long as they are described as above.)

Y

Y

Y

7. Was the scientific quality of the included studies assessed and documented? “A priori” methods of assessment should be provided (eg, for effectiveness studies if the researcher(s) chose to include only randomized, double-blind, placebo-controlled studies, or allocation concealment as inclusion criteria); for other types of studies alternative items will be relevant. (Note. Can include use of a quality scoring tool or checklist, eg, Jadad scale, risk of bias, sensitivity analysis; or a description of quality items, with some kind of result for each study [“low” or “high” is fine, as long as it is clear which studies scored “low” and which scored “high”; a summary score and/or range for all studies is not acceptable]).

N

N

N

8. Was the scientific quality of the included studies used appropriately in formulating conclusions? The results of the methodologic rigor and scientific quality should be considered in the analysis and conclusions of the review, and explicitly stated in formulating recommendations. (Note. Might say something such as “the results should be interpreted with caution due to poor quality of included studies”; cannot score “yes” for this question if scored “no” for question 7.)

N

N

N

9. Were the methods used to combine the findings of studies appropriate? For the pooled results, a test should be done to ensure the studies were combinable, to assess their homogeneity (that is, c2 test for homogeneity). (Note. Indicate “yes” if they mention or describe heterogeneity, ie, if they explain that they cannot pool because of heterogeneity or variability between interventions.)

N

NA

NA

Criteria

Lang and Teich

Al-Amleh et al20

6

Volume

Table IV.

-

Issue

-

(Continued) Assessment of systematic reviews by using Assessment of Multiple Systematic Reviews (AMSTAR)

checklista,b

Answer No. (%) Wang et al18

Triwatana et al19

10. Was the likelihood of publication bias assessed? An assessment of publication bias should include a combination of graphic aids (eg, funnel plot, other available tests), statistical tests (eg, Egger regression test) or both. (Note. If no test values or funnel plot included, score “no”; score “yes” if mentions that publication bias could not be assessed because there were fewer than 10 included studies.)

N

N

Y

11. Was the conflict of interest stated? Potential sources of support should be clearly acknowledged in both the systematic review and the included studies. Note. To get a “yes,” must indicate source of funding or support for the systematic review and for each of the included studies.)

N

N

N

Criteria

Al-Amleh et al20

Y, Yes; N, No; CA, Can not Answer. a Notes from http://amstar.ca/Amstar_Checklist.php. b From Ref. 12. c Question 5. It is not practical to provide list of excluded articles, therefore, yes was assigned if review provided detailed list of included articles and mentions overall number of articles found during search.

affirmative answers each, whereas Triwatana et al19 received only 1 affirmative answer. The results of the evaluation done with the Oxford Systematic Review Appraisal Sheet for each of the 3 systematic reviews are presented in Table V. Wang et al18 and Al-Amleh et al20 met 3 of the 5 criteria for a valid review, whereas Triwatana et al19 only met 2 of the criteria. The rank ordering of the 3 systematic reviews was similar with both questionnaires. One study21 was referenced in all 3 systematic reviews, but each systematic review reported the data differently. One review reported only core fracture (n¼0) and veneering fracture (n¼4) numbers,18 one reported cumulative survival rate (CSR) (92.7%),19 and the third reported failure rates (6%) and CSR (93%).20 The reported sample size was cited differently as well. Wang et al18 reported the sample size as 216 crowns, whereas Triwatana et al19 reported 168 crowns, and Al-Amleh et al20 reported 204 crowns. This finding led the authors of this investigation (LAL and STT) to review the details of the publication by Ortorp et al21 that was included in all 3 systematic reviews (Table VI). Reference to the study varied based on how the researchers of the systematic reviews decided to report the crowns that were

lost to follow-up. In fact, 216 crowns were cemented, 204 crowns were available at the first recall, and 168 were available for examination at the last recall (Table VI). The study reported the number of failures over the course of the investigation, rather than reporting a “failure rate,” it reported the CSR to be 92.7%.21 To determine if a standard Cochran Review protocol pertinent to the reviewed topic would meet the criteria as outlined by AMSTAR,12 the researchers of this study decided to review the protocol designed by Poggio et al.17 Ten of the 11 areas mentioned in the AMSTAR form were specifically addressed. The only area that the protocol did not address was how the studies would present the data; it did not indicate whether a list of included and excluded studies would be provided.

DISCUSSION The methodology for systematic reviews aims to create a standardized approach in summarizing clinical health research, but the quality of the studies generated through this approach has received relatively little attention. Quality can be defined as “the likelihood that the design of a systematic review will generate unbiased results.”13 Although

The Journal of Prosthetic Dentistry

systematic review quality assessment tools were mentioned as early as 1987,25 the majority of these tools did not gain popularity, which can be attributed to the complexity involved in applying these instruments and a variety of weaknesses in the assessment results.12,13 Recently, newer appraisal tools have been designed and validated. These tools are easier to apply and provide meaningful evaluation. The appraisal of the Poggio et al17 protocol with the AMSTAR instrument illustrates that it is possible to develop a methodology that would address all criteria of a valid and reliable systematic review. However, the evaluation of 3 other systematic reviews18-20 mentioned in this study illustrates the variation in how systematic reviews are conducted and how they report research. This variability may significantly influence the conclusions and recommendations generated in these reviews. None of the researchers appeared to have widened their database by searching gray literature (unpublished literature such as dissertations, conference proceedings, and trial registries)26 and appeared to have limited their search to peer-reviewed journals. However, according to Shea et al,12,13 the use of multiple databases by Wang et al18 and Al-Amleh et al20 theoretically strengthened their search.

Lang and Teich

-

2014

Table V.

7 Oxford Systematic Review Appraisal14

Question

Study (response)

What question (PICO [patients, intervention, comparison, outcome]) did the systematic review address? The main question being addressed should be clearly stated; the exposure, such as a therapy or diagnostic test, and the outcome(s) of interest will often be expressed in terms of a simple relationship.

Wang et al18 (yes); Triwatana et al19 (no) (comment: no clear research question formulated); Al-Amhed et al20 (yes)

Is it unlikely that important, relevant studies were missed? The starting point for comprehensive search for all relevant studies is the major bibliographic databases (eg, Medline, Cochrane, EMBASE) but should also include a search of reference lists from relevant studies, and contact with experts, particularly to inquire about unpublished studies; the search should not be limited to English language only; the search strategy should include both MeSH terms and text words.

Wang et al18 (no) (comment: search in English and Chinese; no MeSH terms in search strategy; no gray literature search); Triwatana et al19 (no) (comment: searched only PubMed in English; no gray literature search; no MeSH terms in search strategy); Al-Amhed et al20 (no) (comment: no gray literature search; search limited to English; no MeSH terms in search strategy)

Were the criteria used to select articles for inclusion appropriate? The inclusion or exclusion of studies in a systematic review should be clearly defined a priori; the eligibility criteria used should specify the patients, interventions or exposures, and outcomes of interest; in many cases the type of study design will also be a key component of the eligibility criteria.

Wang et al18 (yes); Triwatana et al19 (yes); Al-Amhed et al20 (yes)

Were the included studies sufficiently valid for the type of question asked? The article should describe how the quality of each study was assessed by using predetermined quality criteria appropriate to the type of clinical question (eg, randomization, blinding, and completeness of follow-up).

Wang et al18 (yes); Triwatana et al19 (yes); Al-Amhed et al20 (yes)

Were the results similar from study to study? Ideally, the results of the different studies should be similar or homogeneous; if heterogeneity exists, the researchers may estimate whether the differences are significant (c2 test); possible reasons for the heterogeneity should be explored.

Wang et al18 (no) (comment: no discussion regarding heterogeneity and no forest plot); Triwatana et al19 (no) (comment: no discussion regarding heterogeneity and no forest plot); Al-Amhed et al20 (no) (comment: no discussion regarding heterogeneity and no forest plot)

Answer options: “Yes” or “No.”

All 3 reviews gathered the data from various studies without performing statistical analyses to determine if these databases should be combined. Because none of the systematic reviews evaluated the homogeneity of the data reported, the data should not be pooled. Wang et al,18 however, chose to pool the fracture data of multiple ceramic materials to get a “credible result.” The rationale was that, “according to previous reviews, IPS Empress, In-Ceram Alumina, and feldspathic porcelain crowns demonstrated similar clinical survival rates irrespective of their position in the mouth.” This pooling caused confusion when

Lang and Teich

the data for ZBSC was evaluated. Furthermore, by including older (and mostly obsolete) ceramic materials, for example, Dicor glass ceramic, the results were skewed, so the conclusion would have been that ceramic crowns in general should not be in clinical use. To neutralize this effect, the authors ungrouped the glass-ceramic fractures from the pool of results and arrived at a more “acceptable” result. Furthermore, the 3 systematic reviews did not evaluate the scientific quality of the included studies in formulating their conclusions. Because neither of these 2 important evaluations was addressed, the results of these

3 systematic reviews should be interpreted cautiously. Wang et al18 focused on the fracture rates of all ceramic crowns without considering any other mode of failure. Core fracture and veneer fracture rates were analyzed separately and as pooled data. Because Wang et al18 had difficulty in distinguishing clear definitions with regard to the degree of veneering material loss (“chipping” vs “veneer fracture”), the authors analyzed all publication data related to the loss of veneering porcelain as “veneer fracture” rather than discerning between those that were failures versus those that were clinically acceptable complications. Al-Amleh

8

Volume

Issue

-

Data from Ortorp et al21

Table VI.

Follow-up Period Study Classification Material (mo) Örtorp et al, 200921

-

Retrospective with clinical follow-up

Procera zirconia

36

Sample Size

Overall Complication Rate Complications

216(reported Core fracture 0%, by Wang ceramic veneer et al18)a; 204 fracture 2%, extraction (reported by 2%, endodontic Al-Amleh treatment 6%, et al20)b; 168 lost retention 7% (reported by Triwatana et al19)c

16%

Failure Definition

CSR

92.7% (3-y) Tooth extraction, remake of crown, pain

CSR, cumulative survival rate. a A total of 216 crowns were placed. b A total of 204 crowns were available for first recall appointment. c A total of 168 crowns were available for examination at last recall.

Table VII.

Definition of failure by publication

Study

Failure Definition

Wang et al,18 2012 Triwatana et al,19 2012

Al-Amleh et al,20 2010

Not clear Ceramic core fracture, secondary caries, veneer fracture that results in remaking crown, extraction of tooth, pain that results in remaking crown, loss of retention that results in remaking crown Ceramic core fracture, veneer fracture that results in remaking crown, extraction of tooth, pain that results in remaking crown

et al20 reported a 6% failure rate and a 3-year CSR of 93%. Triwatana et al,19 by contrast, reported the 3-year CSR at 92.7% as described in the original publication.21 As early as 2007, published recommendations27 defined criteria for failure. In 2010, the World Dental Federation28 published an update of these evaluation criteria. Although these standards existed, they were not necessarily followed uniformly by all investigators. If clinical investigators do not follow professionally agreed upon criteria for failure, then the researchers of the systematic reviews face a challenge to develop scientifically valid conclusions based on a statistical analysis. The various definitions for failure that were used by the studies in each of the systematic reviews are described in Table VII.18-20 None of the 3 reviews mention the failure criteria in a

table or bullet list, and the reader has to read the article carefully to determine what is considered a failure or a complication. The quality of the review is dependent on the original published data. Data may be missing because of editorial review; for example, tests for heterogeneity or publication bias may have been performed but not reported. Quality also is dictated by the heterogeneity of the studies included in the analysis, which may not allow for meta-analysis. Another factor that may affect the review is the evaluation bias in short-term studies. To neutralize this effect, the use of CSRs rather than success or failure rates seems to be essential. The results generated in the search illustrate the challenges that the clinician faces with regard to interpreting the available literature to make clinical decisions.

The Journal of Prosthetic Dentistry

When systematic reviews are found, the reader must depend on the researcher’s definitions and interpretation of the data and outcomes selected from the literature.

CONCLUSIONS The purpose of a systematic review was to help the clinician keep abreast of the plethora of literature and make day-to-day evidence-based clinical decisions. However, throughout the dental literature, systematic reviews vary greatly in their quality. The methodologies used in systematic reviews may significantly influence the researcher’s final conclusions. This investigation attempted to critically appraise the current systematic review process, which results in publications that clinicians use to make an evidenced-based decision. Based on the limitations on this study, the following was concluded: 1. Because of the variation in methodologies, systematic reviews should be interpreted cautiously. Clinicians should critically appraise the quality of the systematic review before accepting its recommendations. The AMSTAR checklist and the Oxford Systematic Review Appraisal Sheet are practical tools to appraise and determine the quality of systematic reviews.

Lang and Teich

-

2014 REFERENCES

1. Sackett DL, Rosenberg WM, Gray JA, Haynes RB, Richardson WS. Evidence based medicine: what it is and what it isn’t. BMJ 1996;312:71-2. 2. Sackett DL, Strauss SE, Richardson WE. Evidence-based medicine: how to practice and teach EBM. 2nd ed.Edinburgh: Churchill Livingston; 2000. p. 261. 3. Glover J, Izzo D, Odata K, Wang L. EBM Pyramid and EBM Page Generatorª Trustees of Dartmouth College and Yale University; 2006. Available at: http://www.dartmouth.edu/ wbiomed/resources.htmld/guides/ebm_ resources.shtml. Accessed February 18, 2013. 4. Rosner AL. Evidence-based medicine: revisiting the pyramid of priorities. J Bodyw Mov Ther 2012;16:42-9. 5. Laney WR. Proceedings of Prosthodontics 21. A national symposium on prosthodontics. J Prosthet Dent 1990;64:391-6. 6. McGivney GP. Editorial: evidence-based dentistry article series. J Prosthet Dent 2000;83:11-2. 7. American Dental Association. Available at: http://ebd.ada.org/About.aspx. Accessed January 3, 2014. 8. Phillips B, Ball C, Sackett D, Badenoch D, Straus S, Haynes B, et al. Oxford Centre for Evidence-based Medicine. Levels of evidence March 2009. Available at: http://www.cebm. net/?o¼1025. Accessed February 18, 2013. 9. Hadorn DC, Baker D, Hodges JS, Hicks N. Rating the quality of evidence for clinical practice guidelines. J Clin Epidemiol 1996;49:749-54. 10. Agency for Healthcare Research and Quality. Description of ideal evaluation methods: assessing the strength of evidence across studies of patient safety practices: assessing the evidence for context-sensitive effectiveness and safety. December 2010. Agency for Healthcare Research and Quality, Rockville, MD. Available at: http://www.ahrq.gov/research/findings/ final-reports/contextsensitive/context13.html. Last accessed January 9, 2014. 11. Higgins JP, Green S. Cochrane handbook for systematic reviews of interventions. Wiley Online Library 2008:672. Available at: www.cochrane-handbook.org. Last accessed January 14, 2014.

Lang and Teich

9 12. Shea BJ, Grimshaw JM, Wells GA, Boers M, Andersson N, Hamel C, et al. Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol 2007;7:10-7. 13. Shea BJ, Hamel C, Wells GA, Bouter LM, Kristjansson E, Grimshaw J, et al. AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. J Clin Epidemiol 2009;62:1013-20. 14. Oxford Centre for Evidence-based Medicine. University of Oxford. 2005. Available at: http://www.cebm.net/index.aspx?o¼1157. Accessed February 18, 2013. 15. American Dental Association Commission on Dental Accreditation. Accreditation Standards for Dental Education Programs. Available at. Chicago, Ill: American Dental Association; 2010. Available at: http://www. ada.org/sections/educationAndCareers/ pdfs/predoc_2013.pdf. Accessed October 16, 2013. 16. National Center for Biotechnology Information. U.S. National Library of Medicine. PubMed clinical queries. Available at: http:// www.ncbi.nlm.nih.gov/pubmed/clinical. Last accessed January 9, 2014. 17. Poggio CE, Ercoli C, Monaco C, Esposito M. Metal-free materials for fixed prosthodontic restorations (Protocol). The Cochrane Library. John Wiley & Sons; 2012. p. 10. Available at: http://onlinelibrary.wiley.com/doi/10.1002/ 14651858.CD009606/abstract. Last accessed January 9, 2014. 18. Wang X, Fan D, Swain MV, Zhao K. A systematic review of all-ceramic crowns: clinical fracture rates in relation to restored tooth type. Int J Prosthodont 2012;25: 441-50. 19. Triwatana P, Nagaviroj N, Tulapornchai C. Clinical performance and failures of zirconia-based fixed partial dentures: a review literature. J Adv Prosthodont 2012;4: 76-83. 20. Al-Amleh B, Lyons K, Swain M. Clinical trials in zirconia: a systematic review. J Oral Rehabil 2010;37:641-52. 21. Ortorp A, Kihl ML, Carlsson GE. A 3-year retrospective and clinical follow-up study of zirconia single crowns performed in a private practice. J Dent 2009;37:731-6.

22. Schmitt J, Wichmann M, Holst S, Reich S. Restoring severely compromised anterior teeth with zirconia crowns and feather-edged margin preparations: a 3-year follow-up of a prospective clinical trial. Int J Prosthodont 2010;23:107-9. 23. Beuer F, Stimmelmayr M, Gernet W, Edelhoff D, Guh JF, Naumann M. Prospective study of zirconia-based restorations: 3-year clinical results. Quintessence Int 2010;41: 631-7. 24. Cehreli MC, Kokat AM, Akca K. CAD/CAM Zirconia vs. slip-cast glass-infiltrated alumina/ Zirconia all-ceramic crowns: 2-year results of a randomized controlled clinical trial. J Appl Oral Sci 2009;17:49-55. 25. Sacks HS, Berrier J, Reitman D, Ancona-Berk V, Chalmers TC. Meta-analyses of randomized controlled trials. New Engl J Med 1987;316:450-5. 26. McAuley L, Pham B, Tugwell P, Moher D. Does the inclusion of grey literature influence estimates of intervention effectiveness reported in meta-analyses? Lancet 2000; 356:1228-31. 27. Hickel R, Roulet JF, Bayne S, Heintze SD, Mjor IA, Peters M, et al. Recommendations for conducting controlled clinical studies of dental restorative materials. Clin Oral Investig 2007;11:5-33. 28. Hickel R, Peschke A, Tyas M, Mjor I, Bayne S, Peters M, et al. FDI World Dental Federation: clinical criteria for the evaluation of direct and indirect restorations-update and clinical examples. Clin Oral Investig 2010; 14:349-66. Corresponding author: Dr Lisa A. Lang Case Western Reserve University School of Dental Medicine Department of Comprehensive Care 2124 Cornell Rd Cleveland, OH 44106. E-mail: [email protected] Copyright ª 2014 by the Editorial Council for The Journal of Prosthetic Dentistry.