558085
research-article2014
AJMXXX10.1177/1062860614558085American Journal of Medical QualityHwang et al
Article
Finding Order in Chaos: A Review of Hospital Ratings
American Journal of Medical Quality 1–9 © The Author(s) 2014 Reprints and permissions: sagepub.com/journalsPermissions.nav DOI: 10.1177/1062860614558085 ajmq.sagepub.com
Wenke Hwang, PhD1, Jordan Derk, BS1, Michelle LaClair, MPH1, and Harold Paz, MD, MS1
Abstract Publicly reported hospital performance data have become widely available to health care consumers in recent years. In response to a growing demand for more readily available health care information, various organizations have begun assessing hospital performance. These performance reporting systems have tremendous potential to aid patients, families, and primary care providers in their clinical decision making. This study takes a systematic approach to review the main features of 9 existing hospital rating systems, each of which is described using 9 areas of evaluation. The hospital rating systems included in this study vary widely in scope, methodology, transparency, and presentation of their results. Their results often present conflicting conclusions regarding the performance of the same hospital. This review of hospital rating systems demonstrates how public reporting may add confusion to patients’ health care decision making. Keywords hospital performance, hospital safety, hospital quality of care, public reporting quality measures
Over the past decade, the Internet has become a more prominent source of information for patients seeking health care advice. The literature has shown an increase in patients searching for information regarding health care provider quality online.1 In response to the growing demand for hospital care information, there has been an increase in the number of organizations conducting hospital ratings or rankings. These hospital performance reporting systems have tremendous potential to aid patients, families, and primary care providers in their choice of hospitals. These organizations select different sets of quality measurements, either with existing data available in the public domain or through proprietary data collection. They focus on different aspects of hospital performance, including specific structural features in the delivery system, process of care, outcomes of care, and patient experience. These organizations also aggregate and report data in different ways. Multiple public reporting Web sites provide a mechanism for users to directly measure and rank a hospital’s quality performance against other hospitals within the same market. One organization provides a rationale for insurance payers to negotiate reimbursement rates based on their hospital rating results.2 All these organizations aim to provide hospital quality and safety information directly to consumers to help them
choose which hospital is appropriate for their health care needs. With increased access to this type of information, consumers are now faced with the challenge of trying to decipher all the competing information available to them on hospital quality and safety. Moreover, some public reporting systems have more transparency in their methodologies than others. Most systems require extensive research to understand their methods, intended audiences, and how best to interpret their results. Studies have shown that there is little consistency in the level of detail and in the results among these Web sites.3 The information on the quality of hospital care is chaotic and inconsistent at best. Hospital performance information disseminated by current public reporting systems has rarely gone through peer review, yet is made widely available to consumers and hospital executives. It is no surprise that previous literature has shown little evidence as to how or if public reporting of hospital quality data has an impact on quality of care4 or patients’ decision 1
Penn State University College of Medicine, Hershey, PA
Corresponding Author: Wenke Hwang, PhD, Department of Public Health Sciences, Penn State University College of Medicine, Mail Code A210, 90 Hope Drive, Suite 2200, Hershey, PA 17033. Email: [email protected]
Downloaded from ajm.sagepub.com by guest on November 15, 2015
2
American Journal of Medical Quality
Table 1. Descriptions of Areas of Evaluation. Area Primary objective Intended audience Domains of measure Composite score Ranking/Rating/List
Geographic context Data sources Specialty performance Transparency of methodology
Description What is the purpose for judging hospital performance? These purposes are quoted directly from the organizations’ own explanations, when possible Who does the organization view as its readers? This may be a specific professional group or a general category. Identified audiences are quoted directly from the organization’s own descriptions, when possible What areas of hospital performance are being judged? Does the organization provide a single composite score for each hospital? If so, in what way is the score presented (eg, numerical score, letter grade). How does the organization display hospital performance? Ranking denotes a system in which all hospitals are arranged in order of ascending performance. A hospital’s final score then denotes its placement among all other hospitals. Rating denotes a system in which hospitals’ quality can be understood without comparison to other hospitals. Listing denotes a publication in which inclusion or exclusion of a hospital identifies quality. Listings do not provide any indication of differences in quality of care between hospitals—either a hospital is on the list or it is not What is the scope of the organization’s rating? Is it meant to be used to determine hospital quality on a regional or national level? Regional is considered to be a county, district, or statelevel focus. National is considered to be a country-level focus From which sources did the organization obtain the data utilized in generating their performance assessment? In addition to providing quality information for the hospital as a whole, does the organization also provide quality information for specific medical specialties? How does the organization explain its methodology? Not Provided denotes a rating system that does not provide any methodology. Provided but not reproducible denotes a rating system that provides a methodology, but one that is not clear enough for an outside party to reproduce the results if given the necessary data. Provided and reproducible denotes a rating system that provides a methodology that is clear enough for an outside party to reproduce the results if given the necessary data
making.5,6 This study takes a systematic approach to review the main features of 9 existing hospital public reporting systems and the context in which the performance information is presented. This article is intended to aid health care consumers and professionals to select and interpret the reported performance results. However, it does not attempt to recommend one system over another.
Methods A total of 9 hospital public reporting systems were selected for inclusion in this study. The systems were evaluated and their main features highlighted across the following 9 areas: primary objective, intended audience, domain of measure, use of composite score, ranking/rating/list, geographic context, data sources, specialty performance, and transparency of methodology. Detailed descriptions of each measure are presented in Table 1. Publicly available information from the Web sites of the 9 systems included in this study was utilized to describe each system in the 9 areas noted. When clarification was needed, contact was made with a representative of the reporting system in question. To highlight the chaotic and often conflicting results presented by the public reporting systems, the U.S. News
and World Report’s 18 Honor Roll hospitals were scored for each system. The Centers for Medicare & Medicaid Services (CMS) Hospital Compare provides critical quality data, but does not produce an aggregated rating or ranking for hospitals; therefore, it is not included in this evaluation. The ratings for the 18 Honor Roll hospitals are compared. The ratings or rankings are presented as they are by the organizations themselves, when possible. Consumer Reports utilizes a scoring system based on “blobs.” For easy interpretation, this study uses numerical ratings (1-5) instead of blob illustrations for this system.
Results All the public reporting systems included in this review were described using 9 areas of evaluation (Table 2).7-18 The following section is a brief overview highlighting specific characteristics of each of the public reporting systems.
Becker’s Hospital Review: 100 Great Hospitals in America Becker’s Hospital Review is a monthly trade publication created by Becker’s Healthcare that reaches approximately
Downloaded from ajm.sagepub.com by guest on November 15, 2015
Downloaded from ajm.sagepub.com by guest on November 15, 2015
3
List Regional/National Unclear
No Not provided
No
List
National
Nominations, U.S. News & World Report, Truven’s 100 Top Hospitals, Healthgrades, Magnet Recognition, Studer Group and Malcolm Baldrige National Quality Award Recipients7 Noa
Not provided
Composite score? Ranking/Rating/ List Geographic context Data sources
Provided and reproducible
No
CMS
Regional/National
Rating
Infections, readmissions, communication, scanning, complications11
Timely and effective care, readmissions, complications, deaths, medical imaging, patient experience, number of Medicare patients, Medicare payment10 No
Provided and reproducible11
No
CMS, state departments of health, AHA11
Regional/National
Rating
Yes (blobs)
Patients
Consumers, providers
To help patients compare hospitals11
Consumer Reports Hospital Ratings To provide “meaningful information about the comparative performance of accredited organizations to the public”12 Public 12
The Joint Commission
Provided but not reproducible
Yes
No Provided and reproducible
Not provided
Leapfrog Hospital Survey, AHRQ, CDC, AHA, and CMS Hospital Compare17
Regional
Rating
Yes (A-F)
Process/Structural, outcomes17
Employers, insurers, patients
To save lives by “reducing injuries, accidents, and preventable medical and medication errors”13
Leapfrog Hospital Safety Score
Yes
Regional/ Regional National CMS, MedPAR, The Joint 19 state Commission departments of health8
Patient safety Risk-adjusted mortality rates, goal compliance, National Quality complication rates across 33 Improvement goals performance, conditions and patient satisfaction, procedures patient mortality12 Yes (1, 3, and 5 No stars) Ratings Rating
Consumers8
“To help consumers evaluate and compare hospital performance”8
Healthgrades Hospital Ratings
Ranking
Yes (0-100)
Structure, process, outcomes18
Patients
To provide a tool to help patients find unusually skilled inpatient care15
US News Best Hospitals
Provided and reproducible
No
Provided and reproducible
Yes
Regional/ National MedPAR, HCRIS, AHA, MedPAR, 14 and CMS U.S. News Physician Surveys18
National
List
No
Clinical quality, efficiency, financial health, patient perception of care14
Hospital administrators
To identify “the U.S. hospitals with the best facility-wide performance”14
Truven 100 Top Hospitals
Abbreviations: AHA, American Hospital Association; AHRQ, Agency for Healthcare Research and Quality; CDC, Centers for Disease Control and Prevention; CMS, Centers for Medicare & Medicaid Services; HCRIS, Healthcare Cost Report Information System; MedPAR, Medicare Provider Analysis and Review. a Becker’s Hospital Review “100 Hospitals With Great Heart Programs” list published separately from “100 Great Hospitals in America.”
Specialty performance Transparency of methodology
No
Unclear
Domains of measure
Hospital executives, hospital’s surrounding community9 Financial viability and plant reinvestment, cost structure, charge structure, quality performance9
Health care industry leaders16
Intended audience
CMS Hospital Compare
To measure a “hospital’s To help patients decide value to [its] where to get health care, community”9 to “encourage hospitals to improve the quality of care”10
To recognize organizations that “are home to medical breakthroughs and act as anchors of health within their respective communities.”7
Cleverley + Associates Community Value Leadership Awards
Primary objective
Becker’s Hospital Review 100 Great Hospitals in America
Table 2. Hospital Rating System Overview.
4
American Journal of Medical Quality
18 500 health care leaders. Their annual “100 Great Hospitals in America” list is composed of community hospitals, academic medical centers, large flagship facilities, and small specialty hospitals.7 100 Great Hospitals in America identifies hospitals that are leaders in the health care industry. The 100 Great Hospitals list is presented alphabetically with no indication of rank or specific level of quality for the hospitals. Consequently, it is impossible to distinguish any difference in quality between the hospitals listed. The Becker’s Hospital Review editorial team accepts nominations for the list, consults other hospital rating systems, and “conducts research”7 in order to compose the list. No methodology is available to determine how each of these components factors in to the selection of the hospitals on the list, nor is there any indication of what is involved in the research conducted by the editorial team.
Cleverley + Associates Community Value Leadership Awards The Community Value Leadership Awards measure the value that hospitals provide to their communities. The awards are based on the Community Value Index (CVI), a metric developed by Cleverley + Associates in 2004. The index determines hospital value using 10 measures in 4 performance areas: financial viability and plant reinvestment, hospital cost structure, hospital charge structure, and hospital quality performance. Cleverley + Associates provides 2 distinct Community Value Leadership Awards: The Community Value 100 and the Community Value Five-Star. To determine award winners, hospitals are divided into one of 5 groups according to their teaching status and size. Hospitals that achieved the highest 20 CVI scores in their respective groups earned the Community Value 100 Award. Hospitals that achieved CVI scores within the top quintile (20%) of all hospitals within their group earned the Community Value Five-Star award. Although measuring hospitals’ value to their communities is a somewhat unique approach to hospital ratings, the award lists are presented alphabetically, with no distinction of relative ranking or quality. Additionally, CVI scores for the listed hospitals are not presented. Consequently, it is impossible for consumers to determine a hospital’s measured value from Cleverley + Associates’ awards. Finally, a lack of transparency in both methodology and source of data make it impossible to determine what exactly determines the CVI score.
CMS Hospital Compare Under the Hospital Quality Initiative, CMS’s Hospital Compare was developed in 2005 “to publicly report valid,
credible, and user-friendly information about the quality of care delivered in the nation’s hospitals.”19 It has since served as a data source for most other hospital quality public reporting organizations. Hospital Compare presents information about the quality of care for more than 4000 Medicare-certified hospitals. Hospital Compare allows patients to easily compare hospitals’ quality information side by side. Hospital Compare does not provide any composite scores. Because consumers are only able to compare hospital quality through dozens of clinical measures, the information presented may be too complex for the average consumer to understand. The measure scores are often presented in easy-to-understand ratings, such as “better than the U.S. national benchmark.” However, the measures themselves are clinically based and may require clinical knowledge to truly understand their meaning.
Consumer Reports Hospital Ratings Instead of focusing on hospital “quality” or “value,” Consumer Reports’ Hospital Ratings help patients compare hospital safety. Hospitals are given a score of 1 to 5 “blobs,” identical to Consumer Reports ratings in other industries. The rating methodology utilizes CMS Hospital Compare data for care outcomes, patient experience, and hospital practices. Additional data are retrieved from state departments of health when they are available. Consumer Reports’ use of blobs helps consumers easily understand level of quality. Not all state departments of health collect the particular measures that Consumer Reports attempts to use. Sometimes, this prevents hospitals from receiving scores. Furthermore, a subscription to Consumer Reports is required to view any of the hospital ratings, limiting patient access.
Healthgrades Hospital Ratings Healthgrades evaluates hospital quality based on 33 conditions or procedures. These measures are rated using a 5-star system to identify patient outcome quality. One star is given for a measure that is “worse than expected.”8 Three stars are given for a measure that is “as expected.” Five stars are given for a measure that is “better than expected.” Healthgrades also gives “Excellence Awards” for various categories of health care quality, such as patient safety, or superior outcomes in various specialties. Consumers can search Healthgrades’ rating Web site for hospitals by name or by geographic area. The search yields a list of hospitals fitting the search criteria, accompanied by their numbers of Healthgrades 5-Star Ratings, Healthgrades Quality Awards, and affiliated providers.
Downloaded from ajm.sagepub.com by guest on November 15, 2015
5
Hwang et al This is the only way to directly compare hospitals on one page because consumers may only access hospital reports one at a time.
The Joint Commission The Joint Commission awards accreditation for health care providers and certification for providers’ individual programs. The Joint Commission conducts site visits and collects federally reported data to determine hospital quality. Accreditation and certification status is provided for each evaluated hospital on the Joint Commission’s Web site, Quality Check. Consumers can compare hospitals side by side using Quality Check. Quality Check presents information only for accredited health care providers and their certified programs. Therefore, consumers are unable to compare nonaccredited hospitals or any noncertified programs. Also, no methodology is provided to consumers to demonstrate exactly how hospitals are evaluated by The Joint Commission. Although it is clear that The Joint Commission uses extensive means to evaluate hospitals, it is impossible to determine exactly how these means are used to measure quality. Furthermore, The Joint Commission does not explicitly define accreditation or certification. There is no publicly available description of what standards must be met to earn accreditation or certification statuses. This may cause some confusion or misunderstanding for consumers.
Leapfrog Hospital Safety Score Now in its third year of existence, Leapfrog’s Hospital Safety Score provides composite scores for more than 2500 acute care hospitals in the United States. The Hospital Safety Score is composed of 28 safety measures that demonstrate a hospital’s ability to minimize care mistakes with patients. Process, structural, and outcome measures are included in the score. Of the measures, 12 are contributed from Leapfrog’s Hospital Survey, an optional survey that hospitals may complete annually, free of charge. The safety score is calculated using established clinical measures only, instead of reputation surveys. The score’s methodology is also peer reviewed, the only hospital rating methodology the authors are aware of that has gone through such a process. Hospitals that do not participate in the Leapfrog Hospital Survey may receive lower scores than they could by participating.20 Leapfrog utilizes a different scoring methodology to account for 8 missing measures for nonparticipants. Although the Leapfrog Hospital Safety Score is one of the most robust and comprehensive rating systems available, its ability to give a hospital 2 different letter grades based on its participation in the Leapfrog Hospital Survey creates uncertainty as to the score’s fairness.
Truven 100 Top Hospitals Using only publicly available data, Truven identifies the 100 top short-term, acute-care, nonfederal hospitals in America using their Balanced Scorecard. The Balanced Scorecard consists of 10 measures distributed across 4 domains—quality, efficiency, finance, and consumer assessment of care. Hospitals are divided into 5 groups, based on their teaching status and size. The top 15 to 25 scoring hospitals are selected from each group to compose the 100 Top Hospitals. Because Truven uses only publicly available data and provides a detailed methodology on its Web site, the Balanced Scorecard can be reproduced by anyone, highlighting the system’s transparency. The 100 Top Hospitals includes the top hospitals from each group instead of the actual top 100 hospitals regardless of teaching status or size. Consequently, a hospital that is actually scored as a top 100 hospital overall may not be included in the list because it falls below the cutoff point within its respective group. Furthermore, the list does not provide any quantifiable measure or rank of hospital quality. The 100 Top Hospitals list is presented alphabetically, with no distinction of relative ranking or quality.
U.S. News & World Report’s Best Hospitals U.S. News & World Report’s Best Hospitals ranking is in its 24th year of existence. As one of the oldest hospital ranking systems, it has become arguably the most recognizable quality reporting system for patients. Rankings are provided for 16 specialties of care, making the rankings relevant to a very wide range of patients. An “Honor Roll” list of hospitals also is presented to identify the nation’s top hospitals across all specialties. The methodology relies heavily on hospitals’ reputation scores for specialty rankings (32.5%). The significant role of reputation scores has been criticized because it reduces the impact of well-established clinical measures.21 Although the Best Hospital list may be broken down into specialties, it only provides a list of the “best” hospitals, instead of providing ratings to all hospitals in the country. This means that many hospitals in a given geographic area will not have a ranking, possibly making the Best Hospitals list less relevant for patients wishing to compare unranked hospitals’ quality of care.
Ratings for U.S. News & World Report’s 18 Honor Roll Hospitals Table 3 presents the ratings for the 18 Honor Roll Hospitals across 8 of the public reporting organizations addressed in this study. Inconsistencies can be seen across the ratings for multiple hospitals. Brigham and Women’s earned Consumer Reports’s worst rating (5) but earned 11
Downloaded from ajm.sagepub.com by guest on November 15, 2015
6
Downloaded from ajm.sagepub.com by guest on November 15, 2015
a
Abbreviations: NL, not listed; NS, not scored. Consumer Reports Scores: 1 = best, 5 = worst.
Barnes-Jewish Brigham and Women’s Cedars-Sinai Cleveland Clinic Duke Hospital of the University of Pennsylvania Indiana University Health Johns Hopkins Massachusetts General Mayo Clinic New York-Presbyterian Northwestern Memorial NYU Langone Thomas Jefferson UCLA UCSF University Hospitals Case Medical Center UPMC (Presbyterian)
Listed Listed Listed Listed Listed Listed Listed Listed Listed Listed Listed Listed Listed Listed Listed Listed Listed Listed
Becker’s (2014) NL NL NL NL NL NL NL NL NL Listed NL NL NL NL NL NL NL NL
Cleverley (2013) 3 5 3 3 3 3 3 3 5 3 1 3 1 5 3 3 2 2
Consumer Reportsa (2013) 4 Five-star, 4 quality 11 Five-star, 5 quality 18 Five -star, 8 quality 6 Five-star, 4 quality 14 Five-star, 7 quality 4 Five-star, 1 quality 22 Five-star, 14 quality 3 Five-star, 1 quality 17 Five-star, 7 quality 9 Five-star, 4 quality 15 Five-star, 10 quality 19 Five-star, 11 quality 16 Five-star, 5 quality 20 Five-star, 9 quality 11 Five-star, 3 quality 2 Five-star, 3 quality 10 Five-star, 2 quality 5 Five-star, 1 quality
Healthgrades (2014)
Table 3. Ratings for U.S. News & World Report’s 18 Honor Roll Hospitals.
Hospital, nursing care center, behavioral health care Hospital, laboratory accreditation program Hospital, behavioral health care Hospital, home care Hospital Hospital, home care, laboratory accreditation program Hospital, behavioral health care, home care Hospital, behavioral health care Hospital, laboratory accreditation program Hospital, ambulatory care Hospital, behavioral health care, laboratory accreditation program Hospital Hospital Hospital Hospital, home care Hospital, home care, laboratory accreditation program Hospital Hospital, behavioral health care
The Joint Commission (2013)
C A B C A A C NS B A C A B B B B A B
Leapfrog (Fall 2013)
NL NL NL NL Listed NL NL NL NL NL NL NL NL NL NL NL NL NL
Truven (2014)
15 9 13 4 12 11 16 1 2 3 7 6 14 17 5 7 18 10
U.S. News (2013-2014)
7
Hwang et al five-star ratings by Healthgrades as well as Leapfrog’s highest grade (A). New York-Presbyterian earned Consumer Reports’s best rating (1), as well as 15 five-star measures and 10 quality awards by Healthgrades but earned a grade of C from Leapfrog. NYU Langone and Thomas Jefferson both earned Bs from Leapfrog but earned the best (1) and worst (5) ratings, respectively, from Consumer Reports. Consumer Reports assigned a rating of 3 to 11 of the 18 Honor Roll Hospitals. Out of these 11 hospitals, Leapfrog assigned 4 As, 3 Bs, and 3 Cs. Inclusion of hospitals on these lists also varied. Johns Hopkins was not scored by Leapfrog because Medicare does not publicly report data for Maryland hospitals. Only one Honor Roll hospital (Duke) was listed by Truven’s 100 Top Hospitals. All 18 hospitals were listed by Becker’s Hospital Review. Only one hospital (Mayo Clinic) was listed by Cleverley + Associates.
Discussion Both the public and private sectors have invested immense resources to collect data and measure hospital quality and patient safety. There is a tremendous potential to make this information useful for consumers to choose hospitals and for hospital executives to improve quality of health services. The current state of public reporting may not be providing much help to consumers or health care executives. This review of 9 important attributes across 9 major public reporting systems found that the systems focus on vastly different quality attributes when measuring hospital performance. This difference in performance measurement can lead to inconsistent and possibly conflicting conclusions. However, this report is not intended to evaluate the relative accuracy or validity of each public reporting system. This review of public reporting systems and examples of inconsistent hospital performance results demonstrate how public reporting may actually add confusion to patients’ health care decision making. Hospitals were often given conflicting ratings and rankings by public reporting organizations. Some hospitals received top marks from one system but low marks from another. Often, hospitals that shared the same rating from one rating organization received very different ratings from the other organizations. In addition to the public reporting systems included in this report, many states are now mandating that certain hospital quality measures be publicly reported. However, the requirements of these mandates vary across states.22,23 Different data submission requirements, methods of risk adjustment, time frames of data collection, and presentation of the data cause further confusion regarding evaluation of hospital quality and safety.
Much of the confusion caused by public reporting systems may be related in part to a lack of a minimum standard for public reporting. Standardized methods for developing quality measures are used by CMS24 and the Agency for Healthcare Research and Quality (AHRQ).25 Unfortunately, these methods only apply to developing individual measures. No standardized methods exist for developing composite scores that combine multiple quality measures. There is no commonly accepted guideline for reporting organizations to use to disseminate quality information. A standardized method for developing methodologies to measure hospital performance, similar to the CMS and AHRQ approaches for developing quality measures, will be of high value. The Association of American Medical Colleges already provides 3 broad principles for public reporting26; however, no specific standards have been developed or accepted yet. Without a commonly accepted minimum standard, public reporting will continue to cause confusion for patients. At the very least, public reporting organizations should more clearly explain the overall purpose of their systems to prevent confusion for patients. For example, Consumer Reports, U.S. News & World Report, and Leapfrog often present conflicting scores to hospitals. One hospital may receive 3 very different scores from these 3 organizations. This should not be a surprise because the 3 organizations measure different aspects of hospital performance. Consumer Reports focuses on patient safety for a few specific surgical procedures. U.S. News measures which hospitals give the most advanced, industry-leading care for the most complicated medical conditions. Leapfrog measures general hospital safety. However, it may be difficult for patients to discern which public reporting system is most appropriate to use for their specific needs. If public reporting systems provided more detailed explanations of their general purpose to users, much of the confusion caused by the conflicting ratings could be avoided. Moreover, many public reporting organizations generate revenue by allowing hospitals to use their various results, awards, certifications, and logos for marketing purposes. As a result, hospital performance information has become an important marketing tool for the health care institutions and media instead of being used as a guide for patients, families, and primary care physicians to select hospitals for needed care or referrals. This may explain why consumers have been shown to heavily rely on the Internet for diagnosis and treatment information but generally lack interest in quality measure reports.27 Current financial models, where public reporting systems generate revenue from their ratings, may raise conflict of interest concerns. Ideally, public reporting should move away from these models in the future to avoid potential conflict of interest.
Downloaded from ajm.sagepub.com by guest on November 15, 2015
8
American Journal of Medical Quality
The issue of multiple, conflicting reports of hospital performance is unlikely to change anytime soon. Therefore, it is imperative that patients and health care professionals understand how to use the systems that already exist. Each system uses its own unique methods to measure “quality” or “safety.” Patients and health care professionals must use the information and interpret with caution. The tables provided in this report can help identify which public reporting system is most appropriate for a given circumstance. Hospital quality ratings are a growing topic of importance in US health care. Further efforts must be made to ensure that the hospital ratings being reported to the public are as accurate and useful as possible. Future public reporting studies should examine how hospitals use and disseminate quality ratings. Recent work suggests that consumers may have little interest in aggregate data and may be more interested in quality and safety information associated with their personal provider.28 Developing a better understanding of the need for, and use of, quality rating systems is essential to guide the improvements of these systems. Declaration of Conflicting Interests The authors declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: Dr Paz was serving as chief executive officer of the Penn State Hershey Medical Center and Health System at the time this study was conducted.
Funding The authors received no financial support for the research, authorship, and/or publication of this article.
References 1. Leonardi MJ, McGory ML, Ko CY. Publicly available hospital comparison web sites: determination of useful, valid, and appropriate information for comparing surgical quality. Arch Surg. 2007;142:863-869. 2. The Leapfrog Group. The hidden surcharge Americans pay for hospital errors. 2013. http://www.leapfroggroup. org/employers_purchasers/HiddenSurchargeCalculator. Accessed March 25, 2014. 3. Halasyamani LK, Davis MM. Conflicting measures of hospital quality: ratings from “Hospital Compare” versus “Best Hospitals.” J Hosp Med. 2007;2:128-134. 4. Hollenbeak CS, Gorton CP, Tabak YP, Jones JL, Milstein A, Johannes RS. Reductions in mortality associated with intensive public reporting of hospital outcomes. Am J Med Qual. 2008;23:279-286. 5. Mazor KM, Dodd KS, Kunches L. Communicating hospital infection data to the public: a study of consumer responses and preferences. Am J Med Qual. 2009;24:108-115. 6. McGuckin M, Govednik J, Hyman D, Black B. Public reporting of health care-associated infection rates: are consumers aware and engaged? Am J Med Qual. 2014;29:83-85.
7. Gamble H, Herman B. 100 Great hospitals in America. 2013. http://www.beckershospitalreview.com/100-greathospitals-2013/100-great-hospitals-in-america-2013.html/. Accessed March 25, 2014. 8. Healthgrades. Mortality and complication outcomes: 2014 methodology. https://d2dcgio3q2u5fb.cloudfront.net/b7/ 4e/543c99af4ec399f4093b7c727cfe/2014-mortality-andcomplications-outcomes-methodology.pdf. Accessed March 25, 2014. 9. Cleverley + Associates. The Community Value Index. http://www.cleverleyassociates.com/CommunityValue/ AboutTheCVI/Default.aspx. Accessed March 25, 2014. 10. Centers for Medicare and Medicaid Services. What is Hospital Compare? http://www.medicare.gov/hospitalcompare/About/What-Is-HOS.html. Accessed March 25, 2014. 11. Consumer Reports Health. How we rate hospitals. http:// www.consumerreports.org/health/resources/pdf/how-werate-hospitals/How%20We%20Rate%20Hospitals.pdf. Accessed March 25, 2014. 12. The Joint Commission. Facts about quality check and quality reports. http://www.qualitycheck.org/help_qc_facts. aspx. Accessed March 25, 2014. 13. The Leapfrog Group. About the Leapfrog Group. http:// www.hospitalsafetyscore.org/about-the-score/about-theleapfrog-group. Accessed March 25, 2014. 14. Truven Health Analytics. 100 Top hospitals: study overview. http://100tophospitals.com/portals/2/assets/100-TopStudy-March-2014.pdf. Accessed March 25, 2014. 15. U.S. News & World Report. FAQ: How we ranked the best hospitals 2013-2014. http://health.usnews.com/healthnews/best-hospitals/articles/2013/07/16/how-we-rankedthe-best-hospitals-2013-14-an-faq. Accessed March 25, 2014. 16. Becker’s Hospital Review. About Becker’s Hospital Review and Becker’s Healthcare. http://www.beckershospitalreview.com/about-us.html. Accessed March 25, 2014. 17. The Leapfrog Group. Scoring methodology. http://www. hospitalsafetyscore.org/media/file/HospitalSafetyScore_ ScoringMethodology_Spring2014_Final.pdf. Accessed March 25, 2014. 18. Olmstead M, Murphy J, Geisen E, et al. Methodology: U.S. News & World Report best hospitals 2013-14. http:// www.usnews.com/pubfiles/BH_Methodology_Report.pdf. Accessed March 25, 2014. 19. Centers for Medicare & Medicaid Services. Hospital Quality Initiative: overview. https://www.cms.gov/Medicare/ Quality-Initiatives-Patient-Assessment-Instruments/ HospitalQualityInits/downloads/HospitalOverview.pdf. Accessed March 25, 2014. 20. Hwang W, Derk J, Laclair M, Paz H. Hospital patient safety grades may misrepresent hospital performance. J Hosp Med. 2014;9:111-115. 21. Sehgal AR. The role of reputation in U.S. News & World Report’s rankings of the top 50 American hospitals. Ann Intern Med. 2010;152:521-525. 22. Aswani MS, Reagan J, Jin L, Pronovost PJ, Goeschel C. Variation in public reporting of central line-associated
Downloaded from ajm.sagepub.com by guest on November 15, 2015
9
Hwang et al bloodstream infections by state. Am J Med Qual. 2011;26: 387-395. 23. Herzig CT, Reagan J, Pogorzelska-Maziarz M, Srinath D, Stone PW. State-mandated reporting of health careassociated infections in the United States: trends over time [published online June 20, 2014]. Am J Med Qual. doi:10.1177/1062860614540200. 24. Centers for Medicare & Medicaid Services. Quality initiatives: general information. http://www.cms.gov/Medicare/ Quality-Initiatives-Patient-Assessment-Instruments/ QualityInitiativesGenInfo/index.html. Accessed March 25, 2014. 25. Agency for Healthcare Research and Quality. Quality indicator measure development, implementation, maintenance and retirement: summary. http://qualityindicators.ahrq.gov/
Downloads/Resources/Publications/2011/QI_Measure_ Development_Implementation_Maintenance_Retirement_ Summary_05-03-11.pdf. Accessed March 25, 2014. 26. Association of American Medical Colleges. Guiding principles for public reporting of provider performance. https:// www.aamc.org/download/370236/data/guidingprinciplesforpublicreporting.pdf. Accessed March 25, 2014. 27. Marshall MN, Shekelle PG, Leatherman S, Brook RH. The public release of performance data: what do we expect to gain? A review of the evidence. JAMA. 2000;283:1866-1874. 28. Smith B, Lynch WD, Markow C, Lifsey S, Slover M. Consumers’ understanding of and interest in provider- versus practice-level quality characteristics: findings from a focus group study [published online April 22, 2014]. Am J Med Qual. doi:10.1177/1062860614531447.
Downloaded from ajm.sagepub.com by guest on November 15, 2015
research-article2014
AJMXXX10.1177/1062860614558085American Journal of Medical QualityHwang et al
Article
Finding Order in Chaos: A Review of Hospital Ratings
American Journal of Medical Quality 1–9 © The Author(s) 2014 Reprints and permissions: sagepub.com/journalsPermissions.nav DOI: 10.1177/1062860614558085 ajmq.sagepub.com
Wenke Hwang, PhD1, Jordan Derk, BS1, Michelle LaClair, MPH1, and Harold Paz, MD, MS1
Abstract Publicly reported hospital performance data have become widely available to health care consumers in recent years. In response to a growing demand for more readily available health care information, various organizations have begun assessing hospital performance. These performance reporting systems have tremendous potential to aid patients, families, and primary care providers in their clinical decision making. This study takes a systematic approach to review the main features of 9 existing hospital rating systems, each of which is described using 9 areas of evaluation. The hospital rating systems included in this study vary widely in scope, methodology, transparency, and presentation of their results. Their results often present conflicting conclusions regarding the performance of the same hospital. This review of hospital rating systems demonstrates how public reporting may add confusion to patients’ health care decision making. Keywords hospital performance, hospital safety, hospital quality of care, public reporting quality measures
Over the past decade, the Internet has become a more prominent source of information for patients seeking health care advice. The literature has shown an increase in patients searching for information regarding health care provider quality online.1 In response to the growing demand for hospital care information, there has been an increase in the number of organizations conducting hospital ratings or rankings. These hospital performance reporting systems have tremendous potential to aid patients, families, and primary care providers in their choice of hospitals. These organizations select different sets of quality measurements, either with existing data available in the public domain or through proprietary data collection. They focus on different aspects of hospital performance, including specific structural features in the delivery system, process of care, outcomes of care, and patient experience. These organizations also aggregate and report data in different ways. Multiple public reporting Web sites provide a mechanism for users to directly measure and rank a hospital’s quality performance against other hospitals within the same market. One organization provides a rationale for insurance payers to negotiate reimbursement rates based on their hospital rating results.2 All these organizations aim to provide hospital quality and safety information directly to consumers to help them
choose which hospital is appropriate for their health care needs. With increased access to this type of information, consumers are now faced with the challenge of trying to decipher all the competing information available to them on hospital quality and safety. Moreover, some public reporting systems have more transparency in their methodologies than others. Most systems require extensive research to understand their methods, intended audiences, and how best to interpret their results. Studies have shown that there is little consistency in the level of detail and in the results among these Web sites.3 The information on the quality of hospital care is chaotic and inconsistent at best. Hospital performance information disseminated by current public reporting systems has rarely gone through peer review, yet is made widely available to consumers and hospital executives. It is no surprise that previous literature has shown little evidence as to how or if public reporting of hospital quality data has an impact on quality of care4 or patients’ decision 1
Penn State University College of Medicine, Hershey, PA
Corresponding Author: Wenke Hwang, PhD, Department of Public Health Sciences, Penn State University College of Medicine, Mail Code A210, 90 Hope Drive, Suite 2200, Hershey, PA 17033. Email: [email protected]
Downloaded from ajm.sagepub.com by guest on November 15, 2015
2
American Journal of Medical Quality
Table 1. Descriptions of Areas of Evaluation. Area Primary objective Intended audience Domains of measure Composite score Ranking/Rating/List
Geographic context Data sources Specialty performance Transparency of methodology
Description What is the purpose for judging hospital performance? These purposes are quoted directly from the organizations’ own explanations, when possible Who does the organization view as its readers? This may be a specific professional group or a general category. Identified audiences are quoted directly from the organization’s own descriptions, when possible What areas of hospital performance are being judged? Does the organization provide a single composite score for each hospital? If so, in what way is the score presented (eg, numerical score, letter grade). How does the organization display hospital performance? Ranking denotes a system in which all hospitals are arranged in order of ascending performance. A hospital’s final score then denotes its placement among all other hospitals. Rating denotes a system in which hospitals’ quality can be understood without comparison to other hospitals. Listing denotes a publication in which inclusion or exclusion of a hospital identifies quality. Listings do not provide any indication of differences in quality of care between hospitals—either a hospital is on the list or it is not What is the scope of the organization’s rating? Is it meant to be used to determine hospital quality on a regional or national level? Regional is considered to be a county, district, or statelevel focus. National is considered to be a country-level focus From which sources did the organization obtain the data utilized in generating their performance assessment? In addition to providing quality information for the hospital as a whole, does the organization also provide quality information for specific medical specialties? How does the organization explain its methodology? Not Provided denotes a rating system that does not provide any methodology. Provided but not reproducible denotes a rating system that provides a methodology, but one that is not clear enough for an outside party to reproduce the results if given the necessary data. Provided and reproducible denotes a rating system that provides a methodology that is clear enough for an outside party to reproduce the results if given the necessary data
making.5,6 This study takes a systematic approach to review the main features of 9 existing hospital public reporting systems and the context in which the performance information is presented. This article is intended to aid health care consumers and professionals to select and interpret the reported performance results. However, it does not attempt to recommend one system over another.
Methods A total of 9 hospital public reporting systems were selected for inclusion in this study. The systems were evaluated and their main features highlighted across the following 9 areas: primary objective, intended audience, domain of measure, use of composite score, ranking/rating/list, geographic context, data sources, specialty performance, and transparency of methodology. Detailed descriptions of each measure are presented in Table 1. Publicly available information from the Web sites of the 9 systems included in this study was utilized to describe each system in the 9 areas noted. When clarification was needed, contact was made with a representative of the reporting system in question. To highlight the chaotic and often conflicting results presented by the public reporting systems, the U.S. News
and World Report’s 18 Honor Roll hospitals were scored for each system. The Centers for Medicare & Medicaid Services (CMS) Hospital Compare provides critical quality data, but does not produce an aggregated rating or ranking for hospitals; therefore, it is not included in this evaluation. The ratings for the 18 Honor Roll hospitals are compared. The ratings or rankings are presented as they are by the organizations themselves, when possible. Consumer Reports utilizes a scoring system based on “blobs.” For easy interpretation, this study uses numerical ratings (1-5) instead of blob illustrations for this system.
Results All the public reporting systems included in this review were described using 9 areas of evaluation (Table 2).7-18 The following section is a brief overview highlighting specific characteristics of each of the public reporting systems.
Becker’s Hospital Review: 100 Great Hospitals in America Becker’s Hospital Review is a monthly trade publication created by Becker’s Healthcare that reaches approximately
Downloaded from ajm.sagepub.com by guest on November 15, 2015
Downloaded from ajm.sagepub.com by guest on November 15, 2015
3
List Regional/National Unclear
No Not provided
No
List
National
Nominations, U.S. News & World Report, Truven’s 100 Top Hospitals, Healthgrades, Magnet Recognition, Studer Group and Malcolm Baldrige National Quality Award Recipients7 Noa
Not provided
Composite score? Ranking/Rating/ List Geographic context Data sources
Provided and reproducible
No
CMS
Regional/National
Rating
Infections, readmissions, communication, scanning, complications11
Timely and effective care, readmissions, complications, deaths, medical imaging, patient experience, number of Medicare patients, Medicare payment10 No
Provided and reproducible11
No
CMS, state departments of health, AHA11
Regional/National
Rating
Yes (blobs)
Patients
Consumers, providers
To help patients compare hospitals11
Consumer Reports Hospital Ratings To provide “meaningful information about the comparative performance of accredited organizations to the public”12 Public 12
The Joint Commission
Provided but not reproducible
Yes
No Provided and reproducible
Not provided
Leapfrog Hospital Survey, AHRQ, CDC, AHA, and CMS Hospital Compare17
Regional
Rating
Yes (A-F)
Process/Structural, outcomes17
Employers, insurers, patients
To save lives by “reducing injuries, accidents, and preventable medical and medication errors”13
Leapfrog Hospital Safety Score
Yes
Regional/ Regional National CMS, MedPAR, The Joint 19 state Commission departments of health8
Patient safety Risk-adjusted mortality rates, goal compliance, National Quality complication rates across 33 Improvement goals performance, conditions and patient satisfaction, procedures patient mortality12 Yes (1, 3, and 5 No stars) Ratings Rating
Consumers8
“To help consumers evaluate and compare hospital performance”8
Healthgrades Hospital Ratings
Ranking
Yes (0-100)
Structure, process, outcomes18
Patients
To provide a tool to help patients find unusually skilled inpatient care15
US News Best Hospitals
Provided and reproducible
No
Provided and reproducible
Yes
Regional/ National MedPAR, HCRIS, AHA, MedPAR, 14 and CMS U.S. News Physician Surveys18
National
List
No
Clinical quality, efficiency, financial health, patient perception of care14
Hospital administrators
To identify “the U.S. hospitals with the best facility-wide performance”14
Truven 100 Top Hospitals
Abbreviations: AHA, American Hospital Association; AHRQ, Agency for Healthcare Research and Quality; CDC, Centers for Disease Control and Prevention; CMS, Centers for Medicare & Medicaid Services; HCRIS, Healthcare Cost Report Information System; MedPAR, Medicare Provider Analysis and Review. a Becker’s Hospital Review “100 Hospitals With Great Heart Programs” list published separately from “100 Great Hospitals in America.”
Specialty performance Transparency of methodology
No
Unclear
Domains of measure
Hospital executives, hospital’s surrounding community9 Financial viability and plant reinvestment, cost structure, charge structure, quality performance9
Health care industry leaders16
Intended audience
CMS Hospital Compare
To measure a “hospital’s To help patients decide value to [its] where to get health care, community”9 to “encourage hospitals to improve the quality of care”10
To recognize organizations that “are home to medical breakthroughs and act as anchors of health within their respective communities.”7
Cleverley + Associates Community Value Leadership Awards
Primary objective
Becker’s Hospital Review 100 Great Hospitals in America
Table 2. Hospital Rating System Overview.
4
American Journal of Medical Quality
18 500 health care leaders. Their annual “100 Great Hospitals in America” list is composed of community hospitals, academic medical centers, large flagship facilities, and small specialty hospitals.7 100 Great Hospitals in America identifies hospitals that are leaders in the health care industry. The 100 Great Hospitals list is presented alphabetically with no indication of rank or specific level of quality for the hospitals. Consequently, it is impossible to distinguish any difference in quality between the hospitals listed. The Becker’s Hospital Review editorial team accepts nominations for the list, consults other hospital rating systems, and “conducts research”7 in order to compose the list. No methodology is available to determine how each of these components factors in to the selection of the hospitals on the list, nor is there any indication of what is involved in the research conducted by the editorial team.
Cleverley + Associates Community Value Leadership Awards The Community Value Leadership Awards measure the value that hospitals provide to their communities. The awards are based on the Community Value Index (CVI), a metric developed by Cleverley + Associates in 2004. The index determines hospital value using 10 measures in 4 performance areas: financial viability and plant reinvestment, hospital cost structure, hospital charge structure, and hospital quality performance. Cleverley + Associates provides 2 distinct Community Value Leadership Awards: The Community Value 100 and the Community Value Five-Star. To determine award winners, hospitals are divided into one of 5 groups according to their teaching status and size. Hospitals that achieved the highest 20 CVI scores in their respective groups earned the Community Value 100 Award. Hospitals that achieved CVI scores within the top quintile (20%) of all hospitals within their group earned the Community Value Five-Star award. Although measuring hospitals’ value to their communities is a somewhat unique approach to hospital ratings, the award lists are presented alphabetically, with no distinction of relative ranking or quality. Additionally, CVI scores for the listed hospitals are not presented. Consequently, it is impossible for consumers to determine a hospital’s measured value from Cleverley + Associates’ awards. Finally, a lack of transparency in both methodology and source of data make it impossible to determine what exactly determines the CVI score.
CMS Hospital Compare Under the Hospital Quality Initiative, CMS’s Hospital Compare was developed in 2005 “to publicly report valid,
credible, and user-friendly information about the quality of care delivered in the nation’s hospitals.”19 It has since served as a data source for most other hospital quality public reporting organizations. Hospital Compare presents information about the quality of care for more than 4000 Medicare-certified hospitals. Hospital Compare allows patients to easily compare hospitals’ quality information side by side. Hospital Compare does not provide any composite scores. Because consumers are only able to compare hospital quality through dozens of clinical measures, the information presented may be too complex for the average consumer to understand. The measure scores are often presented in easy-to-understand ratings, such as “better than the U.S. national benchmark.” However, the measures themselves are clinically based and may require clinical knowledge to truly understand their meaning.
Consumer Reports Hospital Ratings Instead of focusing on hospital “quality” or “value,” Consumer Reports’ Hospital Ratings help patients compare hospital safety. Hospitals are given a score of 1 to 5 “blobs,” identical to Consumer Reports ratings in other industries. The rating methodology utilizes CMS Hospital Compare data for care outcomes, patient experience, and hospital practices. Additional data are retrieved from state departments of health when they are available. Consumer Reports’ use of blobs helps consumers easily understand level of quality. Not all state departments of health collect the particular measures that Consumer Reports attempts to use. Sometimes, this prevents hospitals from receiving scores. Furthermore, a subscription to Consumer Reports is required to view any of the hospital ratings, limiting patient access.
Healthgrades Hospital Ratings Healthgrades evaluates hospital quality based on 33 conditions or procedures. These measures are rated using a 5-star system to identify patient outcome quality. One star is given for a measure that is “worse than expected.”8 Three stars are given for a measure that is “as expected.” Five stars are given for a measure that is “better than expected.” Healthgrades also gives “Excellence Awards” for various categories of health care quality, such as patient safety, or superior outcomes in various specialties. Consumers can search Healthgrades’ rating Web site for hospitals by name or by geographic area. The search yields a list of hospitals fitting the search criteria, accompanied by their numbers of Healthgrades 5-Star Ratings, Healthgrades Quality Awards, and affiliated providers.
Downloaded from ajm.sagepub.com by guest on November 15, 2015
5
Hwang et al This is the only way to directly compare hospitals on one page because consumers may only access hospital reports one at a time.
The Joint Commission The Joint Commission awards accreditation for health care providers and certification for providers’ individual programs. The Joint Commission conducts site visits and collects federally reported data to determine hospital quality. Accreditation and certification status is provided for each evaluated hospital on the Joint Commission’s Web site, Quality Check. Consumers can compare hospitals side by side using Quality Check. Quality Check presents information only for accredited health care providers and their certified programs. Therefore, consumers are unable to compare nonaccredited hospitals or any noncertified programs. Also, no methodology is provided to consumers to demonstrate exactly how hospitals are evaluated by The Joint Commission. Although it is clear that The Joint Commission uses extensive means to evaluate hospitals, it is impossible to determine exactly how these means are used to measure quality. Furthermore, The Joint Commission does not explicitly define accreditation or certification. There is no publicly available description of what standards must be met to earn accreditation or certification statuses. This may cause some confusion or misunderstanding for consumers.
Leapfrog Hospital Safety Score Now in its third year of existence, Leapfrog’s Hospital Safety Score provides composite scores for more than 2500 acute care hospitals in the United States. The Hospital Safety Score is composed of 28 safety measures that demonstrate a hospital’s ability to minimize care mistakes with patients. Process, structural, and outcome measures are included in the score. Of the measures, 12 are contributed from Leapfrog’s Hospital Survey, an optional survey that hospitals may complete annually, free of charge. The safety score is calculated using established clinical measures only, instead of reputation surveys. The score’s methodology is also peer reviewed, the only hospital rating methodology the authors are aware of that has gone through such a process. Hospitals that do not participate in the Leapfrog Hospital Survey may receive lower scores than they could by participating.20 Leapfrog utilizes a different scoring methodology to account for 8 missing measures for nonparticipants. Although the Leapfrog Hospital Safety Score is one of the most robust and comprehensive rating systems available, its ability to give a hospital 2 different letter grades based on its participation in the Leapfrog Hospital Survey creates uncertainty as to the score’s fairness.
Truven 100 Top Hospitals Using only publicly available data, Truven identifies the 100 top short-term, acute-care, nonfederal hospitals in America using their Balanced Scorecard. The Balanced Scorecard consists of 10 measures distributed across 4 domains—quality, efficiency, finance, and consumer assessment of care. Hospitals are divided into 5 groups, based on their teaching status and size. The top 15 to 25 scoring hospitals are selected from each group to compose the 100 Top Hospitals. Because Truven uses only publicly available data and provides a detailed methodology on its Web site, the Balanced Scorecard can be reproduced by anyone, highlighting the system’s transparency. The 100 Top Hospitals includes the top hospitals from each group instead of the actual top 100 hospitals regardless of teaching status or size. Consequently, a hospital that is actually scored as a top 100 hospital overall may not be included in the list because it falls below the cutoff point within its respective group. Furthermore, the list does not provide any quantifiable measure or rank of hospital quality. The 100 Top Hospitals list is presented alphabetically, with no distinction of relative ranking or quality.
U.S. News & World Report’s Best Hospitals U.S. News & World Report’s Best Hospitals ranking is in its 24th year of existence. As one of the oldest hospital ranking systems, it has become arguably the most recognizable quality reporting system for patients. Rankings are provided for 16 specialties of care, making the rankings relevant to a very wide range of patients. An “Honor Roll” list of hospitals also is presented to identify the nation’s top hospitals across all specialties. The methodology relies heavily on hospitals’ reputation scores for specialty rankings (32.5%). The significant role of reputation scores has been criticized because it reduces the impact of well-established clinical measures.21 Although the Best Hospital list may be broken down into specialties, it only provides a list of the “best” hospitals, instead of providing ratings to all hospitals in the country. This means that many hospitals in a given geographic area will not have a ranking, possibly making the Best Hospitals list less relevant for patients wishing to compare unranked hospitals’ quality of care.
Ratings for U.S. News & World Report’s 18 Honor Roll Hospitals Table 3 presents the ratings for the 18 Honor Roll Hospitals across 8 of the public reporting organizations addressed in this study. Inconsistencies can be seen across the ratings for multiple hospitals. Brigham and Women’s earned Consumer Reports’s worst rating (5) but earned 11
Downloaded from ajm.sagepub.com by guest on November 15, 2015
6
Downloaded from ajm.sagepub.com by guest on November 15, 2015
a
Abbreviations: NL, not listed; NS, not scored. Consumer Reports Scores: 1 = best, 5 = worst.
Barnes-Jewish Brigham and Women’s Cedars-Sinai Cleveland Clinic Duke Hospital of the University of Pennsylvania Indiana University Health Johns Hopkins Massachusetts General Mayo Clinic New York-Presbyterian Northwestern Memorial NYU Langone Thomas Jefferson UCLA UCSF University Hospitals Case Medical Center UPMC (Presbyterian)
Listed Listed Listed Listed Listed Listed Listed Listed Listed Listed Listed Listed Listed Listed Listed Listed Listed Listed
Becker’s (2014) NL NL NL NL NL NL NL NL NL Listed NL NL NL NL NL NL NL NL
Cleverley (2013) 3 5 3 3 3 3 3 3 5 3 1 3 1 5 3 3 2 2
Consumer Reportsa (2013) 4 Five-star, 4 quality 11 Five-star, 5 quality 18 Five -star, 8 quality 6 Five-star, 4 quality 14 Five-star, 7 quality 4 Five-star, 1 quality 22 Five-star, 14 quality 3 Five-star, 1 quality 17 Five-star, 7 quality 9 Five-star, 4 quality 15 Five-star, 10 quality 19 Five-star, 11 quality 16 Five-star, 5 quality 20 Five-star, 9 quality 11 Five-star, 3 quality 2 Five-star, 3 quality 10 Five-star, 2 quality 5 Five-star, 1 quality
Healthgrades (2014)
Table 3. Ratings for U.S. News & World Report’s 18 Honor Roll Hospitals.
Hospital, nursing care center, behavioral health care Hospital, laboratory accreditation program Hospital, behavioral health care Hospital, home care Hospital Hospital, home care, laboratory accreditation program Hospital, behavioral health care, home care Hospital, behavioral health care Hospital, laboratory accreditation program Hospital, ambulatory care Hospital, behavioral health care, laboratory accreditation program Hospital Hospital Hospital Hospital, home care Hospital, home care, laboratory accreditation program Hospital Hospital, behavioral health care
The Joint Commission (2013)
C A B C A A C NS B A C A B B B B A B
Leapfrog (Fall 2013)
NL NL NL NL Listed NL NL NL NL NL NL NL NL NL NL NL NL NL
Truven (2014)
15 9 13 4 12 11 16 1 2 3 7 6 14 17 5 7 18 10
U.S. News (2013-2014)
7
Hwang et al five-star ratings by Healthgrades as well as Leapfrog’s highest grade (A). New York-Presbyterian earned Consumer Reports’s best rating (1), as well as 15 five-star measures and 10 quality awards by Healthgrades but earned a grade of C from Leapfrog. NYU Langone and Thomas Jefferson both earned Bs from Leapfrog but earned the best (1) and worst (5) ratings, respectively, from Consumer Reports. Consumer Reports assigned a rating of 3 to 11 of the 18 Honor Roll Hospitals. Out of these 11 hospitals, Leapfrog assigned 4 As, 3 Bs, and 3 Cs. Inclusion of hospitals on these lists also varied. Johns Hopkins was not scored by Leapfrog because Medicare does not publicly report data for Maryland hospitals. Only one Honor Roll hospital (Duke) was listed by Truven’s 100 Top Hospitals. All 18 hospitals were listed by Becker’s Hospital Review. Only one hospital (Mayo Clinic) was listed by Cleverley + Associates.
Discussion Both the public and private sectors have invested immense resources to collect data and measure hospital quality and patient safety. There is a tremendous potential to make this information useful for consumers to choose hospitals and for hospital executives to improve quality of health services. The current state of public reporting may not be providing much help to consumers or health care executives. This review of 9 important attributes across 9 major public reporting systems found that the systems focus on vastly different quality attributes when measuring hospital performance. This difference in performance measurement can lead to inconsistent and possibly conflicting conclusions. However, this report is not intended to evaluate the relative accuracy or validity of each public reporting system. This review of public reporting systems and examples of inconsistent hospital performance results demonstrate how public reporting may actually add confusion to patients’ health care decision making. Hospitals were often given conflicting ratings and rankings by public reporting organizations. Some hospitals received top marks from one system but low marks from another. Often, hospitals that shared the same rating from one rating organization received very different ratings from the other organizations. In addition to the public reporting systems included in this report, many states are now mandating that certain hospital quality measures be publicly reported. However, the requirements of these mandates vary across states.22,23 Different data submission requirements, methods of risk adjustment, time frames of data collection, and presentation of the data cause further confusion regarding evaluation of hospital quality and safety.
Much of the confusion caused by public reporting systems may be related in part to a lack of a minimum standard for public reporting. Standardized methods for developing quality measures are used by CMS24 and the Agency for Healthcare Research and Quality (AHRQ).25 Unfortunately, these methods only apply to developing individual measures. No standardized methods exist for developing composite scores that combine multiple quality measures. There is no commonly accepted guideline for reporting organizations to use to disseminate quality information. A standardized method for developing methodologies to measure hospital performance, similar to the CMS and AHRQ approaches for developing quality measures, will be of high value. The Association of American Medical Colleges already provides 3 broad principles for public reporting26; however, no specific standards have been developed or accepted yet. Without a commonly accepted minimum standard, public reporting will continue to cause confusion for patients. At the very least, public reporting organizations should more clearly explain the overall purpose of their systems to prevent confusion for patients. For example, Consumer Reports, U.S. News & World Report, and Leapfrog often present conflicting scores to hospitals. One hospital may receive 3 very different scores from these 3 organizations. This should not be a surprise because the 3 organizations measure different aspects of hospital performance. Consumer Reports focuses on patient safety for a few specific surgical procedures. U.S. News measures which hospitals give the most advanced, industry-leading care for the most complicated medical conditions. Leapfrog measures general hospital safety. However, it may be difficult for patients to discern which public reporting system is most appropriate to use for their specific needs. If public reporting systems provided more detailed explanations of their general purpose to users, much of the confusion caused by the conflicting ratings could be avoided. Moreover, many public reporting organizations generate revenue by allowing hospitals to use their various results, awards, certifications, and logos for marketing purposes. As a result, hospital performance information has become an important marketing tool for the health care institutions and media instead of being used as a guide for patients, families, and primary care physicians to select hospitals for needed care or referrals. This may explain why consumers have been shown to heavily rely on the Internet for diagnosis and treatment information but generally lack interest in quality measure reports.27 Current financial models, where public reporting systems generate revenue from their ratings, may raise conflict of interest concerns. Ideally, public reporting should move away from these models in the future to avoid potential conflict of interest.
Downloaded from ajm.sagepub.com by guest on November 15, 2015
8
American Journal of Medical Quality
The issue of multiple, conflicting reports of hospital performance is unlikely to change anytime soon. Therefore, it is imperative that patients and health care professionals understand how to use the systems that already exist. Each system uses its own unique methods to measure “quality” or “safety.” Patients and health care professionals must use the information and interpret with caution. The tables provided in this report can help identify which public reporting system is most appropriate for a given circumstance. Hospital quality ratings are a growing topic of importance in US health care. Further efforts must be made to ensure that the hospital ratings being reported to the public are as accurate and useful as possible. Future public reporting studies should examine how hospitals use and disseminate quality ratings. Recent work suggests that consumers may have little interest in aggregate data and may be more interested in quality and safety information associated with their personal provider.28 Developing a better understanding of the need for, and use of, quality rating systems is essential to guide the improvements of these systems. Declaration of Conflicting Interests The authors declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: Dr Paz was serving as chief executive officer of the Penn State Hershey Medical Center and Health System at the time this study was conducted.
Funding The authors received no financial support for the research, authorship, and/or publication of this article.
References 1. Leonardi MJ, McGory ML, Ko CY. Publicly available hospital comparison web sites: determination of useful, valid, and appropriate information for comparing surgical quality. Arch Surg. 2007;142:863-869. 2. The Leapfrog Group. The hidden surcharge Americans pay for hospital errors. 2013. http://www.leapfroggroup. org/employers_purchasers/HiddenSurchargeCalculator. Accessed March 25, 2014. 3. Halasyamani LK, Davis MM. Conflicting measures of hospital quality: ratings from “Hospital Compare” versus “Best Hospitals.” J Hosp Med. 2007;2:128-134. 4. Hollenbeak CS, Gorton CP, Tabak YP, Jones JL, Milstein A, Johannes RS. Reductions in mortality associated with intensive public reporting of hospital outcomes. Am J Med Qual. 2008;23:279-286. 5. Mazor KM, Dodd KS, Kunches L. Communicating hospital infection data to the public: a study of consumer responses and preferences. Am J Med Qual. 2009;24:108-115. 6. McGuckin M, Govednik J, Hyman D, Black B. Public reporting of health care-associated infection rates: are consumers aware and engaged? Am J Med Qual. 2014;29:83-85.
7. Gamble H, Herman B. 100 Great hospitals in America. 2013. http://www.beckershospitalreview.com/100-greathospitals-2013/100-great-hospitals-in-america-2013.html/. Accessed March 25, 2014. 8. Healthgrades. Mortality and complication outcomes: 2014 methodology. https://d2dcgio3q2u5fb.cloudfront.net/b7/ 4e/543c99af4ec399f4093b7c727cfe/2014-mortality-andcomplications-outcomes-methodology.pdf. Accessed March 25, 2014. 9. Cleverley + Associates. The Community Value Index. http://www.cleverleyassociates.com/CommunityValue/ AboutTheCVI/Default.aspx. Accessed March 25, 2014. 10. Centers for Medicare and Medicaid Services. What is Hospital Compare? http://www.medicare.gov/hospitalcompare/About/What-Is-HOS.html. Accessed March 25, 2014. 11. Consumer Reports Health. How we rate hospitals. http:// www.consumerreports.org/health/resources/pdf/how-werate-hospitals/How%20We%20Rate%20Hospitals.pdf. Accessed March 25, 2014. 12. The Joint Commission. Facts about quality check and quality reports. http://www.qualitycheck.org/help_qc_facts. aspx. Accessed March 25, 2014. 13. The Leapfrog Group. About the Leapfrog Group. http:// www.hospitalsafetyscore.org/about-the-score/about-theleapfrog-group. Accessed March 25, 2014. 14. Truven Health Analytics. 100 Top hospitals: study overview. http://100tophospitals.com/portals/2/assets/100-TopStudy-March-2014.pdf. Accessed March 25, 2014. 15. U.S. News & World Report. FAQ: How we ranked the best hospitals 2013-2014. http://health.usnews.com/healthnews/best-hospitals/articles/2013/07/16/how-we-rankedthe-best-hospitals-2013-14-an-faq. Accessed March 25, 2014. 16. Becker’s Hospital Review. About Becker’s Hospital Review and Becker’s Healthcare. http://www.beckershospitalreview.com/about-us.html. Accessed March 25, 2014. 17. The Leapfrog Group. Scoring methodology. http://www. hospitalsafetyscore.org/media/file/HospitalSafetyScore_ ScoringMethodology_Spring2014_Final.pdf. Accessed March 25, 2014. 18. Olmstead M, Murphy J, Geisen E, et al. Methodology: U.S. News & World Report best hospitals 2013-14. http:// www.usnews.com/pubfiles/BH_Methodology_Report.pdf. Accessed March 25, 2014. 19. Centers for Medicare & Medicaid Services. Hospital Quality Initiative: overview. https://www.cms.gov/Medicare/ Quality-Initiatives-Patient-Assessment-Instruments/ HospitalQualityInits/downloads/HospitalOverview.pdf. Accessed March 25, 2014. 20. Hwang W, Derk J, Laclair M, Paz H. Hospital patient safety grades may misrepresent hospital performance. J Hosp Med. 2014;9:111-115. 21. Sehgal AR. The role of reputation in U.S. News & World Report’s rankings of the top 50 American hospitals. Ann Intern Med. 2010;152:521-525. 22. Aswani MS, Reagan J, Jin L, Pronovost PJ, Goeschel C. Variation in public reporting of central line-associated
Downloaded from ajm.sagepub.com by guest on November 15, 2015
9
Hwang et al bloodstream infections by state. Am J Med Qual. 2011;26: 387-395. 23. Herzig CT, Reagan J, Pogorzelska-Maziarz M, Srinath D, Stone PW. State-mandated reporting of health careassociated infections in the United States: trends over time [published online June 20, 2014]. Am J Med Qual. doi:10.1177/1062860614540200. 24. Centers for Medicare & Medicaid Services. Quality initiatives: general information. http://www.cms.gov/Medicare/ Quality-Initiatives-Patient-Assessment-Instruments/ QualityInitiativesGenInfo/index.html. Accessed March 25, 2014. 25. Agency for Healthcare Research and Quality. Quality indicator measure development, implementation, maintenance and retirement: summary. http://qualityindicators.ahrq.gov/
Downloads/Resources/Publications/2011/QI_Measure_ Development_Implementation_Maintenance_Retirement_ Summary_05-03-11.pdf. Accessed March 25, 2014. 26. Association of American Medical Colleges. Guiding principles for public reporting of provider performance. https:// www.aamc.org/download/370236/data/guidingprinciplesforpublicreporting.pdf. Accessed March 25, 2014. 27. Marshall MN, Shekelle PG, Leatherman S, Brook RH. The public release of performance data: what do we expect to gain? A review of the evidence. JAMA. 2000;283:1866-1874. 28. Smith B, Lynch WD, Markow C, Lifsey S, Slover M. Consumers’ understanding of and interest in provider- versus practice-level quality characteristics: findings from a focus group study [published online April 22, 2014]. Am J Med Qual. doi:10.1177/1062860614531447.
Downloaded from ajm.sagepub.com by guest on November 15, 2015
