COMMENTARY
Marshaling the Evidence: The Prioritized Public Health Accreditation Research Agenda A national public health department accreditation program was recently developed and implemented by the Public Health Accreditation Board (PHAB) to improve the quality and performance of public health departments. Because of its potential to transform public health, it is critical that the evidence base around accreditation be strong. With input from public health practitioners and researchers, PHAB developed a research agenda that highlights priority questions related to barriers and facilitators to seeking and obtaining accreditation, the PHAB standards and review process, metrics to determine the impact of accreditation, and benefits and outcomes associated with accreditation for the departments that undergo the process. We present that agenda, discuss the potential challenges of conducting accreditation research, and call on researchers to build a greater base of evidence related to accreditation. (Am J Public Health. 2015;105: S153–S158. doi:10.2105/ AJPH.2014.302247)
Jessica Kronstadt, MPP, Leslie M. Beitsch, MD, JD, and Kaye Bender, RN, PhD
THE NATIONAL VOLUNTARY public health department accreditation program, which is administered by the Public Health Accreditation Board (PHAB), seeks to advance the quality and performance of tribal, state, local, and territorial public health departments.1,2 Because of its potential to transform public health, it is critical that the accreditation program has a firm grounding in evidence.3 From the outset, PHAB established a research and evaluation committee, demonstrating recognition of the critical need to promote accreditation-related research. After providing background information about PHAB, we present a public health accreditation research agenda and describe the process by which the agenda was developed. We then discuss potential challenges with conducting research on accreditation and related efforts to understand the impact of accreditation. We conclude with a call to the public health services and systems research (PHSSR) field to help build the evidence base related to accreditation.
BACKGROUND The 2003 Institute of Medicine report, The Future of the Public’s Health in the 21st Century, recommended that the public health field consider accreditation as a mechanism for strengthening the performance and accountability of public health departments.4 In response, the Robert Wood Johnson Foundation and the Centers for Disease Control and Prevention funded the Exploring
Supplement 2, 2015, Vol 105, No. S2 | American Journal of Public Health
Accreditation project, which determined that voluntary accreditation was both feasible and desirable. One of the Exploring Accreditation workgroups was devoted to research and evaluation,5 reflecting the importance of developing the evidence base to the accreditation endeavor. Following the model developed by Exploring Accreditation, PHAB was incorporated in May 2007 as a nonprofit organization to administer the national voluntary accreditation program. Standards were developed by a committee with broad representation across all sectors of public health, emphasizing practice. Input was solic ited from a series of think tanks, which convened practitioners, academicians, and other experts on specific topic areas to ensure that the measures were applied in the context of public health practice and reflected the evidence base when available and appropriate.6 In total, PHAB engaged more than 400 public health practitioners in developing the accreditation process and the initial standards and measures.7 The PHAB Standards and Measures are organized around 12 domains—10 reflect the essential public health services, and the remaining two describe health department roles related to management and administration and governance.8 Thirty health departments participated in a beta test, which was evaluated to identify improvements before the launch of the accreditation program in 2011.9 The first health departments were accredited in February 2013.
As of June 2014, there are five accredited state health departments, with an additional 20 actively pursuing accreditation. Of the approximately 2300 local health departments, 39 are nationally accredited and more than 220 are in progress of being accredited. Although the national accreditation program is relatively new, there is a growing body of relevant research,10---13 including an issue of the Journal of Public Health Management and Practice dedicated to public health accreditation.14 Further underscoring the importance of research on accreditation, a thematic area of the PHSSR research agenda focuses on “performance measurement, quality improvement, and accreditation.”15
DEVELOPING THE RESEARCH AGENDA The development of an accreditation research agenda has been an iterative process. A foundational step was the development of an initial logic model created by the Exploring Accreditation’s Research and Evaluation workgroup and informed by a literature review of accreditation programs in other fields.5 Joly et al. built upon this effort—incorporating research on performance management in public health and on accreditation in other sectors—and published a revised logic model in 2007 along with some initial research questions.16 That same workgroup recommended that the accrediting body, once it was established, create a research and evaluation committee and develop a research agenda to
Kronstadt et al. | Peer Reviewed | Commentary | S153
COMMENTARY
Research Questions for Public Health Department Accreditation Questions That Are Current Priorities 1. What are the barriers and facilitators to seeking and obtaining accreditation? a. Which incentives for health departments to participate in the voluntary national accreditation program are most effective for state, tribal, local, and territorial health departments? b. What are the barriers to health department participation in accreditation? Are the barriers to health department participation in accreditation real or perceived? c. What health department preparation activities improve readiness for accreditation and successfully achieving accreditation? d. What are the characteristics (knowledge, skills, or background) of a successful accreditation coordinator and accreditation team at a health department? e. What supports do state, tribal, and local health departments, as well as other entities, such as their governing bodies, need to prepare for accreditation? f. How can coordination of accreditation activities at the state and local levels be most effective (e.g., state and local participation at the same time, state participation before local or vice versa, full local statewide participation in a coordinated manner)? Does this differ based on governance structure or typology of the state? g. What are the costs of preparing for accreditation and participating in the accreditation assessment process? h. What impact does the existence of dedicated “core” or infrastructure funding have on the health department’s likelihood to achieve accreditation? i. What is the impact of grants targeted toward accreditation on health department operations and ability to obtain accreditation? j. What role do foundations and other nongovernmental sources play in supporting accreditation efforts by health departments? 2. Do the PHAB standards and the review process capture what is most meaningful for evaluating the performance of health departments? How could they be improved? a. Are there accreditation standards and measures that are missing? b. Are there accreditation standards and measures that could be omitted? c. What are appropriate metrics to study the criterion validity of the measures? d. How is the bar for determining whether accreditation has been achieved set most appropriately? e. Does the PHAB accreditation assessment and review process account sufficiently for contextual differences among health departments seeking accreditation (e.g., limited budget, different governance structures, or other factors outside health department control)? 3. What are the best metrics to determine the impact of accreditation? a. What are the most appropriate metrics for assessing the proximate and intermediate outcomes listed in the logic model? b. How can researchers isolate the impact of accreditation from the selection effect? c. What are the best methods to determine if the national accreditation program demonstrated value to the public and policymakers? d. What are the most appropriate methods to determine the contributions of a high-performing health department to improved health outcomes or reduced health disparities? 4. What are the benefits and outcomes associated with accreditation for the health departments that undergo the process? a. Does accreditation enhance progress in achieving the health department strategic plan? The state or community health improvement plan? b. Does preparation for accreditation encourage the adoption of evidence-based and best practices? c. Does accreditation increase the engagement in quality improvement or enhance quality improvement culture? How? d. How does accreditation affect the health department workforce? Does accreditation influence morale, productivity, workforce retention or satisfaction, achievement of competencies, and seeking advanced degrees? e. Do accredited health departments have greater accountability? f. Does accreditation influence the relationship between local health departments and other system partners, for example, governing entities? g. Do accredited health departments have greater community visibility as a result of accreditation? h. Does accreditation result in greater political support for accredited health departments? i. What impact does accreditation have on tribal health departments? j. Do accredited health departments strengthen their financial status? k. Does accreditation influence or change how health departments use their resources? l. Does accreditation influence progress in achieving Healthy People objectives, GPRA, or other federal benchmarks? m. To what extent does accreditation have value for programmatic areas within the health department? n. Are some essential services more affected by accreditation than others, and if so, why? o. Does accreditation catalyze changes in health department activities, responsibilities, or structure at the state and local level (e.g., regionalization or shift of responsibilities)? p. What are the unintended consequences of health department accreditation? (For example, does accreditation divert attention away from important health department activities?) Questions That Are Not Current Prioritiesa 5. What are the costs and benefits for health departments throughout the accreditation cycle? a. What are the costs for improving performance after accreditation? b. What are the costs associated with maintaining accreditation? c. Does accreditation benefit exceed cost? Continued
S154 | Commentary | Peer Reviewed | Kronstadt et al.
American Journal of Public Health | Supplement 2, 2015, Vol 105, No. S2
COMMENTARY
Continued d. Is performance maintained (or improved?) by accredited health departments between cycles of accreditation? e. Does maintenance of accreditation influence uptake of quality improvement within the health department? 6. How do accredited health departments differ from nonaccredited health departments?b a. What factors are the strongest predictors for being nationally accredited? For example, do differences in leadership, governance, structure, budget, and workforce influence accreditation? b. What characteristics of tribal health departments make them more or less likely to apply for or successfully achieve accreditation? c. Are jurisdictions that have engaged in broader systems initiatives (e.g., NPHPSP, MAPP, SHIP) better positioned to seek and attain accreditation? d. Are health departments that emphasize emergency preparedness and readiness (e.g., Project Public Health Ready) better positioned to seek and attain accreditation? e. Are health departments with well-developed quality improvement systems better prepared to seek and attain accreditation? f. Are local health departments more likely to seek accreditation if surrounding health departments are seeking it or have attained it? g. Are health departments that collaborate with schools of public health (e.g., Academic Health Department programs) more likely to achieve accreditation or meet certain standards? 7. How does accreditation affect the broader public health system, including health departments that do not apply for accreditation? a. Does accreditation help build the evidence base for quality improvement in public health departments? b. Does accreditation result in improved performance of health departments undertaking accreditation, or extend to all health departments in general? c. Does the adoption of national standards for accreditation result in funding changes for public health generally? Does this depend upon where the accreditation bar is set? d. Does accreditation promote or hinder the role of a health department in building a public health system? Does accreditation reinforce that building the public health system is an important role, or does it cause health department leadership and staff to focus efforts internally, away from systems building? e. Does accreditation increase the consistency of health departments’ activities nationally? Does consistency result in higher performing health departments? f. To what extent does accreditation have value for federal programmatic initiatives? 8. What impact, if any, does health department accreditation have on health outcomes? a. Are some health outcomes more readily influenced by health department accreditation? b. Are the improvements in health from accreditation ethically distributed across the population served by accredited health departments? c. If pursuit of accreditation leads to sharing of services between health departments to apply for accreditation, what is the impact on health outcomes and health services? 9. What factors affect the impact of accreditation and how does accreditation’s impact compare with the impact of other initiatives? a. What roles do practitioners, partners, consumers, and others play in influencing the accreditation process and its effect on health department performance? b. Is the impact of accreditation altered by whether it is voluntary or mandatory? c. Is the impact of accreditation altered by whether it is pursued statewide, and whether the state and locals, both or all, seek accreditation in a coordinated fashion? d. Are there any differences between outcomes achieved by accredited health departments versus health departments that do other types of performance improvement activities (e.g., National Public Health Performance Standards Program; Baldrige, etc.)? e. Is health department accreditation more effective than public health system accreditation? Note. GPRA = Government Performance and Results Act; MAPP = Mobilizing for Action through Planning and Partnerships; NPHPSP = National Public Health Performance Standards Program; PHAB = Public Health Accreditation Board; SHIP = state health improvement plan. Source. Public Health Accreditation Board, 2013.21 a These questions may be a higher priority in the future. Because of limited data and the amount of time that the accreditation program has been in place, these are not considered high priority by PHAB at the moment. b This question encompasses both the differences between health departments that apply and those that do not apply and the differences between health departments that apply and are accredited and those that apply and are not accredited.
examine issues related to the importance and value of a voluntary national accreditation program, as well as the desired outcomes as listed in the logic model.5(p39)
In accordance with that recommendation, the PHAB Board of Directors established the Research and Evaluation Committee (R&E Committee) shortly after the organization was incorporated in 2007. Composed of governmental public health practitioners and
researchers, the R&E Committee further refined the logic model and research questions. In 2010, a forum was held during an AcademyHealth conference to solicit suggestions concerning research questions that would be an appropriate focus for the PHSSR field in preparation for the launch of accreditation in 2011.17 A preliminary research agenda was subsequently presented during the 2011 Keeneland Conference, and
Supplement 2, 2015, Vol 105, No. S2 | American Journal of Public Health
participants provided feedback on the proposed research agenda. It was refined and published in 2012.18 As health departments began to apply for and were granted accreditation, PHAB recognized that many of the initial outputs described in the logic model had been achieved. Feedback from the field also indicated that a revision of the logic model and the research questions was needed.
Through its extensive network of committees and workgroups, PHAB learned that there are important aspects of the accreditation experience that were not captured in the earlier efforts to identify research questions. For example, the concept of a quality improvement culture needed to be incorporated into the research agenda to reflect the growing number of studies in this area.19,20 Committee members
Kronstadt et al. | Peer Reviewed | Commentary | S155
COMMENTARY
also indicated that the original list of research questions, which was organized using the same structure as the logic model (i.e., categorized around such concepts as “inputs” and “outputs”), was lengthy and occasionally redundant. Using an iterative process, the PHAB Research Advisory Council—formerly a subgroup of the R&E Committee— added, eliminated, re-categorized, and prioritized questions. The process was guided by the following criteria for selecting research questions that are currently high priority21:
Inputs • Organizational structure • Board, committees, and work groups • Staffing and expertise • Principles for standards, measures, and assessment process • Site visitors • Funders • Partners at national, state, regional, and local levels • Funding • Incentives • Technical Assistance • Researchers and research networks • Interest, buy-in, and commitment to seek accreditation • Appropriate stability, resources, and readiness to apply • Previous quality improvement and assessment experience
d
d
d
Data would likely be available in the near future; Questions are appropriate to study because of the number of health departments that have completed the process; and Questions have the potential to yield information PHAB can take action to address.
d
d
d
Based on those criteria, the following were identified as overarching questions that are current priorities: d
What are the barriers and facilitators to seeking and obtaining accreditation?
Strategies • Market program • Implement the 7 steps of accreditation • Train agencies and site visitors • Develop e-PHAB • Evaluate program and improve quality • Promote research • Promote national accreditation • Encourage agencies to meet national standards and seek accreditation • Support agencies through TA before, during, and after process • Conduct and disseminate research • Participate in training and TA • Assess readiness • Submit application and documentation • Host site visit • Review and share findings • Develop and implement improvement plan • Implement QI • Mentor other agencies • Participate in reaccreditation process
Outputs • Accreditation program: marketed, implemented, evaluated, and improved • e-PHAB developed and data captured • National consensus standards for public health agencies
• Communication efforts delivered • Technical assistance, trainings, and QI tools provided • Research conducted and disseminated
• Agencies are accredited • Report received and acted on • QI efforts are in place • Agencies are mentored • Plans for reaccreditation under way
Do the PHAB standards and the review process capture what is most meaningful for evaluating the performance of health departments? How can they be improved? What are the best metrics to determine the impact of accreditation? What are the benefits and outcomes associated with accreditation for the health departments that undergo the process? 21
The research agenda includes detailed questions pertaining to each of Proximate Outcomes
Strong, credible and sustainable accreditation program in place Standards adopted as performance measures Increased support for accreditation Increased use of benchmarks for evaluating performance Improved communication about public health
Enhanced internal and external collaboration Increased organizational accountability Increased knowledge of organizational strengths and weaknesses Increased awareness of importance of QI and a supportive culture
the preceding topics and a series of questions for exploration in future years. It was approved by the PHAB board of directors in December 2013 (see the box on pages e2---e3). At the same time, the board approved a revised logic model (Figure 1).
UNDERSTANDING THE IMPACT OF ACCREDITATION An important focus of the research agenda is gaining a clear picture of the impact of accreditation on health departments that are accredited, on the broader
Intermediate Outcomes Improved identification and use of evidencebased practices and policies
Increased consistency in practice Increased use of proven QI methods and tools resulting in improvements in practice Standards drive public health transformation Increased interagency and intersectoral collaboration Increased visibility and credibility of public health agencies Increased science base for public health practice
Public health agencies more effectively and efficiently use resources Strengthened organizational capacity and workforce Improved responsiveness to community priorities
Ultimate Outcomes Strengthened public health agencies and systems Improved conditions in which people can be healthy Improved community health indicators /reduced health disparities
Increased capacity for optimal investment in public health Increased public recognition of public health role and value Legend Accrediting Agency Individual Public Health Agencies Stakeholders and Partners Public Health Field
Note. PHAB = Public Health Accreditation Board; QI = quality improvement; TA = technical assistance. Source. Public Health Accreditation Board, Alexandria, VA, 2013.
FIGURE 1—Logic model for public health accreditation system.
S156 | Commentary | Peer Reviewed | Kronstadt et al.
American Journal of Public Health | Supplement 2, 2015, Vol 105, No. S2
COMMENTARY
public health field, and on the communities served by health departments. In addition to encouraging public health services and systems researchers to conduct studies to answer the questions posed in the research agenda, PHAB is undertaking evaluation activities. To supplement the process feedback that PHAB collects systematically from participating health departments and site visitors, PHAB contracted with the National Opinion Research Center at the University of Chicago to conduct a three-year evaluation of the accreditation program. Guided by an evaluation plan that was reviewed by the PHAB Research and Evaluation Committee, this evaluation is both formative and summative in nature. It will gather information on the extent to which accredited health departments have achieved the benefits listed as proximate outcomes on the logic model. In a parallel effort, PHAB will more systematically document the evidence base around their Standards and Measures. Working with its Research Advisory Council, PHAB is developing a plan to track, on an ongoing basis, research relevant to the standards and to extract the salient points from those studies. This effort will help inform future revisions of the standards. For example, if research provides greater clarity on effective public health practice, the PHAB committee responsible for revising the standards may recommend augmenting or altering the guidance for a particular measure. This research will also provide insights about the causal links for how accreditation can strengthen health department performance and may contribute to improved health outcomes. In other words, if evidence suggests that a health department that can demonstrate a particular capacity
—incorporated in the standards and measures—is more likely to effectively promote the health in its community, then greater uptake of that capacity may be one of the ways in which accreditation can strengthen health department performance. Although the study of accreditation’s impact is important, it will be challenging. As Mays noted, “We must insist on patience and caution in pursuing reliable evidence on this topic.”3(p151) Selection bias will pose a significant challenge, because of the voluntary nature of the accreditation program. Researchers must also be attentive to a potential exposure effect—even health departments that are not accredited may benefit from the accreditation program if the national standards become commonly accepted. With regard to understanding the impact of accreditation on the health of the public, such outcomes will have long-term implications. Because of the many determinants of health, disentangling the role of the health department in contributing to population health will be challenging; attributing those benefits to accreditation will prove even more complicated. In light of this, one of the research agenda questions identified as high priority is related to developing the methodologies to tackle these issues once more health departments have been accredited for a longer period of time and their data are available.
A CALL TO RESEARCHERS Already, national public health department accreditation is beginning to deliver on its promise to strengthen health department performance. Case studies from accredited health departments and those going through the process illustrate accomplishments
Supplement 2, 2015, Vol 105, No. S2 | American Journal of Public Health
related to enhanced internal and external collaboration, stronger quality improvement culture, increased accountability, development of new health department policies, and identification of—and actions to address—areas for improvement.22 Robust studies to address the research agenda are needed to connect the dots from these individual stories of success and to more fully understand the impact of accreditation. In addition to augmenting our understanding of the benefits of accreditation, the research agenda is designed to capture key information about the accreditation process as experienced by health departments. By understanding the barriers and facilitators to seeking and obtaining accreditation, national-, regional-, and state-based resources can be leveraged to address priorities. Findings about the standards and measures and the process will continue to inform PHAB as it systematically works to improve and strengthen the accreditation program over time. Feedback from peer site visitors that applicants receive about their conformity with the standards can support continuous quality improvement in those health depart ments. To ensure that the standards continue to foster improved public health department performance, the standards will evolve with the growing evidence base. By establishing a national, consensus set of standards that matter and have an impact on the health of the community, accreditation has the potential to transform the way public health is delivered in this country. Research will play a critical role shaping how accreditation meets that potential. j
About the Authors Jessica Kronstadt and Kaye Bender are with the Public Health Accreditation Board,
Alexandria, VA. Leslie M. Beitsch is with the Department of Behavioral Sciences and Social Medicine, Center for Medicine and Public Health, Florida State University College of Medicine, Tallahassee. Correspondence should be sent to Jessica Kronstadt, MPP, Director of Research and Evaluation, Public Health Accreditation Board, 1600 Duke Street, Suite 200, Alexandria, VA 22314 (e-mail: jkronstadt@ phaboard.org). Reprints can be ordered at http://www.ajph.org by clicking the “Reprints” link. This article was accepted August 7, 2014.
Contributors J. Kronstadt was involved in revising the logic model and research agenda; she took the lead in writing the article. L. M. Beitsch was involved in developing and revising the logic model and research agenda; he contributed to writing and revising the article. K. Bender was involved in developing and revising the logic model and research agenda; she contributed to writing and revising the article.
Acknowledgments This project was supported by funding from the Centers for Disease Control and Prevention and the Robert Wood Johnson Foundation. L. M. Beitsch and K. Bender are on the Board of Directors of the Public Health Accreditation Board. Note. The contents of this article are solely the responsibility of the authors and do not necessarily represent the official views of the Centers for Disease Control and Prevention or the Robert Wood Johnson Foundation. The information contained in this article reflects the opinions of the authors and does not necessarily represent official Public Health Accreditation Board policy.
Human Participant Protection Human participants were not involved in research for this article, so institutional review board approval was not needed.
References 1. Public Health Accreditation Board Web site. Available at: http://www. phaboard.org. Accessed March 31, 2014. 2. Beitsch LM, Riley W, Bender K. Embedding quality improvement into accreditation: evolving from theory to practice. J Public Health Manag Pract. 2014;20(1):57---60. 3. Mays GP. Untangling desirable and undesirable variation in public health practice: accreditation and research working together. J Public Health Manag Pract. 2014;20(1):149---151.
Kronstadt et al. | Peer Reviewed | Commentary | S157
COMMENTARY
4. Institute of Medicine. The Future of the Public’s Health in the 21st Century. Washington, DC: National Academies Press; 2003. 5. Exploring Accreditation Planning Committee. Final Recommendations for a Voluntary National Accreditation Program for State and Local Public Health Departments. Exploring Accreditation Proj ect Report. Washington, DC: Exploring Accreditation; 2006. 6. Ingram RC, Bender K, Wilcox R, Kronstadt J. A consensus-based approach to national public health accreditation. J Public Health Manag Pract. 2014;20 (1):9---13. 7. Bender K, Kronstadt J, Wilcox R, Lee TP. Overview of the Public Health Accreditation Board. J Public Health Manag Pract. 2014;20(1):4---6. 8. Public Health Accreditation Board. Public Health Accreditation Board Standards and Measures, Version 1.0. Alexandria, VA: Public Health Accreditation Board; 2011. 9. NORC at the University of Chicago. Brief report: evaluation of the Public Health Accreditation Board beta test. 2011. Available at: http://www. phaboard.org/wp-content/uploads/ EvaluationofthePHABBetaTest BriefReportAugust2011.pdf. Accessed April 14, 2014. 10. Davis MV, Cannon MM, Corso L, Lenaway D, Baker E. Incentives to encourage participation in the national public health accreditation model: a systematic investigation. Am J Public Health. 2009;99(9):1705---1711. 11. Erwin PC, Knox County Tennessee Work Group on Accreditation. A selfassessment process for accreditation preparedness: a practical example for local health departments. J Public Health Manag Pract. 2009;15(6):503---508.
Foundation, and the National Coordinating Center for Public Health Services and Systems Research. A national research agenda for public health services and systems. Am J Prev Med. 2012;42(5 suppl 1):S72---S78. 16. Joly BM, Polyak G, Davis MV, et al. Linking accreditation and public health outcomes: a logic model approach. J Public Health Manag Pract. 2007;13(4):349--356. 17. The Public Health Systems & Services Research Community Informs the Research Agenda for Accreditation. Washington, DC: AcademyHealth, 2010. Available at: http://www.academyhealth. org/Training/ResourceDetail.cfm? ItemNumber=5358. Accessed April 15, 2014. 18. Riley WJ, Lownik EM, Scutchfield FD, Mays GP, Corso LC, Beitsch LM. Public health department accreditation: setting the research agenda. Am J Prev Med. 2012;42(3):263---271. 19. Beitsch LM, Rider NL, Joly BM, Leep C, Polyak G. Driving a public health culture of quality: how far down the highway have local health departments traveled? J Public Health Manag Pract. 2013;19(6):569---574. 20. Davis MV, Mahanna E, Joly B, et al. Creating quality improvement culture in public health agencies. Am J Public Health. 2014;104(1):e98---e104. 21. Public Health Accreditation Board. Research questions for accreditation. 2013. Available at: http://www. phaboard.org/wp-content/uploads/ Research-agenda-December2013.pdf. Accessed April 1, 2014. 22. Joly B, Davis M. Introduction to case reports: one goal, many journeys. J Public Health Manag Pract. 2014;20(1):64---65.
12. Conley AM, Vagi S, Horney JA. Use of the community assessment for public health emergency response to conduct community health assessments for public health accreditation. J Public Health Manag Pract. 2014;20(5):490---497. 13. Samoff E, Macdonald PD, Fangman MT, Waller AE. Local surveillance practice evaluation in North Carolina and value of new national accreditation measures. J Public Health Manag Pract. 2013;19(2):146---152. 14. Beitsch LM, Corso LC, Davis MV, Joly BM, Kronstadt J, Riley WJ. Transforming public health practice through accreditation (a user guide for the special accreditation issue). J Public Health Manag Pract. 2014;20(1)2---3. 15. A Consortium from Altarum Institute, the Centers for Disease Control and Prevention, the Robert Wood Johnson
S158 | Commentary | Peer Reviewed | Kronstadt et al.
American Journal of Public Health | Supplement 2, 2015, Vol 105, No. S2
Marshaling the Evidence: The Prioritized Public Health Accreditation Research Agenda A national public health department accreditation program was recently developed and implemented by the Public Health Accreditation Board (PHAB) to improve the quality and performance of public health departments. Because of its potential to transform public health, it is critical that the evidence base around accreditation be strong. With input from public health practitioners and researchers, PHAB developed a research agenda that highlights priority questions related to barriers and facilitators to seeking and obtaining accreditation, the PHAB standards and review process, metrics to determine the impact of accreditation, and benefits and outcomes associated with accreditation for the departments that undergo the process. We present that agenda, discuss the potential challenges of conducting accreditation research, and call on researchers to build a greater base of evidence related to accreditation. (Am J Public Health. 2015;105: S153–S158. doi:10.2105/ AJPH.2014.302247)
Jessica Kronstadt, MPP, Leslie M. Beitsch, MD, JD, and Kaye Bender, RN, PhD
THE NATIONAL VOLUNTARY public health department accreditation program, which is administered by the Public Health Accreditation Board (PHAB), seeks to advance the quality and performance of tribal, state, local, and territorial public health departments.1,2 Because of its potential to transform public health, it is critical that the accreditation program has a firm grounding in evidence.3 From the outset, PHAB established a research and evaluation committee, demonstrating recognition of the critical need to promote accreditation-related research. After providing background information about PHAB, we present a public health accreditation research agenda and describe the process by which the agenda was developed. We then discuss potential challenges with conducting research on accreditation and related efforts to understand the impact of accreditation. We conclude with a call to the public health services and systems research (PHSSR) field to help build the evidence base related to accreditation.
BACKGROUND The 2003 Institute of Medicine report, The Future of the Public’s Health in the 21st Century, recommended that the public health field consider accreditation as a mechanism for strengthening the performance and accountability of public health departments.4 In response, the Robert Wood Johnson Foundation and the Centers for Disease Control and Prevention funded the Exploring
Supplement 2, 2015, Vol 105, No. S2 | American Journal of Public Health
Accreditation project, which determined that voluntary accreditation was both feasible and desirable. One of the Exploring Accreditation workgroups was devoted to research and evaluation,5 reflecting the importance of developing the evidence base to the accreditation endeavor. Following the model developed by Exploring Accreditation, PHAB was incorporated in May 2007 as a nonprofit organization to administer the national voluntary accreditation program. Standards were developed by a committee with broad representation across all sectors of public health, emphasizing practice. Input was solic ited from a series of think tanks, which convened practitioners, academicians, and other experts on specific topic areas to ensure that the measures were applied in the context of public health practice and reflected the evidence base when available and appropriate.6 In total, PHAB engaged more than 400 public health practitioners in developing the accreditation process and the initial standards and measures.7 The PHAB Standards and Measures are organized around 12 domains—10 reflect the essential public health services, and the remaining two describe health department roles related to management and administration and governance.8 Thirty health departments participated in a beta test, which was evaluated to identify improvements before the launch of the accreditation program in 2011.9 The first health departments were accredited in February 2013.
As of June 2014, there are five accredited state health departments, with an additional 20 actively pursuing accreditation. Of the approximately 2300 local health departments, 39 are nationally accredited and more than 220 are in progress of being accredited. Although the national accreditation program is relatively new, there is a growing body of relevant research,10---13 including an issue of the Journal of Public Health Management and Practice dedicated to public health accreditation.14 Further underscoring the importance of research on accreditation, a thematic area of the PHSSR research agenda focuses on “performance measurement, quality improvement, and accreditation.”15
DEVELOPING THE RESEARCH AGENDA The development of an accreditation research agenda has been an iterative process. A foundational step was the development of an initial logic model created by the Exploring Accreditation’s Research and Evaluation workgroup and informed by a literature review of accreditation programs in other fields.5 Joly et al. built upon this effort—incorporating research on performance management in public health and on accreditation in other sectors—and published a revised logic model in 2007 along with some initial research questions.16 That same workgroup recommended that the accrediting body, once it was established, create a research and evaluation committee and develop a research agenda to
Kronstadt et al. | Peer Reviewed | Commentary | S153
COMMENTARY
Research Questions for Public Health Department Accreditation Questions That Are Current Priorities 1. What are the barriers and facilitators to seeking and obtaining accreditation? a. Which incentives for health departments to participate in the voluntary national accreditation program are most effective for state, tribal, local, and territorial health departments? b. What are the barriers to health department participation in accreditation? Are the barriers to health department participation in accreditation real or perceived? c. What health department preparation activities improve readiness for accreditation and successfully achieving accreditation? d. What are the characteristics (knowledge, skills, or background) of a successful accreditation coordinator and accreditation team at a health department? e. What supports do state, tribal, and local health departments, as well as other entities, such as their governing bodies, need to prepare for accreditation? f. How can coordination of accreditation activities at the state and local levels be most effective (e.g., state and local participation at the same time, state participation before local or vice versa, full local statewide participation in a coordinated manner)? Does this differ based on governance structure or typology of the state? g. What are the costs of preparing for accreditation and participating in the accreditation assessment process? h. What impact does the existence of dedicated “core” or infrastructure funding have on the health department’s likelihood to achieve accreditation? i. What is the impact of grants targeted toward accreditation on health department operations and ability to obtain accreditation? j. What role do foundations and other nongovernmental sources play in supporting accreditation efforts by health departments? 2. Do the PHAB standards and the review process capture what is most meaningful for evaluating the performance of health departments? How could they be improved? a. Are there accreditation standards and measures that are missing? b. Are there accreditation standards and measures that could be omitted? c. What are appropriate metrics to study the criterion validity of the measures? d. How is the bar for determining whether accreditation has been achieved set most appropriately? e. Does the PHAB accreditation assessment and review process account sufficiently for contextual differences among health departments seeking accreditation (e.g., limited budget, different governance structures, or other factors outside health department control)? 3. What are the best metrics to determine the impact of accreditation? a. What are the most appropriate metrics for assessing the proximate and intermediate outcomes listed in the logic model? b. How can researchers isolate the impact of accreditation from the selection effect? c. What are the best methods to determine if the national accreditation program demonstrated value to the public and policymakers? d. What are the most appropriate methods to determine the contributions of a high-performing health department to improved health outcomes or reduced health disparities? 4. What are the benefits and outcomes associated with accreditation for the health departments that undergo the process? a. Does accreditation enhance progress in achieving the health department strategic plan? The state or community health improvement plan? b. Does preparation for accreditation encourage the adoption of evidence-based and best practices? c. Does accreditation increase the engagement in quality improvement or enhance quality improvement culture? How? d. How does accreditation affect the health department workforce? Does accreditation influence morale, productivity, workforce retention or satisfaction, achievement of competencies, and seeking advanced degrees? e. Do accredited health departments have greater accountability? f. Does accreditation influence the relationship between local health departments and other system partners, for example, governing entities? g. Do accredited health departments have greater community visibility as a result of accreditation? h. Does accreditation result in greater political support for accredited health departments? i. What impact does accreditation have on tribal health departments? j. Do accredited health departments strengthen their financial status? k. Does accreditation influence or change how health departments use their resources? l. Does accreditation influence progress in achieving Healthy People objectives, GPRA, or other federal benchmarks? m. To what extent does accreditation have value for programmatic areas within the health department? n. Are some essential services more affected by accreditation than others, and if so, why? o. Does accreditation catalyze changes in health department activities, responsibilities, or structure at the state and local level (e.g., regionalization or shift of responsibilities)? p. What are the unintended consequences of health department accreditation? (For example, does accreditation divert attention away from important health department activities?) Questions That Are Not Current Prioritiesa 5. What are the costs and benefits for health departments throughout the accreditation cycle? a. What are the costs for improving performance after accreditation? b. What are the costs associated with maintaining accreditation? c. Does accreditation benefit exceed cost? Continued
S154 | Commentary | Peer Reviewed | Kronstadt et al.
American Journal of Public Health | Supplement 2, 2015, Vol 105, No. S2
COMMENTARY
Continued d. Is performance maintained (or improved?) by accredited health departments between cycles of accreditation? e. Does maintenance of accreditation influence uptake of quality improvement within the health department? 6. How do accredited health departments differ from nonaccredited health departments?b a. What factors are the strongest predictors for being nationally accredited? For example, do differences in leadership, governance, structure, budget, and workforce influence accreditation? b. What characteristics of tribal health departments make them more or less likely to apply for or successfully achieve accreditation? c. Are jurisdictions that have engaged in broader systems initiatives (e.g., NPHPSP, MAPP, SHIP) better positioned to seek and attain accreditation? d. Are health departments that emphasize emergency preparedness and readiness (e.g., Project Public Health Ready) better positioned to seek and attain accreditation? e. Are health departments with well-developed quality improvement systems better prepared to seek and attain accreditation? f. Are local health departments more likely to seek accreditation if surrounding health departments are seeking it or have attained it? g. Are health departments that collaborate with schools of public health (e.g., Academic Health Department programs) more likely to achieve accreditation or meet certain standards? 7. How does accreditation affect the broader public health system, including health departments that do not apply for accreditation? a. Does accreditation help build the evidence base for quality improvement in public health departments? b. Does accreditation result in improved performance of health departments undertaking accreditation, or extend to all health departments in general? c. Does the adoption of national standards for accreditation result in funding changes for public health generally? Does this depend upon where the accreditation bar is set? d. Does accreditation promote or hinder the role of a health department in building a public health system? Does accreditation reinforce that building the public health system is an important role, or does it cause health department leadership and staff to focus efforts internally, away from systems building? e. Does accreditation increase the consistency of health departments’ activities nationally? Does consistency result in higher performing health departments? f. To what extent does accreditation have value for federal programmatic initiatives? 8. What impact, if any, does health department accreditation have on health outcomes? a. Are some health outcomes more readily influenced by health department accreditation? b. Are the improvements in health from accreditation ethically distributed across the population served by accredited health departments? c. If pursuit of accreditation leads to sharing of services between health departments to apply for accreditation, what is the impact on health outcomes and health services? 9. What factors affect the impact of accreditation and how does accreditation’s impact compare with the impact of other initiatives? a. What roles do practitioners, partners, consumers, and others play in influencing the accreditation process and its effect on health department performance? b. Is the impact of accreditation altered by whether it is voluntary or mandatory? c. Is the impact of accreditation altered by whether it is pursued statewide, and whether the state and locals, both or all, seek accreditation in a coordinated fashion? d. Are there any differences between outcomes achieved by accredited health departments versus health departments that do other types of performance improvement activities (e.g., National Public Health Performance Standards Program; Baldrige, etc.)? e. Is health department accreditation more effective than public health system accreditation? Note. GPRA = Government Performance and Results Act; MAPP = Mobilizing for Action through Planning and Partnerships; NPHPSP = National Public Health Performance Standards Program; PHAB = Public Health Accreditation Board; SHIP = state health improvement plan. Source. Public Health Accreditation Board, 2013.21 a These questions may be a higher priority in the future. Because of limited data and the amount of time that the accreditation program has been in place, these are not considered high priority by PHAB at the moment. b This question encompasses both the differences between health departments that apply and those that do not apply and the differences between health departments that apply and are accredited and those that apply and are not accredited.
examine issues related to the importance and value of a voluntary national accreditation program, as well as the desired outcomes as listed in the logic model.5(p39)
In accordance with that recommendation, the PHAB Board of Directors established the Research and Evaluation Committee (R&E Committee) shortly after the organization was incorporated in 2007. Composed of governmental public health practitioners and
researchers, the R&E Committee further refined the logic model and research questions. In 2010, a forum was held during an AcademyHealth conference to solicit suggestions concerning research questions that would be an appropriate focus for the PHSSR field in preparation for the launch of accreditation in 2011.17 A preliminary research agenda was subsequently presented during the 2011 Keeneland Conference, and
Supplement 2, 2015, Vol 105, No. S2 | American Journal of Public Health
participants provided feedback on the proposed research agenda. It was refined and published in 2012.18 As health departments began to apply for and were granted accreditation, PHAB recognized that many of the initial outputs described in the logic model had been achieved. Feedback from the field also indicated that a revision of the logic model and the research questions was needed.
Through its extensive network of committees and workgroups, PHAB learned that there are important aspects of the accreditation experience that were not captured in the earlier efforts to identify research questions. For example, the concept of a quality improvement culture needed to be incorporated into the research agenda to reflect the growing number of studies in this area.19,20 Committee members
Kronstadt et al. | Peer Reviewed | Commentary | S155
COMMENTARY
also indicated that the original list of research questions, which was organized using the same structure as the logic model (i.e., categorized around such concepts as “inputs” and “outputs”), was lengthy and occasionally redundant. Using an iterative process, the PHAB Research Advisory Council—formerly a subgroup of the R&E Committee— added, eliminated, re-categorized, and prioritized questions. The process was guided by the following criteria for selecting research questions that are currently high priority21:
Inputs • Organizational structure • Board, committees, and work groups • Staffing and expertise • Principles for standards, measures, and assessment process • Site visitors • Funders • Partners at national, state, regional, and local levels • Funding • Incentives • Technical Assistance • Researchers and research networks • Interest, buy-in, and commitment to seek accreditation • Appropriate stability, resources, and readiness to apply • Previous quality improvement and assessment experience
d
d
d
Data would likely be available in the near future; Questions are appropriate to study because of the number of health departments that have completed the process; and Questions have the potential to yield information PHAB can take action to address.
d
d
d
Based on those criteria, the following were identified as overarching questions that are current priorities: d
What are the barriers and facilitators to seeking and obtaining accreditation?
Strategies • Market program • Implement the 7 steps of accreditation • Train agencies and site visitors • Develop e-PHAB • Evaluate program and improve quality • Promote research • Promote national accreditation • Encourage agencies to meet national standards and seek accreditation • Support agencies through TA before, during, and after process • Conduct and disseminate research • Participate in training and TA • Assess readiness • Submit application and documentation • Host site visit • Review and share findings • Develop and implement improvement plan • Implement QI • Mentor other agencies • Participate in reaccreditation process
Outputs • Accreditation program: marketed, implemented, evaluated, and improved • e-PHAB developed and data captured • National consensus standards for public health agencies
• Communication efforts delivered • Technical assistance, trainings, and QI tools provided • Research conducted and disseminated
• Agencies are accredited • Report received and acted on • QI efforts are in place • Agencies are mentored • Plans for reaccreditation under way
Do the PHAB standards and the review process capture what is most meaningful for evaluating the performance of health departments? How can they be improved? What are the best metrics to determine the impact of accreditation? What are the benefits and outcomes associated with accreditation for the health departments that undergo the process? 21
The research agenda includes detailed questions pertaining to each of Proximate Outcomes
Strong, credible and sustainable accreditation program in place Standards adopted as performance measures Increased support for accreditation Increased use of benchmarks for evaluating performance Improved communication about public health
Enhanced internal and external collaboration Increased organizational accountability Increased knowledge of organizational strengths and weaknesses Increased awareness of importance of QI and a supportive culture
the preceding topics and a series of questions for exploration in future years. It was approved by the PHAB board of directors in December 2013 (see the box on pages e2---e3). At the same time, the board approved a revised logic model (Figure 1).
UNDERSTANDING THE IMPACT OF ACCREDITATION An important focus of the research agenda is gaining a clear picture of the impact of accreditation on health departments that are accredited, on the broader
Intermediate Outcomes Improved identification and use of evidencebased practices and policies
Increased consistency in practice Increased use of proven QI methods and tools resulting in improvements in practice Standards drive public health transformation Increased interagency and intersectoral collaboration Increased visibility and credibility of public health agencies Increased science base for public health practice
Public health agencies more effectively and efficiently use resources Strengthened organizational capacity and workforce Improved responsiveness to community priorities
Ultimate Outcomes Strengthened public health agencies and systems Improved conditions in which people can be healthy Improved community health indicators /reduced health disparities
Increased capacity for optimal investment in public health Increased public recognition of public health role and value Legend Accrediting Agency Individual Public Health Agencies Stakeholders and Partners Public Health Field
Note. PHAB = Public Health Accreditation Board; QI = quality improvement; TA = technical assistance. Source. Public Health Accreditation Board, Alexandria, VA, 2013.
FIGURE 1—Logic model for public health accreditation system.
S156 | Commentary | Peer Reviewed | Kronstadt et al.
American Journal of Public Health | Supplement 2, 2015, Vol 105, No. S2
COMMENTARY
public health field, and on the communities served by health departments. In addition to encouraging public health services and systems researchers to conduct studies to answer the questions posed in the research agenda, PHAB is undertaking evaluation activities. To supplement the process feedback that PHAB collects systematically from participating health departments and site visitors, PHAB contracted with the National Opinion Research Center at the University of Chicago to conduct a three-year evaluation of the accreditation program. Guided by an evaluation plan that was reviewed by the PHAB Research and Evaluation Committee, this evaluation is both formative and summative in nature. It will gather information on the extent to which accredited health departments have achieved the benefits listed as proximate outcomes on the logic model. In a parallel effort, PHAB will more systematically document the evidence base around their Standards and Measures. Working with its Research Advisory Council, PHAB is developing a plan to track, on an ongoing basis, research relevant to the standards and to extract the salient points from those studies. This effort will help inform future revisions of the standards. For example, if research provides greater clarity on effective public health practice, the PHAB committee responsible for revising the standards may recommend augmenting or altering the guidance for a particular measure. This research will also provide insights about the causal links for how accreditation can strengthen health department performance and may contribute to improved health outcomes. In other words, if evidence suggests that a health department that can demonstrate a particular capacity
—incorporated in the standards and measures—is more likely to effectively promote the health in its community, then greater uptake of that capacity may be one of the ways in which accreditation can strengthen health department performance. Although the study of accreditation’s impact is important, it will be challenging. As Mays noted, “We must insist on patience and caution in pursuing reliable evidence on this topic.”3(p151) Selection bias will pose a significant challenge, because of the voluntary nature of the accreditation program. Researchers must also be attentive to a potential exposure effect—even health departments that are not accredited may benefit from the accreditation program if the national standards become commonly accepted. With regard to understanding the impact of accreditation on the health of the public, such outcomes will have long-term implications. Because of the many determinants of health, disentangling the role of the health department in contributing to population health will be challenging; attributing those benefits to accreditation will prove even more complicated. In light of this, one of the research agenda questions identified as high priority is related to developing the methodologies to tackle these issues once more health departments have been accredited for a longer period of time and their data are available.
A CALL TO RESEARCHERS Already, national public health department accreditation is beginning to deliver on its promise to strengthen health department performance. Case studies from accredited health departments and those going through the process illustrate accomplishments
Supplement 2, 2015, Vol 105, No. S2 | American Journal of Public Health
related to enhanced internal and external collaboration, stronger quality improvement culture, increased accountability, development of new health department policies, and identification of—and actions to address—areas for improvement.22 Robust studies to address the research agenda are needed to connect the dots from these individual stories of success and to more fully understand the impact of accreditation. In addition to augmenting our understanding of the benefits of accreditation, the research agenda is designed to capture key information about the accreditation process as experienced by health departments. By understanding the barriers and facilitators to seeking and obtaining accreditation, national-, regional-, and state-based resources can be leveraged to address priorities. Findings about the standards and measures and the process will continue to inform PHAB as it systematically works to improve and strengthen the accreditation program over time. Feedback from peer site visitors that applicants receive about their conformity with the standards can support continuous quality improvement in those health depart ments. To ensure that the standards continue to foster improved public health department performance, the standards will evolve with the growing evidence base. By establishing a national, consensus set of standards that matter and have an impact on the health of the community, accreditation has the potential to transform the way public health is delivered in this country. Research will play a critical role shaping how accreditation meets that potential. j
About the Authors Jessica Kronstadt and Kaye Bender are with the Public Health Accreditation Board,
Alexandria, VA. Leslie M. Beitsch is with the Department of Behavioral Sciences and Social Medicine, Center for Medicine and Public Health, Florida State University College of Medicine, Tallahassee. Correspondence should be sent to Jessica Kronstadt, MPP, Director of Research and Evaluation, Public Health Accreditation Board, 1600 Duke Street, Suite 200, Alexandria, VA 22314 (e-mail: jkronstadt@ phaboard.org). Reprints can be ordered at http://www.ajph.org by clicking the “Reprints” link. This article was accepted August 7, 2014.
Contributors J. Kronstadt was involved in revising the logic model and research agenda; she took the lead in writing the article. L. M. Beitsch was involved in developing and revising the logic model and research agenda; he contributed to writing and revising the article. K. Bender was involved in developing and revising the logic model and research agenda; she contributed to writing and revising the article.
Acknowledgments This project was supported by funding from the Centers for Disease Control and Prevention and the Robert Wood Johnson Foundation. L. M. Beitsch and K. Bender are on the Board of Directors of the Public Health Accreditation Board. Note. The contents of this article are solely the responsibility of the authors and do not necessarily represent the official views of the Centers for Disease Control and Prevention or the Robert Wood Johnson Foundation. The information contained in this article reflects the opinions of the authors and does not necessarily represent official Public Health Accreditation Board policy.
Human Participant Protection Human participants were not involved in research for this article, so institutional review board approval was not needed.
References 1. Public Health Accreditation Board Web site. Available at: http://www. phaboard.org. Accessed March 31, 2014. 2. Beitsch LM, Riley W, Bender K. Embedding quality improvement into accreditation: evolving from theory to practice. J Public Health Manag Pract. 2014;20(1):57---60. 3. Mays GP. Untangling desirable and undesirable variation in public health practice: accreditation and research working together. J Public Health Manag Pract. 2014;20(1):149---151.
Kronstadt et al. | Peer Reviewed | Commentary | S157
COMMENTARY
4. Institute of Medicine. The Future of the Public’s Health in the 21st Century. Washington, DC: National Academies Press; 2003. 5. Exploring Accreditation Planning Committee. Final Recommendations for a Voluntary National Accreditation Program for State and Local Public Health Departments. Exploring Accreditation Proj ect Report. Washington, DC: Exploring Accreditation; 2006. 6. Ingram RC, Bender K, Wilcox R, Kronstadt J. A consensus-based approach to national public health accreditation. J Public Health Manag Pract. 2014;20 (1):9---13. 7. Bender K, Kronstadt J, Wilcox R, Lee TP. Overview of the Public Health Accreditation Board. J Public Health Manag Pract. 2014;20(1):4---6. 8. Public Health Accreditation Board. Public Health Accreditation Board Standards and Measures, Version 1.0. Alexandria, VA: Public Health Accreditation Board; 2011. 9. NORC at the University of Chicago. Brief report: evaluation of the Public Health Accreditation Board beta test. 2011. Available at: http://www. phaboard.org/wp-content/uploads/ EvaluationofthePHABBetaTest BriefReportAugust2011.pdf. Accessed April 14, 2014. 10. Davis MV, Cannon MM, Corso L, Lenaway D, Baker E. Incentives to encourage participation in the national public health accreditation model: a systematic investigation. Am J Public Health. 2009;99(9):1705---1711. 11. Erwin PC, Knox County Tennessee Work Group on Accreditation. A selfassessment process for accreditation preparedness: a practical example for local health departments. J Public Health Manag Pract. 2009;15(6):503---508.
Foundation, and the National Coordinating Center for Public Health Services and Systems Research. A national research agenda for public health services and systems. Am J Prev Med. 2012;42(5 suppl 1):S72---S78. 16. Joly BM, Polyak G, Davis MV, et al. Linking accreditation and public health outcomes: a logic model approach. J Public Health Manag Pract. 2007;13(4):349--356. 17. The Public Health Systems & Services Research Community Informs the Research Agenda for Accreditation. Washington, DC: AcademyHealth, 2010. Available at: http://www.academyhealth. org/Training/ResourceDetail.cfm? ItemNumber=5358. Accessed April 15, 2014. 18. Riley WJ, Lownik EM, Scutchfield FD, Mays GP, Corso LC, Beitsch LM. Public health department accreditation: setting the research agenda. Am J Prev Med. 2012;42(3):263---271. 19. Beitsch LM, Rider NL, Joly BM, Leep C, Polyak G. Driving a public health culture of quality: how far down the highway have local health departments traveled? J Public Health Manag Pract. 2013;19(6):569---574. 20. Davis MV, Mahanna E, Joly B, et al. Creating quality improvement culture in public health agencies. Am J Public Health. 2014;104(1):e98---e104. 21. Public Health Accreditation Board. Research questions for accreditation. 2013. Available at: http://www. phaboard.org/wp-content/uploads/ Research-agenda-December2013.pdf. Accessed April 1, 2014. 22. Joly B, Davis M. Introduction to case reports: one goal, many journeys. J Public Health Manag Pract. 2014;20(1):64---65.
12. Conley AM, Vagi S, Horney JA. Use of the community assessment for public health emergency response to conduct community health assessments for public health accreditation. J Public Health Manag Pract. 2014;20(5):490---497. 13. Samoff E, Macdonald PD, Fangman MT, Waller AE. Local surveillance practice evaluation in North Carolina and value of new national accreditation measures. J Public Health Manag Pract. 2013;19(2):146---152. 14. Beitsch LM, Corso LC, Davis MV, Joly BM, Kronstadt J, Riley WJ. Transforming public health practice through accreditation (a user guide for the special accreditation issue). J Public Health Manag Pract. 2014;20(1)2---3. 15. A Consortium from Altarum Institute, the Centers for Disease Control and Prevention, the Robert Wood Johnson
S158 | Commentary | Peer Reviewed | Kronstadt et al.
American Journal of Public Health | Supplement 2, 2015, Vol 105, No. S2
