Stakeholder engagement in comparative effectiveness research: how will we measure success?

PERSPECTIVE For reprint orders, please contact: [email protected]

Stakeholder engagement in comparative effectiveness research: how will we measure success? Stakeholder engagement in comparative effectiveness research continues to gain national attention. While various methods are used to gather stakeholder expertise and form recommendations, evaluation of the stakeholder experience is often missing. The lack of evaluation prohibits assessing how effective and meaningful engagement practices are for enhancing research efforts and limits the ability to identify areas for future improvement. We propose that an evaluation plan of engagement processes be developed before stakeholder involvement begins and be required as part of a request for proposal or research grant where stakeholder input is being sought. Furthermore, we recommend the inclusion of six meta-criteria that represent normative goals of multiple studies: respect, trust, legitimacy, fairness, competence and accountability. To aid in the development of future evaluations, we have developed definitions for and matched specific examples of measuring each meta-criterion to serve a guide for others in the field.

Danielle C Lavallee1, Carla J Williams2, Ellen S Tambor3 & Patricia A Deverka*3 Surgical Outcomes Research Center, University of Washington, 1107 NE 45th Street, Suite 502 Seattle, WA 98105, USA 2 Johns Hopkins School of Medicine, 733 N Broadway, Baltimore, MD 21205, USA 3 Center for Medical Technology Policy, 401 E Pratt Street, Suite 631, Baltimore, MD 21201, USA *Author for correspondence: Tel.: +1 410 547 2687 Fax: +1 410 547 5088 [email protected] 1

Keywords: conceptual model n evaluation n process outcomes n public participation n research design n stakeholder engagement

The creation of the Patient-Centered Outcomes Research Institute (PCORI) is the most recent national effort in the USA to further promote the involvement of stakeholders in research-related activities such as topic prioritization, study design, and dissemination of results [1]. Established by the 2010 Patient Protection and Affordable Care Act, the framework for PCORI specifically includes measures for patient and other stakeholder involvement to ensure that funded research will lead to evidence to help these stakeholder groups make more informed healthcare decisions and ultimately improve health outcomes [2]. This mirrors similar expectations for research funded by the Agency for Healthcare Research and Quality’s (AHRQ) Effective Healthcare Program and NIH Clinical and Translational Science Awards (CTSA) [3]. Increasing investments in and recommendations for stakeholder involvement in comparative effectiveness research (CER) will likely shift the traditional ‘scientistled’ research enterprise to one that is more inclusive and collaborative with patients, clinicians, policy-makers and payers. This will require a greater emphasis on methods for evaluating stakeholder engagement to quantify impact, describe best practices and recommend strategies for improvement. Stakeholder involvement in CER

For the purposes of CER, we define stakeholders as those individuals, organizations or communities that have a direct interest in the process and outcomes of a project, policy or research endeavor – including patients and consumers, healthcare providers, payers and purchasers, policy-makers and regulators, industry representatives, researchers and research funders [4]. The rationale for stakeholder involvement derives from an analysis of problems with the clinical research enterprise that may be partially remedied by stakeholder participation. Despite the billions of dollars invested by both the public and private sectors in clinical research, systematic reviews frequently reveal a lack of

10.2217/CER.12.44 © 2012 Future Medicine Ltd

1(5), 397–407 (2012)

part of

ISSN 2042-6305

397

Perspective

Lavallee, Williams, Tambor & Deverka

necessary evidence to inform decision-making. An investigator- or funder-driven approach to research priority setting and study design has produced outcomes that seem ‘out of sync’ with the priorities and information needs of patients, clinicians and payers [5]. Research findings are used to inform policy development, coverage decisions, clinical practice guidelines, patient decision-making, clinician decision-making and regulatory decisions. Being responsive to these diverse evidentiary needs requires researchers to understand the context for decision-making, what research needs exist, how research questions should be framed and what outcomes are most important to capture. Involving stakeholders in the research process both for their technical knowledge as well as their explicit input of values and insights as part of deliberative discussions is a key focus of CER and an approach that is being rapidly embraced [4,6,101]. Similarly, based on our work, we have defined stakeholder engagement as an iterative process of actively soliciting knowledge, experience, judgment and values of individuals selected to represent a broad range of direct interests in a particular issue for the dual purposes of creating a shared understanding and making relevant, transparent and effective decisions [4]. The process of deliberation is fundamental to this definition of stakeholder engagement, emphasizing collective problem solving, consensus building and active participation by all stakeholders in a process that seeks to minimize status differentials [7]. Such engagement represents a significant change from traditional clinical research, where the generation, design and conduct of research are primarily guided by scientifically trained professionals. In light of limited evidence defining what constitutes meaningful stakeholder engagement in CER and what impact involving stakeholders has on evidence generation, there exists a real possibility that the true benefits of stakeholder engagement could go unrealized [101]. Without defining and measuring successful stakeholder engagement in CER, there is no ability to quantify impact, describe best practices or recommend strategies for improvement. Recognizing the resources and time invested by funders, researchers and the stakeholders themselves in this evolving field, it is imperative that methods for evaluation are established and implemented. Over time, these assessments will contribute to a public evidence base for what constitutes meaningful stakeholder engagement [8] that can be

398

J. Compar. Effect. Res. (2012) 1(5)

used to plan future engagement activities and increase the likelihood of achieving the goals of CER. Conceptual model for stakeholder engagement in CER

To facilitate the design, implementation and evaluation of stakeholder engagement activities in CER, we developed a conceptual model that illustrates the dual importance of evidence collection and deliberation by stakeholders in arriving at decisions and recommendations (Figure 1) [4]. Importantly, it includes both process and CER outcome criteria that can be measured as part of an evaluation. In CER, outcome end points include improved health, more efficient use of healthcare resources, and more useful evidence for clinical and health policy decisionmaking. Evaluating CER outcomes directly is critical since they correspond to the desired aims of the exercise, and developing methods to effectively evaluate these outcomes will be a priority as the field of CER moves forward. As depicted in Figure 1, process outcomes represent the more immediate outcomes of an engagement activity, and include stakeholder ratings of effective engagement, changes in knowledge or project decisions and increased pathways for implementation [4]. Process measures are proposed as a surrogate for outcomes with the rationale that an effective process has met the goals and intent of the activity, and is therefore more likely to have a positive impact on outcomes. Lessons learned from other fields

While relatively new to the field of CER, stakeholder engagement has been practiced for some time in the areas of environmental planning, public policy, community-based participatory research (CBPR) and, more recently, health technology assessment [7–20]. These related efforts demonstrate how experts in these fields conceptualized the relationship between stakeholder engagement and project goals and provide insight on factors related to successful stakeholder engagement that can be applied to other fields, including CER. CBPR is a model of involving community members in full partnership with researchers in an effort to reduce health disparities [18]. The goals of CBPR resemble those of CER in several respects, including an emphasis on the participation of individuals outside the traditional scientific paradigm in a process of creating knowledge

future science group

Stakeholder engagement in CER: how will we measure success?

Perspective

Types of: • • • •

Inputs

Values Research Professional experience Patient and consumer knowledge and experience Methods of combining:

Quantitative • Questionnaires • Delphi method • Multicriteria • Mapping • Value of information modeling

Analytic–deliberative model

Method

Qualitative • Nominal group technique • Facilitated workshops/meetings • Stakeholder decision analysis

Decisions • • • •

Topic generation Research priorities Study designs Evidentiary thresholds for clinical and health policy decision-making • Implementation strategies

Output

Outcomes

Process

CER

• Meta-criteria: trust, respect, accountability, legitimacy, fairness and competence • Change in knowledge/attitudes • Change in CER project decisions (e.g., choice of interventions, study design and funding priorities) • More useful evidence for clinical and health policy decision-making • More efficient use of healthcare resources • Improved health outcomes

Figure 1. Conceptual model for stakeholder engagement in comparative effectiveness research. CER: Comparative effectiveness research.

that is characterized by mutual respect and trust, and an ultimate goal of improving health outcomes. However, the characteristics that define successful CBPR differ markedly from those that define successful stakeholder engagement in CER. The nine principles for effective CBPR that are widely accepted within the field focus largely on issues related to the context, infrastructure and sustainability of CBPR programs [18]. Although facilitating collaborative, equitable partnerships is one principle of CBPR, the examination of deliberative processes as they relate to decision-making has not been a focus of CBPR evaluation efforts to date [19,20]. Deliberative processes have been at the core of efforts to involve the public in decisions involving health and environmental policy, where


the motivations include both ideological and practical goals. Practical goals for stakeholder engagement include resolving conflict among competing interests, achieving public support for potentially unpopular or controversial issues and restoring public trust in public institutions. Such goals are common motivations for involving the public in decisions involving controversial environmental issues such as nuclear power or advances in biotechnology (e.g., genetic engineering) [8,16,17]. Ideological goals for stakeholder engagement focus on the democratic ideals of an informed and involved citizenry engaged in collaborative problem-solving with decision-makers in processes characterized by transparency, accountability and legitimacy [7]. As these

www.futuremedicine.com

399

Perspective


decisions (e.g., priority setting of new healthcare technologies or radioactive waste management) are complex, the discussions must be deliberative – in other words, differing views and knowledge claims are openly aired and discussed in a respectful manner in a spirit of searching for a common resolution after a reasoned exchange. The normative presumption is that the outcome of an effective deliberative process will be ‘better’ because of the shared understanding of differing perspectives achieved by participants in their pursuit of a common objective. Emphasis is placed on the design of the deliberation process, the quality of the facilitation and how the stakeholders relate to each other, in the service of supporting public involvement. A ‘good’ process is one that incorporates these democratic values in robust methods that lead to an outcome that is acceptable to the majority of participants [21]. There is some evidence in the environmental literature that the quality of the deliberative process is highly correlated with ‘successful’ program outcomes [22]. What links all of these efforts to involve stakeholders (primarily the public) in decisions that were formerly controlled by technical experts (scientists, policy-makers and government officials) is the belief that ‘technocratic’ approaches have failed to produce the best solutions because they have not effectively engaged the knowledge, values and experience of stakeholders. We conclude that these efforts have produced generalizable insights that are highly relevant to the current situation in CER, where a similar rationale exists that involving stakeholders in the CER research agenda will produce more relevant information for healthcare decisionmaking. Although past efforts have focused on engaging the public, we propose that the principles of effective engagement described in this paper apply to other CER stakeholder groups as well (e.g., healthcare providers, payers and policy-makers). Evaluating ‘effective’ engagement

The theoretical framework for an ‘effective’ engagement process has its roots in the normative theory of public participation and deliberation and work of the German philosopher Jürgen Habermas from the early 1970s. He specified certain conditions that must be present during a deliberative process in order to level the playing field and ensure all participants had equal chances to act. According to Habermas, communication

400


is rational if it establishes mutual understanding via free and uncoerced discussion among all parties and allows a consensus to be reached [23]. Renn and Webler revised Habermas’ principles and proposed a normative theory of public participation that emphasized fairness and competence and operationalized these two principles as standard criteria that can be used to drive the design and evaluation of such processes [24,25]. The Renn and Webler framework of fairness and competence has been adapted for use in a number of different settings, and basic elements from this framework were used in the creation of an evaluation for public participation in the healthcare sector [7]. Abelson combined these two meta-criteria (fairness and competence) with the work of Beierle [22] to recommend the use of four key components in evaluating deliberative processes (fairness, competency, legitimacy and accountability) [7]. In reviewing the literature, we have identified a number of additional meta-criteria, or general principles, that are considered indicative of successful stakeholder and public engagement. Two of the additional meta-criteria, respect and trust, were recommended by multiple authors. We propose that a set of six meta-criteria should be considered as part of stakeholder engagement evaluation efforts: respect, trust, legitimacy, fairness, competence and accountability. These meta-criteria, indicative of the democratic and practical goals of public deliberation, provide a useful framework for evaluating the immediate outcomes of stakeholder engagement efforts. Meta-criteria for evaluation

These six meta-criteria that we consider important for evaluation of stakeholder engagement activities are not consistently defined in the literature. Therefore, drawing on the available literature and our experience leading stakeholder engagements efforts in CER, we developed clear definitions for each criterion that should promote shared understanding and consistent evaluation of stakeholder engagement in future CER projects (Table 1). ■■ Respect

A common theme in the engagement literature is that of respect. While many authors agree that stakeholder ideas should be heard and respected, very few offer a clear definition or provide indicators to measure whether a process was respectful. This meta-criterion is perhaps best measured by



Perspective

Table 1. Meta-criteria for stakeholder engagement in comparative effectiveness research. Meta-criterion

Definition

Application to stakeholder engagement in CER

Respect

Regard for stakeholders and their ideas is consistently demonstrated

Process is collaborative with an atmosphere conducive to open airing of diverse viewpoints. There is a two-way flow of information that facilitates mutual learning and understanding different viewpoints. Engagement is an ongoing, sustainable process

Trust

Stakeholders are confident that project outcomes reflect the discussions and decisions reached through a deliberative process

Opportunities for face-to-face contact are available. Engagement process promotes a sense of community and increased stakeholder confidence in the outcomes. Partnerships among stakeholders and between stakeholders and the sponsoring organization are established

Legitimacy

The process allows a diverse, balanced and informed group of stakeholders to arrive at a shared understanding of the issues and make decisions based on this collective understanding

A balanced representation of relevant stakeholders is achieved. Conflicts and disagreements are well managed. Decisions are made by group, with opportunities to publicly state minority opinions

Fairness

The process is free from bias, just to all stakeholders and consistent with logic

Stakeholders understand the purpose and process of engagement as well as their individual roles. Engagement activities are scheduled at a time and place that allows for stakeholder attendance. Facilitator is independent of the sponsoring agency

Competence

Meaningful participation is supported by providing stakeholders with appropriate information, ensuring that they understand the information and are able to contribute to discussions and decision-making

Briefings and other background materials are provided throughout the process that meet the diverse information needs of stakeholders. Stakeholder understanding of process, sponsoring organization or topic is improved. Decisionmaking is driven by accurate and relevant information. Decision-making is enabled by skilled facilitation

Accountability

The sponsoring agency is responsive to stakeholder input in achieving the goals of a process and takes responsible and justifiable actions based on project outcomes

The process is transparent. Stakeholder input is incorporated into the final decision. Results of the engagement process and rationales are disseminated to stakeholders and the wider community

CER: Comparative effectiveness research.

asking stakeholders at the end of the process to report whether they felt respected and whether they have more respect for others and others’ ideas as a result [9,26–28]. Respect is closely related to the theme of mutual learning, whereby stakeholders listen to each other, alter their own views in light of their learning or increase their tolerance of others’ paradigms [9,29]. To assess whether mutual learning is achieved, Carnes et al. suggest measuring the ability of stakeholders to identify each others’ concerns as well as understand the basis for those concerns [26]. Others suggest that a respectful process will result in continued dialogue and lasting relationships among participants and between participants and the sponsoring agency [12,30,31]. A respectful process also provides for constructive interactions among participants where all ideas are heard, all participants feel comfortable contributing and no one person dominates the decision-making [11,13,32–36]. Finally, respect is


demonstrated by reporting how stakeholders contributed to and how their ideas were incorporated into the final decision [37]. Our definition of respect is: regard for stakeholders and their ideas is consistently demonstrated (Table 1). Respect is integral to the very definition of stakeholder engagement, which specifically states that engagement should result in a shared understanding among participants [4]. Engagement implies that there is a two-way flow of information between participants and that participants alter their ways of thinking or become more tolerant of others’ viewpoints [38]. To actively solicit knowledge, values and experiences from participants successfully, a process must be open to and respectful of different viewpoints. ■■ Trust

Trust is closely related to the theme of respect and is an important element for participants to


401

Perspective


feel comfortable contributing ideas [16,39,40] and for building strong and lasting relationships among participants, especially those with opposing viewpoints [10,16,30,34,41,42]. However, trust is distinct from the meta-criterion of respect in that the former is more outcome-oriented. Respect is demonstrated during the engagement process, while trust is earned slowly over time as a result of an ongoing, participatory process [33,35,38,43]. According to Burton et al., it is through mature relationships involving collaboration that the gradual development of trust and shared ownership in the outcomes is achieved [44]. Trust is strengthened by increasing transparency, providing opportunities for open, twoway dialogue between stakeholders, and ensuring that communication between stakeholders and the sponsoring agency is well-coordinated and systematic [16,26,31,34,43,45,102]. The sponsoring agency should have a regular presence during the engagement process and should take contributions seriously [26,46]. Pedersen notes that a lack of consensus between stakeholders and the sponsoring agency can undermine trust and create a gap between the results of dialogue and observable practice [47]. Trust can also be increased by incorporating stakeholder input into the final decision and communicating to the wider community how stakeholder input was or was not used [11,22]. Holding in-person meetings, working in small groups and providing opportunities for socializing can help to build trust among stakeholders [30,48,102], while adhering to a predetermined process can enhance stakeholder trust and confidence in the process [49]. A successful process should increase the confidence of each party in the outcomes of discussions, in the overall conclusions and in the sponsoring agency [9,22,26,28,39,41,44,45,50]. Our definition of trust is: stakeholders are confident that project outcomes reflect the discussions and decisions reached through a deliberative process (Table 1). Trust is built by involving stakeholders in CER activities and communicating how their input, knowledge and experience are incorporated into decisions made within the scope of the project. Stakeholder engagement is often an iterative process occurring over time as a research project develops and evolves. Building meaningful relationships with and among participants is crucial for a trustworthy process [38].

402


■■ Legitimacy

For a process to be legitimate, stakeholder groups should be inclusive, should adequately reflect the diverse perspectives of those who are potentially affected by the outcomes and should contain balanced numbers of representatives from all key stakeholder groups [16,26,27,32,33,39,51]. Patients or other nontechnical participants should be empowered to participate, in order to eliminate the perceived power imbalance that sometimes exists between them and experts [32,39,52]. One way of achieving such equality, and therefore legitimacy, is to increase the number of patients in the group relative to the number of experts [32]. Many authors agree that a legitimate process uses formal processes for decision-making: it allows all participants to discuss and debate issues in an open forum with the purpose of arriving at a common view [9,12,16,22,31,41]. In order to reach a shared understanding, the process cannot collapse into political or personal disagreements and, thus, a legitimate process is one that manages conflict well [12,22,53]. Abelson and others suggest that legitimacy of the process can be assessed by considering the point in the decision-making process at which stakeholder input is being sought, who is listening and ultimately responding to stakeholder input, and to what extent stakeholder input is incorporated into the final decision [7,53]. All participants must be able to express their viewpoints well before any recommendations or decisions are finalized [31]. Caron-Flinterman et al. add that throughout the process, participants must be treated equally and have equal access to information [32]. Other measures of legitimacy include the communication of decisions and their rationales [9,16], the transparency of the process [16], the neutrality and independence of facilitators from the sponsoring agency [32], the degree to which a process builds trust and ownership of the outcomes [9,53] and the extent to which the process is driven by evidence [12,54]. Our definition of legitimacy is: the process allows a diverse, balanced and informed group of stakeholders to make decisions based on a collective understanding of the issue(s). Legitimacy is ensured during the stakeholder selection process. Thoughtful consideration should go into deciding which stakeholder perspectives should be represented and what specific input they will be asked to provide. The process for selecting stakeholders should be transparent to participants and clearly articulated in methods of research.



■■ Fairness

[7,9,14,21,31,46,52,56]. A fair process should be highly

There is extensive mention of fairness in the engagement literature, yet a clear definition of the term is lacking. Some authors define a fair process as one that is ‘perceived as fair by the disputants and the community at large’ [50,51], while others define a fair process as one that meets the criteria for accountability of reasonableness: the decisions and rationales are made public, rest on reasons that stakeholders think are relevant, are revisable in light of new evidence and enforcement measures exist to ensure these previous three conditions are met [55,56]. Research by Lauber and Knuth found that citizens’ perceptions of fairness are highly correlated with the sponsoring agency’s knowledge, reasoning and receptivity, with the influence citizens had on the decision, and with the relationships that were built between stakeholders and between stakeholders and the sponsoring agency [28]. They also concluded that perceptions of fairness are related to satisfaction with the process and with the sponsoring agency and noted that in-person meetings gave people a more favorable impression [28]. Gibson et al. suggest that fairness in priority setting can be assessed by asking whether stakeholders understand the process and felt engaged, whether decisions were justified and whether ‘winners’ and ‘losers’ both felt as though they had been fairly treated [51]. We define fairness as: a process free from bias, just to all stakeholders and consistent with logic. Most importantly, this definition captures the theme that all participants must have equal access to the process [9,12,21,31,53], equal access to information [50], equal participation in the interaction [7,8,9,12,21,28] and equal opportunities to influence the final decision or outcome [12,21,28,41]. A fair process is one where there is shared decisionmaking power and no one individual or group dominates the discussion [9,12,41,52]. There needs to be open dialogue and efficient two-way communication so that all concerns and issues can be brought forth and discussed in order to create an active understanding among participants, to lead to shared decisions and to build relationships [9,14,47,52,56]. Open dialogue also means there is adequate time and facilities for stakeholders to voice their views and opinions [48]. For this atmosphere to exist, there needs to be skilled, professional facilitation by a neutral mediator [9,12,28]. In addition, a fair process has selection criteria established ensuring a diverse, balanced and committed group of stakeholders is convened

structured, explicit and assign specific roles to each contributor [21,52,56]. Members of the group should be informed and educated about the topic, and decisions should be based upon the best available information [28,56]. Finally, fairness is ensured by using formal decision-making methods such as the Nominal Group Technique and Delphi methods to ensure each stakeholder has an equal opportunity to participate [57,58].


Perspective

■■ Competence

Competence refers to the condition of being informed and capable of participating in a process. To achieve this meta-criterion, appropriate procedures must be used to select information that will be used in the process, and stakeholder understanding of that knowledge must be achieved through access to information as well as interpretations of the information [7,12,31]. A competent process uses the best available information and ensures stakeholder understanding of terms, concepts, definitions and language used [9,52]. Stakeholder learning is highlighted as a key component of competence, and an evaluation should assess what participants may need in order to contribute satisfactorily to the exercise and discuss technical matters [9,14,21,40]. In addition, a competent process is one where all key issues are addressed, discussions go deeper than headline level, communication skills are used to articulate, evaluate and refute arguments about an issue, mutual understandings are developed and a decision is reached based on those understandings [9,10,52]. Authors stress the importance of a skilled facilitator who successfully resolves disputes and who facilitates constructive personal behaviors [9,12,15,31]. Our definition of competence is: meaningful participation is supported by providing stakeholders with appropriate information, ensuring that they understand the information and are able to contribute to discussions and decision-making. Competence is demonstrated through providing adequate information (relevant research studies, values and professional and patient expertise, among others) and support (training and meeting facilitation) to stakeholders to ensure meaningful involvement. ■■ Accountability

Accountability concerns the extent to which a process is receptive to stakeholder input, achieves its end goals, and results in responsible and


403

Perspective


justifiable actions. Abelson et al. distinguish between three types of accountability: sanction, answerability and structural. Sanction consists of laws and penalties to ensure that agreed-upon expectations are met. Answerability emphasizes transparency, trust and active dialogue and obliges decision-makers to provide a rationale for actions taken. Finally, structural accountability focuses on building relationships between stakeholders and the sponsoring agency, which can be achieved through communication [16]. They suggest that the following indicators should be measured to assess accountability: the extent to which public input was incorporated into the final decision, the extent to which decisions and public input into those decisions were communicated back to the public, and the degree to which decision-making authority was found to respond to the public’s input [7]. Documentation of stakeholder advice, involvement in decision-making and use of mass media to disseminate results is important to demonstrate these principles [37]. In addition, an accountable process should result in participant satisfaction and a more informed citizenry [7]. Other authors agree that an accountable process is one where the sponsoring agency listens to stakeholder input and is responsive to it, builds consensus, allows participants to change the course of discussion or outcome by making a compelling argument, and leads to improved decision-making [31,32,43,53]. Tuler and Webler add that decision-making should be informed by social or natural science and local knowledge rather than driven by politics [53]. Others add that transparency is important; the sponsoring agency needs to make decisions public and provide justification for them. Without information about the implementation of the decisions from stakeholder dialogue, it is impossible to evaluate whether the process has been a truly participatory approach to problem solving or just a public relations exercise [21,31,47,51,55]. Finally, an accountable process is one that manages conflicts, creates stakeholder identification with the outcomes of the organization’s activities (buy-in), builds trust in the sponsoring organization, and makes the responsibilities and roles of all those involved explicit [43,50,52]. We define accountability as: the sponsoring agency and research team are responsive to stakeholder input in achieving the goals of a process and take responsible and justifiable actions based on project outcomes. Accountability is assessed in the outcomes by measuring the extent to

404


which better or different decisions were made, and how attitudes and knowledge changed as a result of engagement [4]. Advancing CER through evaluation

We believe these six meta-criteria are critical components in measuring the success of efforts to engage stakeholders in CER. Although this paper intentionally focuses on evaluating the immediate outcomes of the stakeholder engagement process, it is not meant to minimize the importance of assessing health outcomes, changes in policy or clinical decisions as CER strives to develop meaningful evidence. An evaluation serves to increase the accountability of the engagement process [8,9,10,11] and allows time for reflection so that both participants and project organizers can learn from their experiences, feeding those insights back into the process design to improve it for the future [12]. Evaluations can measure process-based and/or outcomes-based criteria [9,14,49]. For example, Sibbald et al. created a framework consisting of process and outcome measures for stakeholder engagement in priority setting. Process concepts included quality of stakeholder engagement, use of an explicit process, management of information, consideration of values and context, and the presence of revision or appeal mechanisms. Outcome concepts included improved stakeholder understanding, shifted priorities or reallocated resources, improved decision-making quality, stakeholder satisfaction and positive externalities [49]. Ideally, the evaluation criteria and tools are designed and made explicit before the engagement process begins, ensuring that all parties involved (the sponsoring agency, researchers, facilitators and the stakeholders) understand what criteria are being used to establish the goals of the activity and define successful and meaningful engagement [13,14]. Evaluation methodologies need to be both general and replicable to compare across processes, over time or across agencies, but should also be adapted to local methods and needs [41]. We recognize that while an engagement event would ideally achieve the six meta-criteria defined in Table 1, the specific approach to determining success may vary based on the unique aspects of the activity. We recommend clearly operationalizing each meta-criterion before an engagement event begins and choosing specific and relevant determinants to measure how effectively a process achieves that criterion. Table 1 provides relevant examples from the literature for measuring



meta-criteria for stakeholder engagement in CER. While we recognize overlap in how metacriteria have been evaluated, (e.g., managing conflicts has been used to assess legitimacy, fairness, competence and accountability), we suggest each meta-criterion be evaluated by a specific question rather than attempting to measure multiple criteria simultaneously. Thus far, we have focused on important criteria to include in stakeholder engagement evaluations. The design and approach of an evaluation requires some specific extensions and modifications. For example, the timing and structure of an evaluation needs to be considered [9]. If an evaluation is conducted immediately after a stakeholder engagement meeting, the process and immediate outcomes will be easily remembered by participants, but there will be little time for thought and contemplation. Ideally, an evaluation should be given at a time that both maximizes memory and leaves adequate time for reflection. Also, since engagement should be a continuous process, it may be beneficial to conduct a series of evaluations over the life time of a project, allowing opportunities to adjust activities in response to feedback. The format of an evaluation should be considered such that it will maximize the response rate and provide accurate and adequate feedback from stakeholders. For CER projects that are longer-term and involve numerous engagement activities, using multiple methods for evaluation

Perspective

is recommended, such as a written survey, document analysis with follow-up or semi-structured phone interviews [14,59]. Conclusion

In this paper, we have identified six meta-criteria for effective stakeholder engagement and offered clear definitions for each as they relate to CER. We believe that these meta-criteria provide a strong foundation for the design of evaluations to measure the success of stakeholder engagement activities. It is important to emphasize that routine evaluation should be part of any engagement process, and we recommend that an evaluation be required as part of requests for proposals or research grants where stakeholder input is being sought. As evaluations become a routine part of stakeholder engagement processes, we will continue to refine and adapt this evaluation process to specifically meet the needs of stakeholder participation in CER. Future perspective

CER funded by agencies such as PCORI, AHRQ and NIH is now required to involve stakeholders as collaborators in the research process. Similarly, there is a growing recognition by life sciences companies that new product development programs will need to account for CER expectations and stakeholder perspectives. Incorporating rigorous evaluation and reporting of engagement

Executive summary Stakeholder involvement in comparative effectiveness research ■■ To date there is no agreement about what constitutes successful stakeholder engagement in comparative effectiveness research (CER), thus limiting the ability to conduct consistent evaluations of the process to assess impact and improve the quality and efficiency of engagement. ■■ Given that significant resources and time are invested by researchers, funders and stakeholders themselves in engagement activities, it is critical that methods for evaluation are established and implemented. Lessons learned from other fields ■■ Published evaluation research of public deliberation efforts in science, environmental planning and health policy provides an opportunity to identify relevant process measures for assessing overarching principles that typify successful stakeholder engagement. ■■ These principles or meta-criteria include: respect, trust, legitimacy, fairness, competence and accountability. Meta-criteria for evaluation ■■ The meta-criteria were inconsistently defined in the literature; we defined and operationalized these concepts in the context of stakeholder engagement in CER. ■■ Correspondingly, the meta-criteria seem appropriate as process measures in CER, as illustrated in a conceptual model for stakeholder engagement that also includes outcomes measures. Advancing CER through evaluation ■■ Whereas stakeholder engagement in CER is routinely advocated, evidence of the relative effectiveness of various strategies for engaging stakeholders given particular study objectives is just beginning to develop. ■■ We encourage routine evaluation and publication of stakeholder engagement activities in CER, using at a minimum the meta-criteria and ideally the conceptual framework, to further advance the field.



405

Perspective


activities will enable objective assessment of current strategies and ongoing improvement in engagement methodologies. As results from these studies become available over the next few years, the CER community will learn valuable information about the impact of stakeholder engagement on the process and outcomes of healthcare research. Such findings will further advance the development and refinement of best practices for involving diverse stakeholders throughout the research continuum. Use of standard outcome measures of effective engagement such as

n

2

Washington AE, Lipstein SH. The PatientCentered Outcomes Research Institute – promoting better information, decisions, and health. N. Engl. J. Med. 365(15), e31 (2011). Manchikanti L, Falco F, Benyamin R, Helm S, Parr A, Hirsch J. The impact of comparative effectiveness research on interventional pain management: evolution from Medicare Modernization Act to Patient Protection and Affordable Care Act and the Patient-Centered Outcomes Research Institute. Pain Physician 14, 249–282 (2011).

3

Helfand M, Tunis S, Whitlock EP et al. A CTSA agenda to advance methods for comparative effectiveness research. Clin. Transl. Sci. 4(3), 188–198 (2011).

4

Deverka P, Lavallee D, Desai PJ et al. Stakeholder participation in comparative effectiveness research: defining a framework for effective engagement. J. Compar. Effect. Res. 1(2), 181–194 (2012).

n

8

n

1

n

5

6

7

406

Development of definitions for stakeholder engagement in comparative effectiveness research and a conceptual model for engaging stakeholders. Sox HC, Greenfield S. Comparative effectiveness research: a report from the Institute of Medicine. Ann. Intern. Med. 151(3), 203–205 (2009). Hoffman A, Montgomery R, Aubry W, Tunis SR. How best to engage patients, doctors, and other stakeholders in designing comparative effectiveness studies. Health Aff. 29(10), 1834–1841 (2010). Abelson J, Forest P-G, Eyles J, Smith P, Martin E, Gauvin F-P. Deliberations about deliberative methods: issues in the design and

Financial & competing interests disclosure The authors have no relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript. This includes employment, consultancies, honoraria, stock ownership or options, expert testimony, grants or patents received or pending, or royalties. No writing assistance was utilized in the production of this manuscript.

evaluation of public participation processes. Soc. Sci. Med. 57(2), 239–251 (2003).

References Papers of special note have been highlighted as: of considerable interest

these six meta-criteria should help accelerate this process.

9

Chess C. Evaluating environmental public participation: methodological questions. J. Environ. Plan. Manage. 43(6), 769–784 (2000). Oels A. Evaluating stakeholder dialogues. In: Stakeholder Dialogues in Natural Resources Management. Stollkleemann S, Welp M (Eds.). Springer Berlin Heidelberg, Germany, 117–151 (2006).

10 Butterfoss FD. Process evaluation for

community participation. Annu. Rev. Public Health 27(1), 323–340 (2006). 11 Rowe G, Frewer LJ. Public participation

methods: a framework for evaluation. Sci. Technol. Human Values 25(1), 3–29 (2000). 12 Webler T, Tuler S, Krueger ROB. What is a

good public participation process? Five perspectives from the public. Environ. Manage. 27(3), 435–450 (2001). 13 Gardner J, Dowd A, Mason C, Ashworth P.

A framework for stakeholder engagement on climate adaptation. CSIRO Climate Adaptation Flagship Working Paper (2009). 14 Rowe G, Marsh R, Frewer LJ. Evaluation of a

deliberative conference. Sci. Technol. Human Values 29(1), 88–121 (2004). n

n

Use of a questionnaire, evaluation checklist and telephone interviews to evaluate the success of a deliberative conference according to acceptance criteria (representativeness, early involvement, influence and transparency) and process criteria (resource accessibility, task definition, structured decision making and cost–effectiveness).

15 Rowe G, Frewer LJ. Evaluating public-

participation exercises: a research agenda. Sci. Technol. Human Values 29(4), 512–556 (2004).

technology assessment and coverage policy decisions: from principles to practice. Health Policy 82(1), 37–50 (2007). 17 Hailey D, Nordwall M. Survey on the

involvement of consumers in health technology assessment programs. Int. J. Technol. Assess. Health Care 22(4), 497–499 (2006). 18 Israel BA, Schulz AJ, Parker EA, Becker AB.

Review of community-based research: assessing partnership approaches to improve Public Health. Annu. Rev. Public Health 19, 173–202 (1998). 19 Sandoval JA, Lucero J, Oetzel J et al. Process

and outcome constructs for evaluating community-based participatory research projects: a matrix of existing measures. Health. Educ. Res. 27(4), 680–690 (2011). 20 Braun KL, Nguyen TT, Tanjasiri SP.

Operationalization of community-based participatory research principles: assessment of the National Cancer Institute’s Community Network Programs. Am. J. Public Health 102(6), 1195–1203 (2011). 21 Burgess J, Stirling A, Clark J et al.

Deliberative mapping: a novel analyticdeliberative methodology to support contested science-policy decisions. Public Understand. Sci. 16(3), 299–322 (2007). 22 Beierle TC, Konisky DM. Values, conflict,

and trust in participatory environmental planning. J. Policy Anal. Manage. 19(4), 587–602 (2000). 23 Santos SL, Chess C. Evaluating citizen

advisory boards: the importance of theory and participant-based criteria and practical implications. Risk Analysis 23(2), 269–279 (2003). 24 Renn O. Risk communication: towards a

rational discourse with the public. J. Hazard. Mater. 29, 465–519 (1992). 25 Webler T. ‘Right’ discourse in citizen

F-P. Bringing “the public” into health

participation: an evaluative yardstick. In: Fairness and Competence in Citizen Participation: Evaluating Models for



16 Abelson J, Giacomini M, Lehoux P, Gauvin


Environmental Discourse. Renn O, Webler T, Wiedelmann P (Eds). Kluwar Academic Press, Boston, MA, USA, 35–86 (1995).

n

n

26 Carnes SA, Schweitzer M, Peelle EB, Wolfe

AK, Munro JF. Measuring the success of public participation on environmental restoration and waste management activities in the US Department of Energy. Technol. Soc. 20(4), 385–406 (1998). 27 Halvorsen KE. Assessing public participation

39 Bruni RA, Laupacis A, Martin DK, Group

citizen participation: a case study of moose management. Soc. Nat. Resources 12, 19–37 (1999). 29 Franche R-L, Baril R, Shaw W, Nicholas M,

Loisel P. Workplace-based return-to-work interventions: optimizing the role of stakeholders in implementation and research. J. Occup. Rehabil. 15(4), 525–542 (2005). 30 Reed MS. Stakeholder participation for

40 Tompkins EL, Few R, Brown K. Scenario-

in citizen participation. Admin. Soc. 32(5), 566–595 (2000).

participation. J. Plan. Edu. Res. 28(3), 293–309 (2009). Collaborative partnerships at the state level: promoting systems changes in public health infrastructure. J. Public Health Manage. Prac. 10(3), 251–257 (2004). 43 Amaeshi KM, Crane A. Stakeholder

engagement: a mechanism for sustainable aviation. Corp. Soc. Responsibil. Environ. Manage. 13(5), 245–260 (2006). Bäck P. Developing stakeholder involvement for introducing public health genomics into public policy. Public Health Genomics 12(1), 11–19 (2009). 45 Rowe G, Horlick-Jones T, Walls J, Pidgeon N.

Difficulties in evaluating public engagement initiatives: reflections on an evaluation of the UK GM Nation? Public debate about transgenic crops. Public Understand. Sci. 14(4), 331–352 (2005).

38 Mitton C, Smith N, Peacock S, Evoy B,

Abelson J. Public participation in health care priority setting: a scoping review. Health Policy 91(3), 219–228 (2009).


participants expect of a public participation process. Soc. Nat. Resources 12, 437–453 (1999). 54 Renn O. A model for an analytic-deliberative

process in risk management. Environ. Sci. Technol. 33(18), 3049–3055 (1999). 55 Daniels N, Sabin JE. Accountability for

reasonableness: an update. BMJ 337 (2008). 56 Menon D, Stafinski T, Martin D. Priority-

setting for healthcare: who, how, and is it fair? Health Policy 84(2–3), 220–233 (2007). 57 Jones J, Hunter D. Consensus methods for

medical and health services research. BMJ 311, 376–380(1995). 58 Delbecq AL, Van de Ven AH. A group process

model for problem identification and program planning. J. Appl. Behav. Sci. 7(4), 466–492(1971). 59 Chappel D, Bailey J, Stacy R, Rodgers H,

Thomson R. Implementation and evaluation of local-level priority setting for stroke. Public Health 115(1), 21–29 (2001). n

47 Pedersen ER. Making corporate social

responsibility (CSR) operable: how companies translate stakeholder dialogue into practice. Business Soc. Rev. 111(2), 137–163 (2006). 48 Hanson CE, Palutikof JP, Dlugolecki A,

Giannakopoulos C. Bridging the gap between science and the stakeholder: the case of climate change research. Climate Res. 31(1), 121–133 (2006).

challenges in global health: community engagement in research in developing countries. PLoS Med. 4(9), e273 (2007). SUPPORT tools for evidence-informed health Policymaking (STP) 15: engaging the public in evidence-informed policymaking. Health Res. Policy Syst. 7, 1478–4505 (2009).

53 Tuler S, Webler T. Voices from the forest: what

forest planning: 8 attributes of success. J. Forestry 95, 17–19 (1997).

36 Tindana PO, Singh JA, Tracy CS et al. Grand

37 Oxman A, Lewin S, Lavis J, Fretheim A.

Heaney C, Landsbergis P. The intervention research process in occupational safety and health: an overview from the National Occupational Research Agenda Intervention Effectiveness Research Team. J. Occup. Environ. Med. 43(7), 616–622 (2001).

46 Shindler B, Neburka J. Public participation in

engagement: a three-phase methodology. Environ. Qual. Manage. 14(3), 9–24 (2005). Community engagement as a component of revitalization: lessons learned from the Technical Outreach Services to Communities programme. Local Environ. 11, 515–535 (2006).

52 Goldenhar LM, LaMontagne AD, Katz T,

44 Burton H, Adams M, Bunton R, Schröder-

stakeholder engagement. Stanford Soc. Innov. Rev. 40–45 (2010).

35 Ellerbusch F, Gute D, Desmarais A, Woodin M.

in health care organizations: criteria, processes, and parameters of success. BMC Health Serv. Res. 4(1), 25 (2004).

42 Padgett SM, Bekemeier B, Berkowitz B.

33 Enright K, Bourns C. The case for

34 Gable C, Shireman B. Stakeholder

51 Gibson J, Martin D, Singer P. Setting priorities

41 Laurian L, Shaw MM. Evaluation of public

32 Caron-Flinterman J, Broerse J, Teerling J

et al. Stakeholder participation in health research agenda setting: the case of asthma and COPD research in the Netherlands. Sci. Public Policy 33, 291–304 (2006).

integrating analytical results into environmental decisions involving multiple stakeholders. Risk Analysis 18(5), 621–634 (1998).

based stakeholder engagement: incorporating stakeholders preferences into coastal planning for climate change. J. Environ. Manage. 88(4), 1580–1592 (2008).

environmental management: a literature review. Biolog. Conserv. 141(10), 2417–2431 (2008). 31 Webler T, Tuler S. Fairness and competence

50 Apostolakis GE, Pickett SE. Deliberation:

for the U of TPS in HCR. Public engagement in setting priorities in health care. Can. Med. Assoc. J. 179(1), 15–18 (2008).

techniques for comfort, convenience, satisfaction, and deliberation. Environ. Manage. 28(2), 179–186 (2001). 28 Lauber TB, Knuth BA. Measuring fairness in

Recognizes that deliberative approaches to engaging the public are becoming more common but formal evaluation of public engagement efforts are rare.

49 Sibbald S, Singer P, Upshur R, Martin D.

Priority setting: what constitutes success? A conceptual framework for successful priority setting. BMC Health Serv. Res. 9(1), 43 (2009). n

n

Development of a framework for successful stakeholder priority setting in healthcare that relates to both process and outcome components by asking relevant stakeholders what they understand successful priority setting to mean.


Perspective

n

Semi-structured interviews used to evaluate stakeholder engagement in a priority setting process with a discussion of areas of success and areas for improvement.

■■ Websites 101 O’Haire C, McPheeters M, Nakamoto E et al.

Engaging stakeholders to identify and prioritize future research needs. Methods Future Research Needs Report No 4 (2011). www.effectivehealthcare.ahrq.gov/ehc/ products/200/698/MFRNGuide04-Engaging_Stakeholders--6–10–2011.pdf 102 Preskill H, Jones N. A practical guide for

engaging stakeholders in developing evaluation questions (2009). www.rwjf.org/pr/product.jsp?id=49951

407

How do we measure the success of operational research?

Stakeholder engagement for comparative effectiveness research in cancer care: experience of the DEcIDE Cancer Consortium.

A systematic review of stakeholder engagement in comparative effectiveness and patient-centered outcomes research.

Editorial Commentary: Pseudomonas aeruginosa Eradication: How Do We Measure Success?

Stakeholder assessment of comparative effectiveness research needs for Medicaid populations.

Disparity in organizational research: How should we measure it?

The ECOUTER methodology for stakeholder engagement in translational research.

Towards Building a Bridge between Community Engagement in Research (CEnR) and Comparative Effectiveness Research (CER).

IMPACCT Kids' Care: a real-world example of stakeholder involvement in comparative effectiveness research.

Fungi in homes--how do we measure?

How we can measure quality in colonoscopy?

How do we encourage patient engagement?

Comparative effectiveness research in the U.S.A.: when will there be an impact on healthcare decision-making?

Chronic toxicity: How can we measure it?

Documenting malnutrition--checking how we measure up.

How to measure mood in nutrition research.

Comparative effectiveness research.

Building a "Deep Fund of Good Will": Reframing Research Engagement.

From methods to policy: past as prologue: how comparative effectiveness research became patient-centered outcomes research.

How we will treat chronic myeloid leukemia in 2016.

Stakeholder engagement in dredged material management decisions.

Quantifying Success: Using Control Groups to Measure Program Effectiveness.

Accelerating aging research: how can we measure the rate of biologic aging?

How can we improve oral contraceptive success in obese women?