doi: 10.1111/hex.12213

Patient involvement in research programming and implementation A responsive evaluation of the Dialogue Model for research agenda setting Tineke A. Abma RN Msc PhD,* Carina A.C.M. Pittens MSc PhD,† Merel Visse MSc PhD,* Janneke E. Elberse MSc PhD† and Jacqueline E.W. Broerse MSc PhD† *Professor and Research Director, Researcher, EMGO+ Institute for Health and Care Research, VU University Medical Centre, Amsterdam and †Researcher, Professor and Director, Faculty of Earth and Life Science Athena Institute for Research on Innovation and Communication in Health and Life Sciences, VU University, Amsterdam, The Netherlands

Abstract Correspondence Tineke A. Abma RN Msc PhD Professor EMGO+ Institute for Health and Care Research VU University Medical Centre Van der Boechorststraat 7 1081 BT Amsterdam The Netherlands E-mail: [email protected] Accepted for publication 23 April 2014 Keywords: Dialogue Model, priority setting, research agenda setting, research programming and implementation, responsive evaluation

Background The Dialogue Model for research agenda-setting, involving multiple stakeholders including patients, was developed and validated in the Netherlands. However, there is little insight into whether and how patient involvement is sustained during the programming and implementation of research agendas. Aim To understand how the Dialogue Model can be optimised by focusing on programming and implementation, in order to stimulate the inclusion of (the perspectives of) patients in research. Methods A responsive evaluation of the programming and implementation phases of nine agenda-setting projects that had used the Dialogue Model for agenda-setting was conducted. Fifty-four semi-structured interviews were held with different stakeholders (patients, researchers, funding agencies). Three focus groups with patients, funding agencies and researchers (16 participants) were organized to validate the findings. Results Patient involvement in programming and implementation of the research agendas was limited. This was partly related to poor programming and implementation, partly to pitfalls in earlier phases of the agenda-setting. Optimization of the Dialogue Model is possible by attending to the nature of the agenda and its intended use in earlier phases. Attention should also be given to the ambassadors and intended users of agenda topics. Support is needed during programming and implementation to organize patient involvement and adapt organizational structures like review procedures. In all phases the attitude to patient involvement, stakeholder participation, especially of researchers, and formal and informal relationships between parties need to be addressed to build a strong relationship with a shared goal. Conclusion Patient involvement in agenda-setting is not automatically followed by patient involvement in programming and

ª 2014 John Wiley & Sons Ltd Health Expectations, 18, pp.2449–2464

2449

2450

Involvement in programming and implementation, T A Abma et al.

implementation. More attention should be paid, in earlier stages, to the attitude and engagement of researchers and funding agencies.

Introduction Researchers and funding agencies have, for a long time, been the ones to set research agendas for medical and health research, while the research process itself was mainly controlled by researchers. Two decades ago this became part of a public debate in various Western countries; the awareness of patients’ rights increased. The history of public involvement in research (agenda-setting) goes partly back to the campaigning of the disability and advocacy movements for more control over the process of knowledge production.1 Transitions in society, including consumerism, public accountability and democratisation of science, further prompted disputes over the benefits and pitfalls of patient involvement, including the validity of experiential knowledge.2–9 In the field of research agenda-setting, also referred to as priority setting, various models and strategies have been developed since then, ranging from reasonably well structured approaches, from a one-off consultation to regular advice from beginning to end.1,2 In the UK, for instance, the James Lind Alliance3 has proposed a number of methods to engage patients, carers and clinicians in dialogue about uncertainties in medical treatment. The Delphi Method, originally used for expert consensus, has also been translated to agenda setting.4,5 The Nominal Group Technique, also used in the UK, is a combination of the Delphi Method and focus group discussions to identify, discuss and prioritize research topics.6 In The Netherlands public involvement in research agenda-setting became an issue after 2000. Several local charity/government funding agencies, in some occasions after pressure from local patient organizations, started to initiate (and fund) projects in which patients were engaged in agenda setting. These projects were facilitated by academic departments in the field

of social sciences who collaborated in the development and validation of the Dialogue Model.7 The model operationalizes consultation of and collaboration among various stakeholders, and is grounded in the notion that involvement is an interactive process between stakeholders. Since its inception, the Dialogue Model has been used on fourteen different occasions.8–24 What all these agenda setting methodologies have in common is the belief that patients should be involved in the actual agenda setting, and that their involvement should be sustained in the programming and implementation phases. However, the systematic evaluation of patient involvement in these phases is lacking. Patients have been engaged in agenda-setting,7 but it is unclear whether they are involved in later phases.25 Nor do we know under what circumstances and how the research agenda set by patients has actually been programmed and implemented. We also lack studies that reveal whether (limited) patient involvement and uptake of patient priorities is a matter of (poor) programming, or (poor) planning in earlier phases of the agenda setting process. Little is therefore known about the factors influencing patient involvement in programming and implementation. The aim of this article is to contribute towards the knowledge gap concerning patient involvement in the phases of programming and implementation of agenda setting. The article specifically focuses on optimizing the Dialogue Model in order to sustain patient involvement as well as to provide tools to improve the programming and implementation of patients’ priorities. To this end, a responsive evaluation of nine agenda-setting projects, which involved patients and used the Dialogue Model, has been conducted. The findings from this study are situated in The Netherlands and therefore context laden, but we expect that lessons learned can be transferred to other contexts.

ª 2014 John Wiley & Sons Ltd Health Expectations, 18, pp.2449–2464

Involvement in programming and implementation, T A Abma et al.

Dialogue Model The Dialogue Model is a multi-phased participatory approach. The Model has been developed within academia in order to facilitate the involvement of patients in research agendasetting.7 It is based on the methodology of Responsive Evaluation26–32 and the Interactive Learning and Action (ILA) approach, which enables end-users to have a role in decisionmaking of innovation processes in various societal domains.33 The aim of the Dialogue Model is to create equal partnership among patients and other stakeholders (e.g. health care professionals, researchers) over the agenda-setting process. Dialogue is seen as an ethical and fruitful way for agenda-setting as it helps professionals understand the societal impact of their research interests, and integrates patients’ experiential knowledge and voice.22,30 Dialogue fosters a mutual learning process which leads to a new shared perspective that is acceptable and recognizable for all involved.17,27 Dialogue potentially stimulates the innovation of ideas and a sense of co-ownership.14,19,28,29 The Dialogue Model acknowledges power differentials and assumes that patients first need the safe environment of their own group to develop their own voice; patients’ priorities are not given, and patients require deliberation if they are to become aware of their priorities.22,30 This deliberation equips patients for dialogue with professionals.22 There are four phases to the Dialogue Model: exploration, consultation, prioritization, and integration. They lead to an integrated and shared research agenda, followed by two phases of programming and implementation: 1. Exploration: the project team is established, key stakeholders are identified, an initial list of issues, perspectives and ideas of patients and other stakeholders is drawn up, and the social conditions for collaboration are created. 2. Consultation: the different stakeholder groups are consulted separately to develop a list of research topics from the perspective of each stakeholder group.

ª 2014 John Wiley & Sons Ltd Health Expectations, 18, pp.2449–2464

3. Prioritization: the stakeholders prioritize the research topics identified in the previous phase. 4. Integration: the prioritized research topics of each stakeholder group are integrated into one integral research agenda. 5. Programming: the integral research agenda is translated into a funding programme or plan. 6. Implementation: the research agenda is actively used, action is taken and results are evaluated. The Dialogue Model uses a mix of methods, including interviews and focus groups in the consultation phase, and questionnaires or the Delphi Method in the prioritization phase. In the integration phase, dialogue is used to integrate the different perspectives and encourage mutual learning. An external facilitator creates the conditions for dialogue, stimulates mutual learning and, if necessary, acts as a mediator.

Methods This article is based on the responsive evaluation of nine agenda-setting processes carried out along Dialogue Model in the Netherlands since 2003 (Table 1). We followed a purposeful sampling strategy, and selected processes that varied in terms of: (i) type of funding agency (governmental; charity); (ii) number of stakeholder groups involved (at least patients); and (iii) disease-specific research agendas. The agendasetting processes should also have been completed at least 1 year ago, otherwise there would have been insufficient time for programming and implementation. The following agenda-setting processes were included: spinal cord injury; asthma and COPD (and revision); diabetes; neuromuscular diseases; renal failure; congenital heart disease; burns, and people with intellectual disabilities. The evaluated projects used external facilitators from academic institutes for the agenda setting. No external facilitator was present for programming and implementation. The evaluation took place between May 2010 and April 2012 and was carried out by five academic researchers (author team).

2451

Dutch Burns Foundation

ZonMw

Intellectual disabilities (December 2005–September 2006)

Burns (February 2006–February 2007)

Kidney Fund

Renal failure (December 2005–May 2006)

Dutch Diabetes Foundation

Lung Foundation

Lung Foundation

Revision Asthma/COPD and rare lung diseases (March 2009–September 2009)

Diabetes (March 2006–July 2006)

Lung Foundation

Lung Foundation and ZonMw

Asthma/COPD (August 2003–June 2004)

Dutch Burns Foundation

Dutch Diabetes Foundation

Kidney Fund and renal patient organization ZonMw

ZonMw

ZonMw

Spinal cord injuries (April 2003–April 2004)

Initiator

Funding agency

Disease

Table 1 Overview of research agenda setting projects which used the Dialogue Model

Patients, health care professionals and researchers

Patients

Patients

Patients

Patients, health care professionals and researchers

Patients, health care professionals and researchers

Patients

Consulted

Yes, one-on-one translation, resulting in funding program ‘Voice of the patient’ (Stem van de patient) Yes, one-on-one translation in the funding program of the foundation

Yes, one-on-one translation, resulting in funding program active from 2005 to 2008 Yes, one-on-one translation, resulting in new funding program active from 2009 to 2013 Yes, integration of research themes in their policy and program structure No

No, Though, one research topic of the agenda received funding

Programmed

Within ZonMw patient involvement activities have been organised, e.g. patient partners in funded research, patient reviewers, patients in advisory committees ‘Reviewing committee’ (in which patient representatives are included) assesses relevance of submitted research for patients None

None

Reviewing of research proposals by patient reviewers

Within ZonMw patient involvement activities have been organised, e.g. patient partners in funded research, patient reviewers, patients in advisory committees Reviewing of research proposals by patient reviewers

Role patients in implementation

2452 Involvement in programming and implementation, T A Abma et al.

ª 2014 John Wiley & Sons Ltd Health Expectations, 18, pp.2449–2464

Heart Foundation Congenital heart diseases (June 2007–October 2007)

Heart Foundation

Patients, health care professionals and researchers

Yes, In 2008 a call for research proposals has been financed on research topics of the agenda. In 2012 new call will be organized

Within ZonMw patient involvement activities have been organised, e.g. patient partners in funded research, patient reviewers, patients in advisory committees ‘Selection committee’ (in which one patient representative is included) assesses research proposals No, but, one research topic of the agenda received funding Patients and researchers ZonMw Neuromuscular diseases (April 2006–December 2007)

Patient organization neuromuscular diseases and Dutch Neuromuscular Research Centre

Programmed Funding agency Disease

Table 1. Continued

Initiator

Consulted

Role patients in implementation

Involvement in programming and implementation, T A Abma et al.

ª 2014 John Wiley & Sons Ltd Health Expectations, 18, pp.2449–2464

Design Responsive evaluation is based on a constructivist theory, and assumes that all human beings endow meaning to their experiences.26–32 Construction is influenced by personal background, but outcomes of a socio-cultural process are also influenced by socio-structural and stakeholder positions. Together, the various complementary stakeholder perspectives will lead to a more informed understanding of what is evaluated. Within responsive evaluation the relation between science, policy and activists is understood as a complex interrelation of many stakeholders and contextual factors. In order to generate a holistic understanding of this complex interaction and the meanings endowed to these processes responsive evaluation generates multiple stakeholders perspectives located in a particular time-and-place constellation. The evaluation consisted of four, iterative phases, and results from previous phases were input for later phases. In the first, exploratory phase the nine agenda-setting projects (phases 1–4 of the Dialogue Model) were described using desk study and eight interviews. In the second phase semi-structured interviews were held with stakeholders involved in the agenda-setting projects and/or subsequent programming and implementation phases (patients, policymakers, researchers, project leaders) to gain a detailed understanding of their experiences, the consequences and pitfalls of programming and implementation and the role of patients. Results were validated in the third phase through three focus groups (patients, funding agencies, researchers). Also improvements for the Dialogue Model were identified. Results were disseminated in the final phase at an invitational conference for participants and external audiences. A steering group of six people from science, research councils and patient organizations followed the study with a critical eye. Sample, recruitment and participants The sample is based on maximum variation. We wanted to involve input from as many

2453

2454

Involvement in programming and implementation, T A Abma et al.

stakeholders as possible in the research agenda-setting processes. Inclusion criteria for the interviews were: (i) being involved in one of the nine agenda-setting projects, and/or (ii) having knowledge of the phases after the agenda-setting. The main inclusion criteria for the focus groups were: (i) not having taken part in an interview; (ii) affiliation with a patient organization, health funding agency or research; (iii) having experience with, knowledge of and interest in agenda setting processes. An overview of the 54 respondents is given in Table 2. The research team recruited the participants for the interviews and the focus groups, who were identified during the desk study and exploratory interviews. All participants, except one, agreed to participate. She felt she would not be able to provide any relevant information.

Data collection A topic list was developed for semi-structured interviews. The topic list covered the following topics: (i) experiences with initiating and formulating the research agenda (first four Dialogue Model phases); (ii) verification of the data obtained by the desk study; (iii) experiences with programming and implementation; and (iv) barriers and facilitators for patient involvement in programming and implementation. The interviews were conducted by the second and third authors, who had not been involved in the agenda-setting projects. The interviews were held in a location indicated by the respondent, usually at work or home and lasted approximately 1.5 to 2 h. The focus groups covered three parts: (i) presentation of preliminary results; (ii) inventory of current

Table 2 Consultation overview Research agenda

No. of explorative interviews

No. of in-depth interviews

Spinal cord injuries

Patient organization (p): 1

Asthma/COPD (including update and extension)

Health specific funding agency: 1

Renal failure

Health specific funding agency: 1

Intellectual disabilities

Governmental funding agency: 1

Diabetes

Health specific funding agency: 1

Burns

Health specific funding agency: 1

Neuromuscular diseases

Patient organization (np): 1

Congenital heart diseases

Health specific funding agency: 1

General

0

Total

8

Patient organization (np): 1 Governmental funding agency: 1 Facilitator research agenda: 1 Health specific funding agency: 1 Researcher (members scientific committee): 2 Patient organization (p): 1 Patient organization (np):1 Patient organization (np): 1 Patient organization (p): 1 Facilitator research agenda: 2 Governmental funding agency: 2 Facilitator research agenda: 1 Health specific funding agency: 1 Patient organization (np): 1 Health specific funding agency: 2 Facilitator research agenda: 2 Patient organization (p): 4 Researcher: 6 Governmental funding agency: 2 Facilitator research agenda: 2 Researcher: 2 Facilitator research agenda: 1 Health specific funding agency: 1 Patient organization (np): 2 Researcher: 2 Governmental funding agency: 2 Facilitator research agenda: 1 46

‘Patient organization (p)’ refers to respondents who are patients themselves and are aligned to a patient organization; ‘patient organization (np)’ refers to respondents who are no patients themselves, but who work as a policy maker for a patient organization.

ª 2014 John Wiley & Sons Ltd Health Expectations, 18, pp.2449–2464

2460 Involvement in programming and implementation, T A Abma et al. Table 5 Guidelines for funding agencies and patient organizations Starting position Open attitude towards Willingness towards Adequate resources Expertise Collaboration between fund and patient organization with middle and long-term goals Funding agency

Patient organization

Appoint policy maker for inclusion patient perspective

Appoint patient representative to work with fund, and to provide patient perspective

Programming Include patient representatives in program committee Identify priorities based on research agenda Write scientific article on outcomes research agenda Communicate the funding program and inclusion of patient perspective

Participate in program committee Participate in identification of priorities based on research agenda Co-author in scientific article on outcomes research agenda Review and adjust program text

Implementation Appointment patient reviewers Set up patient assessment criteria for reviewing Involve patients in writing and disseminating call for proposals Pay deliberate attention to patient issues Request lay summary in proposals Inform and train researchers about working with patient research partners Evaluate patient involvement in process of programming and implementation Disseminate evaluation to broad public

activities like patient involvement in agendasetting. Patient organizations and funding agencies should also consider building a relationship with each other, and set middle and long-term goals together. Organizations lacking abovementioned conditions may first invest in optimizing them, and/or should start with small initiatives regarding patient involvement, and build on these initiatives as their position continues to strengthen.

Discussion Research agenda-setting with and for patients is gaining in popularity. Several formal methodologies have been adopted3,4,6,36–40 including

Recruit patient reviewers Provide input and feedback on patient assessment criteria for reviewing Support patient reviewers

Organize pool of patient research partners Evaluate patient involvement in programming and implementation from patient perspective Disseminate evaluation to patients community

the Dialogue Model.7 Scientific literature on the programming and implementation of agendas is still rare. This evaluation study is among the first to assess the influence of agendasetting through the Dialogue Model on the programming and implementation phases. We conclude on the basis of nine agenda setting processes that patient involvement in programming and implementation and the uptake of the patient priorities did not happen automatically, and was influenced by a wide range of factors relating to the context and the process of programming and implementation. Six out of the nine agenda-setting processes initiated by disease-specific charity funds led to formal programming and implementation with

ª 2014 John Wiley & Sons Ltd Health Expectations, 18, pp.2449–2464

Involvement in programming and implementation, T A Abma et al.

limited patient involvement. The other three agenda setting processes, granted by a governmental fund, involved more participatory activities, but less formal programming and implementation. The pattern of no formal programming, but stimulation of patient involvement can be partly explained by the many policy layers of the governmental fund; the policy makers who commissioned the research agendas and were dedicated to patient involvement, were not necessarily directly involved in the related research programmes. It is likely that this pattern is also related to the broader scope of this governmental fund compared to disease-specific charity funds. Further research is required to verify these explanations. Many differences in programming and implementation were observed. A commonality was, however, the unfamiliarity of funding agencies with involving patients in agenda-setting. Beforehand, commissioners did not consider what they wanted to do with the patient priorities and how to sustain patient involvement, resulting in limited results. Patient involvement activities in the evaluated projects were mainly restricted to a few activities, but we did observe attempts to improve the degree of patient involvement. Although this is far from ideal, there was a heightened awareness among participants that standard procedures and organizational routines needed to be adjusted in order to guarantee patient involvement. This is in line with the findings presented by O’Donnell (2004) about the involvement of patients in charity funds in the UK.41,42 They report that many funding agencies (two-thirds) somehow involved patients in research aspects with a wide variety of involvement activities, such as proposal development and review, patients as members in committees and dissemination of results, but that there were many differences in involvement activities. Based on the above findings, suggestions have been put forward to optimize the Dialogue Model. Attention should first be given to the intended use of the agenda in phase 1 (Exploration). In phase 4 (Integration) more attention should be paid to potential users of

ª 2014 John Wiley & Sons Ltd Health Expectations, 18, pp.2449–2464

the agenda. The relevance of advance assessment, the intended use and potential users of findings in decision-making procedures is wellknown in the utilization literature.44,45 Tailoring the agenda-setting process to intended use and the needs of potential users will foster the uptake of the outcomes. Adjusting the timelines and procedures of agenda setting to the decision-making processes of policymakers is also well-known in the utilization literature.46 Furthermore, the research agenda should be formulated comprehensively enough to leave some room for reinterpretation by researchers,47,48 and simultaneously be specific enough to preserve the patient perspective. Deliberate attention needs to be paid in all phases of the Dialogue Model to the attitude towards patient involvement, stakeholder participation and formal and informal relationships. The Dialogue Model was originally grounded in the notion of partnership and dialogue. In order to balance power asymmetries between researchers and patients the model pays a lot of attention to generating a genuine patient agenda and empowerment in a safe environment.22,30 Until now a lot of attention and time in those trajectories has been given to the least heard group. We should, however, bear in mind that researchers and funding agencies also need to be involved in those processes in order to change through it. This is in line with other literature emphasizing ‘Don’t forget the professional’ in processes of patient involvement in research.49 Engaging these stakeholders actively in discussions on patient involvement early in the agenda-setting process and keeping them informed about the outcomes help to build enduring partnerships. During the programming and implementation phases support is needed to involve patients in writing the programme, in the programme committee and in adjusting review procedures. Guidelines have been developed to assist funding agencies and patient organizations in the process. Although the data in this article pertain to the Dutch context, and the experiences only involve agenda-setting in line with the

2461

Involvement in programming and implementation, T A Abma et al. Table 3 Factors influencing programming and implementation of research agendas including the perspectives of patients Context Prior attitude

Stakeholder participation

Time and timing

Organization

Process of programming and implementation Nature of research agenda

Research proposal assessment

dedicated to patient involvement were able to make a difference. For instance, in the Dutch Lung Foundation one person was able to promote patient involvement. Such change agents positively stimulated a higher commitment for patient involvement in programming and implementation (See: Quotes A). Existing partnerships between funding agencies, patient organizations and the research community were also found to be helpful for programming and implementation.

Table 4 Quotes of respondents Quotes A

Quotes B

Quote C

Quotes D

Quotes E

Quotes F

Quotes G

‘You should lose the thought that patients have nothing to tell. (. . .) They [patients] are not stupid!’ (policy maker funding agency, acting as change agent) ‘It [the involvement of patients in agenda-setting] completely depends on individuals’. (researcher) ‘One patient was member of the committee. This person had useful comments and questions, but a long-term vision on research was discussed too little. That hampered the discussion. Patients are positive when there is a very concrete outcome, for example a questionnaire measuring the burden of disease. The long-term vision should include this type of short-term topics’. (member of a Scientific Advisory Committee) ‘In the scientific committee 20 to 25 people voted for, because they thought it was a good research proposal. As a patient I voted against that proposal, but that doesn’t have any effect. They voted and I had a say, but they didn’t count my input [. . .] One vote doesn’t have any effect’. (patient representative). ‘We supposed to integrate themes [based on the outcomes of the research agenda] in the research programme. This did not happen. That could also be an improvement. I think this did not happen, because the phases [of the research agenda setting project and the development of the research programme] were not running parallel. (. . .) It [the research agenda setting project] was somewhat delayed and we had to finish the programme proposal slightly earlier’. (policy maker funding agency) ‘We try to involve a minimum of employees in the agenda setting project. But if you enter such a trajectory, it is very labour intensive’. (policy maker funding agency) ‘If it is ending up in the drawers, that will absolutely not be a matter of unwillingness. It rather relates to the translation [of the research topics into a programme]’. (staff funding agency) ‘You should keep in mind that patient organizations do not receive subsidies. They do not have money. So, a whole bunch of stuff is only possible on project basis’. (patient representative patient organization) ‘I assume that there were plans [for implementation], otherwise we wouldn’t have participated in the agenda-setting. So I guess it is useful to join in, but then you face the fact that you do not have enough people [to carry out those plans]’ (staff patient organization) ‘When you formulate detailed research topics, things [research topics] become fragmented. The question is whether you want that. We have a four year policy mission. We have a horizon that is very distant, and that is where we are heading. All we are doing should contribute to that [the horizon]’. (policy maker funding agency) ‘It was a luminous idea and ultimately we made it very clear in our first agenda-setting project that we wanted to gain more insight in the critical moments patients’ experienced. In their lives and medical history, because we expected that those are the moments when questions arise, and also issues for policy-making’ (staff funding agency) ‘You should take care of the patient volunteers. If you ask people too much in too short a time, then people tune out. Not for just one moment, but then you lose them for your organization. When you, as a patient organization, would like to be an equal partner, then you probably need at some points a slightly modified schedule’. (patient reviewer) ‘Reviewing research proposals sometimes resulted in problems. Especially, when it concerned biomedical research. Researchers often use complicated jargon for patients. As a result the quality of the reviews of patients had a wide range’. (researcher)

ª 2014 John Wiley & Sons Ltd Health Expectations, 18, pp.2449–2464

2457

2458

Involvement in programming and implementation, T A Abma et al.

Stakeholder participation: Crucial for programming and implementation was the involvement of researchers and their willingness to collaborate in multidisciplinary research projects and to focus on topics not directly related to their research profile. Furthermore, when researchers felt co-owners of the agenda, there was a greater chance of implementation. The support and commitment of researchers in decision-making positions, like members of scientific advisory boards, turned out to be crucial. If these boards were critical towards the involvement of patients this often hampered the programming and implementation (See: Quotes B). Time and timing: A misfit between the timelines of agenda-setting and programming, like in the example of spinal cord injuries, hampered the uptake of agenda topics (See: Quote C) Organization: Many funding agencies did not have sufficient resources to support researchers to develop proposals in line with priorities set by patients. This struggle of funding agencies is emphasized in quotes D. Moreover, adaptations of internal structures and procedures were often found necessary for the research agenda to be implemented. This involved also effort from patient organizations, who not always had the resources (See: Quotes E).

Factors related to the process of programming and implementation: In addition, we identified factors relating to the process of programming and implementation, including the nature of the agenda and research proposal assessment. We found that a broader definition of the agenda created more opportunities for multidisciplinary collaboration. However, when research topics became more specific, the chance increased that research proposals were more needs-oriented. It appeared a challenge for funding agencies to find the right balance between a broad formulation of research topics to engage

researchers, and specific research topics reflecting the priorities of patients (See: Quotes F). Furthermore, factors relating to research proposal assessment were identified. When patients reviewed research proposals this fostered the implementation of the patient agenda. A patient reviewer indicated that patients should be supported in reviewing by having assessment criteria specifically drawn up for that purpose. However, such support was often lacking (See: Quotes G). Implications for the Dialogue Model The above factors indicate that both contextual factors and factors relating to the nature of the agenda and review procedures could be steered in the direction of more patient involvement in programming and implementation. In this section we discuss implications for the Dialogue Model. We found that in most instances participants and commissioners had, beforehand, thought little about how to programme and implement the research agenda. The involved organizations were initially not familiar with patient involvement in agenda setting. They were enthusiastic, but did not know what to expect and what aspects required specific attention. One important lesson for the Dialogue Model is therefore to pay attention to programming and implementation right from the start of agenda setting. Stakeholders in phase 1 (Exploration) should deliberate and determine how the research agenda will be used in phase 5 (Programming). In these deliberations the nature of the agenda should be taken into consideration. Topics might be listed one-by-one in a call, and/or agenda topics can be used in a more general sense to steer the programme. If the options are more explicit and discussed at the beginning we expect fewer agendas will remain unutilized. In addition, it is crucial that timelines and decision-making procedures are matched to connect the agenda to the programming phase. Balancing between a general and specific patient agenda appeared to be difficult for

ª 2014 John Wiley & Sons Ltd Health Expectations, 18, pp.2449–2464

doi: 10.1111/hex.12213

Patient involvement in research programming and implementation A responsive evaluation of the Dialogue Model for research agenda setting Tineke A. Abma RN Msc PhD,* Carina A.C.M. Pittens MSc PhD,† Merel Visse MSc PhD,* Janneke E. Elberse MSc PhD† and Jacqueline E.W. Broerse MSc PhD† *Professor and Research Director, Researcher, EMGO+ Institute for Health and Care Research, VU University Medical Centre, Amsterdam and †Researcher, Professor and Director, Faculty of Earth and Life Science Athena Institute for Research on Innovation and Communication in Health and Life Sciences, VU University, Amsterdam, The Netherlands

Abstract Correspondence Tineke A. Abma RN Msc PhD Professor EMGO+ Institute for Health and Care Research VU University Medical Centre Van der Boechorststraat 7 1081 BT Amsterdam The Netherlands E-mail: [email protected] Accepted for publication 23 April 2014 Keywords: Dialogue Model, priority setting, research agenda setting, research programming and implementation, responsive evaluation

Background The Dialogue Model for research agenda-setting, involving multiple stakeholders including patients, was developed and validated in the Netherlands. However, there is little insight into whether and how patient involvement is sustained during the programming and implementation of research agendas. Aim To understand how the Dialogue Model can be optimised by focusing on programming and implementation, in order to stimulate the inclusion of (the perspectives of) patients in research. Methods A responsive evaluation of the programming and implementation phases of nine agenda-setting projects that had used the Dialogue Model for agenda-setting was conducted. Fifty-four semi-structured interviews were held with different stakeholders (patients, researchers, funding agencies). Three focus groups with patients, funding agencies and researchers (16 participants) were organized to validate the findings. Results Patient involvement in programming and implementation of the research agendas was limited. This was partly related to poor programming and implementation, partly to pitfalls in earlier phases of the agenda-setting. Optimization of the Dialogue Model is possible by attending to the nature of the agenda and its intended use in earlier phases. Attention should also be given to the ambassadors and intended users of agenda topics. Support is needed during programming and implementation to organize patient involvement and adapt organizational structures like review procedures. In all phases the attitude to patient involvement, stakeholder participation, especially of researchers, and formal and informal relationships between parties need to be addressed to build a strong relationship with a shared goal. Conclusion Patient involvement in agenda-setting is not automatically followed by patient involvement in programming and

ª 2014 John Wiley & Sons Ltd Health Expectations, 18, pp.2449–2464

2449

2460 Involvement in programming and implementation, T A Abma et al. Table 5 Guidelines for funding agencies and patient organizations Starting position Open attitude towards Willingness towards Adequate resources Expertise Collaboration between fund and patient organization with middle and long-term goals Funding agency

Patient organization

Appoint policy maker for inclusion patient perspective

Appoint patient representative to work with fund, and to provide patient perspective

Programming Include patient representatives in program committee Identify priorities based on research agenda Write scientific article on outcomes research agenda Communicate the funding program and inclusion of patient perspective

Participate in program committee Participate in identification of priorities based on research agenda Co-author in scientific article on outcomes research agenda Review and adjust program text

Implementation Appointment patient reviewers Set up patient assessment criteria for reviewing Involve patients in writing and disseminating call for proposals Pay deliberate attention to patient issues Request lay summary in proposals Inform and train researchers about working with patient research partners Evaluate patient involvement in process of programming and implementation Disseminate evaluation to broad public

activities like patient involvement in agendasetting. Patient organizations and funding agencies should also consider building a relationship with each other, and set middle and long-term goals together. Organizations lacking abovementioned conditions may first invest in optimizing them, and/or should start with small initiatives regarding patient involvement, and build on these initiatives as their position continues to strengthen.

Discussion Research agenda-setting with and for patients is gaining in popularity. Several formal methodologies have been adopted3,4,6,36–40 including

Recruit patient reviewers Provide input and feedback on patient assessment criteria for reviewing Support patient reviewers

Organize pool of patient research partners Evaluate patient involvement in programming and implementation from patient perspective Disseminate evaluation to patients community

the Dialogue Model.7 Scientific literature on the programming and implementation of agendas is still rare. This evaluation study is among the first to assess the influence of agendasetting through the Dialogue Model on the programming and implementation phases. We conclude on the basis of nine agenda setting processes that patient involvement in programming and implementation and the uptake of the patient priorities did not happen automatically, and was influenced by a wide range of factors relating to the context and the process of programming and implementation. Six out of the nine agenda-setting processes initiated by disease-specific charity funds led to formal programming and implementation with

ª 2014 John Wiley & Sons Ltd Health Expectations, 18, pp.2449–2464

Involvement in programming and implementation, T A Abma et al.

limited patient involvement. The other three agenda setting processes, granted by a governmental fund, involved more participatory activities, but less formal programming and implementation. The pattern of no formal programming, but stimulation of patient involvement can be partly explained by the many policy layers of the governmental fund; the policy makers who commissioned the research agendas and were dedicated to patient involvement, were not necessarily directly involved in the related research programmes. It is likely that this pattern is also related to the broader scope of this governmental fund compared to disease-specific charity funds. Further research is required to verify these explanations. Many differences in programming and implementation were observed. A commonality was, however, the unfamiliarity of funding agencies with involving patients in agenda-setting. Beforehand, commissioners did not consider what they wanted to do with the patient priorities and how to sustain patient involvement, resulting in limited results. Patient involvement activities in the evaluated projects were mainly restricted to a few activities, but we did observe attempts to improve the degree of patient involvement. Although this is far from ideal, there was a heightened awareness among participants that standard procedures and organizational routines needed to be adjusted in order to guarantee patient involvement. This is in line with the findings presented by O’Donnell (2004) about the involvement of patients in charity funds in the UK.41,42 They report that many funding agencies (two-thirds) somehow involved patients in research aspects with a wide variety of involvement activities, such as proposal development and review, patients as members in committees and dissemination of results, but that there were many differences in involvement activities. Based on the above findings, suggestions have been put forward to optimize the Dialogue Model. Attention should first be given to the intended use of the agenda in phase 1 (Exploration). In phase 4 (Integration) more attention should be paid to potential users of

ª 2014 John Wiley & Sons Ltd Health Expectations, 18, pp.2449–2464

the agenda. The relevance of advance assessment, the intended use and potential users of findings in decision-making procedures is wellknown in the utilization literature.44,45 Tailoring the agenda-setting process to intended use and the needs of potential users will foster the uptake of the outcomes. Adjusting the timelines and procedures of agenda setting to the decision-making processes of policymakers is also well-known in the utilization literature.46 Furthermore, the research agenda should be formulated comprehensively enough to leave some room for reinterpretation by researchers,47,48 and simultaneously be specific enough to preserve the patient perspective. Deliberate attention needs to be paid in all phases of the Dialogue Model to the attitude towards patient involvement, stakeholder participation and formal and informal relationships. The Dialogue Model was originally grounded in the notion of partnership and dialogue. In order to balance power asymmetries between researchers and patients the model pays a lot of attention to generating a genuine patient agenda and empowerment in a safe environment.22,30 Until now a lot of attention and time in those trajectories has been given to the least heard group. We should, however, bear in mind that researchers and funding agencies also need to be involved in those processes in order to change through it. This is in line with other literature emphasizing ‘Don’t forget the professional’ in processes of patient involvement in research.49 Engaging these stakeholders actively in discussions on patient involvement early in the agenda-setting process and keeping them informed about the outcomes help to build enduring partnerships. During the programming and implementation phases support is needed to involve patients in writing the programme, in the programme committee and in adjusting review procedures. Guidelines have been developed to assist funding agencies and patient organizations in the process. Although the data in this article pertain to the Dutch context, and the experiences only involve agenda-setting in line with the

2461

2462

Involvement in programming and implementation, T A Abma et al.

Dialogue Model, we believe readers might benefit from what is presented here, and transfer some ideas to their own context, based on patterns they recognize or consider relevant. Moreover, international literature describes corresponding challenges in research agenda-setting and follow-up; in most approaches for agendasetting, patient involvement is limited to the actual agenda-setting and there is little understanding of what happens next1,2 and/or how to shape patient involvement activities in follow-up phases.41 The objective of the presented study was to add to that emerging knowledge base, and open new horizons for patient involvement in programming and implementation of patientdriven research agendas.

Acknowledgements We would like to thank the advisory committee for their reflection on and support of our work. Most of all we would like to extend our thanks to all patient representatives, policymakers and researchers who participated in this research. Via interviews and discussions they shared their experiences, opinions and ideas with us.

Funding This work was supported by the Netherlands Organization for Health Research and Development, ZonMw (415011003).

Conflicts of interest The authors declare that they have no competing interests.

References 1 Stewart RJ, Caird J, Oliver K, Oliver S. Patients’ and clinicians’ research priorities. Health Expectations, 2011; 14: 439–448. 2 Mitton C, Smith N, Peacock S, Evoy B, Abelson J. Public participation in health care priority setting: a scoping review. Health Policy, 2009; 91: 219–228. 3 Alliance JL. The James Lind Alliance Guidebook. Oxford: James Lind Alliance, 2010.

4 Malcolm C, Knighting K, Forbat L, Kearney N. Prioritisation of future research topics for children’s hospice care by its key stakeholders: a Delphi study. Palliative Medicine, 2009; 23: 398–405. 5 Owens C, Ley A, Aitken P. Do different stakeholder groups share mental health research priorities? A four-arm Delphi study. Health Expectations, 2008; 11: 418–431. 6 Jones J, Hunter D. Consensus methods for medical and health-services research. British Medical Journal, 1995; 311: 376–380. 7 Abma TA, Broerse JEW. Patient participation as dialogue: setting research agendas. Health Expectations, 2010; 13: 160–173. 8 Abma TA. Patient participation in health research: research with and for people with spinal cord injuries. Qualitative Health Research, 2005; 15: 11310–11328. 9 Teerling J, Caron-Flinterman F, Broerse JEW. Programmering Wetenschappelijk Onderzoek Astma en COPD 2005–2008: ‘De Maatschappelijke Agenda’. Amsterdam: VU University, 2004. 10 Caron-Flinterman JF, Broerse JEW, Teerling J, Bunders JFG. Patients’ priorities concerning health research: the case of asthma and COPD research in the Netherlands. Health Expectations, 2005; 8: 253–263. 11 Caron-Flinterman JF, Broerse JEW, Teerling J et al. Stakeholder participation in health research agenda setting: the case of asthma and COPD research in the Netherlands. Science and Public Policy, 2006; 33: 291–304. 12 RGO. Advies Onderzoeksagenda Medische Biotechnologie. Advisory report. Den Haag: Raad van Gezondheidsonderzoek, 2006 April 2006. Report No., Contract No., publicatie 53. 13 Elberse J, Laan D, de Cock Buning T, Teunissen T, Broerse J, de Boer W. Patient involvement in agenda setting for respiratory research in the Netherlands. European Respiratory Journal, 2012; 40: 508–510. 14 Nierse CJ, Schipper K, van Zadelhoff E, van de Griendt J, Abma TA. Collaboration and coownership in research: dynamics and dialogues between patient research partners and professional researchers in a research team. Health Expectations, 2011; 15: 242–254. 15 Nierse C, Abma T, Broerse JEW et al. Engaging people with intellectual disabilities in health research agenda setting. Journal of Applied Research in Intellectual Disabilities, 2007; 33: 84–96. 16 Broerse JEW, Zweekhorst MBM, De Groot J, Oud AMM. Patientenraadpleging Diabetes Fonds. Amsterdam: Athena Instituut, VU University, 2006. 17 Broerse JEW, Zweekhorst MBM, van Rensen AJML, de Haan MJM. Involving burn survivors in

ª 2014 John Wiley & Sons Ltd Health Expectations, 18, pp.2449–2464

Involvement in programming and implementation, T A Abma et al.

18

19

20

21

22

23

24

25

26

27

28 29

30

agenda setting on burn research: an added value? Burns, 2010; 36: 217–231. Nierse CJ, Abma TA, Horemans AMC, van Engelen BGM. Research priorities of patients with neuromuscular disease. Disability and Rehabilitation, 2012; 35: 405–412. Elberse JE, Caron-Flinterman JF, Broerse JEW. Patient–expert partnerships in research: how to stimulate inclusion of patient perspectives. Health Expectations, 2011; 14: 225–239. Elberse JE, Pittens C, Buning TDC, Broerse JEW. Patient involvement in a scientific advisory process: setting the research agenda for medical products. Health Policy, 2012; 107: 231–242. Konijn W, Elberse JE, Pittens CACM, Broerse JEW. Onderzoeksagenda Dementie. Behoeften van Mensen met Dementie, Mantelzorgers, Burgers en Onderzoekers. [Research Agenda Dementia Needs of People With Dementia, Informal Carers, Citizen and Researchers]. Amsterdam: VU University, in opdracht van Stichting Alzheimer Nederland, 2011. Baart ILMA, Abma TA. Patient participation in fundamental psychiatric genomics research: a Dutch case study. Health Expectations, 2011; 14: 240–249. Schipper K, Abma TA. Coping, family and mastery: top-priorities for social-scientific research by kidney patients. Nephrology, Dialysis & Transplantation, 2011; 4: 1–7. Schipper K, Dauwerse L, Hendrikx A, Leedekerken JW, Abma TA. Living with Parkinson’s Disease: priorities for research suggested by patients, Parkinsonism, accepted. Elberse JE, Broerse JEW. De rol van pati€enten in onderzoek. In: Slager M, Dedding C (eds) Rafels van Participatie in de Gezondheidszorg. Den Haag: Boom Lemma uitgevers, 2013: 87–110. Abma TA. Responsive evaluation: its meaning and special contribution to health promotion. Evaluation and Program Planning, 2005; 28: 279–289. Stake RE. To evaluate an arts program. In: Stake RE (ed.) Evaluating the Arts in Education: A Responsive Approach. Columbus, OH: Merrill, 1975: 13–31. Guba EG, Lincoln YS. Fourth Generation Evaluation. Beverly Hills, CA: Sage, 1989. Widdershoven GAM, Abma TA. Hermeneutic ethics between practice and theory. In: Ashcroft RE, Dawson A, Draper H, McMillan JR (eds) Principles of Health Care Ethics. West Sussex: Wiley, 2007: 215–222. Nierse CJ, Abma TA. Developing voice and empowerment: the first step towards a broad consultation in research agenda setting. Journal of Intellectual Disability Research, 2011; 55: 411–421.

ª 2014 John Wiley & Sons Ltd Health Expectations, 18, pp.2449–2464

31 Abma TA, Widdershoven GAM. Evaluation as a relationally responsive practice practice. In: Denzin N, Lincoln YS (eds) Qualitative Research. Los Angelos, CA: Sage Publications, 2011: 669–680. 32 Abma TA, Nierse CJ, Widdershoven GAM. Patients as partners in responsive research: methodological notions for collaborations in mixed research teams. Qualitative Health Research, 2009; 19: 401–415. 33 Broerse JEW, Bunders JFG. Requirements for biotechnology development: the necesity of an interactive and participatory innovative development. International Journal of Biotechnology, 2000; 2: 275–296. 34 Teunissen GJ, Visse MA, de Boer WI, Abma TA. Structuring patient advocates’ appraisal and evaluation of health research and quality of care. Journal of Participatory Medicine, 2013; 5: e16. 35 Pittens CACM, Elberse JE, Visse M, Abma TA, Broerse JEW. Research agendas involving patients: factors that facilitate or impede translation of patients’ perspectives in programming and implementation. Social Policy and Society, 2014. doi: 10.1093/scipol/scu010. 36 Elwyn G, Crowe S, Fenton M et al. Identifying and prioritizing uncertainties: patient and clinician engagement in the identification of research questions. Journal of Evaluation in Clinical Practice, 2010; 16: 627–631. 37 Lloyd K, White J. Democratizing clinical research. Nature, 2011; 474: 277–278. 38 Gallagher M, Hares T, Spencer J, Bradshaw C, Webb I. The nominal group technique - a research tool for general-practice. Family Practice, 1993; 10: 76–81. 39 Dewar A, White M, Posade ST, Dillon W. Using nominal group technique to assess chronic pain, patients’ perceived challenges and needs in a community health region. Health Expectations: An International Journal of Public Participation in Health Care and Health Policy, 2003; 6: 44–52. 40 Nasser M, Welch V, Ueffing E, Crowe S, Oliver S, Carlo R. The Cochrane Collaboration review prioritization projects show that a variety of approaches successfully identify high-priority topics. Journal of Clinical Epidemiology, 2012; 66: 474–482. 41 O’Donnell M, Entwistle V. Consumer involvement in decisions about what health-related research is funded. Health Policy, 2004; 70: 281–290. 42 O’Donnell M, Entwistle V. Consumer involvement in research projects: the activities of research funders. Health Policy, 2004; 69: 229–238. 43 Broerse JEW, Elberse JE, Caron-Flinterman JF, Zweekhorst MBM. Enhancing a transition towards a needs-oriented health research system through patient participation. In: Broerse JEW, Bunders

2463

2464 Involvement in programming and implementation, T A Abma et al. JFG (eds) Transitions in Health Systems: Dealing With Persistent Problems. Amsterdam: VU University Press, 2010. 44 Weiss C. If program decisions hinged only on information: a response to Patton. Evaluation Practice, 1998; 9: 15–28. 45 Patton MQ. Utilization-Focused Evaluation. Beverly Hills, CA: Sage, 1978. 46 Stufflebeam DL, Foley WJ, Gephart WJ et al. Educational Evaluation and Decision Making. Itasca, IL: F.E. Peacock, 1971.

47 Mackey ER. Street-Level Bureaucrats and the Shaping of University Housing Policy. Fayetteville, AR: University of Arkansas Press, 2008. 48 Lipsky M. Street-Level Bureaucracy: Dilemmas of the Individual in Public Services. New York: Russell Sage Foundation, 1980. 49 de Wit MPT, Elberse JE, Broerse JEW, Abma TA. Do not forget the professional – the value of the FIRST model for guiding the structural involvement of patients in rheumatology research. Health Expectations, 2013.

ª 2014 John Wiley & Sons Ltd Health Expectations, 18, pp.2449–2464

Patient involvement in research programming and implementation: A responsive evaluation of the Dialogue Model for research agenda setting.

The Dialogue Model for research agenda-setting, involving multiple stakeholders including patients, was developed and validated in the Netherlands. Ho...
131KB Sizes 0 Downloads 3 Views