Conflict of Interest and Evaluation Research: Should We Do Effectiveness Studies of Our Own Educational Programs?


Although little is known about the extent of conflict of interest (COI) in the field of continuing education in the health professions, my reading of the literature suggests that most of the evaluation research reported in the literature is conducted by investigators who were involved in the design and implementation of the program. The result is an increased potential for an actual or perceived COI. Internal evaluators often have a stake in the success of the intervention, increasing the risk of unintentional or even intentional bias. Unfortunately, we lack data on the extent of the problem and the association between COI and evaluation results in our field. In the field of criminology, however, Petrosino and Soydan1 reviewed meta-analyses of the impact of mixing the roles of program developer and program evaluator in interventions aimed at reducing criminal recidivism. Of the 12 analyses that addressed the issue, 11 showed that “effect size increased positively, sometimes substantially so, when evaluators were influential or involved in the treatment setting.”1,p 435 They followed up with their own meta-analysis of 300 randomized field trials assessing the outcomes of offender treatment programs. They found that developersas-evaluators reported “consistently and substantially larger effect sizes”1,p 442 when the evaluator had a high degree of influence in the program setting. Petrosino and Soydan emphasized that there are two possible explanations for these effects. One is the high fidelity view, which assumes that high involvement of researchers leads to better outcomes due to factors such as better training for the faculty, closer monitoring of implementation, or Disclosures: The author reports none. Dr. Olson: Editor-in-Chief, Journal of Continuing Education in the Health Professions. Correspondence: Curtis A. Olson, Center for Continuing Education in the Health Professions, One Medical Center Drive, Lebanon, NH 03766; e-mail: [email protected] © 2013 The Alliance for Continuing Education in the Health Professions, the Society for Academic Continuing Medical Education, and the Council on Continuing Medical Education, Association for Hospital Medical Education. • Published online in Wiley Online Library (wileyonlinelibrary.com). DOI: 10.1002/chp.21192

the infectious enthusiasm of the investigator. The other is the cynical view, in which the investigator feels pressured to present their program in the best possible light. Pressure may come from concerns about prospects for future funding, gaining or maintaining prestige, or a desire to enhance the marketability of the program. Eliminating COIs in evaluation research may be neither desirable nor even possible. As Winch and Sinnott observed, COI policies and regulations are linked to the enlightenment notion that progress in human affairs is gained “through the application of value-free knowledge gained in an objective, scientific way.”2,p 390 However, these policies are sometimes construed in a simplistic, dualistic way (eg, conflict of interest is bad; having neutral, detached investigators in evaluation research is good). The problem is that as a practical matter, many or even most evaluations involve persons with significant potential or real conflicts of interest.3 Furthermore, as a matter of best practice and principle, participatory approaches to evaluation purposefully involve people who have a vested interest in the program, such as program planners and faculty.4 This is done for a variety of reasons such as enhancing evaluation utilization, enhancing self-determinism, building evaluation capacity, and deepening understanding of a program, its context, and impacts. An alternative to eliminating COIs is to begin with the assumption that investigators have multiple and varying interests and that a potential conflict is not, in itself, grounds for excluding someone from participation in an evaluation. The task then becomes identifying and managing potential conflicts. This editorial explores sources of conflicts of interest, critically examines several potential strategies for dealing with COIs, and, drawing on the evaluation standards literature,3,5 offers some general guidelines for identifying and managing COIs in the conduct and reporting of evaluation research studies.

Sources of COI There are many sources of conflicts of interest. Some of the more widely recognized sources are as follows:



• Financial relationships. The evaluator’s livelihood or ability to get future grants may be affected by the outcome of the evaluation. • Personal or organizational relationships. Evaluators may have personal relationships with people involved in the program or have themselves been heavily involved. • Personal beliefs and experiences. A person who has been either the victim or a perpetrator of domestic violence, for example, may find it difficult to render a balanced assessment of a program designed to teach health care providers to recognize and support victims among their patients.

How can situations like these be managed? Problematic Strategies for Avoiding Conflicts There are several strategies that initially seem promising do not hold up under scrutiny. Using External Evaluators Although it may appear to be a solution to the COI problem, recruiting an external evaluator does not eliminate potential conflicts of interest. Assuming that independent, nationally known experts are free of conflicts is a common error.3 External evaluators may shape an evaluation to help them obtain contracts with future clients. They may also have relationships with people in competing organizations. External evaluators may be chosen because they are expected to provide a favorable assessment. Furthermore, external evaluators may lack essential knowledge and experience, such as an in-depth understanding of the organization or the program being evaluated or personal experience with the problems that the program’s participants confront, limiting their ability to make useful recommendations.6

ing disclosure does not ensure that all relevant conflicts are identified. Identifying and Managing Conflicts Given that many evaluation studies involve investigators with potential COIs, the problem is less how to avoid them than to deal with them.3 There are some mechanisms in place to safeguard against the influence of COIs in works that are published in the scholarly literature. Journals expect authors to disclose potential sources of COI and manuscripts are subject to peer and editorial review. However, investigators also have a responsibility to proactively address COI issues in their everyday practice.5 The remainder of this editorial will present several principles, drawn primarily from the evaluation standards literature, that address the investigator’s role. The overarching goal served by these principles is that “Conflict of interest should be dealt with openly and honestly, so that it does not compromise the evaluation processes and results.”5 It is important to note that these principles do not apply to every situation and there are times when not following a principle is justifiable. Planning the Study 1. Identify and describe COIs in the early stages of the evaluation planning process.3 Early identification is necessary if COIs are to be proactively managed. 2. Develop a written plan for procedures to be used to protect against problems.3 Formalizing the plan provides transparency and helps ensure that potential conflicts are addressed. 3. Seek input from persons who have different perspectives on the evaluation to stay open to alternative evaluation approaches.3 This step can help identify unconscious bias in the choice of methods.

Reliance on Methodology Another common error is to assume that following wellestablished evaluation research procedures will eliminate conflicts of interest.3 Investigators must inevitably make several key decisions that CAN influence the findings of a study (eg, phrasing of survey questions); there is potential for COI to exert influence at each decision point. Interpretation of results is similarly vulnerable. Disclosure A third error is believing that calling attention to a real or potential COI is sufficient to address the problem.3 There are many ways that a conflict can influence an evaluation, and even an informed, critical reader of an evaluation report may overlook problems, especially when they are problems of omission rather than commission. In addition, requir204

Implementation 1. To the extent possible, isolate investigators with potential high-risk conflicts from key decisions. These decisions should be made by less conflicted investigators or reviewed by a less conflicted individual or group. 2. Document and disclose any changes in the evaluation plan and the reasons why. This practice mitigates concerns that unfavorable results may have been excluded from the analysis or that the methodology was altered to produce more favorable results.

Reporting 1. Disclose sources of potential COI for each investigator in evaluation reports.7 The disclosure should include a description of each investigator’s role in the development and implementation of the intervention.


Conflict of Interest and Evaluation Research

2. Disclose all sources of financial support for the evaluation. The source of the request for the evaluation should also be reported.3 3. In published reports of the evaluation, describe evaluation procedures, data, and the basis for conclusions clearly and in sufficient detail, so that they can be judged by other independent evaluators and peer reviewers.3 Doing so documents the grounds for interpretations and conclusions and enhances their defensibility. 4. Provide a balanced assessment of the intervention, addressing the strengths and weaknesses of the intervention being evaluated. For example, in describing participant comments about the intervention, indicate the extent to which a viewpoint was shared by others (eg, “most participants described changing their approach to assessing risk of opioid abuse”) and include dissenting perspectives (eg, “two participants stated they found the opioid risk abuse assessment process cumbersome and difficult to implement”). 5. Within reason, evaluators should attempt to prevent or correct misuse of the evaluation report by others.5 Some readers will lack the ability to make an independent judgment of the quality of the evaluation or will draw conclusions that are not supported by the study. Investigators should ensure that their evaluation report includes a discussion of the most important limitations of the data and phrase their interpretations and conclusions in a way that discourages overgeneralization.

To return to the question posed in the title of this editorial, I believe we can serve as both program developers

and evaluators of program effectiveness, provided appropriate safeguards are in place. Persons who are uniquely qualified to be involved in the evaluation need not be excluded solely due to fear of conflict of interest allegations.3 No one is completely free of conflict and bias; however, that reality need not undermine the integrity of an evaluation.6 Our obligation as evaluators, peer reviewers, and editors is to be vigilant about potential conflicts of interest and take steps to ensure they are dealt with in an explicit and transparent manner. References 1. Petrosino A, Soydan H. The impact of program developers as evaluators on criminal recidivism: Results from meta-analyses of experimental and quasi-experimental research. Journal of Experimental Criminology. 2005;1(4):435–450. 2. Winch S, Sinnott M. Toward a sociology of conflict of interest in medical research. J Bioeth Inq. 2011;8:389–391. 3. Joint Committee on Standards for Educational Evaluation. The Program Evaluation Standards: A Guide for Evaluators and Evaluation Users. 3rd ed. Thousand Oaks, CA: Sage; 2011. 4. Whitmore E, ed. Understanding and Practicing Participatory Evaluation. San Francisco, CA: Wiley; 1998. New Directions for Evaluation (Issue #80). 5. American Evaluation Association. Guiding principles for evaluators. 2005. http://www.eval.org/p/cm/ld/fid=51. Accessed September 5, 2013. 6. Fitzpatrick JL. Conflict of interest. In: Mathison S, ed. Encyclopedia of Evaluation. Thousand Oaks, CA: Sage; 2005:25.



Conflict of interest and evaluation research: should we do effectiveness studies of our own educational programs?

Conflict of interest and evaluation research: should we do effectiveness studies of our own educational programs? - PDF Download Free
60KB Sizes 0 Downloads 0 Views