Journal of Clinical Epidemiology

-

(2015)

-

ORIGINAL ARTICLE

Elements of informed consent and decision quality were poorly correlated in informed consent documents Jamie C. Brehauta,b,*, Kelly Carrolla, Glyn Elwync, Raphael Saginurd, Jonathan Kimmelmane, Kaveh Shojaniaf, Ania Syrowatkag, Trang Nguyenh, Dean Fergussona a

Clinical Epidemiology Program, Ottawa Hospital Research Institute, The Ottawa Hospital, General Campus, 501 Smyth Road, Centre for Practice-Changing Research, Box 201B, Ottawa, Ontario, Canada K1H 8L6 b School of Epidemiology, Public Health and Preventive Medicine, University of Ottawa, 451 Smyth Road, Ottawa, Ontario, Canada K1H 8M5 c The Dartmouth Center for Health Care Delivery Science and The Dartmouth Institute for Health Policy and Clinical Practice, Dartmouth College, 37 Dewey Field Road, Hanover, NH 03755, USA d Clinical Epidemiology Program, Ottawa Hospital Research Institute, The Ottawa Hospital, Civic Campus, 1053 Carling Avenue, Ottawa, Ontario, Canada K1Y 4E9 e Biomedical Ethics Unit, McGill University, 3647 Peel Street, Montreal, Quebec, Canada H3A 1X1 f Department of Medicine, Sunnybrook Health Sciences Centre and University of Toronto Centre for Patient Safety, 2075 Bayview Avenue, Toronto, Ontario, Canada M4N 3M5 g Department of Epidemiology, Biostatistics and Occupational Health, McGill University, 1020 Pine Avenue West, Montreal, Quebec, Canada H3A 1A2 h Population Health Modeling, Social Determinants and Science Integration Directorate, Public Health Agency of Canada, 785 Carling Ave, Ottawa, Ontario, Canada K1A 0K9 Accepted 3 March 2015; Published online xxxx

Abstract Objectives: Although informed consent (IC) documents must contain specific elements, inclusion of these elements may be insufficient to encourage high-quality decision making. We assessed the extent to which documents conform to IC standards and how well conformity to decision quality (DQ) standards can be predicted by IC standards, IC document characteristics, and study characteristics. Study Design and Setting: We obtained 139 IC documents for trials registered with ClinicalTrials.gov from study investigators. Using a four-point scale, two raters independently assessed each IC document on 36 IC standard items and 9 DQ items. Results: Overall agreement between raters across all 45 items was 93%. Across the 36 IC standards items, conformity was generally quite high but variable, with 20 items showing conformity of 80% or more and seven items showing conformity of 50% or lower. IC standards concordance, overall length of the IC document, and country of study were all significant predictors of DQ standards but together accounted for less than 20% of the variance in DQ standards. Conclusion: Conformity to recommendations for improving IC documents was relatively high but variable. The extent to which an IC document conformed to these recommendations was only moderately related to whether it conformed to recommendations for improving DQ. Existing IC regulations may not describe the optimal approach to helping people make good study participation decisions. Ó 2015 Elsevier Inc. All rights reserved. Keywords: Informed consent; Decision quality; Clinical trials; Research participants; Trial participation; Decision support

Ethics approval: This work was reviewed by the Ottawa Health Science Network Research Ethics Board (OHSN REB). As Chair of the OHSN REB, R.S. recused himself from all consideration of this study. Funding: This work was supported by the Canadian Institutes of Health Research through a CIHR New Investigator Award in Knowledge Translation (to J.C.B.) and an Operating grant (CIHR #MOP-86753; J.C.B. PI). The funding agency had no role in the study design, collection, analysis, or interpretation of data, writing of the manuscript, or in the decision to submit the manuscript for publication. J.K. also holds a http://dx.doi.org/10.1016/j.jclinepi.2015.03.002 0895-4356/Ó 2015 Elsevier Inc. All rights reserved.

CIHR New Investigator Salary Award. R.S. is recipient of a research salary award from the Department of Medicine, University of Ottawa. K.S. holds a Canada Research Chair in Patient Safety and Quality Improvement. A.S. and T.N. both received a CIHR Frederick Banting & Charles Best Canada Graduate Scholarships Master’s Award in 2009 and 2010, respectively. Conflict of interest: None. * Corresponding author. Tel.: 613-737-8899x73820; fax: 613-7396938. E-mail address: [email protected] (J.C. Brehaut).

2

J.C. Brehaut et al. / Journal of Clinical Epidemiology

1. Introduction The informed consent (IC) process is the chief source of information for potential health research participants making difficult decisions about study participation. Recently, debate has formed around a decision by the U.S. Office of Human Research Protections to question the adequacy of IC in a highly prominent neonatal trial (the SUPPORT trial) [1e7]. This trial, despite having been reviewed and approved by no less than 23 different research ethics boards [7], was determined not to meet IC regulations. Such uncertainty around basic IC issues helps explain why many study participants do not understand even basic components of the studies they have agreed to join [8e10]. Core criteria for IC include a voluntary decision, capacity to understand the relevant information, disclosure of all relevant information, and comprehension of the information [11]. Increasingly, however, a fundamental tension between the latter two criteria has been identified; pressures toward disclosing more relevant information may come at a cost to overall comprehension. Although evidence suggests that disclosure of relevant information has increased over recent years [12,13], so too has the length of IC documents [12,14,15], and many have argued that provision of too much detail in IC documents leads to a poorer process overall [14,16,17]. Efforts to assess IC documents [9,12,13,18e21] have most commonly focused on presence of specific elements within the documents, including (1) core study elements (e.g., rationale, procedures involved, risks/benefits), (2) specific rights (e.g., rights to withdraw, protection of confidentiality) [22e26], and (3) formatting recommendations (e.g., readability, nontechnical language) [23,25,26]. We believe such information is insufficient to ensure IC. Instead, we have argued for an alternative model, where a core goal of the process is ensuring a high-quality decision, defined as one that involves demonstrable knowledge of key aspects of the decision, accurate perceptions of the probabilities of outcomes, and a match between preferred outcomes and the choice made [27,28]. The patient decision aid literature has sought to achieve these ends in the context of treatment and screening decisions [28], and the current work continues our exploration into whether this approach can lead to a better IC process. As a first step, we assessed IC documents from a random sample of clinical trials in light of a group of recommendations for creating documents that encourage good-quality decisions [decision quality (DQ) or DQ Standards]. These recommendations were based on the International Patient Decision Aids Standards instrument (IPDASi [29e31]), a set of validated, empirically supported standards describing how to encourage good-quality decision making. A sample of 139 IC documents showed less than 10% concordance with 15 of the 32 standards [32]. Despite poor concordance overall, there was considerable variation among IC documents in how well they

-

(2015)

-

conformed to individual DQ standards. Different possible explanations for this variation exist. One hypothesis is that concordance with DQ standards is simply determined by how detailed a IC document is; the more information a document provides, the more it is likely to include components conducive to encouraging a better-quality decision. If so, longer IC documents should be correlated with conformity to DQ standards. Another might be that IC documents that conform to existing IC recommendations will lead to better quality decision making; in essence, the degree to which an IC document serves to encourage a high-quality decision-making process is related to the extent to which it conforms to existing IC recommendations. If so, then documents that meet more IC recommendations will conform to more DQ standards. This study describes the extent to which a sample of IC documents conformed to a range of IC standards (IC Standards; that is, recommendations for the content of IC documents suggested by policy and the IC literature ([22e25]) and whether conformity to these IC standards varied with IC document characteristics (length, readability) and study characteristics. We also sought to understand how well conformity to DQ standards were predicted by IC standards, IC document characteristics (length, readability), or study characteristics (phase of study, clinical area, country of origin).

2. Methods Analyses for this study are based on the same sample of IC documents described in our previous work [32]. DQ items are described in more detail in the previous study. The present study describes the IC items and their association with a subset of DQ items. 2.1. Item identification We detailed the process of DQ item identification in our previous study [32]. In brief, we began with an initial set of 47 items organized in 10 domains comprising the IPDAS instrument (IPDASi) by Elwyn et al. [30], an instrument originally designed and validated to evaluate the quality of decision support tools designed for treatment and screening decisions. We went through an iterative process to identify items that fell outside the scope of our study, did not apply well to the IC document application, or overlapped with other items. A total of 17 items were dropped based on this process. Another seven items were added based on team discussions around how to apply IPDAS principles in the IC document context. Several items were split into multiple items, or merged into a single item, to improve clarity for coders. The result was 32 items, organized into five separate domains. Based on previously reported findings [32], a number of the items showed little or no concordance across our sample of IC documents and so were not included in the current

J.C. Brehaut et al. / Journal of Clinical Epidemiology

(2015)

-

3

literature [39]. Two reviewers examined these materials for recommendations about information to include in IC documents or guidance about how to present that information. Specifically, we sought recommendations that were neither disease nor discipline specific and that would thus be generalizable to a broad sample of IC documents drawn from different clinical disciplines and study types. For example, recommendations about research-related injury compensation were excluded due to variability across various consent forms. After an iterative process of item evaluation similar to the one discussed previously, we settled on 36 items organized into four categories for descriptive purposes: regulatory statements (11 items), design and procedures (12 items), formatting/style (7 items), and items recommended by both IC and DQ standards (6 items).

What is new? Key findings  The extent to which an informed consent (IC) document encourages high-quality decision making is related to how well it conforms to existing IC standards but only moderately. What this adds to what was known?  The results suggest that a good-quality decisionmaking process is not implied given existing IC regulations. What is the implication and what should change now?  Decision quality standards provide novel, clear, and testable hypotheses about how to help potential research participants make difficult study participation decisions.

2.2. Sampling IC documents We have detailed our sample and sampling strategy elsewhere [32]. Briefly, we selected studies registered with ClinicalTrials.gov, excluding (1) studies originating outside Canada, the United States, Australia, or the United Kingdom; (2) studies listed as funded solely by industry; (3) studies involving proxy decision makers or children; (4) study designs without clear interventions; (5) phase 4 trials, many of which would not involve IC; (6) studies listed as withdrawn; and (7) trials with a start date before year 2000. The resulting sample was 8,302 trials. Our goal was to obtain 100 IC documents, approximately half from early (phase 1, 2, or 1e2) and half from later phase trials (2e3, 3). Based on pilot data suggesting we might expect a 20% response rate, we identified a random sample of 500 studies to contact (250 early and 250 late). We obtained the name and contact information of the principal investigators and contacted them via e-mail asking them to send us

analyses. The nine items depicted in Fig. 1 are those that were unique to the DQ item set and that showed at least 10% concordance in the current sample of IC documents. Item identification for IC standards first involved systematically examining regulatory guidelines including Health Canada [23], the Tri-Council Policy Statement [22], the US Code of Federal Regulations (45 CFR 46.116) [25], the Food and Drug Administration [24], templates and guidance from key funding bodies [26], research ethics board guidance ([33,34], the UK General Medical Council document [35], existing scales assessing components of IC [13,14,19,36e38], and select articles from the ethics 0%

-

20%

40%

60%

80%

100%

% SA & A

Decision Quality Standards (9 items) Describes that a participation decision needs to be made

54%

Describes disadv. of participation in a consistent order

37%

Describes the health condition or problem

32%

Comparision of probabilities using same denominator

19%

Provides info. on probabilities for all adv/disadv

19%

Specifies probabilities in natural frequencies

14%

Provides info. about levels of uncertainty around probabilities

13%

Provides more than one way of viewing probabilities

13%

Specifies populations that yielded the probabilities

10%

Strongly Agree

Agree

Neutral

Disagree

Strongly Disagree

Fig. 1. Percentage ratings of the 139 informed consent documents for each of the DQ standard items.

4

J.C. Brehaut et al. / Journal of Clinical Epidemiology

an example of their IC documents. All identifying information (investigator names, contact information, institutions) was redacted by the research coordinator. 2.3. Assessment Each of the 45 items (9 DQ items and 36 IC items) was independently rated by two trained individuals on a fourpoint scale (0, strongly disagree; 1, disagree; 2, agree; and 3, strongly agree), consistent with the original IPDASi instrument. After achieving reliability with the trainer on a series of pilot IC documents, two of five possible coders rated the IC documents; consensus meetings resolved discrepancies of two or more points on the scale for the first 60 IC documents, and after assessment of score changes showed items means changing minimally after consensus meetings, mean of the two ratings was taken thereafter. Ratings of 2 or 2.5 were rated as agree; only a rating of 3 was counted as strongly agree. In addition to the ratings, we extracted study descriptive information from the ClinicalTrials.gov database. This information included phase of the trial (early: phases 1, 2, or 1/2; late: phases 2, 3, or 2/3), health condition (cancer related, other), country (United States, other), study recruitment status (trial listed as completed, active but not recruiting, or suspended/terminated), funding sources (industry involved, National Institutes of Health (NIH) with no industry involved, other), and number of months between trial registration and data extraction (January 2011). We also assessed document length (number of words) and readability (Using Microsoft Word’s Flesch Reading Ease).

-

(2015)

-

registration, and funding source. We excluded the recruitment status variable a priori because there was no clear prediction about how the variable might relate to the DQ outcome. Overall R2adj showed the amount of variance predicted by all variables in predicting the DQ percentage score.

3. Results Response rate, sample frame characteristics, and analyses of bias are presented in detail as part of our previous study [32]. In brief, our sample frame ended up being equally split across early/late phase trials (by design) and cancer/noncancer studies (by chance). In addition, random selection resulted in a sample frame that involved primarily US trials (85.8%), a mix of completed (44.8%), active (45.8%), and suspended/terminated trials (9.4%); 25.4% of the trials in the sample frame had some industry involvement, 47.8% were funded by NIH alone, and 26.8% involved other funding bodies. Overall, we were unable to contact 11 investigators and obtained 162 IC documents, for a response rate of 32.4% (162 of 489). Late phase trials, noncancer trials, and more recent trials were more likely to provide an IC document; study recruitment status, funding source, and country of study origin were not related to IC provision. Of the 162, one was a pediatrics study involving proxy consent and 22 were template forms and were therefore excluded, leaving us with a final sample of 139. Table 1 describes our final sample of 139 complete consent forms. The majority were later phase, noncancer trials

2.4. Analyses

Table 1. Description of sample of 139 IC documents

We used descriptive statistics, chi-square tests (for categorical variables), and t-tests (for continuous variables) to compare whether studies with different characteristics were more or less likely to provide us with a consent form, using a 0.05 alpha level of significance. The extent to which the 139 IC documents conformed to the individual items was summarized with percentages, means, and ranges. To explore the relationship between IC standards, IC document characteristics, and study characteristics on concordance with DQ standards, we summed the ratings scores of the nine DQ standards and converted them into a percentage summary score, where 100% corresponded to a strongly agree rating of 3 for all 9 items in the subsection score (i.e., 27 of 27 5 100%). We computed a similar total IC summary score. Bivariate correlations were described with Pearson or Spearman correlations as appropriate, using Cohen’s descriptors of 0.1 as weak, 0.3 as moderate, and O0.5 as large [40]. As a final analysis, we regressed DQ scores on total IC summary scores, reading ease scores, word count, phase of trial, health condition, country, months since trial

Trial descriptor Phase of trial Early Late Health condition Cancer Other Country United States Other Study recruitment status Completed Active, not recruiting Suspended/terminated Funding Industry involved NIH involved, no industry Other Months since trial registration (mean, range) Readability score (mean, range) Word count (mean, range)

# of documents

% Of 139 documents

62 77

44.6 55.4

58 81

41.7 58.3

116 23

83.5 16.5

63 67 9

45.3 48.2 6.5

36 54 49 76.33

25.9 38.8 35.3 29e133

51.63 3,608

33.40e70.60 499e8,464

Abbreviations: IC, informed consent; NIH, U.S. National Institutes of Health.

J.C. Brehaut et al. / Journal of Clinical Epidemiology

based in the United States. Mean length of time since trial registration to initial analyses was 76 months (range, 29e133). Mean Flesch Reading Ease scores across all IC documents were 51.6 (range, 33.4e70.6), whereas mean word count was 3,601 (range, 499e8,464). Overall agreement between two raters before consensus across all 36 IC and 9 DQ items and all 139 IC documents, with agreement defined as no more than a one-point difference on the four-point scale, was 93.0%. Item-specific agreement ranged from 79% (for one item) to 99.3% (for two items). 3.1. Concordance with IC standards Fig. 2 describes the extent to which our sample of 139 IC documents conformed to each of 36 IC standard items. Among the 11 regulatory statements, concordance was highly variable, with IC documents conforming very well

-

(2015)

-

5

to some items (e.g., ability to withdraw from study at any time, participation in the study is voluntary, the study involves research; ratings of strongly agree (SA) or agree (A) for over 90% of IC documents) but variably to others (reasons why participation may be terminated describes how personal health information will be protected; SA or A in !60% of IC documents). Only one item (compensation for patient enrollment) showed uniformly low concordance (SA or A in 6% of IC documents). Among the 12 items focused on design and procedures, concordance was similarly variable, with six items showing over 70% concordance (different groups to which the participants can be assigned, rationale for the research, procedures are explained, randomization explained, expected number of participants in the study; SA or A for O70% of IC documents), although three items showed very little concordance (reversibility of side effects, expected duration

Informed Consent Standards 0%

Regulatory Statements (11 items)

20%

40%

60%

80%

100%

% SA & A

Statement about the withdrawal from the study at any time

98%

Statement that participation is voluntary

95%

Statement that the study involves research

92%

Describes who will have access to personal information

89%

Statement about the informing of new evidence

81%

Statement that participation may benefit others/society

78%

Describes risks to an embryo, fetus, or nursing infant (n = 135)

78%

Name of the research ethics board that approved the study

65%

Describes reasons why participation may be terminated

57%

Describes how personal health info. will be protected

33% 6%

Statement about compensation for patient enrolment Design & Procedures (12 items) Describes group assignement (n = 92)

93%

The rationale for the research is explained

92%

All procedures are explained

91%

Expected number of participants in study is stated

84%

The term randomization is explained (n = 87)

77%

Expected duration of the PPs participation is stated

72%

Statement about why the PP is being asked to participate

69%

Describes the other options available

68%

Describes which procedures are solely for research purposes

39%

Describes reversibility of side effects of participation

7%

Expected duration of the study as a whole is stated

6%

Describes time of onset of side effects of participation

2%

Formatting/Style (7 items) Spelling is error-free

100%

All font sizes are 11 point or greater

98%

Abbreviations & acronyms are minimally used and well-defined

97%

Grammar is error-free

96%

Language is as non-technical as is feasible

93%

No need to remove repetition

93%

Graphical elements are clear and clearly labeled (n = 37)

86%

Items Recomended by BOTH IC & DQ Standards (6 items) Provides name & contact information for questions

99%

Describes advantages of participation

92%

Name, credentials & contact info. of the PI(s)

88%

Describes disadvantages of participation

85%

Provides info. about funding sources for the study

66%

Describes the intervention

50% Strongly Agree

Agree

Neutral

Disagree

Strongly Disagree

Fig. 2. Percentage ratings of the 139 ICDs, for each of the 36 items (PPs, potential participants; IC, informed consent; DQ, decision quality).

6

J.C. Brehaut et al. / Journal of Clinical Epidemiology

of the study as a whole, time of onset of side effects; SA or A for !10% of IC documents). Formatting/style items showed good concordance, with all seven items receiving SA or A ratings for over 85% of IC documents. Among the six items recommended by both IC and DQ standards, concordance was high with four items receiving SA or A ratings for over 85% of IC documents (name and contact information of someone to whom questions can be asked, advantages of participation described, disadvantages of participation described, name, credentials, and contact information of the principal investigator(s) provided). Two items showed only moderate concordance (information about funding sources: 66%; describes intervention: 50%). 3.2. Concordance with DQ standards As reported previously [32], concordance with all 32 DQ items was quite low, with the exception of items that were also recommended as IC standards (see above). For the purposes of examining the relationship between presence of IC and DQ standards, we retained nine DQ items that showed 10% concordance or better from our previous work. Fig. 1 describes conformity with those items. Conformity with only one item was in the moderate range (describes that a participation decision needs to be made: 54%), although all other items ranged between 10% and 37%. 3.3. Subsection scores Table 2 describes the distributions of summary scores across the 139 IC documents for the different subsections of items. Scores were generally high for items focused on regulatory statements, formatting and style, and items common to both IC and DQ standards. Scores were somewhat lower for items focused on design and procedures and the DQ items. Range across IC documents was substantial for all five subsections. 3.4. Predicting DQ standards Table 3 describes the bivariate correlations between all the factors to be entered into the final regression. Table 4 describes the results of the regression predicting DQ summary scores. Adjusted R2 for the full regression model was 0.19. Significant predictors of DQ standards included IC standards (standardized b 5 0.26; t 5 2.59; P 5 0.01), word count (b 5 0.29; t 5 3.00; P 5 0.003), and country (b 5 0.21;

-

(2015)

-

t 5 2.32; P 5 0.02); IC documents from the United States scored slightly lower on DQ standards. No other variables approached statistical significance. 4. Discussion We have previously shown that existing IC documents are variable and generally poor at providing the conditions for good-quality decision making, as measured by a set of 32 standards derived from the patient decision aid literature [32]. The current work describes the extent to which this sample of IC documents conforms to a different set of recommendations derived from the IC literature itself, recommendations describing the conditions for adequate IC. In addition, we explored the extent to which variation in concordance with DQ standards was associated with IC standards concordance. IC documents vary in the extent to which they conform to individual IC standards. Our study showed clear variation in nearly all the IC standards we measured, a finding consistent with other studies [9,12,13,20,37]. Note that we cannot claim that all such variation indicates a poorer document, that each item is similarly generalizable, or that conformity to each individual item will have equivalent impact on the IC process. Although we sought to identify IC standards items that would be applicable and recommended across a broad range of IC documents, the relation between each individual item and overall measures of IC could not be assessed in this study. What we can say is lower conformity on any individual item corresponds to a lack of implementation of recommended practice within this sample of IC documents and that those developing IC documents should at least consider incorporation of each of these IC standards. More work is needed to understand the extent to which conformity on these items corresponds to risks to those participating in the consent process. Furthermore, a scale for assessing IC documents might be derived from this pool of items, although such a scale would have to demonstrate benefit over existing instruments (e.g., [9,37]). We showed previously [32] that with the exception of six items recommended by both DQ and IC standards, concordance with DQ standards was variable but low. Among the nine items for which there was some variation (as opposed to the 17 items dropped from the current analyses because of nearly uniform nonconcordance), concordance ranged

Table 2. Distribution of summary scores across the IC documents for the different subsections of items Subsection Regulatory statements Design and procedures Formatting/style Both IC and DQ standards DQ items

# Of items 11 12 7 6 9

Mean (SD) percentage 70.82 59.34 88.97 79.66 31.95

Abbreviations: IC, informed consent; SD, standard deviation; DQ, decision quality.

(15.38) (12.48) (7.21) (12.68) (15.94)

Range percentage 12.12e96.67 23.33e84.72 55.56e100.00 25.00e100.00 8.33e77.78

J.C. Brehaut et al. / Journal of Clinical Epidemiology

-

(2015)

-

7

Table 3. Bivariate correlations among decision quality (DQ) scores, informed consent (IC) scores, reading ease scores, word count, and study characteristics Variable 1. 2. 3. 4. 5. 6. 7. 8. 9.

DQ score IC summary score Reading ease score Word count Months since trial registration Country (United States vs. Other) Funding source Phase of trial Health condition

1 d 0.36*** 0.03 0.40*** 0.17* 0.04 0.10 0.05 0.10

2 d 0.12 0.57*** 0.21* 0.23** 0.16 0.13 0.35***

3

4

d 0.05 0.10 0.12 0.12 0.10 0.23*

d 0.24** 0.23** 0.16 0.15 0.22*

5

d 0.01 0.14 0.10 0.03

6

7

8

9

d 0.14 0.24** 0.26**

d 0.02 0.21*

d 0.44***

d

*P ! 0.05; **P ! 0.01; ***P ! 0.001.

from 54% (‘‘describes that a participation decision needs to be made’’) to 10% (‘‘specifies populations that yielded the probabilities’’). Despite low concordance, we see no issues that would preclude implementation of any of these items within the IC documents we reviewed. The documents are not concordant with these items because they were developed within the context of a disclosure rather than a decision making model. We believe that each of the DQ items makes a clear and testable prediction about how to ensure that IC documents help trial participants work although a difficult decision-making process. More generally, a DQ perspective leads one to consider using the considerable work on shared decision making and patient decision aids to design more effective IC tools and processes [41,42]. This work is ongoing [43,44]. We hypothesized that if DQ standards provide a conceptually distinct approach to describing IC documents, the association between IC standards and DQ standards might be relatively low. Our finding that a relatively small amount of variance in DQ standards is explained by IC standard concordance was consistent with that hypothesis. This finding can only be described as preliminary, however, given that neither set of standards has been evaluated psychometrically. Furthermore, this correlation would have increased somewhat had we included the group of six items common to both sets of standards, but we sought instead to test the association of the two sets of standards without this methodological artifact. Future work to develop validated measures of IC documents and DQ, perhaps using a larger sample of IC documents that could support exploration of the factor structure underlying the different standards, would be worthwhile. Concordance with DQ standards was statistically associated with the length of the IC document; longer documents conformed to more DQ standards. This result may not be surprising, as the DQ standards used in this study require the provision of additional information not typically required in IC documents (e.g., ‘‘Describe the health condition or problem’’; ‘‘Provides information about levels of uncertainty around probabilities’’). Lengthier IC may be more likely to include more of this information that is considered ‘‘additional’’ within the standard IC model. We note, however, that not all DQ standards require the

provision of additional information and so the idea that an IC document that is concordant with all DQ standards must be longer (with all the inconvenience that entails) needs to be evaluated empirically. Finally, concordance with DQ standards was associated with country, with US documents scoring slightly lower than non-US documents. This association was small, and so we hesitate to read much into it. We have previously shown that US investigators were not more or less likely to respond to our plea for IC documents, so a selection bias explanation seems unlikely. Post hoc exploration showed that US documents tended also to score higher on IC standards, leading to speculation that increased regulation in the United States around IC standards [16] may have resulted in greater concordance with IC guidelines at the expense of efforts to add detail consistent with the DQ standards measured in this study. 4.1. Limitations Some limitations of this study warrant consideration. As previously mentioned, the subset of nine DQ items cannot be considered a validated scale of DQ. Because previous work had shown near perfect nonconformity to 17 of the original 32 items resulting in their exclusion from this study, this subset of nine items likely suffers from incomplete construct coverage; in essence, our outcome was not Table 4. Regression predicting decision quality (DQ) summary scores Predictor

Standardized coefficients (beta)

Constant IC summary scores Reading ease Word count Phase of trial Health condition Country Months since trial registration Funding source Industry (dummy coded) Other funding (dummy coded) Abbreviation: IC, informed consent. R2adj 5 0.19; *P ! 0.05; **P ! 0.01.

t

0.23 0.26* 0.02 0.29** 0.01 0.20 0.21* 0.05

0.01 2.59 0.30 3.00 0.14 0.21 2.32 0.62

0.05 0.04

0.51 0.37

8

J.C. Brehaut et al. / Journal of Clinical Epidemiology

measuring all factors that contribute to DQ. Because of this, we cannot be certain that the lack of association between IC items and DQ items does not reflect simply a poor measure of DQ, as opposed to a lack of association between IC standards and DQ. In this analysis, we are assuming that a higher DQ summary score is associated with more indications of the conditions for encouraging good-quality decisions. Additional work to validate a scale based on DQ items would be an important step toward assessing the conditions for quality decision making created by IC documents. Similarly, neither the whole group of 36 IC items nor the different subsections should be considered validated measures of IC. Although we chose these items carefully from clear, evidence-based sources, we did not have enough power (with only 139 IC documents) to assess the extent to which individual items within the groups were correlated, formed independent factors, and so on. For this reason, we chose not to incorporate subsection scores into the regression analysis, only the total score. As with the DQ items, in this analysis we are assuming that greater IC summary scores correlate with more indications of a goodquality IC document based on current recommendations. Response bias issues also need to be considered in the context of these findings. As reported in our previous work [32], IC documents provided to us were more likely to be more recent, late phase, and noncancer related. As such, we cannot be sure these results will generalize to the full range of randomized control trial IC documents. Such response bias might be predicted to inflate overall IC and DQ scores, as older, early phase trials might be expected to score more poorly on relatively recent standards. However, it seems unlikely that such bias would result in dramatic improvements in the correlation between IC and DQ standards, and so, the central result of a relatively low correlation between the two sets of standards seems likely to be robust to such bias. 4.2. Conclusion In a random sample of IC documents, conformity to recommendations for improving IC documents was relatively high but variable. We examined whether this variability (as well as other study characteristics) could explain variability in concordance with recommendations for improving DQ. Although DQ item concordance was statistically related to IC recommendations, length of IC document, and country of study, these factors combined explained less than 20% of the variability. Existing IC regulations may not describe the optimal approach to helping people make good study participation decisions. Acknowledgments The authors thank Erica Hoe and Mathieu Chalifoux for help with data extraction.

-

(2015)

-

Authors’ contributions: J.C.B., G.E., R.S., J.K., and K.S. contributed to the conception and design of the study. J.C.B. drafted the article. J.C.B. and K.C. conducted the analysis of the data. J.C.B., K.C., A.S., T.N., and Erica Hoe collected the data. All authors contributed to the interpretation of the data, commented on the first draft, and revised the article critically for important intellectual content. All authors approved the final version. All authors had full access to all the data (including statistical reports and tables) in the study and can take responsibility for the integrity of the data and the accuracy of the data analysis. J.C.B. accepts full responsibility for the work and/or the conduct of the study, had access to the data, and controlled the decision to publish. References [1] Stenson B, Brocklehurst P, Tarnow-Mordi W. Increased 36-week survival with high oxygen saturation target in extremely preterm infants. N Engl J Med 2011;364:1680e2. [2] Drazen JM, Solomon CG, Greene MF. Informed consent and SUPPORT. N Engl J Med 2013;368:1929e31. [3] Hunter D. Can the regulatory response to SUPPORT be supported? Am J Bioeth 2013;13(12):37e9. [4] King NM. There’s a lot we don’t know (and we ought to say so). Am J Bioeth 2013;13(12):20e1. [5] Merz JF, Yerramilli D. SUPPORT asked the wrong question. Am J Bioeth 2013;13(12):25e6. [6] Magnus D. The SUPPORT controversy and the debate over research within the standard of care. Am J Bioeth 2013;13(12):1e2. [7] Wilfond BS. Quality improvement ethics: lessons from the support study. Am J Bioeth 2013;13(12):14e9. [8] Howard JM, DeMets D. How informed is informed consent: the BHAT experience. Control Clin Trials 1981;2:287e303. [9] Joffe S, Cook EF, Cleary PD, Clark JW, Weeks JC. Quality of informed consent in cancer clinical trials: a cross-sectional survey. Lancet 2001;358:1772e7. [10] Bergler JH, Pennington AC, Metcalfe M, Freis ED. Informed consent: how much does the patient understand? Clin Pharmacol Ther 1980;27:435e40. [11] Beauchamp T, Childress J. Principles of biomedical ethics. (4th ed.). New York: Oxford University Press; 1994. [12] Albala I, Doyle M, Appelbaum PS. The evolution of consent forms for research: a quarter century of changes. IRB 2010;32(3):7e11. [13] Horng S, Emanuel EJ, Wilfond B, Rackoff J, Martz K, Grady C. Description of benefits and risks in consent forms for phase I oncology trials. N Engl J Med 2002;347:2134e40. [14] Berger O, Gronberg BH, Sand K, Kaasa S, Loge JH. The length of consent documents in oncological trials is doubled in twenty years. Ann Oncol 2009;20:379e85. [15] Beardsley E, Jefford M, Mileshkin L. Longer consent forms for clinical trials compromise patient understanding: so why are they lengthening? J Clin Oncol 2007;25:e13e4. [16] Dunlop AL, Graham T, Leroy Z, Glanz K, Dunlop B. The impact of HIPAA authorization on willingness to participate in clinical research. Ann Epidemiol 2007;17:899e905. [17] Kass NE, Chaisson L, Taylor HA, Lohse J. Length and complexity of US and international HIV consent forms from federal HIV network trials. J Gen Intern Med 2011;26:1324e8. [18] Jefford M, Moore R. Improvement of informed consent and the quality of consent documents. Lancet Oncol 2008;9(5):485e93. [19] Sugarman J, Lavori PW, Boeger M, Cain C, Edsond R, Morrison V, et al. Evaluating the quality of informed consent. Clin Trials 2005;2: 34e41.

J.C. Brehaut et al. / Journal of Clinical Epidemiology [20] Resnik DB, Patrone D, Peddada S. Evaluating the quality of information about alternatives to research participation in oncology consent forms. Contemp Clin Trials 2010;31:18e21. [21] White LJ, Jones JS, Felton CW, Pool LC. Informed consent for medical research: common discrepancies and readability. Acad Emerg Med 1996;3:745e50. [22] Canadian Institutes of Health Research; Natural Sciences Engineering Research Council of Canada; Social Sciences Humanities Research Council of Canada. Tri-Council Policy Statement (TCPS2): ethical conduct for research involving humans. 2010. [23] Health Canada. Good clinical practice: consolidated guidelines, international conference on harmonization of technical requirements for the registration of pharmaceuticals for human use 1997: Available at http://www.hc-sc.gc.ca/dhp-mps/alt_formats/hpfb-dgpsa/pdf/ prodpharma/e6-eng.pdf Accessed October 20, 2014. [24] Food and Drug Administration. A guide to informed consentinformation sheet: guidance for institutional review boards and clinical investigators. 21 CFR 50.20e27: Available at http://www. fda.gov/RegulatoryInformation/Guidances/ucm126431.htm Accessed October 20, 2014. [25] Code of Federal Regulations for the Protection of Human Subjects in Research: General requirements for informed consent. Code of federal regulations, 2009 45 CFR 46.116. [26] National Cancer Institute. Simplification of informed consent documents 2009: Available at http://www.cancer.gov/clinicaltrials/ education/simplification-of-informed-consent-docs/page2 Accessed October 20, 2014. [27] O’Connor AM, Stacey D, Entwistle V, Llewellyn-Thomas H, Rovner D, Holmes-Rovner M, et al. Decision aids for people facing health treatment or screening decisions. Cochrane Libr 2003;1. [28] Stacey D, Legare F, Col NF, Bennett CL, Barry MJ, Eden KB, et al. Decision aids for people facing health treatment or screening decisions. Cochrane Database Syst Rev 2014;1:CD001431. [29] Elwyn G, O’Connor A, Stacey D, Volk R, Edwards A, Coulter A, et al. International Patient Decision Aids Standards (IPDAS) Collaboration. Developing a quality criteria framework for patient decision aids: online international Delphi consensus process. BMJ 2006;333:417. [30] Elwyn G, O’Connor AM, Bennett C, Newcombe RG, Politi M, Durand MA, et al. Assessing the quality of decision support technologies using the International Patient Decision Aid Standards instrument (IPDASi). PLoS One 2009;4:e4705.

-

(2015)

-

9

[31] Volk R, Llewellyn-Thomas H, Stacey D, Elwyn G. Ten years of the International Patient Decision Aid Standards Collaboration: evolution of the core dimensions for assessing the quality of patient decision aids. BMC Med Inform Decis Making 2013;13:S1. [32] Brehaut JC, Carroll K, Elwyn G, Saginur R, Kimmelman J, Shojania K, et al. Informed consent documents do not encourage good-quality decision making. J Clin Epidemiol 2012;65:708e24. [33] Ottawa Health Science Network Research Ethics Board. Sample consent form. Ottawa, Ontario, Canada. Version 2009. [34] Hamilton Integrated Research Ethics Board (HiREB). Consent checklist. Hamilton, Ontario, Canada. Version 2008. [35] General Medical Council. Consent: patients and doctors making decisions together 2008: Available at http://www.gmc-uk.org/guidance/ ethical_guidance/consent_guidance_index.asp Accessed October 20, 2014. [36] Joffe S, Cook EF, Cleary PD, Clark JW, Weeks JC. Quality of informed consent: a new measure of understanding among research subjects. J Natl Cancer 2001;93:139e47. [37] Padhy BM, Gupta P, Gupta YK. Analysis of the compliance of informed consent documents with good clinical practice guideline. Contemp Clin Trials 2011;32:662e6. [38] Miller CK, O’Donnell DC, Searight HR, Barbarash RA. The Deaconess informed consent comprehension test: an assessment tool for clinical research subjects. Pharmacotherapy 1996;16:872e8. [39] Koyfman SA, McCabe MS, Emanuel EJ, Grady C. A consent form template for phase I oncology trials. IRB 2009;31(4):1e8. [40] Cohen J. A coefficient of agreement for nominal scales. Educ Psychol Meas 1960;20:37e46. [41] Brehaut JC, Fergusson DA, Kimmelman J, Shojania KG, Saginur R, Elwyn G. Using decision aids may improve informed consent for research. Contemp Clin Trials 2010;31:218e20. [42] Brehaut J, Saginur R, Elwyn G. Informed consent documentation necessary but not sufficient. Contemp Clin Trials 2009;30:388e9. [43] Juraskova I, Butow P, Lopez A, Seccombe M, Coates A, Boyle F, et al. Improving informed consent: pilot of a decision aid for women invited to participate in a breast cancer prevention trial. Health Expect 2008;11:252e62. [44] Gillies K, Skea Z, Politi M, Brehaut J. Decision support interventions for people making decisions about participation in clinical trials. Cochrane Database Syst Rev 2012;3:CD009736.