AJPH RESEARCH
Replicating Reducing the Risk: 12-Month Impacts of a Cluster Randomized Controlled Trial Meredith Kelsey, PhD, Michelle Blocklin, PhD, Jean Layzer, MEd, Cristofer Price, ScM, Randall Juras, PhD, and Lesley Freiman, MPP Objectives. To test the effectiveness of Reducing the Risk, an evidence-based sexual health curriculum designed to help prevent adolescent pregnancy and sexually transmitted infections, on youth sexual behavior and intermediate outcomes thought to lead to these behaviors. Methods. Classes within schools in St. Louis, Missouri; Austin, Texas; and San Diego, California; were randomly assigned to receive Reducing the Risk or “business as usual.” Youths completed Web-based surveys at baseline (preintervention, August 2012–January 2014) and 12 months later (August 2013–January 2015). Intent-to-treat analyses were conducted across sites; we tested for differences in impacts between sites and other subgroups. Results. The program had no overall impact on sexual behaviors. However, at 1 site, program participants were significantly less likely to have engaged in recent sexual intercourse than were control group members. There were positive overall impacts on intermediate outcomes (e.g., knowledge, attitudes). Conclusions. After 12 months, Reducing the Risk was unsuccessful at changing sexual behaviors. Other results were mixed, but promising evidence (e.g., behavioral impacts at 1 site, impacts on intermediate outcomes) suggests potential for more widespread behavioral impacts over a longer term. (Am J Public Health. 2016;106:S45–S52. doi: 10.2105/AJPH.2016.303409) See editorials, p. S5–S31.
F
inding ways to reduce adolescent pregnancy is a priority for the US Department of Health and Human Services (HHS). To achieve this goal, HHS is investing in evidence-based pregnancy reduction interventions. The federal Teen Pregnancy Prevention Program, administered by the Office of Adolescent Health (OAH) within HHS, acknowledges the need for additional research on evidence-based programs given the scarcity of independent evaluations and the limited number of program replications.1 Accordingly, the program includes funding to replicate evidence-based adolescent pregnancy prevention models. This study examines the effectiveness of Reducing the Risk by examining 3 OAH-funded replications. Reducing the Risk is an evidence-based sexual health curriculum developed in the early 1990s that was designed to help prevent pregnancy and sexually transmitted infections (STIs) in adolescents aged 13 to 19 years. The program is intended for use in school
Supplement 1, 2016, Vol 106, No. S1
AJPH
classrooms with students of all ethnicities. Its 16 units can be delivered separately in 45-minute sequential sessions or grouped in eight 90-minute sessions. Figure 1 shows the program elements,2 intended outcomes, and pathways by which the program seeks to achieve these outcomes. Taken together, the program sessions are intended to improve students’ attitudes toward protection; increase knowledge and understanding of sexual health issues; improve motivation and intentions to avoid risk; encourage values and beliefs that support abstinence and avoidance of unprotected sex; and improve negotiation, refusal, and condom use skills. These intermediate outcomes
are believed to lead to the behavioral outcomes the program seeks to achieve: correct and consistent use of condoms and other contraceptives, abstinence from sex, and a reduction in sexual activity and number of partners. Prevention of or reduction in sexually risky behavior is ultimately expected to reduce rates of STIs, unwanted pregnancies, and births among adolescents. The evidence for the effectiveness of this program is from a single quasi-experimental study completed 25 years ago in rural and urban areas of northern California with primarily White high school students.3 Although this study found that the program had significant positive effects, further evaluation is critical to determine whether the program remains effective years later in different settings. This study was guided by the following research questions to examine the effectiveness of Reducing the Risk after 12 months: 1. Did the program reduce risky sexual behavior? 2. Did the program have positive effects on intermediate outcomes thought to lead to behavior change (attitudes, knowledge, motivation, intentions, and skills)?
METHODS The study used an experimental design in which classes were randomly assigned to receive the intervention or business as usual, that is, the regular curriculum or offering in the assigned class.
ABOUT THE AUTHORS Meredith Kelsey, Michelle Blocklin, Cristofer Price, Randall Juras, and Lesley Freiman are with Abt Associates, Inc., Cambridge, MA. Jean Layzer is with Belmont Research Associates, Belmont, MA. Correspondence should be sent to Meredith Kelsey, PhD, 55 Wheeler Street, Cambridge, MA 02138 (e-mail: Meredith_Kelsey@ abtassoc.com). Reprints can be ordered at http://www.ajph.org by clicking the “Reprints” link. This article was accepted July 20, 2016. doi: 10.2105/AJPH.2016.303409
Kelsey et al.
Peer Reviewed
Research
S45
AJPH RESEARCH
Inputs
Process Establish psychologically safe environment
Trained teacher Deliver curriculum modules in participatory, interactive sessions
Active participation in discussion, sharing ideas, feelings
Intermediate Outcomes
Outcomes
Positive attitudes toward protection Improved knowledge and understanding
Safer sexual behavior
Improved motivation/intention to avoid risk
Consistent use of contraceptives
C istent effective i Cons condom use
Abstinence
16-unit curriculum
Reinforce norms for abstinence or protected sex
Encourage youths to personalize information, identify vulnerabilities and values
Role-play, practice to improve skills
Values and beliefs supportive of abstinence and avoidance of unprotected sex
Improved negotiation/refusal/ condom use skills
Reduction in sexual activity Reduction in no. of sexual partners
Reduction in STIs Reduction in unwanted pregnancies Reduction in adolescent births
Teach youths to anticipate and prepare for risky situations Note. STI = sexually transmitted infection.
FIGURE 1—Logic Model: Reducing the Risk
Setting Reducing the Risk was implemented by 3 grantees and their partners. These replications, all implemented with careful adherence to the original program model under OAH’s guidance, provided a unique opportunity to study the impact of the program model in more geographically and demographically diverse settings than are typically present in a single replication site. In all 3 replications, external health educators, employed by grantees and their partners, delivered the program in public school classrooms as part of the regular school day. One grantee delivered the program in ninth-grade classes (with some students from higher grades) in 6 public high schools in St. Louis, Missouri; St. Louis County, Missouri; and St. Clair County, Illinois; we refer to this site as St. Louis. Another grantee implemented the program in
S46
Research
Peer Reviewed
Kelsey et al.
mixed 9th- and 10th-grade classes (with some older students) in 5 public high schools in the Austin Independent School District, Austin, Texas. The third grantee implemented the program in eighth- or ninth-grade classes in 6 public middle, junior high, or high schools in San Diego County, California.
Sample Once schools agreed to participate, school staff identified classes to participate in the study. Students in identified classes were recruited for the study without knowing whether their class would receive the program or the “business as usual” curriculum. Program staff distributed consent forms and study brochures. All students who obtained parental permission were enrolled in the study. As shown in Figure A (available as a supplement to the online version of this
article at http://www.ajph.org), 3314 students in 150 classes obtained parental permission. Of those, 2689 (81%) in 150 classes completed the 12-month follow-up survey and were included in the analytic sample. Differential attrition between the treatment and control groups was less than 1% (0.52%), and we found no evidence of systematic differences in baseline characteristics between dropouts in the treatment and control groups. This low rate of differential attrition combined with an overall attrition rate of 19% mitigates the threat of bias.4 At baseline (Table 1), students were (on average) aged almost 15 years, and about half were female. One third were Black; almost half were Hispanic; the remainder were almost equally White or other race. There was some variation across sites (Table A, available as supplement to the online version of this
AJPH
Supplement 1, 2016, Vol 106, No. S1
AJPH RESEARCH
TABLE 1—Baseline Characteristics of the Analytic Sample: Reducing the Risk; St. Louis, MO, Austin, TX, and San Diego County, CA; August 2012–January 2014 Baseline Characteristic
Rangea
No.
Treatment Meanb
Control Mean
Group Differencec
Age
13–20
2689
14.50
14.56
–0.07
Grade
8–12
Female, %d
P .20
2689
9.25
9.27
–0.02
.63
2689
49.10
49.10
0.00
‡ .99
Race/ethnicity, %d Hispanic
2689
46.09
47.12
–1.04
.52
Non-Hispanic Black
2689
33.10
32.96
0.14
.90
Non-Hispanic White
2689
11.34
10.73
0.61
.62
Othere
2689
9.54
9.18
0.36
.79
Lives with biological parents Feels very close to and cared for by father
2613 2368
93.06 45.46
92.24 46.74
0.82 –1.27
.48 .56
Feels very close to and cared for by mother
2592
63.38
65.98
–2.61
.17
Ever smoked cigarettes
2631
21.01
20.63
0.38
.83
Ever drank alcohol
2634
45.73
45.05
0.69
.74
Ever used marijuana
2632
31.23
30.00
1.23
.52
1–4
2652
3.04
3.04
0.00
.91
Knowledge of pregnancy risk
0–100
2653
51.84
50.61
1.22
.45
Knowledge of STI risk
0–100
2655
44.42
43.46
0.96
.41
Intentions to have sexual intercourse in the next 12 mo
2588
41.14
39.16
1.98
.31
Intentions to have oral sex in the next 12 mo
2581
30.00
30.09
–0.08
.97
Intentions to use a condom if they were to have sexual
2592
94.59
94.17
0.42
.65
2558
89.41
90.79
–1.38
.26
Ever sexually activeh Currently sexually active in past 90 dh
2602 2590
30.57 18.37
31.32 20.79
–0.75 –2.42
.68 .18
Sexual intercourse in the past 90 d
2590
16.53
17.99
–1.46
.41
Oral sex in the past 90 d
2582
12.12
14.56
–2.44
.11
Sexual intercourse without birth control in the past 90 d
2590
5.36
6.71
–1.35
.25
Sexual intercourse without a condom in the past 90 d
2590
8.04
9.85
–1.80
.19
Oral sex without a condom in the past 90 d
2582
10.40
11.77
–1.37
.35
d
Family structure and relationships, %
Risk behavior, %d
f
Attitudes toward protection Knowledgeg
d
Intentions, %
intercourse Intentions to use birth control if they were to have sexual intercourse Sexual behavior, %d
d
Sexual risk, %
Note. The baseline treatment–control difference was estimated in a 2-level multilevel model with random intercept terms for classes. The dependent variable was the baseline measure, and the only independent variables included in the model were the treatment group indicator and terms for the randomization blocks. a For continuous variables, we present the range. All other variables are dichotomous. b The treatment mean was calculated as the sum of the control group mean and the model estimated treatment–control difference (group difference). c The group difference is the treatment–control difference. For outcomes reported as percentages, the group difference is expressed in percentage points. For scale outcomes, the group difference is expressed in the original metric of the outcome variable. Because of rounding, reported group differences may differ from differences between reported means for the treatment and control groups. d For dichotomous variables, we present the percentage of respondents who responded affirmatively. e Defined as Asian, American Indian or Alaska Native, Native Hawaiian or other Pacific Islander, multiracial, or undisclosed. f Attitudes variable is a composite scale score with higher scores indicating more positive attitudes. g Knowledge variables are composite scale scores representing the percentage of items answered correctly. h Sexual activity was defined differently across grantees. At 1 site, sexual activity referred to sexual intercourse, oral sex, or anal sex. Youths were not asked about anal sex at 2 of the sites.
Supplement 1, 2016, Vol 106, No. S1
AJPH
Kelsey et al.
Peer Reviewed
Research
S47
AJPH RESEARCH
article): students in San Diego were younger (more than a year, on average), and students in St. Louis were primarily Black, whereas the majority of students in Austin and San Diego were Hispanic. Students in San Diego were also less likely to have engaged in risky behaviors (both sexual and nonsexual).
Random Assignment Design Participating classes in each school were randomly assigned by evaluation staff to receive Reducing the Risk or business as usual (Figure A). The random assignment ratio varied across sites and schools, depending on school and program preferences, with more classes assigned to treatment overall. Schools were not informed of random assignment results until after students completed a baseline survey, described later. Students in classrooms assigned to Reducing the Risk received the program as described in the introduction. At all 3 replication sites, grantees delivered all 16 sessions of the program. To ensure each replication was implemented with fidelity, OAH required grantees to request permission for any adaptations. In Austin, a session on reproductive anatomy and another optional Reducing the Risk session, which reinforced messages about pregnancy and STI prevention, were added and the condom demonstration was dropped to conform to school district policy. At 1 school in St. Louis, the program was delivered to male and female students separately, by instructors of the same gender. In the same school, the agency also replaced the condom demonstration with a video (for boys) and a mini-lecture (for girls). To accommodate larger class sizes, the San Diego site received permission to deliver the program with 2 facilitators. In Austin, students in the control group received the regularly scheduled health class. In St. Louis and San Diego, the class attended by control group students differed by school but was the regularly scheduled class displaced by Reducing the Risk and included physical education, ROTC, health, homeroom or guidance, and science classes.
Data Collection All youths completed a 30-minute survey at baseline (preintervention; August 2012– January 2014) and 12 months after baseline (August 2013–January 2015). At both time
S48
Research
Peer Reviewed
Kelsey et al.
points, a Web-based audio computer-assisted self-interview system was used to capture and store survey responses. The baseline survey was completed in group sessions in schools, using school computers when possible or tablets provided by field staff. Hard copies of the baseline survey (in Spanish and English) were available as backup. The same Webbased system was used for the 12-month follow-up. When possible, the follow-up survey was administered in group sessions in schools. When necessary, students completed the survey on their own time, using personal tablets or computers, school or library computers, or even smartphones. Participants received gift cards after completing each survey (baseline and followup). Of the 3314 randomized students, 98% completed the baseline survey, and 81% completed the follow-up survey.
Measures We collected data on implementation, participant demographic characteristics, and outcomes. Student surveys measured 2 sets of outcomes: (1) sexual behavior outcomes and (2) intermediate outcomes believed to lead to behavioral outcomes. We briefly describe these measures here. Additional details on the construction of these measures and their reliability are presented in Appendix A (available as a supplement to the online version of this article). Implementation. We conducted a series of site visits and focus groups to measure implementation. To measure fidelity, the funding agency provided fidelity checklists, which health educators completed for every session of the program. In addition, grantee-determined supervisory staff were required to observe, record, and report on the quality of program delivery for a sample of sessions. Program staff collected attendance data. Behavioral outcomes. To address the study’s key questions about the impact of the intervention, we identified 7 measures in the domain of youth sexual behavior: 1. 2. 3. 4. 5.
ever sexually active, sexually active in the past 90 days, sexual intercourse in the past 90 days, oral sex in the past 90 days, sexual intercourse without birth control (defined as a condom or other contraceptive) in the past 90 days,
6. sexual intercourse without a condom in the past 90 days, and 7. oral sex without a condom in the past 90 days. Sexually active refers to sexual intercourse, oral sex, or anal sex, though youths were not asked about anal sex at 2 of the sites. Responses for each behavior were dichotomous (yes–no). Intermediate outcomes. We constructed composite measures to assess several intermediate outcomes. Attitudes toward protection were measured on a scale ranging from 1 to 4, with higher scores indicating more positive attitudes. Attitudes toward risky behavior were measured on a scale ranging from 0 to 100, reflecting the percentage of statements with which a respondent agreed; higher scores represented greater support for risky behavior. Knowledge of pregnancy risk and knowledge of STI risk scales both ranged from 0 to 100 and reflected the percentage of items answered correctly. Motivation to delay childbearing was measured on a scale ranging from 1 to 4, with higher scores indicating greater motivation. Condom negotiation skills and refusal skills were both measured on scales ranging from 1 to 4, with higher scores indicating stronger skills. To assess intentions to engage in sexual behaviors and use protection, we examined 4 dichotomous (yes–no) items separately.
Baseline demographic and behavioral characteristics. We collected information on sexual risk behaviors and demographic characteristics at baseline. These measures included ever sexually active, substance use (ever smoked a cigarette, drank alcohol, used marijuana), pregnancy and STI risk knowledge, and intentions to have intercourse and oral sex. Additional demographic measures included age, grade, and race/ethnicity. To measure race/ethnicity, students responded to 2 questions: (1) “Are you Hispanic/Latino (yes–no)?” and (2) “What is your race (American Indian or Alaska Native, Asian, Black or African American, Native Hawaiian or Other Pacific Islander, White, or some other race)?” We classified students as Hispanic, non-Hispanic White, non-Hispanic Black, or non-Hispanic other race (i.e., Asian, American Indian or Alaska Native, Native Hawaiian or other Pacific Islander, multiracial, and undisclosed race).
AJPH
Supplement 1, 2016, Vol 106, No. S1
AJPH RESEARCH
Analytic Approach As part of our analytic strategy, we prespecified that the primary inferences regarding program effectiveness would be based on a pooled analysis (across program sites) of 2 confirmatory outcomes: sexually active in the past 90 days and sexual intercourse without birth control in the past 90 days. Other analyses were considered exploratory, or secondary. Impacts were pooled across sites to limit the number of tests of program impact, to increase power to detect effects, and to support more generalizable statements about program impacts. Given the emphasis on fidelity and consistent implementation across replications, we are confident that pooling across the 3 sites provided a valid test of program effectiveness. To estimate impacts of Reducing the Risk on each of the study’s outcomes, we compared the outcomes for treatment and control group members using a regression framework; for continuous outcomes, these are traditional regression models, and for dichotomous outcomes, these are linear probability models. We included the baseline characteristics described in the preceding section and preintervention measures of the outcome variable, when available, as covariates to increase statistical precision (i.e., reduce the standard errors) of the impact estimates for a given sample size5 and reduce attrition bias from missing data.6 Randomization blocking variables, which included site, school, year, semester, and gender, were also included. To maintain the integrity of the random assignment design, we conducted an intent-to-treat analysis, using assignment status as the key variable of interest regardless of actual program participation. To address missing data, we used case deletion for the few instances of missing outcome data; we used dummy-variable adjustment in regression models to account for missing covariates.6 Specifying 2 confirmatory outcomes, as noted earlier in this section, addresses potential problems resulting from multiple comparisons (i.e., the potential for spurious findings when many hypotheses are tested). To control for having 2 confirmatory outcomes, we applied a formal Benjamini-Hochberg correction, which controls for the false positive rate by adjusting P-value thresholds.4 We considered all other outcomes and all subgroup analyses, described later, to be exploratory and did not adjust for multiple comparisons.7
Supplement 1, 2016, Vol 106, No. S1
AJPH
In addition to the overall pooled impacts, to gain insight into what worked for whom we conducted exploratory subgroup analyses by including subgroup indicators and Treatment · Subgroup interaction terms in the model and by testing for the significance of the interaction term. Only when there was a significant difference in impacts between groups do we discuss impacts for individual subgroups, which helps guard against overinterpretation of potentially spurious findings across the large number of subgroup by outcome tests. Because subgroup analyses were exploratory, we did not adjust for multiple comparisons. Subgroups tested included replication site, gender, age, race/ethnicity, and sexual experience at baseline.
RESULTS In the sections that follow, we briefly summarize our findings on program implementation, compare the treatment and control groups at baseline, and present impacts on behaviors followed by impacts on intermediate outcomes.
Program Implementation Reducing the Risk was well implemented across the 3 replication sites. Grantees hired health educators with appropriate background experience and skills to deliver the program; all received training approved by the program developer, and the program was delivered with fidelity (adherence to its core elements and without modifications that threatened those core elements). Attendance varied by replication site: In San Diego, 85% of students attended at least 75% of the sessions, compared with 73% of students in St. Louis and 47% of students in Austin.
Comparability of the 2 Groups at Baseline We assessed baseline equivalence with a series of models with the same structural components as the impact models. At baseline, no differences were found between the treatment and control groups (Table 1).
Program Impacts on Behavior As shown in Table 2, Reducing the Risk had no impact on either of the 2 confirmatory sexual behavior outcomes (currently sexually
TABLE 2—12-Month Impacts of Reducing the Risk on Sexual Behavior: St. Louis, MO, Austin, TX, and San Diego County, CA; August 2013–January 2015
Outcome
No. Treatment,a % Control, %
Group Differenceb
P
Sexual behavior Ever sexually activec
2670
42.99
41.71
1.28
.41
Currently sexually active (in past 90 d)c,d
2665
28.02
28.14
–0.11
.95
Sexual intercourse in the past 90 d
2667
23.66
24.37
–0.72
.67
Oral sex in the past 90 d
2661
19.24
19.50
–0.26
.87
Sexual risk Sexual intercourse without birth control in the past 90 dd 2667
8.73
8.99
-0.25
.86
Sexual intercourse without a condom in the past 90 d 2667
13.57
15.38
–1.81
.18
Oral sex without a condom in the past 90 d
16.20
17.33
–1.13
.44
2661
Note. All outcomes are dichotomous. a The treatment group percentage is regression adjusted, calculated as the sum of the control group percentage and the regression-adjusted impact estimate (group difference). b The group difference is the treatment–control difference expressed in percentage points. Because of rounding, reported group differences may differ from differences between reported percentages for the treatment and control groups. c Sexual activity was defined differently across grantees. At 1 site, sexual activity referred to sexual intercourse, oral sex, or anal sex. Youths were not asked about anal sex at 2 of the sites. Impacts for anal sex are not reported here. d Confirmatory outcome; for these outcomes, statistical significance at P < .05, P < .01, or P < .001 implies statistical significance at those levels after applying a Benjamini-Hochberg adjustment for multiple comparisons.
Kelsey et al.
Peer Reviewed
Research
S49
AJPH RESEARCH
TABLE 3—12-Month Impacts of Reducing the Risk on Intermediate Outcomes: St. Louis, MO, Austin, TX, and San Diego County, CA; August 2013–January 2015 Rangea
Outcome
No.
Treatment Meanb
Control Mean
Group Differencec
SESd
P
0.13
£ .001
e
Attitudes Attitudes toward protection Attitudes toward risky behavior
1–4
2688
3.18
3.13
0.05
0–100
2675
5.32
4.53
0.80
.16
e
Knowledge
Knowledge of pregnancy risk
0–100
2689
65.55
61.55
4.01
£ .001
Knowledge of STI risk
0–100
2689
60.47
56.21
4.26
£ .001
1–4
2683
3.68
3.68
–0.01
Sexual intercourse Oral sex
2660 2654
52.67 42.41
50.69 43.27
1.97 –0.86
Use a condom if they were to have sexual intercourse
2667
91.21
92.11
–0.90
.40
Use birth control if they were to have sexual intercourse
2662
90.39
89.67
0.72
.54
Motivation to delay childbearinge
–0.01
.74
Intentions to engage in the following behaviors in the next 12 mo, %f .28 .63
Skillse Perceived refusal skills
1–4
2681
3.12
3.08
0.04
0.06
.13
Perceived condom negotiation skills
1–4
2685
3.53
3.50
0.03
0.06
.18
Note. SES = standardized effect size of the difference; STI = sexually transmitted infection. a For continuous variables, we present the range. All other variables are dichotomous. b The treatment group mean is regression adjusted, calculated as the sum of the control group mean and the regression-adjusted impact estimate (group difference). c The group difference is the treatment–control difference. For outcomes reported as percentages, the group difference is expressed in percentage points. For scale outcomes, the group difference is expressed in the original metric of the outcome variable. Because of rounding, reported group differences may differ from differences between reported means for the treatment and control groups. d For outcomes that are not dichotomous or measured on a 0–100 scale, the SES is the group difference divided by the pooled standard deviation of the treatment and control groups. e Composite scale scores. f Dichotomous variables, reported as percentage of respondents who responded affirmatively.
active or recent sexual intercourse without birth control), nor did it have an impact on other related sexual behaviors. Subgroup analyses largely revealed no significant differences in impacts on sexual risk behaviors by subgroup. An important exception was a significant effect of the program on students in the St. Louis sample (Table B, available as supplement to online version of this article). Compared with control group members, program participants in St. Louis were 7 percentage points (> 15%) less likely to report engaging in sexual intercourse in the previous 90 days (P = .011).
Program Impacts on Intermediate Outcomes The program had positive impacts on attitudes and knowledge. No overall effects on motivation, intentions, or skills were found.
S50
Research
Peer Reviewed
Kelsey et al.
Attitudes. As shown in Table 3, the program had a small but statistically significant impact on students’ attitudes toward protection (birth control and condoms). Students in the treatment group had more positive (and protective) attitudes than did those in the control group. Subgroup analyses revealed a difference in impact by site, suggesting that the San Diego replication largely accounted for the overall impact. The impact on attitudes toward protection was moderate (an increase of 0.12 points on a 1–4 scale; standardized effect size = 0.30) and highly significant in San Diego (P < .001); no significant impacts on these attitudes were found at the other 2 sites (Table B). The program had no statistically significant impacts on attitudes toward risky behavior overall (Table 3). However, subgroup analyses showed that among older
students (aged ‡ 15 y) and among White students, those in the treatment group were more likely to express positive attitudes toward risky behavior than were those in the control group (Table B). Among older students, those in the treatment group agreed with 50% more items (P = .008) regarding the acceptability of risky sexual behavior than did those in the control group. Among White students, those in the treatment group agreed with 75% more items (P = .001) regarding the acceptability of risky sexual behavior. Knowledge. The program had statistically significant positive effects on knowledge of pregnancy risk and knowledge of STI risk (Table 3). Compared with control group students, treatment group students scored 4 percentage points (about 7%) higher on knowledge
AJPH
Supplement 1, 2016, Vol 106, No. S1
AJPH RESEARCH
of pregnancy risk and knowledge of STI risk. Motivation to delay childbearing. Students in both the treatment and control groups were highly motivated to delay childbearing. Accordingly, there were no overall differences between the 2 groups (Table 3). However, subgroup analyses revealed that, in the St. Louis sample, treatment students were slightly less motivated to delay childbearing than were control group students (0.10 points lower on a 1–4 scale; standardized effect size = –0.18; P = .014; Table B). Intentions. No overall impact was found on intentions to engage in sexual activity (Table 3). Equal proportions of students in the treatment and control groups expected to engage in sexual intercourse or oral sex in the next year. Nearly all students said they intended to use condoms or birth control if they engaged in sexual intercourse. There was 1 significant site-level effect: In St. Louis, students in the treatment group were 7 percentage points (about 15%) less likely than those in the control group to report intentions to engage in oral sex (P = .011; Table B). Skills. No overall impacts were found on perceived condom negotiation or refusal skills (Table 3). However, subgroup analyses revealed that in the San Diego sample, treatment group students reported slightly stronger (0.10 on a 1–4 scale, standardized effect size = 0.19) condom negotiation skills than did control group students (P = .008; Table B).
DISCUSSION Building on the original study of the effectiveness of Reducing the Risk, this study provides additional evidence regarding the program’s effectiveness in different settings 25 years later. Because Reducing the Risk did not affect either of the study’s confirmatory outcomes, we found no evidence of effectiveness after 12 months. However, this does not rule out the possibility of longerterm effects. Impacts on exploratory outcomes were mixed, but some provide grounds for optimism. A significant program effect was found on sexual behavior at 1 site, and the program achieved its
Supplement 1, 2016, Vol 106, No. S1
AJPH
intended effects on some intermediate outcomes, which might lead to more widespread impacts over time. We found no effects on our confirmatory outcomes of current sexual activity and sexual intercourse without birth control. However, the original study found no effects on any sexual risk behavior 6 months after the program ended (which corresponds roughly to our 12-month followup). At 1 site, we found a sizable, significant effect on 1 aspect of sexual behavior after 12 months. Program participants in St. Louis were less likely to have engaged in recent sexual intercourse than were members of the control group. Compared with the original study,3 we found even stronger impacts on intermediate outcomes. Although effects on intermediate outcomes were considered exploratory, given the low number of tests in each domain it is unlikely that these findings emerged as a result of chance. Consistent with the earlier study, we found that Reducing the Risk increased knowledge about sexual risk behaviors. Unlike the earlier study, however, we also found positive effects on attitudes toward protection. This impact on attitudes was largely attributable to the San Diego replication, where students in the sample were younger and less likely to have engaged in risky sexual behavior than were students at the other 2 sites. This finding suggests that these younger students who were not yet engaging in risk behaviors may have been more open to receiving the program’s messages. Given the program’s emphasis on developing and practicing skills, it was surprising to find no overall effects on skills. However, the significant impact on perceived condom negotiation skills among students in San Diego again suggests a more open attitude toward the curriculum among these students. Impacts on intermediate outcomes, particularly among the sample of students in San Diego who were younger and less engaged in risky behaviors, suggest the potential for these impacts to translate to impacts on behavior over time. Despite these promising exploratory impacts, other exploratory findings may be cause for concern. Notably, older and White
students who participated in Reducing the Risk had more positive attitudes toward risky behavior, and students in St. Louis who participated in the program were less motivated to delay childbearing as a result of the program. The unfavorable impact on motivation, along with the lack of impacts on other intermediate outcomes in St. Louis, with the exception of knowledge and intentions, is curious given the significant impact on sexual behavior at this site. One possibility is that the intermediate outcomes examined may not lead to behavior change as expected. Examining impacts over a longer time period will provide further insight into both the effectiveness of Reducing the Risk and potential pathways to behavioral change. As an examination of the effectiveness of evidence-based programs and what happens when they are taken to scale, replicated with different populations, and offered in different settings, this study provides important information on the effectiveness of Reducing the Risk. Although we did not find evidence of effectiveness after 12 months, the exploratory findings were mixed. Some may be cause for concern, and others suggest a potential for favorable longer-term effects. Additional follow-up data collection will provide evidence of the effectiveness of Reducing the Risk over time. From these data, we will be able to determine which, if any, impacts on intermediate outcomes after 12 months translate to broader behavior change over time. CONTRIBUTORS M. Kelsey and J. Layzer conceptualized and designed the study, interpreted the data, and drafted and revised the content. M. Blocklin and L. Freiman drafted and revised the content. C. Price and R. Juras conceptualized and designed the study and analyzed and interpreted the data.
ACKNOWLEDGMENTS This study was conducted by Abt Associates as part of the Teen Pregnancy Prevention Replication Study, a national replication study of 3 evidence-based teen pregnancy prevention reduction strategies funded by the Office of Adolescent Health (OAH) and the Office of the Assistant Secretary for Planning and Evaluation, US Department of Health and Human Services (HHS; contract no. HHSP23320095624WC, order no. HHSP23337011T, awarded in September 2011). The study’s clinical trial registration number is NCT02540304. This article is based on a larger report located on the HHS Web site at https://aspe.hhs.gov/teen-pregnancyprevention-tpp-replication-study. We thank each of the participating grantees for their guidance and assistance.
Kelsey et al.
Peer Reviewed
Research
S51
AJPH RESEARCH
Note. The views expressed in this report are those of the authors and do not necessarily represent the policies of HHS or the OAH.
HUMAN PARTICIPANT PROTECTION The evaluation research design, consent, assent, and data collection procedures were approved by the Abt Associates institutional review board for all sites and by additional local institutional review boards associated with the study sites as needed.
REFERENCES 1. Goesling B, Colman C, Trenholm C, Terzian M, Moore K. Programs to reduce teen pregnancy, sexually transmitted infections, and associated risk behaviors: a systematic review. J Adolesc Health. 2014;54(5):499–507. 2. Lezin N, Rolleri LA, Wilson MM, Fuller TR, FirpoTriplett R. Reducing the Risk Adaptation Kit. Santa Cruz, CA: ETR Associates; 2010. 3. Kirby D, Barth RP, Leland N, Fetro JV. Reducing the Risk: impact of a new curriculum on sexual risk-taking. Fam Plann Perspect. 1991;23(6):253–263. 4. What Works Clearinghouse. WWC Procedures and Standards Handbook Version 3.0. Washington, DC: What Works Clearinghouse, Institute of Education Sciences, US Department of Education; 2014. Available from: http://ies.ed.gov/ncee/wwc/pdf/reference_resources/ wwc_procedures_v3_0_standards_handbook.pdf. Accessed March 7, 2016. 5. Orr L. Social Experiments: Evaluating Public Programs With Experimental Methods. Thousand Oaks, CA: Sage; 1999. 6. Puma MJ, Olsen RB, Bell SH, Price C. What to Do When Data Are Missing in Group Randomized Controlled Trials. Report No. NCEE 2009-0049. Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, US Department of Education; 2009. 7. Schochet PZ. Technical Methods Report: Guidelines for Multiple Testing in Impact Evaluations. Report No.: NCEE 2008-4018. Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, US Department of Education; 2008.
S52
Research
Peer Reviewed
Kelsey et al.
AJPH
Supplement 1, 2016, Vol 106, No. S1
Replicating Reducing the Risk: 12-Month Impacts of a Cluster Randomized Controlled Trial Meredith Kelsey, PhD, Michelle Blocklin, PhD, Jean Layzer, MEd, Cristofer Price, ScM, Randall Juras, PhD, and Lesley Freiman, MPP Objectives. To test the effectiveness of Reducing the Risk, an evidence-based sexual health curriculum designed to help prevent adolescent pregnancy and sexually transmitted infections, on youth sexual behavior and intermediate outcomes thought to lead to these behaviors. Methods. Classes within schools in St. Louis, Missouri; Austin, Texas; and San Diego, California; were randomly assigned to receive Reducing the Risk or “business as usual.” Youths completed Web-based surveys at baseline (preintervention, August 2012–January 2014) and 12 months later (August 2013–January 2015). Intent-to-treat analyses were conducted across sites; we tested for differences in impacts between sites and other subgroups. Results. The program had no overall impact on sexual behaviors. However, at 1 site, program participants were significantly less likely to have engaged in recent sexual intercourse than were control group members. There were positive overall impacts on intermediate outcomes (e.g., knowledge, attitudes). Conclusions. After 12 months, Reducing the Risk was unsuccessful at changing sexual behaviors. Other results were mixed, but promising evidence (e.g., behavioral impacts at 1 site, impacts on intermediate outcomes) suggests potential for more widespread behavioral impacts over a longer term. (Am J Public Health. 2016;106:S45–S52. doi: 10.2105/AJPH.2016.303409) See editorials, p. S5–S31.
F
inding ways to reduce adolescent pregnancy is a priority for the US Department of Health and Human Services (HHS). To achieve this goal, HHS is investing in evidence-based pregnancy reduction interventions. The federal Teen Pregnancy Prevention Program, administered by the Office of Adolescent Health (OAH) within HHS, acknowledges the need for additional research on evidence-based programs given the scarcity of independent evaluations and the limited number of program replications.1 Accordingly, the program includes funding to replicate evidence-based adolescent pregnancy prevention models. This study examines the effectiveness of Reducing the Risk by examining 3 OAH-funded replications. Reducing the Risk is an evidence-based sexual health curriculum developed in the early 1990s that was designed to help prevent pregnancy and sexually transmitted infections (STIs) in adolescents aged 13 to 19 years. The program is intended for use in school
Supplement 1, 2016, Vol 106, No. S1
AJPH
classrooms with students of all ethnicities. Its 16 units can be delivered separately in 45-minute sequential sessions or grouped in eight 90-minute sessions. Figure 1 shows the program elements,2 intended outcomes, and pathways by which the program seeks to achieve these outcomes. Taken together, the program sessions are intended to improve students’ attitudes toward protection; increase knowledge and understanding of sexual health issues; improve motivation and intentions to avoid risk; encourage values and beliefs that support abstinence and avoidance of unprotected sex; and improve negotiation, refusal, and condom use skills. These intermediate outcomes
are believed to lead to the behavioral outcomes the program seeks to achieve: correct and consistent use of condoms and other contraceptives, abstinence from sex, and a reduction in sexual activity and number of partners. Prevention of or reduction in sexually risky behavior is ultimately expected to reduce rates of STIs, unwanted pregnancies, and births among adolescents. The evidence for the effectiveness of this program is from a single quasi-experimental study completed 25 years ago in rural and urban areas of northern California with primarily White high school students.3 Although this study found that the program had significant positive effects, further evaluation is critical to determine whether the program remains effective years later in different settings. This study was guided by the following research questions to examine the effectiveness of Reducing the Risk after 12 months: 1. Did the program reduce risky sexual behavior? 2. Did the program have positive effects on intermediate outcomes thought to lead to behavior change (attitudes, knowledge, motivation, intentions, and skills)?
METHODS The study used an experimental design in which classes were randomly assigned to receive the intervention or business as usual, that is, the regular curriculum or offering in the assigned class.
ABOUT THE AUTHORS Meredith Kelsey, Michelle Blocklin, Cristofer Price, Randall Juras, and Lesley Freiman are with Abt Associates, Inc., Cambridge, MA. Jean Layzer is with Belmont Research Associates, Belmont, MA. Correspondence should be sent to Meredith Kelsey, PhD, 55 Wheeler Street, Cambridge, MA 02138 (e-mail: Meredith_Kelsey@ abtassoc.com). Reprints can be ordered at http://www.ajph.org by clicking the “Reprints” link. This article was accepted July 20, 2016. doi: 10.2105/AJPH.2016.303409
Kelsey et al.
Peer Reviewed
Research
S45
AJPH RESEARCH
Inputs
Process Establish psychologically safe environment
Trained teacher Deliver curriculum modules in participatory, interactive sessions
Active participation in discussion, sharing ideas, feelings
Intermediate Outcomes
Outcomes
Positive attitudes toward protection Improved knowledge and understanding
Safer sexual behavior
Improved motivation/intention to avoid risk
Consistent use of contraceptives
C istent effective i Cons condom use
Abstinence
16-unit curriculum
Reinforce norms for abstinence or protected sex
Encourage youths to personalize information, identify vulnerabilities and values
Role-play, practice to improve skills
Values and beliefs supportive of abstinence and avoidance of unprotected sex
Improved negotiation/refusal/ condom use skills
Reduction in sexual activity Reduction in no. of sexual partners
Reduction in STIs Reduction in unwanted pregnancies Reduction in adolescent births
Teach youths to anticipate and prepare for risky situations Note. STI = sexually transmitted infection.
FIGURE 1—Logic Model: Reducing the Risk
Setting Reducing the Risk was implemented by 3 grantees and their partners. These replications, all implemented with careful adherence to the original program model under OAH’s guidance, provided a unique opportunity to study the impact of the program model in more geographically and demographically diverse settings than are typically present in a single replication site. In all 3 replications, external health educators, employed by grantees and their partners, delivered the program in public school classrooms as part of the regular school day. One grantee delivered the program in ninth-grade classes (with some students from higher grades) in 6 public high schools in St. Louis, Missouri; St. Louis County, Missouri; and St. Clair County, Illinois; we refer to this site as St. Louis. Another grantee implemented the program in
S46
Research
Peer Reviewed
Kelsey et al.
mixed 9th- and 10th-grade classes (with some older students) in 5 public high schools in the Austin Independent School District, Austin, Texas. The third grantee implemented the program in eighth- or ninth-grade classes in 6 public middle, junior high, or high schools in San Diego County, California.
Sample Once schools agreed to participate, school staff identified classes to participate in the study. Students in identified classes were recruited for the study without knowing whether their class would receive the program or the “business as usual” curriculum. Program staff distributed consent forms and study brochures. All students who obtained parental permission were enrolled in the study. As shown in Figure A (available as a supplement to the online version of this
article at http://www.ajph.org), 3314 students in 150 classes obtained parental permission. Of those, 2689 (81%) in 150 classes completed the 12-month follow-up survey and were included in the analytic sample. Differential attrition between the treatment and control groups was less than 1% (0.52%), and we found no evidence of systematic differences in baseline characteristics between dropouts in the treatment and control groups. This low rate of differential attrition combined with an overall attrition rate of 19% mitigates the threat of bias.4 At baseline (Table 1), students were (on average) aged almost 15 years, and about half were female. One third were Black; almost half were Hispanic; the remainder were almost equally White or other race. There was some variation across sites (Table A, available as supplement to the online version of this
AJPH
Supplement 1, 2016, Vol 106, No. S1
AJPH RESEARCH
TABLE 1—Baseline Characteristics of the Analytic Sample: Reducing the Risk; St. Louis, MO, Austin, TX, and San Diego County, CA; August 2012–January 2014 Baseline Characteristic
Rangea
No.
Treatment Meanb
Control Mean
Group Differencec
Age
13–20
2689
14.50
14.56
–0.07
Grade
8–12
Female, %d
P .20
2689
9.25
9.27
–0.02
.63
2689
49.10
49.10
0.00
‡ .99
Race/ethnicity, %d Hispanic
2689
46.09
47.12
–1.04
.52
Non-Hispanic Black
2689
33.10
32.96
0.14
.90
Non-Hispanic White
2689
11.34
10.73
0.61
.62
Othere
2689
9.54
9.18
0.36
.79
Lives with biological parents Feels very close to and cared for by father
2613 2368
93.06 45.46
92.24 46.74
0.82 –1.27
.48 .56
Feels very close to and cared for by mother
2592
63.38
65.98
–2.61
.17
Ever smoked cigarettes
2631
21.01
20.63
0.38
.83
Ever drank alcohol
2634
45.73
45.05
0.69
.74
Ever used marijuana
2632
31.23
30.00
1.23
.52
1–4
2652
3.04
3.04
0.00
.91
Knowledge of pregnancy risk
0–100
2653
51.84
50.61
1.22
.45
Knowledge of STI risk
0–100
2655
44.42
43.46
0.96
.41
Intentions to have sexual intercourse in the next 12 mo
2588
41.14
39.16
1.98
.31
Intentions to have oral sex in the next 12 mo
2581
30.00
30.09
–0.08
.97
Intentions to use a condom if they were to have sexual
2592
94.59
94.17
0.42
.65
2558
89.41
90.79
–1.38
.26
Ever sexually activeh Currently sexually active in past 90 dh
2602 2590
30.57 18.37
31.32 20.79
–0.75 –2.42
.68 .18
Sexual intercourse in the past 90 d
2590
16.53
17.99
–1.46
.41
Oral sex in the past 90 d
2582
12.12
14.56
–2.44
.11
Sexual intercourse without birth control in the past 90 d
2590
5.36
6.71
–1.35
.25
Sexual intercourse without a condom in the past 90 d
2590
8.04
9.85
–1.80
.19
Oral sex without a condom in the past 90 d
2582
10.40
11.77
–1.37
.35
d
Family structure and relationships, %
Risk behavior, %d
f
Attitudes toward protection Knowledgeg
d
Intentions, %
intercourse Intentions to use birth control if they were to have sexual intercourse Sexual behavior, %d
d
Sexual risk, %
Note. The baseline treatment–control difference was estimated in a 2-level multilevel model with random intercept terms for classes. The dependent variable was the baseline measure, and the only independent variables included in the model were the treatment group indicator and terms for the randomization blocks. a For continuous variables, we present the range. All other variables are dichotomous. b The treatment mean was calculated as the sum of the control group mean and the model estimated treatment–control difference (group difference). c The group difference is the treatment–control difference. For outcomes reported as percentages, the group difference is expressed in percentage points. For scale outcomes, the group difference is expressed in the original metric of the outcome variable. Because of rounding, reported group differences may differ from differences between reported means for the treatment and control groups. d For dichotomous variables, we present the percentage of respondents who responded affirmatively. e Defined as Asian, American Indian or Alaska Native, Native Hawaiian or other Pacific Islander, multiracial, or undisclosed. f Attitudes variable is a composite scale score with higher scores indicating more positive attitudes. g Knowledge variables are composite scale scores representing the percentage of items answered correctly. h Sexual activity was defined differently across grantees. At 1 site, sexual activity referred to sexual intercourse, oral sex, or anal sex. Youths were not asked about anal sex at 2 of the sites.
Supplement 1, 2016, Vol 106, No. S1
AJPH
Kelsey et al.
Peer Reviewed
Research
S47
AJPH RESEARCH
article): students in San Diego were younger (more than a year, on average), and students in St. Louis were primarily Black, whereas the majority of students in Austin and San Diego were Hispanic. Students in San Diego were also less likely to have engaged in risky behaviors (both sexual and nonsexual).
Random Assignment Design Participating classes in each school were randomly assigned by evaluation staff to receive Reducing the Risk or business as usual (Figure A). The random assignment ratio varied across sites and schools, depending on school and program preferences, with more classes assigned to treatment overall. Schools were not informed of random assignment results until after students completed a baseline survey, described later. Students in classrooms assigned to Reducing the Risk received the program as described in the introduction. At all 3 replication sites, grantees delivered all 16 sessions of the program. To ensure each replication was implemented with fidelity, OAH required grantees to request permission for any adaptations. In Austin, a session on reproductive anatomy and another optional Reducing the Risk session, which reinforced messages about pregnancy and STI prevention, were added and the condom demonstration was dropped to conform to school district policy. At 1 school in St. Louis, the program was delivered to male and female students separately, by instructors of the same gender. In the same school, the agency also replaced the condom demonstration with a video (for boys) and a mini-lecture (for girls). To accommodate larger class sizes, the San Diego site received permission to deliver the program with 2 facilitators. In Austin, students in the control group received the regularly scheduled health class. In St. Louis and San Diego, the class attended by control group students differed by school but was the regularly scheduled class displaced by Reducing the Risk and included physical education, ROTC, health, homeroom or guidance, and science classes.
Data Collection All youths completed a 30-minute survey at baseline (preintervention; August 2012– January 2014) and 12 months after baseline (August 2013–January 2015). At both time
S48
Research
Peer Reviewed
Kelsey et al.
points, a Web-based audio computer-assisted self-interview system was used to capture and store survey responses. The baseline survey was completed in group sessions in schools, using school computers when possible or tablets provided by field staff. Hard copies of the baseline survey (in Spanish and English) were available as backup. The same Webbased system was used for the 12-month follow-up. When possible, the follow-up survey was administered in group sessions in schools. When necessary, students completed the survey on their own time, using personal tablets or computers, school or library computers, or even smartphones. Participants received gift cards after completing each survey (baseline and followup). Of the 3314 randomized students, 98% completed the baseline survey, and 81% completed the follow-up survey.
Measures We collected data on implementation, participant demographic characteristics, and outcomes. Student surveys measured 2 sets of outcomes: (1) sexual behavior outcomes and (2) intermediate outcomes believed to lead to behavioral outcomes. We briefly describe these measures here. Additional details on the construction of these measures and their reliability are presented in Appendix A (available as a supplement to the online version of this article). Implementation. We conducted a series of site visits and focus groups to measure implementation. To measure fidelity, the funding agency provided fidelity checklists, which health educators completed for every session of the program. In addition, grantee-determined supervisory staff were required to observe, record, and report on the quality of program delivery for a sample of sessions. Program staff collected attendance data. Behavioral outcomes. To address the study’s key questions about the impact of the intervention, we identified 7 measures in the domain of youth sexual behavior: 1. 2. 3. 4. 5.
ever sexually active, sexually active in the past 90 days, sexual intercourse in the past 90 days, oral sex in the past 90 days, sexual intercourse without birth control (defined as a condom or other contraceptive) in the past 90 days,
6. sexual intercourse without a condom in the past 90 days, and 7. oral sex without a condom in the past 90 days. Sexually active refers to sexual intercourse, oral sex, or anal sex, though youths were not asked about anal sex at 2 of the sites. Responses for each behavior were dichotomous (yes–no). Intermediate outcomes. We constructed composite measures to assess several intermediate outcomes. Attitudes toward protection were measured on a scale ranging from 1 to 4, with higher scores indicating more positive attitudes. Attitudes toward risky behavior were measured on a scale ranging from 0 to 100, reflecting the percentage of statements with which a respondent agreed; higher scores represented greater support for risky behavior. Knowledge of pregnancy risk and knowledge of STI risk scales both ranged from 0 to 100 and reflected the percentage of items answered correctly. Motivation to delay childbearing was measured on a scale ranging from 1 to 4, with higher scores indicating greater motivation. Condom negotiation skills and refusal skills were both measured on scales ranging from 1 to 4, with higher scores indicating stronger skills. To assess intentions to engage in sexual behaviors and use protection, we examined 4 dichotomous (yes–no) items separately.
Baseline demographic and behavioral characteristics. We collected information on sexual risk behaviors and demographic characteristics at baseline. These measures included ever sexually active, substance use (ever smoked a cigarette, drank alcohol, used marijuana), pregnancy and STI risk knowledge, and intentions to have intercourse and oral sex. Additional demographic measures included age, grade, and race/ethnicity. To measure race/ethnicity, students responded to 2 questions: (1) “Are you Hispanic/Latino (yes–no)?” and (2) “What is your race (American Indian or Alaska Native, Asian, Black or African American, Native Hawaiian or Other Pacific Islander, White, or some other race)?” We classified students as Hispanic, non-Hispanic White, non-Hispanic Black, or non-Hispanic other race (i.e., Asian, American Indian or Alaska Native, Native Hawaiian or other Pacific Islander, multiracial, and undisclosed race).
AJPH
Supplement 1, 2016, Vol 106, No. S1
AJPH RESEARCH
Analytic Approach As part of our analytic strategy, we prespecified that the primary inferences regarding program effectiveness would be based on a pooled analysis (across program sites) of 2 confirmatory outcomes: sexually active in the past 90 days and sexual intercourse without birth control in the past 90 days. Other analyses were considered exploratory, or secondary. Impacts were pooled across sites to limit the number of tests of program impact, to increase power to detect effects, and to support more generalizable statements about program impacts. Given the emphasis on fidelity and consistent implementation across replications, we are confident that pooling across the 3 sites provided a valid test of program effectiveness. To estimate impacts of Reducing the Risk on each of the study’s outcomes, we compared the outcomes for treatment and control group members using a regression framework; for continuous outcomes, these are traditional regression models, and for dichotomous outcomes, these are linear probability models. We included the baseline characteristics described in the preceding section and preintervention measures of the outcome variable, when available, as covariates to increase statistical precision (i.e., reduce the standard errors) of the impact estimates for a given sample size5 and reduce attrition bias from missing data.6 Randomization blocking variables, which included site, school, year, semester, and gender, were also included. To maintain the integrity of the random assignment design, we conducted an intent-to-treat analysis, using assignment status as the key variable of interest regardless of actual program participation. To address missing data, we used case deletion for the few instances of missing outcome data; we used dummy-variable adjustment in regression models to account for missing covariates.6 Specifying 2 confirmatory outcomes, as noted earlier in this section, addresses potential problems resulting from multiple comparisons (i.e., the potential for spurious findings when many hypotheses are tested). To control for having 2 confirmatory outcomes, we applied a formal Benjamini-Hochberg correction, which controls for the false positive rate by adjusting P-value thresholds.4 We considered all other outcomes and all subgroup analyses, described later, to be exploratory and did not adjust for multiple comparisons.7
Supplement 1, 2016, Vol 106, No. S1
AJPH
In addition to the overall pooled impacts, to gain insight into what worked for whom we conducted exploratory subgroup analyses by including subgroup indicators and Treatment · Subgroup interaction terms in the model and by testing for the significance of the interaction term. Only when there was a significant difference in impacts between groups do we discuss impacts for individual subgroups, which helps guard against overinterpretation of potentially spurious findings across the large number of subgroup by outcome tests. Because subgroup analyses were exploratory, we did not adjust for multiple comparisons. Subgroups tested included replication site, gender, age, race/ethnicity, and sexual experience at baseline.
RESULTS In the sections that follow, we briefly summarize our findings on program implementation, compare the treatment and control groups at baseline, and present impacts on behaviors followed by impacts on intermediate outcomes.
Program Implementation Reducing the Risk was well implemented across the 3 replication sites. Grantees hired health educators with appropriate background experience and skills to deliver the program; all received training approved by the program developer, and the program was delivered with fidelity (adherence to its core elements and without modifications that threatened those core elements). Attendance varied by replication site: In San Diego, 85% of students attended at least 75% of the sessions, compared with 73% of students in St. Louis and 47% of students in Austin.
Comparability of the 2 Groups at Baseline We assessed baseline equivalence with a series of models with the same structural components as the impact models. At baseline, no differences were found between the treatment and control groups (Table 1).
Program Impacts on Behavior As shown in Table 2, Reducing the Risk had no impact on either of the 2 confirmatory sexual behavior outcomes (currently sexually
TABLE 2—12-Month Impacts of Reducing the Risk on Sexual Behavior: St. Louis, MO, Austin, TX, and San Diego County, CA; August 2013–January 2015
Outcome
No. Treatment,a % Control, %
Group Differenceb
P
Sexual behavior Ever sexually activec
2670
42.99
41.71
1.28
.41
Currently sexually active (in past 90 d)c,d
2665
28.02
28.14
–0.11
.95
Sexual intercourse in the past 90 d
2667
23.66
24.37
–0.72
.67
Oral sex in the past 90 d
2661
19.24
19.50
–0.26
.87
Sexual risk Sexual intercourse without birth control in the past 90 dd 2667
8.73
8.99
-0.25
.86
Sexual intercourse without a condom in the past 90 d 2667
13.57
15.38
–1.81
.18
Oral sex without a condom in the past 90 d
16.20
17.33
–1.13
.44
2661
Note. All outcomes are dichotomous. a The treatment group percentage is regression adjusted, calculated as the sum of the control group percentage and the regression-adjusted impact estimate (group difference). b The group difference is the treatment–control difference expressed in percentage points. Because of rounding, reported group differences may differ from differences between reported percentages for the treatment and control groups. c Sexual activity was defined differently across grantees. At 1 site, sexual activity referred to sexual intercourse, oral sex, or anal sex. Youths were not asked about anal sex at 2 of the sites. Impacts for anal sex are not reported here. d Confirmatory outcome; for these outcomes, statistical significance at P < .05, P < .01, or P < .001 implies statistical significance at those levels after applying a Benjamini-Hochberg adjustment for multiple comparisons.
Kelsey et al.
Peer Reviewed
Research
S49
AJPH RESEARCH
TABLE 3—12-Month Impacts of Reducing the Risk on Intermediate Outcomes: St. Louis, MO, Austin, TX, and San Diego County, CA; August 2013–January 2015 Rangea
Outcome
No.
Treatment Meanb
Control Mean
Group Differencec
SESd
P
0.13
£ .001
e
Attitudes Attitudes toward protection Attitudes toward risky behavior
1–4
2688
3.18
3.13
0.05
0–100
2675
5.32
4.53
0.80
.16
e
Knowledge
Knowledge of pregnancy risk
0–100
2689
65.55
61.55
4.01
£ .001
Knowledge of STI risk
0–100
2689
60.47
56.21
4.26
£ .001
1–4
2683
3.68
3.68
–0.01
Sexual intercourse Oral sex
2660 2654
52.67 42.41
50.69 43.27
1.97 –0.86
Use a condom if they were to have sexual intercourse
2667
91.21
92.11
–0.90
.40
Use birth control if they were to have sexual intercourse
2662
90.39
89.67
0.72
.54
Motivation to delay childbearinge
–0.01
.74
Intentions to engage in the following behaviors in the next 12 mo, %f .28 .63
Skillse Perceived refusal skills
1–4
2681
3.12
3.08
0.04
0.06
.13
Perceived condom negotiation skills
1–4
2685
3.53
3.50
0.03
0.06
.18
Note. SES = standardized effect size of the difference; STI = sexually transmitted infection. a For continuous variables, we present the range. All other variables are dichotomous. b The treatment group mean is regression adjusted, calculated as the sum of the control group mean and the regression-adjusted impact estimate (group difference). c The group difference is the treatment–control difference. For outcomes reported as percentages, the group difference is expressed in percentage points. For scale outcomes, the group difference is expressed in the original metric of the outcome variable. Because of rounding, reported group differences may differ from differences between reported means for the treatment and control groups. d For outcomes that are not dichotomous or measured on a 0–100 scale, the SES is the group difference divided by the pooled standard deviation of the treatment and control groups. e Composite scale scores. f Dichotomous variables, reported as percentage of respondents who responded affirmatively.
active or recent sexual intercourse without birth control), nor did it have an impact on other related sexual behaviors. Subgroup analyses largely revealed no significant differences in impacts on sexual risk behaviors by subgroup. An important exception was a significant effect of the program on students in the St. Louis sample (Table B, available as supplement to online version of this article). Compared with control group members, program participants in St. Louis were 7 percentage points (> 15%) less likely to report engaging in sexual intercourse in the previous 90 days (P = .011).
Program Impacts on Intermediate Outcomes The program had positive impacts on attitudes and knowledge. No overall effects on motivation, intentions, or skills were found.
S50
Research
Peer Reviewed
Kelsey et al.
Attitudes. As shown in Table 3, the program had a small but statistically significant impact on students’ attitudes toward protection (birth control and condoms). Students in the treatment group had more positive (and protective) attitudes than did those in the control group. Subgroup analyses revealed a difference in impact by site, suggesting that the San Diego replication largely accounted for the overall impact. The impact on attitudes toward protection was moderate (an increase of 0.12 points on a 1–4 scale; standardized effect size = 0.30) and highly significant in San Diego (P < .001); no significant impacts on these attitudes were found at the other 2 sites (Table B). The program had no statistically significant impacts on attitudes toward risky behavior overall (Table 3). However, subgroup analyses showed that among older
students (aged ‡ 15 y) and among White students, those in the treatment group were more likely to express positive attitudes toward risky behavior than were those in the control group (Table B). Among older students, those in the treatment group agreed with 50% more items (P = .008) regarding the acceptability of risky sexual behavior than did those in the control group. Among White students, those in the treatment group agreed with 75% more items (P = .001) regarding the acceptability of risky sexual behavior. Knowledge. The program had statistically significant positive effects on knowledge of pregnancy risk and knowledge of STI risk (Table 3). Compared with control group students, treatment group students scored 4 percentage points (about 7%) higher on knowledge
AJPH
Supplement 1, 2016, Vol 106, No. S1
AJPH RESEARCH
of pregnancy risk and knowledge of STI risk. Motivation to delay childbearing. Students in both the treatment and control groups were highly motivated to delay childbearing. Accordingly, there were no overall differences between the 2 groups (Table 3). However, subgroup analyses revealed that, in the St. Louis sample, treatment students were slightly less motivated to delay childbearing than were control group students (0.10 points lower on a 1–4 scale; standardized effect size = –0.18; P = .014; Table B). Intentions. No overall impact was found on intentions to engage in sexual activity (Table 3). Equal proportions of students in the treatment and control groups expected to engage in sexual intercourse or oral sex in the next year. Nearly all students said they intended to use condoms or birth control if they engaged in sexual intercourse. There was 1 significant site-level effect: In St. Louis, students in the treatment group were 7 percentage points (about 15%) less likely than those in the control group to report intentions to engage in oral sex (P = .011; Table B). Skills. No overall impacts were found on perceived condom negotiation or refusal skills (Table 3). However, subgroup analyses revealed that in the San Diego sample, treatment group students reported slightly stronger (0.10 on a 1–4 scale, standardized effect size = 0.19) condom negotiation skills than did control group students (P = .008; Table B).
DISCUSSION Building on the original study of the effectiveness of Reducing the Risk, this study provides additional evidence regarding the program’s effectiveness in different settings 25 years later. Because Reducing the Risk did not affect either of the study’s confirmatory outcomes, we found no evidence of effectiveness after 12 months. However, this does not rule out the possibility of longerterm effects. Impacts on exploratory outcomes were mixed, but some provide grounds for optimism. A significant program effect was found on sexual behavior at 1 site, and the program achieved its
Supplement 1, 2016, Vol 106, No. S1
AJPH
intended effects on some intermediate outcomes, which might lead to more widespread impacts over time. We found no effects on our confirmatory outcomes of current sexual activity and sexual intercourse without birth control. However, the original study found no effects on any sexual risk behavior 6 months after the program ended (which corresponds roughly to our 12-month followup). At 1 site, we found a sizable, significant effect on 1 aspect of sexual behavior after 12 months. Program participants in St. Louis were less likely to have engaged in recent sexual intercourse than were members of the control group. Compared with the original study,3 we found even stronger impacts on intermediate outcomes. Although effects on intermediate outcomes were considered exploratory, given the low number of tests in each domain it is unlikely that these findings emerged as a result of chance. Consistent with the earlier study, we found that Reducing the Risk increased knowledge about sexual risk behaviors. Unlike the earlier study, however, we also found positive effects on attitudes toward protection. This impact on attitudes was largely attributable to the San Diego replication, where students in the sample were younger and less likely to have engaged in risky sexual behavior than were students at the other 2 sites. This finding suggests that these younger students who were not yet engaging in risk behaviors may have been more open to receiving the program’s messages. Given the program’s emphasis on developing and practicing skills, it was surprising to find no overall effects on skills. However, the significant impact on perceived condom negotiation skills among students in San Diego again suggests a more open attitude toward the curriculum among these students. Impacts on intermediate outcomes, particularly among the sample of students in San Diego who were younger and less engaged in risky behaviors, suggest the potential for these impacts to translate to impacts on behavior over time. Despite these promising exploratory impacts, other exploratory findings may be cause for concern. Notably, older and White
students who participated in Reducing the Risk had more positive attitudes toward risky behavior, and students in St. Louis who participated in the program were less motivated to delay childbearing as a result of the program. The unfavorable impact on motivation, along with the lack of impacts on other intermediate outcomes in St. Louis, with the exception of knowledge and intentions, is curious given the significant impact on sexual behavior at this site. One possibility is that the intermediate outcomes examined may not lead to behavior change as expected. Examining impacts over a longer time period will provide further insight into both the effectiveness of Reducing the Risk and potential pathways to behavioral change. As an examination of the effectiveness of evidence-based programs and what happens when they are taken to scale, replicated with different populations, and offered in different settings, this study provides important information on the effectiveness of Reducing the Risk. Although we did not find evidence of effectiveness after 12 months, the exploratory findings were mixed. Some may be cause for concern, and others suggest a potential for favorable longer-term effects. Additional follow-up data collection will provide evidence of the effectiveness of Reducing the Risk over time. From these data, we will be able to determine which, if any, impacts on intermediate outcomes after 12 months translate to broader behavior change over time. CONTRIBUTORS M. Kelsey and J. Layzer conceptualized and designed the study, interpreted the data, and drafted and revised the content. M. Blocklin and L. Freiman drafted and revised the content. C. Price and R. Juras conceptualized and designed the study and analyzed and interpreted the data.
ACKNOWLEDGMENTS This study was conducted by Abt Associates as part of the Teen Pregnancy Prevention Replication Study, a national replication study of 3 evidence-based teen pregnancy prevention reduction strategies funded by the Office of Adolescent Health (OAH) and the Office of the Assistant Secretary for Planning and Evaluation, US Department of Health and Human Services (HHS; contract no. HHSP23320095624WC, order no. HHSP23337011T, awarded in September 2011). The study’s clinical trial registration number is NCT02540304. This article is based on a larger report located on the HHS Web site at https://aspe.hhs.gov/teen-pregnancyprevention-tpp-replication-study. We thank each of the participating grantees for their guidance and assistance.
Kelsey et al.
Peer Reviewed
Research
S51
AJPH RESEARCH
Note. The views expressed in this report are those of the authors and do not necessarily represent the policies of HHS or the OAH.
HUMAN PARTICIPANT PROTECTION The evaluation research design, consent, assent, and data collection procedures were approved by the Abt Associates institutional review board for all sites and by additional local institutional review boards associated with the study sites as needed.
REFERENCES 1. Goesling B, Colman C, Trenholm C, Terzian M, Moore K. Programs to reduce teen pregnancy, sexually transmitted infections, and associated risk behaviors: a systematic review. J Adolesc Health. 2014;54(5):499–507. 2. Lezin N, Rolleri LA, Wilson MM, Fuller TR, FirpoTriplett R. Reducing the Risk Adaptation Kit. Santa Cruz, CA: ETR Associates; 2010. 3. Kirby D, Barth RP, Leland N, Fetro JV. Reducing the Risk: impact of a new curriculum on sexual risk-taking. Fam Plann Perspect. 1991;23(6):253–263. 4. What Works Clearinghouse. WWC Procedures and Standards Handbook Version 3.0. Washington, DC: What Works Clearinghouse, Institute of Education Sciences, US Department of Education; 2014. Available from: http://ies.ed.gov/ncee/wwc/pdf/reference_resources/ wwc_procedures_v3_0_standards_handbook.pdf. Accessed March 7, 2016. 5. Orr L. Social Experiments: Evaluating Public Programs With Experimental Methods. Thousand Oaks, CA: Sage; 1999. 6. Puma MJ, Olsen RB, Bell SH, Price C. What to Do When Data Are Missing in Group Randomized Controlled Trials. Report No. NCEE 2009-0049. Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, US Department of Education; 2009. 7. Schochet PZ. Technical Methods Report: Guidelines for Multiple Testing in Impact Evaluations. Report No.: NCEE 2008-4018. Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, US Department of Education; 2008.
S52
Research
Peer Reviewed
Kelsey et al.
AJPH
Supplement 1, 2016, Vol 106, No. S1
