AJPH RESEARCH
Replicating ¡Cuídate!: 6-Month Impact Findings of a Randomized Controlled Trial Meredith Kelsey, PhD, Carolyn Layzer, PhD, Jean Layzer, MEd, Cristofer Price, ScM, Randall Juras, PhD, Michelle Blocklin, PhD, and Jacqueline Mendez, MA Objectives. To test whether ¡Cuídate!, a program culturally adapted for Hispanic youths, affects sexual risk behavior. Methods. We evaluated 3 replications of ¡Cuídate! in California, Arizona, and Massachusetts in a randomized controlled trial (registry no. NCT02540304) in which 2169 primarily Hispanic participants were randomly assigned to an intervention (n = 1326) or a control (n = 870) group.Youths were surveyed at baseline (September 2012–April 2014) and 6 months postbaseline (March 2013–October 2014). We estimated pooled and subgroup impacts using a regression framework with baseline covariates to increase statistical precision (1216 youths analyzed in the treatment group, 806 analyzed in the control group). Results. We found no impacts on the study’s primary outcomes of recent sexual activity or recent unprotected sexual activity. However, ¡Cuídate! improved knowledge (10%–20% increase; P < .001), attitudes (effect size = .24; P < .001), and skills (effect size = .14; P = .002). Exploratory subgroup analyses suggest potentially problematic effects for some groups. Conclusions. Findings suggest that ¡Cuídate! was effective in improving youths’ knowledge and attitudes. However, after 6 months, these changes did not translate to improvements in reported sexual risk behaviors. (Am J Public Health. 2016;106:S70–S77. doi:10.2105/AJPH.2016.303371) See editorials, p. S5–S31.
R
educing unplanned adolescent pregnancy is a priority for the US Department of Health and Human Services. The federal Teen Pregnancy Prevention Program, administered by the Office of Adolescent Health (OAH), funds programs that address the high rate of adolescent pregnancy by replicating specific evidence-based programs. The evidence for many of these programs rests on a single study conducted more than a decade ago.1 There is also interest in examining these programs beyond their original setting or target population. This study is one of several currently underway that will provide updated evidence on these programs. This study examines the effect of the ¡Cuídate! program model on adolescent sexual behavior and on intermediate outcomes such as knowledge, attitudes, motivation, intentions, and skills. Developed more than a decade ago in response to concerns about disparities in the rates
S70
Research
Peer Reviewed
Kelsey et al.
of pregnancy and HIV among Hispanic youths, ¡Cuídate! is among a handful of evidence-based programs that grantees can use to address the issue of sexual risk behavior among Hispanic adolescents.2 Aside from the single research study of ¡Cuídate! cited in the OAH evidence review,3 few studies of pregnancy prevention have focused specifically on Hispanic youths, although some have included a substantial number of Hispanic adolescents. Although pregnancy rates among Hispanic adolescents have declined at nearly the same rate as in other groups, the rate remains more than double that for non-Hispanic White adolescents.4 Reasons suggested for the
disparity are that Hispanic adolescents are less likely to use condoms than are White and Black youths and are more likely to report multiple sexual partners.5 ¡Cuídate! (which means “take care of yourself ”) was adapted from Be Proud, Be Responsible!,6 an HIV reduction program shown to be effective with Black youths, and was originally tested by the developer among mostly Puerto Rican youths in Philadelphia, Pennsylvania. ¡Cuídate! is culturally tailored for Hispanic youths; materials used in the sessions emphasize core Hispanic values such as traditional gender roles and the importance of family and link them to safer sexual behavior. Six 60-minute curriculum modules are delivered in participatory, interactive sessions that weave in the theme of taking care of oneself and one’s partner, family, and community. The modules are delivered by a trained adult facilitator, bilingual in English and Spanish, to small groups of 6 to 10 youths. ¡Cuídate! was originally tested in an after-school setting on consecutive weekends, but it can be delivered in other settings and on different schedules. ¡Cuídate! is intended to provide information and affect potential mediating factors such as knowledge and understanding of reproductive health and avoidance of sexual risk, attitudes toward using protection, motivation to delay pregnancy, intentions to become sexually active and use protection, and skills needed to avoid sexual risk. These factors are thought to lead to safer sexual behavior such as increased condom use and abstinence, as well as decreased sexual activity and number of partners. The initial test of ¡Cuídate! found that youths who
ABOUT THE AUTHORS Meredith Kelsey, Carolyn Layzer, Cristofer Price, Randall Juras, Michelle Blocklin and Jacqueline Mendez are with Abt Associates, Inc., Cambridge, MA. Jean Layzer is with Belmont Research Associates, Belmont, MA. Correspondence should be sent to Meredith Kelsey, PhD, 55 Wheeler Street, Cambridge, MA 02138 (e-mail: Meredith_Kelsey@ abtassoc.com). Reprints can be ordered at http://www.ajph.org by clicking the “Reprints” link. This article was accepted July 4, 2016. doi: 10.2105/AJPH.2016.303371
AJPH
Supplement 1, 2016, Vol 106, No. S1
AJPH RESEARCH
Inputs
Trained facilitator
6-unit curriculum (flexible delivery schedule)
Process
Intermediate Outcomes
Incorporate theme of taking care—of self, partner, family, community throughout program sessions
Improved knowledge and understanding of how to avoid unsafe sexual behaviors
Active participation in discussion, sharing ideas, feelings
Deliver curriculum modules in participatory, interactive sessions
Use materials to show and emphasize core Latino values and link to safer sexual behavior
Culturally-and linguistically appropriate materials
Demonstrate condom use and teach negotiation/refusal skills
Small-group sessions
Attitudes, values, and beliefs supportive of abstinence and avoidance of unprotected sex Increased motivation to delay childbearing
Role-play, practice to improve skills
Intentions to abstain from sexual activity and to use protection when sexually active
Improved negotiation/refusal/ condom use skills
Outcomes
Safer sexual behavior Consistent effective condom use Abstinence Reduction in sexual activity Reduction in no. of sexual partners
Reduction in STIs Reduction in unplanned pregnancies Reduction in adolescent births
Note. STI = sexually transmitted infection.
FIGURE 1—¡Cuídate! logic model
completed the program reported fewer incidents of sexual intercourse, fewer sex partners, and fewer days of unprotected intercourse than youths assigned to a health promotion program.2 These safer sexual behaviors are ultimately expected to result in a reduction in the rates of sexually transmitted infections (STIs) and unwanted pregnancies and births among adolescents. Figure 1 shows the program elements, the hypothesized outcomes, and the pathways by which the program seeks to achieve these outcomes. This study focused on program effects after 6 months and was guided by the following research questions: 1. Did ¡Cuídate! improve adolescents’ knowledge and understanding of pregnancy risks and prevention and the transmission and prevention of STIs? 2. Did ¡Cuídate! have positive effects on adolescents’ attitudes toward sexual activity, birth control, and condom use?
Supplement 1, 2016, Vol 106, No. S1
AJPH
3. Did ¡Cuídate! increase adolescents’ motivation to delay childbearing? 4. Did ¡Cuídate! increase adolescents’ intentions to avoid risky sexual behavior? 5. Did ¡Cuídate! increase adolescents’ confidence in their ability to refuse unwanted sex and to negotiate safe sex? 6. Did ¡Cuídate! lead adolescents to delay sexual initiation and reduce risky sexual behaviors?
METHODS We used an experimental design in which students were randomly assigned to a group that received the ¡Cuídate! intervention or to a control group that did not. The clinical trial registry number is NCT02540304. From among the Teen Pregnancy Prevention Program grants awarded in 2010, we selected 3 grantees that we felt could provide the strongest test of ¡Cuídate!—1 in a small city
in Southern California; 1 in Phoenix, Arizona; and 1 in Boston, Massachusetts. By examining all 3 sites together, we were able to assess the program’s impact in more diverse settings and with a more diverse population than is typically available at a single replication site. Close oversight by OAH ensured consistency of implementation and fidelity to the original program model across the 3 sites. Minor variations approved in advance by OAH were permitted.7 All 3 replication sites added a brief review of reproductive anatomy. The California site added 2 sessions to comply with state requirements governing sex education—1 on contraception and a second on STIs other than HIV. The Phoenix site added a session on pregnancy prevention, and it also received approval from OAH to deliver the program in larger groups of 20 students with 2 health educators rather than 1. These modifications were not considered changes to the core elements of the model.
Kelsey et al.
Peer Reviewed
Research
S71
AJPH RESEARCH
Setting Settings varied across replication sites. In California, the program was implemented with 10th graders in 3 public high schools during the regular school day while control group youths remained in their regular physical education class. In Phoenix, it was implemented with eighth graders in singlegender classes in 11 public kindergarten– eighth-grade schools during the regular school day while control group youths received a curriculum focused on health and well-being. In Boston, the program was delivered to youths aged 13 to 19 years in a range of settings, including public high schools (traditional, vocational–technical, and charter; during and after school), a summer youth employment program, and a summer youth sports program, while control group youths received the usual programming—regular physical education, regular health class, or other regular activities.
Sample Beginning in September 2012, students at each replication site were recruited for the program and the study. School staff identified classes or time slots for the program (small groups of students would be pulled out for the program). Program staff distributed consent forms and study brochures and provided small incentives for the return of consent forms. Students with parental permission were included in the random assignment conducted by the independent evaluator. At each replication site, students were stratified by gender and randomly assigned within class period within school. At the 2 replication sites that offered ¡Cuídate! in small groups in schools or in other settings, individual students were randomly assigned using a 2:1 ratio (i.e., for every 2 students assigned to the treatment group, 1 was assigned to the control group). At the 3rd replication site, where the program was delivered to single-gender groups, random assignment was done by gender using a 1:1 ratio. Students completed the baseline survey and were subsequently told their assignment. In total, 2198 eligible students obtained permission and were randomly assigned (1328 to the treatment group and 870 to the control group). Of these, 2022 completed the 6-month survey (1216 in the treatment group and 806 in the control group) and are
S72
Research
Peer Reviewed
Kelsey et al.
included in our analytic sample (Figure A, available as a supplement to the online version of this article at http://www.ajph.org).
Data Collection Youths completed a 30-minute survey at baseline (before the intervention began) and 6 months after baseline to assess the shortterm impacts of ¡Cuídate!. (Study participants were also surveyed 18 months after baseline; these analyses will be used to assess the longer term effectiveness of ¡Cuídate!.) At all time points, a Web-based Audio Computer-Assisted Self-Interview system was used to capture and store survey responses. The baseline survey was completed in group sessions in schools or other settings, on school computers when possible, or on tablets provided by field staff. For the 6-month survey, youths could access and complete the survey using personal tablets or computers, school or library computers, or even their smartphones. Participants received a gift card on completion of each survey. We collected data on implementation through interviews, focus groups, and site visits. We obtained data on fidelity and attendance from the grantees, who were required to collect and report performance measures to the funder. In addition, supervisory staff were required to observe, record, and report on the quality of program delivery for a sample of sessions.
Measures The surveys collected information from students on a variety of topics, including questions that allowed us to measure 2 sets of outcomes: (1) intermediate outcomes hypothesized to lead to behavioral outcomes and (2) sexual activity and risk behaviors. We briefly describe these measures here. Details on their construction are presented in Appendix A (available as a supplement to the online version of this article at http://www.ajph.org). Intermediate outcomes. We constructed composite measures within 4 domains to assess intermediate outcomes: (1) knowledge of pregnancy risk and STI risk, (2) attitudes toward use of condoms and other birth control methods (a = .79) and attitudes that reflect social norms about extreme sexual risk behaviors (a = .81), (3) motivation to delay childbearing (a = .87), and (4) condom
negotiation (a = .83) and refusal (a = .87) skills. In a fifth domain, we analyzed 4 items that measured intentions to engage in sexual intercourse in the next year and to use protection when sexually active. Behavioral outcomes. To address the study’s most important question about the impact of the intervention on sexual activity, we identified 9 measures in the domain of youth sexual behavior: ever sexually active, sexually active in the past 90 days, sexual intercourse in the past 90 days, oral sex in the past 90 days, anal sex in the past 90 days, sexual intercourse without birth control in the past 90 days, sexual intercourse without a condom in the past 90 days, oral sex without a condom in the past 90 days, and anal sex without a condom in the past 90 days.
Baseline behavioral and demographic characteristics. Participant characteristics were measured via baseline survey items and included race/ethnicity, age, grade, living with a biological parent, ever sexually active, ever smoked a cigarette, ever had an alcoholic drink, ever used marijuana, pregnancy and STI risk knowledge, and intentions to have oral sex and sexual intercourse. Respondents were asked to classify their own race/ethnicity by responding to 2 questions: 1. Are you Hispanic/Latino? (yes–no) 2. What is your race? You may select more than one answer (American Indian or Alaska Native; Asian; Black or African American; Native Hawaiian or other Pacific Islander; White; some other race [please specify]). On the basis of these responses, youths were classified as Hispanic, non-Hispanic White, non-Hispanic Black, or non-Hispanic other race (i.e., Asian, American Indian or Alaska Native, Native Hawaiian or other Pacific Islander, multiracial, and undisclosed race).
Analytic Approach To test the impact of ¡Cuídate! after 6 months on each of the study’s outcomes, we compared the outcomes of treatment and control group participants using a regression framework; for continuous outcomes, these were linear regression models; for dichotomous outcomes, these were linear probability models. We included the baseline
AJPH
Supplement 1, 2016, Vol 106, No. S1
AJPH RESEARCH
characteristics described earlier and baseline measures of the outcome as covariates (when available) to increase statistical precision of the impact estimates for a given sample size8 and reduce attrition bias from missing data.9 Randomization blocking variables were also included. To maintain the integrity of the random assignment design, we conducted an intent-to-treat analysis, using assignment status as the key variable of interest, regardless of actual program participation. In other words, all participants who completed the 6-month follow-up survey were included in the analysis with the assignment they were given at the time of randomization (i.e., treatment or control), even if youths assigned to treatment did not show up for the program or youths assigned to the control condition attended some program sessions. Data for the impact estimates were pooled across the 3 replication sites for the main analysis. This study had low overall attrition (8.0% lost to follow-up), and differential attrition across the treatment and control groups was also quite low (1.0%), minimizing the potential for attrition bias.10 To address missing data, we used case deletion for the few instances of missing outcome data and dummy variable adjustment to account for missing covariates (i.e., missing values were recoded to a constant, and dummy variables were included to indicate whether the value was originally missing).9 To mitigate multiple comparison problems (i.e., reduce the chance of spurious findings from a large number of tests), before analyzing data we prespecified 2 confirmatory outcomes in our analysis plan as the main indicators of program effectiveness: sexual activity in the past 90 days and sexual intercourse without birth control in the past 90 days. To control for having 2 confirmatory outcomes, we applied a formal BenjaminiHochberg correction, which controls for the false positive rate by adjusting P-value thresholds.10 We considered all other outcomes to be exploratory; thus, they did not warrant a multiple comparisons adjustment.11 In addition to the overall pooled impacts, we conducted exploratory subgroup analyses for behavioral outcomes to gain some insight into what worked for whom by including subgroup indicators and treatment · subgroup interaction terms in the model and testing for the significance of the interaction term. Subgroups tested included replication
Supplement 1, 2016, Vol 106, No. S1
AJPH
site, gender, age, race/ethnicity, and sexual experience at baseline. In evaluating the effectiveness of ¡Cuídate!, an intervention specifically tailored to Hispanic youths, it was particularly important to examine whether the intervention’s effects differed for youths by reported race/ethnicity. Only when there was a significant difference in impacts between subgroups did we present impact estimates for individual subgroups, to guard against overinterpretation of potentially spurious findings by reducing the number of results presented in tables. Because subgroup analyses are exploratory, we did not adjust for multiple comparisons.
RESULTS Table 1 shows the distribution of the analytic sample on key measures at baseline. At baseline, youths in the study sample were aged 14.4 years, on average. However, average age varied across the replication sites; where the program was implemented only in eighth-grade classrooms, the average age of students was only 13 years. Adolescent girls constituted more than half of the sample. More than 70% were Hispanic, 18% were White, and the remaining 10% were divided between Black and other race. More than one fifth of the sample had engaged in sexual intercourse before the study began. About half had ever used alcohol, one fifth had smoked cigarettes, and a quarter had used marijuana. Baseline equivalence was assessed using a series of models with the same structural components as the impact models. At baseline, very few significant differences were found between the treatment and control groups (Table 1). The treatment group was at a slightly higher grade level, was less likely to be Black, was more likely to live with their biological parents, and had higher intention to have sexual intercourse in the next 12 months. These 4 variables were subsequently included in all impact models as covariates.
Program Implementation ¡Cuídate! was well implemented across all 3 replication sites. The 3 grantees hired staff with appropriate background experience and skills to deliver the program; all received training provided by the curriculum
distributor. Each of the grantees successfully delivered the program with fidelity (adherence to its core elements and without modifications that threatened those core elements). At all 3 replication sites, a majority of students received at least 75% of the sessions offered. More information on program implementation will be available in a forthcoming implementation report.12
Program Impacts on Intermediate Outcomes After 6 months, ¡Cuídate! had a positive impact on youths’ knowledge, attitudes, and skills (Table 2). No program effects were found on motivation or on intentions to engage in sexual behaviors in the following year. Compared with control group students, treatment group students scored 6 percentage points higher on the composite measure of knowledge of pregnancy risk; this difference was statistically significant. ¡Cuídate! also had a large and statistically significant positive impact on a composite measure of knowledge of STI risk. Treatment students scored 11% points higher, on average, than control students. ¡Cuídate! had a small to moderate (standardized effect size of the difference = .24) but statistically significant impact on students’ attitudes toward using protection. For the composite measure of attitudes toward protection (encompassing both birth control and condoms), students in the treatment group had more positive (and protective) attitudes. ¡Cuídate! had no statistically significant impact on student attitudes toward risky behavior. On measures of student views on perceived social norms about risky behavior, students in both the treatment and the control groups typically rejected the view that risky behavior was acceptable. Students in both the treatment and the control groups were also highly motivated to delay childbearing: Participants across both groups indicated a belief in the importance of delaying childbearing until personal goals have been achieved. ¡Cuídate! did not affect student intentions to engage in sexual activity or perceived refusal skills, but the program had a statistically significant impact on perceived condom negotiation skills. Program participants were more confident that they could successfully
Kelsey et al.
Peer Reviewed
Research
S73
AJPH RESEARCH
TABLE 1—Baseline Characteristics of the Analytic Sample: ¡Cuídate!; Southern California; Phoenix, AZ; and Boston, MA; September 2012–April 2014 Rangea
No.b
Treatment Meanc
Control Mean
Group Differenced
P
Age
11–20
2022
14.39
14.37
0.02
.63
Grade
6–13
2022
9.19
9.13
0.05
2022
52.98
52.98
0.00
‡ .99
Hispanic
2022
71.40
70.10
1.30
.51
Black
2022
3.73
5.83
–2.10
.027
White
2022
17.96
18.49
–0.53
.75
Other
2022
6.91
5.58
1.32
.23
Lives with biological parents
1945
94.30
90.71
3.60
.003
Feels very close to and cared for by father
1800
44.45
46.06
–1.61
.50
Feels very close to and cared for by mother
1930
59.13
60.29
–1.16
.61
Ever smoked cigarettes
1975
18.64
18.41
0.23
.90
Ever drank alcohol
1976
46.06
48.08
–2.03
.36
Ever used marijuana
1974
25.32
25.90
–0.58
.76 .83
Outcome Demographic characteristics
Female, %e
.015
Race/ethnicity, %e,f
e
Family structure and relationships, %
Risk behavior, %e
Knowledge
g
Knowledge of pregnancy risk
0–100
1994
48.37
47.98
0.39
Knowledge of STI risk
0–100
1995
38.61
39.36
–0.75
.56
1–4
1988
3.07
3.06
0.00
.83
Intentions to have sexual intercourse in the next 12 mo
1946
31.91
27.50
4.41
.023
Intentions to have oral sex in the next 12 mo
1939
25.10
22.76
2.34
.21
Intentions to use a condom if they were to have sexual intercourse
1944
92.99
94.72
–1.73
.13
Intentions to use birth control if they were to have sexual intercourse
1923
92.07
91.99
0.08
.95
Ever sexually activei
1969
25.22
21.99
3.23
.07
Currently sexually active in the past 90 di
1959
17.38
14.69
2.69
.10
Sexual intercourse in the past 90 d
1962
14.49
12.37
2.12
.16
Oral sex in the past 90 d
1959
12.66
10.34
2.32
.11
Anal sex in the past 90 dg
1143
4.25
3.42
0.84
.48
Sexual intercourse without birth control in the past 90 d
1962
4.38
3.48
0.90
.33
Sexual intercourse without a condom in the past 90 d
1962
8.33
6.44
1.89
.11
Oral sex without a condom in the past 90 d
1959
11.31
9.17
2.14
.12
Anal sex without a condom in the past 90 di
1143
2.77
2.05
0.72
.46
Attitudes toward protectionh Intentions, %e
Sexual behavior, %e
Sexual risk, %e
Note. STI = sexually transmitted infection. The baseline treatment–control difference was estimated by using the baseline measure as the dependent variable and the treatment group indicator and the terms for the randomization blocks as independent variables. a For continuous variables, we present the range. All other variables are dichotomous. b Sample sizes vary slightly across outcomes because of missing outcome data. We report the sample sizes for each outcome. c The treatment mean was calculated as the sum of the control group mean and the model estimated treatment–control difference (group difference). d The group difference is the treatment–control difference. For outcomes reported as percentages, the group difference is expressed in percentage points. For scale outcomes, the group difference is expressed in the original metric of the outcome variable. Because of rounding, reported group differences may differ from differences between reported means for the treatment and control groups. e For dichotomous variables, we present the percentage of respondents who responded affirmatively. f Racial/ethnic categories include Hispanic, Black non-Hispanic, White non-Hispanic, and other race non-Hispanic, where “other” is defined as Asian, American Indian or Alaska Native, Native Hawaiian or other Pacific Islander, multiracial, or undisclosed. g Knowledge variables are composite scale scores representing the percentage of items answered correctly. h Attitude variable is a composite scale score with higher scores indicating more positive attitudes. i Sexual activity was defined differently across grantees. At 2 sites, sexual activity referred to sexual intercourse, oral sex, and anal sex. Youths were not asked about anal sex at 1 site.
S74
Research
Peer Reviewed
Kelsey et al.
AJPH
Supplement 1, 2016, Vol 106, No. S1
AJPH RESEARCH
TABLE 2—Six-Month Impacts of ¡Cuídate! on Intermediate Outcomes: Southern California; Phoenix, AZ; and Boston, MA; March 2013–October 2014 Rangea
Outcome
Nb
Treatment Meanc
Control Mean
Group Differenced
SESe
P
Knowledgef Knowledge of pregnancy risk
0–100
2022
67.07
60.95
6.12
< .001
Knowledge of STI risk
0–100
2022
63.67
53.01
10.66
< .001
f
Attitudes
1–4
2022
3.24
3.14
0.10
0–100
2011
3.12
3.33
–0.21
1–4
2015
3.69
3.69
0.00
Sexual intercourse
2003
40.38
39.07
1.31
Oral sex
1997
37.16
36.60
0.56
.76
Use a condom if they were to have sexual intercourse
2005
92.89
92.74
0.15
.90
Use birth control if they were to have sexual intercourse
1996
93.23
92.42
0.80
.49
f
Motivation to delay childbearing
.24
< .001
Attitudes toward protection Support for risky behavior
.69 –.01
.91
Intentions to engage in the following behaviors in the next 12 mo, %g
Skills
.47
f
Perceived refusal skills
1–4
2015
3.19
3.13
0.06
.08
.06
Perceived condom negotiation skills
1–4
2016
3.53
3.46
0.07
.14
.002
Note. SES = standardized effect size of the difference; STI = sexually transmitted disease. a For continuous variables, we present the range. All other variables are dichotomous. b Sample sizes vary slightly across outcomes because of missing outcome data. We report the sample sizes for each outcome. c The treatment group mean is regression adjusted, calculated as the sum of the control group mean and the regression-adjusted impact estimate (group difference). d The group difference is the treatment–control difference. For outcomes reported as percentages, the group difference is expressed in percentage points. For scale outcomes, the group difference is expressed in the original metric of the outcome variable. Because of rounding, reported group differences may differ from differences between reported means for the treatment and control groups. e For outcomes that are not dichotomous or measured on a 0–100 scale, the SES is the group difference divided by the pooled standard deviation of the treatment and control groups. f Composite scale scores. g Dichotomous variables, reported as percentage of respondents who responded affirmatively.
negotiate condom use with a partner (standardized effect size of the difference = 0.14) than were those in the control group.
Program Impacts on Behavior Despite program impacts on youths’ knowledge, attitudes, and skills, ¡Cuídate! had no statistically significant impact on the confirmatory behavioral outcome measures (i.e., sexually active in the past 90 days and sexual intercourse without birth control in the past 90 days; Table 3). Consistent with these findings, we also found no statistically significant impacts on other related sexual behaviors. Although behavioral impact estimates were not statistically significant for the full sample, we found some variation by subgroup, and for some subgroups, an adverse program effect was observed (Table A, available as a supplement to the online version of this article at http://www.ajph.org).
Supplement 1, 2016, Vol 106, No. S1
AJPH
We found variation in program impact on sexual intercourse in the past 90 days, depending on sexual experience at baseline. A significant adverse program effect was observed for participants who had been sexually active at baseline, with treatment group members significantly more likely (> 7 percentage points; P = .010) to report having had recent sexual intercourse than their control group counterparts. We also found variation in program impacts on oral sex and oral sex without a condom in the past 90 days by respondent race/ethnicity. Significant adverse program effects were observed for White treatment group members, who were about 9 percentage points more likely to report having had oral sex (P = .005) and oral sex without a condom (P = .006) in the past 90 days than their control group counterparts. We found no significant effects for Hispanic, Black, or other-race participants. No variation in impacts on sexual behavior by site or other subgroup emerged.
DISCUSSION Findings on the effectiveness of ¡Cuídate! after 6 months were mixed. The program achieved impacts on some exploratory intermediate outcomes, such as knowledge, attitudes toward protection, and condom negotiation skills, suggesting that health educators were effective in communicating program content. Nevertheless, these early results provide no evidence that ¡Cuídate! favorably affected either of the study’s 2 confirmatory behavioral outcomes or any other sexual risk behaviors. The program significantly increased participants’ knowledge and understanding of sexual risk behavior and ways to prevent outcomes such as unplanned pregnancy and STIs. Participants’ attitudes toward use of birth control and condoms were also significantly more positive as a result of the program. Program participants were also significantly more confident in their ability to
Kelsey et al.
Peer Reviewed
Research
S75
AJPH RESEARCH
TABLE 3—Six-Month Impacts of ¡Cuídate! on Sexual Behavior: Southern California; Phoenix, AZ; and Boston, MA; March 2013–October 2014 Outcome
No.a
Treatment, %b
Control, %
Group Differencec
P
Sexual behavior Ever sexually actived
.29
2012
29.31
27.68
1.63
Currently sexually active in the past 90 dd,e
2011
18.79
17.83
0.96
.52
Sexual intercourse in the past 90 d
2012
15.48
14.09
1.39
.31
Oral sex in the past 90 d
2009
14.69
13.13
1.56
.27
Anal sex in the past 90 dd
1173
2.48
2.87
–0.39
.70
Sexual intercourse without birth control in
2012
5.77
4.86
0.90
.38
the past 90 de Sexual intercourse without a condom in
2012
9.81
8.10
1.70
.16
Oral sex without a condom in the past 90 d
2009
1.46
1.99
–0.53
.21
Anal sex without a condom in the past 90 df
1173
5.77
4.86
0.90
.53
Sexual risk
the past 90 d
Note. All outcomes are dichotomous. a Sample sizes vary slightly across outcomes because of missing outcome data. We report the sample sizes for each outcome. b The treatment group percentage is regression adjusted, calculated as the sum of the control group percentage and the regression-adjusted impact estimate (group difference). c The group difference is the treatment–control difference expressed in percentage points. Because of rounding, reported group differences may differ from differences between reported percentages for the treatment and control groups. d Sexual activity was defined differently across grantees. At 2 sites, sexual activity referred to sexual intercourse, oral sex, or anal sex. Youths were not asked about anal sex at 1 site. e For the 2 confirmatory outcomes, statistical significance at P < .05, P < .01, and P < .001 implies statistical significance at those levels after applying a Benjamini-Hochberg adjustment for multiple comparisons. f Items asking about anal sex were not included in the survey administered to participants in 1 site.
negotiate the use of condoms. However, the program had no significant effects on motivation to delay pregnancy or intentions with respect to future sexual behavior. The primary goal of this and other adolescent pregnancy prevention programs is to change behaviors. The ¡Cuídate! logic model posits that changes in knowledge, attitudes, motivation, intentions, and skills will lead to changes in behavior. After 6 months, however, ¡Cuídate! had no favorable impacts on any behavioral outcomes, although the study was well powered to detect such impacts.13 Standard errors allow us to confidently rule out positive impacts substantially larger than 0.1 standard deviation for behavioral outcomes and for intermediate outcomes that were found to be statistically insignificant. To the contrary, findings suggest the program may have had adverse impacts on some behavioral outcomes for certain subgroups. Six months may not have been long enough for impacts on intermediate outcomes to
S76
Research
Peer Reviewed
Kelsey et al.
translate to beneficial behavioral changes. To better understand the lack of favorable behavioral changes, it will be important to see whether patterns seen after 6 months change or persist over time. The program is specifically designed for Hispanic youths, broadly defined, although the developer maintains that its messages are appropriate for youths of all ethnicities, because it is solidly based on another evidence-based program that is intended for a broader population. All 3 replication sites successfully recruited Hispanic youths, although they also served non-Hispanic youths. Subgroup analyses indicated some differences between racial/ethnic groups, but impacts were not significant among Hispanic youths. Additional sensitivity analyses, in which we ran the full set of analyses on Hispanic youths only, indicated findings similar to those for the full sample. Thus, the lack of favorable behavioral impacts is unlikely to be a result of the inclusion of non-Hispanic youths.
This study was designed to address important research and policy questions about the effectiveness of an evidence-based program taken to scale and replicated with different populations and in different settings. This article provides important information on the early effectiveness of ¡Cuídate! However, the findings after 6 months are not intended to provide comprehensive evidence about behavioral outcomes. A final assessment of the program’s effectiveness will incorporate the findings from this study with the findings from the 18-month follow-up survey. However, these findings suggest that, after 6 months, positive program effects on some of the intermediate outcomes hypothesized to influence sexual behaviors did not translate into overall improvements in the sexual behavior of youths. CONTRIBUTORS M. Kelsey and J. Layzer conceptualized and designed the study, interpreted the data, and drafted and revised the content. C. Layzer and M. Blocklin drafted and revised the content. C. Price and R. Juras conceptualized and designed the study and analyzed and interpreted the data. J. Mendez drafted and revised the content.
ACKNOWLEDGMENTS This study was conducted by Abt Associates as part of the Teen Pregnancy Prevention Replication Study, a national replication study of 3 evidence-based teen pregnancy prevention reduction strategies funded by OAH and the Office of the Assistant Secretary for Planning and Evaluation in the US Department of Health and Human Services (HHS; contract no. HHSP23320095624WC, order no. HHSP23337011T, awarded in September 2011). This article is based on a larger report located on the HHS Web site at https://aspe.hhs.gov/teen-pregnancyprevention-tpp-replication-study. We thank each of the participating grantees for their guidance and assistance. Note. The views expressed in this report are those of the authors and do not necessarily represent the policies of HHS or the OAH.
HUMAN PARTICIPANT PROTECTION The study was approved by the institutional review board at Abt Associates for all sites and by additional local institutional review boards associated with the study sites as needed.
REFERENCES 1. Goesling B, Colman C, Trenholm C, Terzian M, Moore K. Programs to reduce teen pregnancy, sexually transmitted infections, and associated risk behaviors: a systematic review. J Adolesc Health. 2014;54(5):499–507. 2. Office of Adolescent Health, Office of Public Health and Science. Teenage Pregnancy Prevention: Replication of Evidence-Based Programs. Funding Opportunity Announcement and Application Instructions. Washington, DC: US Department of Health and Human Services; 2010. 3. Villarruel AM, Jemmott JB, Jemmott LS. A randomized controlled trial testing an HIV prevention intervention for
AJPH
Supplement 1, 2016, Vol 106, No. S1
AJPH RESEARCH
Latino youth. Arch Pediatr Adolesc Med. 2006;160(8): 772–777. 4. Kost K, Henshaw S. US teenage pregnancies, births and abortions, 2010: national and state trends by age, race, and ethnicity. Available at: http://www.guttmacher.org/ pubs/USTPtrends10.pdf. Accessed November 5, 2015. 5. Centers for Disease Control and Prevention. HIV Surveillance Report, 2003. Vol 15. Atlanta, GA: Centers for Disease Control and Prevention; 2004. 6. Jemmott LS, Jemmott JB III, McCaffree K. Be Proud! Be Responsible! Strategies to Empower Youth to Reduce Their Risk for AIDS. New York, NY: Select Media; 1995. 7. Kelsey M, Layzer J. Implementing three evidencebased program models: early lessons from the Teen Pregnancy Prevention Replication Study. J Adolesc Health. 2014;54(3, suppl):S45–S52. 8. Orr LL. Social Experiments: Evaluating Public Programs With Experimental Methods. Thousand Oaks, CA: Sage; 1999. 9. Puma MJ, Olsen RB, Bell SH, Price C. What to Do When Data Are Missing in Group Randomized Controlled Trials. Report No. NCEE 2009-0049. Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, US Department of Education; 2009. 10. What Works Clearinghouse. WWC Procedures and Standards Handbook Version 3.0. Washington, DC: What Works Clearinghouse, Institute of Education Sciences, US Department of Education; 2014. Available at: http:// ies.ed.gov/ncee/wwc/pdf/reference_resources/ wwc_procedures_v3_0_standards_handbook.pdf. Accessed March 7, 2016. 11. Schochet PZ. Technical Methods Report: Guidelines for Multiple Testing in Impact Evaluations. Report No. NCEE 2008-4018. Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, US Department of Education; 2008. 12. Layzer J, Layzer C, Kelsey M. Teen Pregnancy Prevention Replication Study: ¡Cuídate! Implementation Report. US Department of Health and Human Services, Office of Adolescent Health, Office of the Assistant Secretary for Planning and Evaluation. Available at: http://www.hhs. gov/ash/oah/oah-initiatives/evaluation/grantee-ledevaluation/grantees-2010-2014.html. Accessed September 30, 2016. 13. Kelsey M, Layzer J, Juras R. Teen Pregnancy Prevention Replication Study: Impact Study Design Report. US Department of Health and Human Services, Office of Adolescent Health, Office of the Assistant Secretary for Planning and Evaluation; 2012.
Supplement 1, 2016, Vol 106, No. S1
AJPH
Kelsey et al.
Peer Reviewed
Research
S77
Replicating ¡Cuídate!: 6-Month Impact Findings of a Randomized Controlled Trial Meredith Kelsey, PhD, Carolyn Layzer, PhD, Jean Layzer, MEd, Cristofer Price, ScM, Randall Juras, PhD, Michelle Blocklin, PhD, and Jacqueline Mendez, MA Objectives. To test whether ¡Cuídate!, a program culturally adapted for Hispanic youths, affects sexual risk behavior. Methods. We evaluated 3 replications of ¡Cuídate! in California, Arizona, and Massachusetts in a randomized controlled trial (registry no. NCT02540304) in which 2169 primarily Hispanic participants were randomly assigned to an intervention (n = 1326) or a control (n = 870) group.Youths were surveyed at baseline (September 2012–April 2014) and 6 months postbaseline (March 2013–October 2014). We estimated pooled and subgroup impacts using a regression framework with baseline covariates to increase statistical precision (1216 youths analyzed in the treatment group, 806 analyzed in the control group). Results. We found no impacts on the study’s primary outcomes of recent sexual activity or recent unprotected sexual activity. However, ¡Cuídate! improved knowledge (10%–20% increase; P < .001), attitudes (effect size = .24; P < .001), and skills (effect size = .14; P = .002). Exploratory subgroup analyses suggest potentially problematic effects for some groups. Conclusions. Findings suggest that ¡Cuídate! was effective in improving youths’ knowledge and attitudes. However, after 6 months, these changes did not translate to improvements in reported sexual risk behaviors. (Am J Public Health. 2016;106:S70–S77. doi:10.2105/AJPH.2016.303371) See editorials, p. S5–S31.
R
educing unplanned adolescent pregnancy is a priority for the US Department of Health and Human Services. The federal Teen Pregnancy Prevention Program, administered by the Office of Adolescent Health (OAH), funds programs that address the high rate of adolescent pregnancy by replicating specific evidence-based programs. The evidence for many of these programs rests on a single study conducted more than a decade ago.1 There is also interest in examining these programs beyond their original setting or target population. This study is one of several currently underway that will provide updated evidence on these programs. This study examines the effect of the ¡Cuídate! program model on adolescent sexual behavior and on intermediate outcomes such as knowledge, attitudes, motivation, intentions, and skills. Developed more than a decade ago in response to concerns about disparities in the rates
S70
Research
Peer Reviewed
Kelsey et al.
of pregnancy and HIV among Hispanic youths, ¡Cuídate! is among a handful of evidence-based programs that grantees can use to address the issue of sexual risk behavior among Hispanic adolescents.2 Aside from the single research study of ¡Cuídate! cited in the OAH evidence review,3 few studies of pregnancy prevention have focused specifically on Hispanic youths, although some have included a substantial number of Hispanic adolescents. Although pregnancy rates among Hispanic adolescents have declined at nearly the same rate as in other groups, the rate remains more than double that for non-Hispanic White adolescents.4 Reasons suggested for the
disparity are that Hispanic adolescents are less likely to use condoms than are White and Black youths and are more likely to report multiple sexual partners.5 ¡Cuídate! (which means “take care of yourself ”) was adapted from Be Proud, Be Responsible!,6 an HIV reduction program shown to be effective with Black youths, and was originally tested by the developer among mostly Puerto Rican youths in Philadelphia, Pennsylvania. ¡Cuídate! is culturally tailored for Hispanic youths; materials used in the sessions emphasize core Hispanic values such as traditional gender roles and the importance of family and link them to safer sexual behavior. Six 60-minute curriculum modules are delivered in participatory, interactive sessions that weave in the theme of taking care of oneself and one’s partner, family, and community. The modules are delivered by a trained adult facilitator, bilingual in English and Spanish, to small groups of 6 to 10 youths. ¡Cuídate! was originally tested in an after-school setting on consecutive weekends, but it can be delivered in other settings and on different schedules. ¡Cuídate! is intended to provide information and affect potential mediating factors such as knowledge and understanding of reproductive health and avoidance of sexual risk, attitudes toward using protection, motivation to delay pregnancy, intentions to become sexually active and use protection, and skills needed to avoid sexual risk. These factors are thought to lead to safer sexual behavior such as increased condom use and abstinence, as well as decreased sexual activity and number of partners. The initial test of ¡Cuídate! found that youths who
ABOUT THE AUTHORS Meredith Kelsey, Carolyn Layzer, Cristofer Price, Randall Juras, Michelle Blocklin and Jacqueline Mendez are with Abt Associates, Inc., Cambridge, MA. Jean Layzer is with Belmont Research Associates, Belmont, MA. Correspondence should be sent to Meredith Kelsey, PhD, 55 Wheeler Street, Cambridge, MA 02138 (e-mail: Meredith_Kelsey@ abtassoc.com). Reprints can be ordered at http://www.ajph.org by clicking the “Reprints” link. This article was accepted July 4, 2016. doi: 10.2105/AJPH.2016.303371
AJPH
Supplement 1, 2016, Vol 106, No. S1
AJPH RESEARCH
Inputs
Trained facilitator
6-unit curriculum (flexible delivery schedule)
Process
Intermediate Outcomes
Incorporate theme of taking care—of self, partner, family, community throughout program sessions
Improved knowledge and understanding of how to avoid unsafe sexual behaviors
Active participation in discussion, sharing ideas, feelings
Deliver curriculum modules in participatory, interactive sessions
Use materials to show and emphasize core Latino values and link to safer sexual behavior
Culturally-and linguistically appropriate materials
Demonstrate condom use and teach negotiation/refusal skills
Small-group sessions
Attitudes, values, and beliefs supportive of abstinence and avoidance of unprotected sex Increased motivation to delay childbearing
Role-play, practice to improve skills
Intentions to abstain from sexual activity and to use protection when sexually active
Improved negotiation/refusal/ condom use skills
Outcomes
Safer sexual behavior Consistent effective condom use Abstinence Reduction in sexual activity Reduction in no. of sexual partners
Reduction in STIs Reduction in unplanned pregnancies Reduction in adolescent births
Note. STI = sexually transmitted infection.
FIGURE 1—¡Cuídate! logic model
completed the program reported fewer incidents of sexual intercourse, fewer sex partners, and fewer days of unprotected intercourse than youths assigned to a health promotion program.2 These safer sexual behaviors are ultimately expected to result in a reduction in the rates of sexually transmitted infections (STIs) and unwanted pregnancies and births among adolescents. Figure 1 shows the program elements, the hypothesized outcomes, and the pathways by which the program seeks to achieve these outcomes. This study focused on program effects after 6 months and was guided by the following research questions: 1. Did ¡Cuídate! improve adolescents’ knowledge and understanding of pregnancy risks and prevention and the transmission and prevention of STIs? 2. Did ¡Cuídate! have positive effects on adolescents’ attitudes toward sexual activity, birth control, and condom use?
Supplement 1, 2016, Vol 106, No. S1
AJPH
3. Did ¡Cuídate! increase adolescents’ motivation to delay childbearing? 4. Did ¡Cuídate! increase adolescents’ intentions to avoid risky sexual behavior? 5. Did ¡Cuídate! increase adolescents’ confidence in their ability to refuse unwanted sex and to negotiate safe sex? 6. Did ¡Cuídate! lead adolescents to delay sexual initiation and reduce risky sexual behaviors?
METHODS We used an experimental design in which students were randomly assigned to a group that received the ¡Cuídate! intervention or to a control group that did not. The clinical trial registry number is NCT02540304. From among the Teen Pregnancy Prevention Program grants awarded in 2010, we selected 3 grantees that we felt could provide the strongest test of ¡Cuídate!—1 in a small city
in Southern California; 1 in Phoenix, Arizona; and 1 in Boston, Massachusetts. By examining all 3 sites together, we were able to assess the program’s impact in more diverse settings and with a more diverse population than is typically available at a single replication site. Close oversight by OAH ensured consistency of implementation and fidelity to the original program model across the 3 sites. Minor variations approved in advance by OAH were permitted.7 All 3 replication sites added a brief review of reproductive anatomy. The California site added 2 sessions to comply with state requirements governing sex education—1 on contraception and a second on STIs other than HIV. The Phoenix site added a session on pregnancy prevention, and it also received approval from OAH to deliver the program in larger groups of 20 students with 2 health educators rather than 1. These modifications were not considered changes to the core elements of the model.
Kelsey et al.
Peer Reviewed
Research
S71
AJPH RESEARCH
Setting Settings varied across replication sites. In California, the program was implemented with 10th graders in 3 public high schools during the regular school day while control group youths remained in their regular physical education class. In Phoenix, it was implemented with eighth graders in singlegender classes in 11 public kindergarten– eighth-grade schools during the regular school day while control group youths received a curriculum focused on health and well-being. In Boston, the program was delivered to youths aged 13 to 19 years in a range of settings, including public high schools (traditional, vocational–technical, and charter; during and after school), a summer youth employment program, and a summer youth sports program, while control group youths received the usual programming—regular physical education, regular health class, or other regular activities.
Sample Beginning in September 2012, students at each replication site were recruited for the program and the study. School staff identified classes or time slots for the program (small groups of students would be pulled out for the program). Program staff distributed consent forms and study brochures and provided small incentives for the return of consent forms. Students with parental permission were included in the random assignment conducted by the independent evaluator. At each replication site, students were stratified by gender and randomly assigned within class period within school. At the 2 replication sites that offered ¡Cuídate! in small groups in schools or in other settings, individual students were randomly assigned using a 2:1 ratio (i.e., for every 2 students assigned to the treatment group, 1 was assigned to the control group). At the 3rd replication site, where the program was delivered to single-gender groups, random assignment was done by gender using a 1:1 ratio. Students completed the baseline survey and were subsequently told their assignment. In total, 2198 eligible students obtained permission and were randomly assigned (1328 to the treatment group and 870 to the control group). Of these, 2022 completed the 6-month survey (1216 in the treatment group and 806 in the control group) and are
S72
Research
Peer Reviewed
Kelsey et al.
included in our analytic sample (Figure A, available as a supplement to the online version of this article at http://www.ajph.org).
Data Collection Youths completed a 30-minute survey at baseline (before the intervention began) and 6 months after baseline to assess the shortterm impacts of ¡Cuídate!. (Study participants were also surveyed 18 months after baseline; these analyses will be used to assess the longer term effectiveness of ¡Cuídate!.) At all time points, a Web-based Audio Computer-Assisted Self-Interview system was used to capture and store survey responses. The baseline survey was completed in group sessions in schools or other settings, on school computers when possible, or on tablets provided by field staff. For the 6-month survey, youths could access and complete the survey using personal tablets or computers, school or library computers, or even their smartphones. Participants received a gift card on completion of each survey. We collected data on implementation through interviews, focus groups, and site visits. We obtained data on fidelity and attendance from the grantees, who were required to collect and report performance measures to the funder. In addition, supervisory staff were required to observe, record, and report on the quality of program delivery for a sample of sessions.
Measures The surveys collected information from students on a variety of topics, including questions that allowed us to measure 2 sets of outcomes: (1) intermediate outcomes hypothesized to lead to behavioral outcomes and (2) sexual activity and risk behaviors. We briefly describe these measures here. Details on their construction are presented in Appendix A (available as a supplement to the online version of this article at http://www.ajph.org). Intermediate outcomes. We constructed composite measures within 4 domains to assess intermediate outcomes: (1) knowledge of pregnancy risk and STI risk, (2) attitudes toward use of condoms and other birth control methods (a = .79) and attitudes that reflect social norms about extreme sexual risk behaviors (a = .81), (3) motivation to delay childbearing (a = .87), and (4) condom
negotiation (a = .83) and refusal (a = .87) skills. In a fifth domain, we analyzed 4 items that measured intentions to engage in sexual intercourse in the next year and to use protection when sexually active. Behavioral outcomes. To address the study’s most important question about the impact of the intervention on sexual activity, we identified 9 measures in the domain of youth sexual behavior: ever sexually active, sexually active in the past 90 days, sexual intercourse in the past 90 days, oral sex in the past 90 days, anal sex in the past 90 days, sexual intercourse without birth control in the past 90 days, sexual intercourse without a condom in the past 90 days, oral sex without a condom in the past 90 days, and anal sex without a condom in the past 90 days.
Baseline behavioral and demographic characteristics. Participant characteristics were measured via baseline survey items and included race/ethnicity, age, grade, living with a biological parent, ever sexually active, ever smoked a cigarette, ever had an alcoholic drink, ever used marijuana, pregnancy and STI risk knowledge, and intentions to have oral sex and sexual intercourse. Respondents were asked to classify their own race/ethnicity by responding to 2 questions: 1. Are you Hispanic/Latino? (yes–no) 2. What is your race? You may select more than one answer (American Indian or Alaska Native; Asian; Black or African American; Native Hawaiian or other Pacific Islander; White; some other race [please specify]). On the basis of these responses, youths were classified as Hispanic, non-Hispanic White, non-Hispanic Black, or non-Hispanic other race (i.e., Asian, American Indian or Alaska Native, Native Hawaiian or other Pacific Islander, multiracial, and undisclosed race).
Analytic Approach To test the impact of ¡Cuídate! after 6 months on each of the study’s outcomes, we compared the outcomes of treatment and control group participants using a regression framework; for continuous outcomes, these were linear regression models; for dichotomous outcomes, these were linear probability models. We included the baseline
AJPH
Supplement 1, 2016, Vol 106, No. S1
AJPH RESEARCH
characteristics described earlier and baseline measures of the outcome as covariates (when available) to increase statistical precision of the impact estimates for a given sample size8 and reduce attrition bias from missing data.9 Randomization blocking variables were also included. To maintain the integrity of the random assignment design, we conducted an intent-to-treat analysis, using assignment status as the key variable of interest, regardless of actual program participation. In other words, all participants who completed the 6-month follow-up survey were included in the analysis with the assignment they were given at the time of randomization (i.e., treatment or control), even if youths assigned to treatment did not show up for the program or youths assigned to the control condition attended some program sessions. Data for the impact estimates were pooled across the 3 replication sites for the main analysis. This study had low overall attrition (8.0% lost to follow-up), and differential attrition across the treatment and control groups was also quite low (1.0%), minimizing the potential for attrition bias.10 To address missing data, we used case deletion for the few instances of missing outcome data and dummy variable adjustment to account for missing covariates (i.e., missing values were recoded to a constant, and dummy variables were included to indicate whether the value was originally missing).9 To mitigate multiple comparison problems (i.e., reduce the chance of spurious findings from a large number of tests), before analyzing data we prespecified 2 confirmatory outcomes in our analysis plan as the main indicators of program effectiveness: sexual activity in the past 90 days and sexual intercourse without birth control in the past 90 days. To control for having 2 confirmatory outcomes, we applied a formal BenjaminiHochberg correction, which controls for the false positive rate by adjusting P-value thresholds.10 We considered all other outcomes to be exploratory; thus, they did not warrant a multiple comparisons adjustment.11 In addition to the overall pooled impacts, we conducted exploratory subgroup analyses for behavioral outcomes to gain some insight into what worked for whom by including subgroup indicators and treatment · subgroup interaction terms in the model and testing for the significance of the interaction term. Subgroups tested included replication
Supplement 1, 2016, Vol 106, No. S1
AJPH
site, gender, age, race/ethnicity, and sexual experience at baseline. In evaluating the effectiveness of ¡Cuídate!, an intervention specifically tailored to Hispanic youths, it was particularly important to examine whether the intervention’s effects differed for youths by reported race/ethnicity. Only when there was a significant difference in impacts between subgroups did we present impact estimates for individual subgroups, to guard against overinterpretation of potentially spurious findings by reducing the number of results presented in tables. Because subgroup analyses are exploratory, we did not adjust for multiple comparisons.
RESULTS Table 1 shows the distribution of the analytic sample on key measures at baseline. At baseline, youths in the study sample were aged 14.4 years, on average. However, average age varied across the replication sites; where the program was implemented only in eighth-grade classrooms, the average age of students was only 13 years. Adolescent girls constituted more than half of the sample. More than 70% were Hispanic, 18% were White, and the remaining 10% were divided between Black and other race. More than one fifth of the sample had engaged in sexual intercourse before the study began. About half had ever used alcohol, one fifth had smoked cigarettes, and a quarter had used marijuana. Baseline equivalence was assessed using a series of models with the same structural components as the impact models. At baseline, very few significant differences were found between the treatment and control groups (Table 1). The treatment group was at a slightly higher grade level, was less likely to be Black, was more likely to live with their biological parents, and had higher intention to have sexual intercourse in the next 12 months. These 4 variables were subsequently included in all impact models as covariates.
Program Implementation ¡Cuídate! was well implemented across all 3 replication sites. The 3 grantees hired staff with appropriate background experience and skills to deliver the program; all received training provided by the curriculum
distributor. Each of the grantees successfully delivered the program with fidelity (adherence to its core elements and without modifications that threatened those core elements). At all 3 replication sites, a majority of students received at least 75% of the sessions offered. More information on program implementation will be available in a forthcoming implementation report.12
Program Impacts on Intermediate Outcomes After 6 months, ¡Cuídate! had a positive impact on youths’ knowledge, attitudes, and skills (Table 2). No program effects were found on motivation or on intentions to engage in sexual behaviors in the following year. Compared with control group students, treatment group students scored 6 percentage points higher on the composite measure of knowledge of pregnancy risk; this difference was statistically significant. ¡Cuídate! also had a large and statistically significant positive impact on a composite measure of knowledge of STI risk. Treatment students scored 11% points higher, on average, than control students. ¡Cuídate! had a small to moderate (standardized effect size of the difference = .24) but statistically significant impact on students’ attitudes toward using protection. For the composite measure of attitudes toward protection (encompassing both birth control and condoms), students in the treatment group had more positive (and protective) attitudes. ¡Cuídate! had no statistically significant impact on student attitudes toward risky behavior. On measures of student views on perceived social norms about risky behavior, students in both the treatment and the control groups typically rejected the view that risky behavior was acceptable. Students in both the treatment and the control groups were also highly motivated to delay childbearing: Participants across both groups indicated a belief in the importance of delaying childbearing until personal goals have been achieved. ¡Cuídate! did not affect student intentions to engage in sexual activity or perceived refusal skills, but the program had a statistically significant impact on perceived condom negotiation skills. Program participants were more confident that they could successfully
Kelsey et al.
Peer Reviewed
Research
S73
AJPH RESEARCH
TABLE 1—Baseline Characteristics of the Analytic Sample: ¡Cuídate!; Southern California; Phoenix, AZ; and Boston, MA; September 2012–April 2014 Rangea
No.b
Treatment Meanc
Control Mean
Group Differenced
P
Age
11–20
2022
14.39
14.37
0.02
.63
Grade
6–13
2022
9.19
9.13
0.05
2022
52.98
52.98
0.00
‡ .99
Hispanic
2022
71.40
70.10
1.30
.51
Black
2022
3.73
5.83
–2.10
.027
White
2022
17.96
18.49
–0.53
.75
Other
2022
6.91
5.58
1.32
.23
Lives with biological parents
1945
94.30
90.71
3.60
.003
Feels very close to and cared for by father
1800
44.45
46.06
–1.61
.50
Feels very close to and cared for by mother
1930
59.13
60.29
–1.16
.61
Ever smoked cigarettes
1975
18.64
18.41
0.23
.90
Ever drank alcohol
1976
46.06
48.08
–2.03
.36
Ever used marijuana
1974
25.32
25.90
–0.58
.76 .83
Outcome Demographic characteristics
Female, %e
.015
Race/ethnicity, %e,f
e
Family structure and relationships, %
Risk behavior, %e
Knowledge
g
Knowledge of pregnancy risk
0–100
1994
48.37
47.98
0.39
Knowledge of STI risk
0–100
1995
38.61
39.36
–0.75
.56
1–4
1988
3.07
3.06
0.00
.83
Intentions to have sexual intercourse in the next 12 mo
1946
31.91
27.50
4.41
.023
Intentions to have oral sex in the next 12 mo
1939
25.10
22.76
2.34
.21
Intentions to use a condom if they were to have sexual intercourse
1944
92.99
94.72
–1.73
.13
Intentions to use birth control if they were to have sexual intercourse
1923
92.07
91.99
0.08
.95
Ever sexually activei
1969
25.22
21.99
3.23
.07
Currently sexually active in the past 90 di
1959
17.38
14.69
2.69
.10
Sexual intercourse in the past 90 d
1962
14.49
12.37
2.12
.16
Oral sex in the past 90 d
1959
12.66
10.34
2.32
.11
Anal sex in the past 90 dg
1143
4.25
3.42
0.84
.48
Sexual intercourse without birth control in the past 90 d
1962
4.38
3.48
0.90
.33
Sexual intercourse without a condom in the past 90 d
1962
8.33
6.44
1.89
.11
Oral sex without a condom in the past 90 d
1959
11.31
9.17
2.14
.12
Anal sex without a condom in the past 90 di
1143
2.77
2.05
0.72
.46
Attitudes toward protectionh Intentions, %e
Sexual behavior, %e
Sexual risk, %e
Note. STI = sexually transmitted infection. The baseline treatment–control difference was estimated by using the baseline measure as the dependent variable and the treatment group indicator and the terms for the randomization blocks as independent variables. a For continuous variables, we present the range. All other variables are dichotomous. b Sample sizes vary slightly across outcomes because of missing outcome data. We report the sample sizes for each outcome. c The treatment mean was calculated as the sum of the control group mean and the model estimated treatment–control difference (group difference). d The group difference is the treatment–control difference. For outcomes reported as percentages, the group difference is expressed in percentage points. For scale outcomes, the group difference is expressed in the original metric of the outcome variable. Because of rounding, reported group differences may differ from differences between reported means for the treatment and control groups. e For dichotomous variables, we present the percentage of respondents who responded affirmatively. f Racial/ethnic categories include Hispanic, Black non-Hispanic, White non-Hispanic, and other race non-Hispanic, where “other” is defined as Asian, American Indian or Alaska Native, Native Hawaiian or other Pacific Islander, multiracial, or undisclosed. g Knowledge variables are composite scale scores representing the percentage of items answered correctly. h Attitude variable is a composite scale score with higher scores indicating more positive attitudes. i Sexual activity was defined differently across grantees. At 2 sites, sexual activity referred to sexual intercourse, oral sex, and anal sex. Youths were not asked about anal sex at 1 site.
S74
Research
Peer Reviewed
Kelsey et al.
AJPH
Supplement 1, 2016, Vol 106, No. S1
AJPH RESEARCH
TABLE 2—Six-Month Impacts of ¡Cuídate! on Intermediate Outcomes: Southern California; Phoenix, AZ; and Boston, MA; March 2013–October 2014 Rangea
Outcome
Nb
Treatment Meanc
Control Mean
Group Differenced
SESe
P
Knowledgef Knowledge of pregnancy risk
0–100
2022
67.07
60.95
6.12
< .001
Knowledge of STI risk
0–100
2022
63.67
53.01
10.66
< .001
f
Attitudes
1–4
2022
3.24
3.14
0.10
0–100
2011
3.12
3.33
–0.21
1–4
2015
3.69
3.69
0.00
Sexual intercourse
2003
40.38
39.07
1.31
Oral sex
1997
37.16
36.60
0.56
.76
Use a condom if they were to have sexual intercourse
2005
92.89
92.74
0.15
.90
Use birth control if they were to have sexual intercourse
1996
93.23
92.42
0.80
.49
f
Motivation to delay childbearing
.24
< .001
Attitudes toward protection Support for risky behavior
.69 –.01
.91
Intentions to engage in the following behaviors in the next 12 mo, %g
Skills
.47
f
Perceived refusal skills
1–4
2015
3.19
3.13
0.06
.08
.06
Perceived condom negotiation skills
1–4
2016
3.53
3.46
0.07
.14
.002
Note. SES = standardized effect size of the difference; STI = sexually transmitted disease. a For continuous variables, we present the range. All other variables are dichotomous. b Sample sizes vary slightly across outcomes because of missing outcome data. We report the sample sizes for each outcome. c The treatment group mean is regression adjusted, calculated as the sum of the control group mean and the regression-adjusted impact estimate (group difference). d The group difference is the treatment–control difference. For outcomes reported as percentages, the group difference is expressed in percentage points. For scale outcomes, the group difference is expressed in the original metric of the outcome variable. Because of rounding, reported group differences may differ from differences between reported means for the treatment and control groups. e For outcomes that are not dichotomous or measured on a 0–100 scale, the SES is the group difference divided by the pooled standard deviation of the treatment and control groups. f Composite scale scores. g Dichotomous variables, reported as percentage of respondents who responded affirmatively.
negotiate condom use with a partner (standardized effect size of the difference = 0.14) than were those in the control group.
Program Impacts on Behavior Despite program impacts on youths’ knowledge, attitudes, and skills, ¡Cuídate! had no statistically significant impact on the confirmatory behavioral outcome measures (i.e., sexually active in the past 90 days and sexual intercourse without birth control in the past 90 days; Table 3). Consistent with these findings, we also found no statistically significant impacts on other related sexual behaviors. Although behavioral impact estimates were not statistically significant for the full sample, we found some variation by subgroup, and for some subgroups, an adverse program effect was observed (Table A, available as a supplement to the online version of this article at http://www.ajph.org).
Supplement 1, 2016, Vol 106, No. S1
AJPH
We found variation in program impact on sexual intercourse in the past 90 days, depending on sexual experience at baseline. A significant adverse program effect was observed for participants who had been sexually active at baseline, with treatment group members significantly more likely (> 7 percentage points; P = .010) to report having had recent sexual intercourse than their control group counterparts. We also found variation in program impacts on oral sex and oral sex without a condom in the past 90 days by respondent race/ethnicity. Significant adverse program effects were observed for White treatment group members, who were about 9 percentage points more likely to report having had oral sex (P = .005) and oral sex without a condom (P = .006) in the past 90 days than their control group counterparts. We found no significant effects for Hispanic, Black, or other-race participants. No variation in impacts on sexual behavior by site or other subgroup emerged.
DISCUSSION Findings on the effectiveness of ¡Cuídate! after 6 months were mixed. The program achieved impacts on some exploratory intermediate outcomes, such as knowledge, attitudes toward protection, and condom negotiation skills, suggesting that health educators were effective in communicating program content. Nevertheless, these early results provide no evidence that ¡Cuídate! favorably affected either of the study’s 2 confirmatory behavioral outcomes or any other sexual risk behaviors. The program significantly increased participants’ knowledge and understanding of sexual risk behavior and ways to prevent outcomes such as unplanned pregnancy and STIs. Participants’ attitudes toward use of birth control and condoms were also significantly more positive as a result of the program. Program participants were also significantly more confident in their ability to
Kelsey et al.
Peer Reviewed
Research
S75
AJPH RESEARCH
TABLE 3—Six-Month Impacts of ¡Cuídate! on Sexual Behavior: Southern California; Phoenix, AZ; and Boston, MA; March 2013–October 2014 Outcome
No.a
Treatment, %b
Control, %
Group Differencec
P
Sexual behavior Ever sexually actived
.29
2012
29.31
27.68
1.63
Currently sexually active in the past 90 dd,e
2011
18.79
17.83
0.96
.52
Sexual intercourse in the past 90 d
2012
15.48
14.09
1.39
.31
Oral sex in the past 90 d
2009
14.69
13.13
1.56
.27
Anal sex in the past 90 dd
1173
2.48
2.87
–0.39
.70
Sexual intercourse without birth control in
2012
5.77
4.86
0.90
.38
the past 90 de Sexual intercourse without a condom in
2012
9.81
8.10
1.70
.16
Oral sex without a condom in the past 90 d
2009
1.46
1.99
–0.53
.21
Anal sex without a condom in the past 90 df
1173
5.77
4.86
0.90
.53
Sexual risk
the past 90 d
Note. All outcomes are dichotomous. a Sample sizes vary slightly across outcomes because of missing outcome data. We report the sample sizes for each outcome. b The treatment group percentage is regression adjusted, calculated as the sum of the control group percentage and the regression-adjusted impact estimate (group difference). c The group difference is the treatment–control difference expressed in percentage points. Because of rounding, reported group differences may differ from differences between reported percentages for the treatment and control groups. d Sexual activity was defined differently across grantees. At 2 sites, sexual activity referred to sexual intercourse, oral sex, or anal sex. Youths were not asked about anal sex at 1 site. e For the 2 confirmatory outcomes, statistical significance at P < .05, P < .01, and P < .001 implies statistical significance at those levels after applying a Benjamini-Hochberg adjustment for multiple comparisons. f Items asking about anal sex were not included in the survey administered to participants in 1 site.
negotiate the use of condoms. However, the program had no significant effects on motivation to delay pregnancy or intentions with respect to future sexual behavior. The primary goal of this and other adolescent pregnancy prevention programs is to change behaviors. The ¡Cuídate! logic model posits that changes in knowledge, attitudes, motivation, intentions, and skills will lead to changes in behavior. After 6 months, however, ¡Cuídate! had no favorable impacts on any behavioral outcomes, although the study was well powered to detect such impacts.13 Standard errors allow us to confidently rule out positive impacts substantially larger than 0.1 standard deviation for behavioral outcomes and for intermediate outcomes that were found to be statistically insignificant. To the contrary, findings suggest the program may have had adverse impacts on some behavioral outcomes for certain subgroups. Six months may not have been long enough for impacts on intermediate outcomes to
S76
Research
Peer Reviewed
Kelsey et al.
translate to beneficial behavioral changes. To better understand the lack of favorable behavioral changes, it will be important to see whether patterns seen after 6 months change or persist over time. The program is specifically designed for Hispanic youths, broadly defined, although the developer maintains that its messages are appropriate for youths of all ethnicities, because it is solidly based on another evidence-based program that is intended for a broader population. All 3 replication sites successfully recruited Hispanic youths, although they also served non-Hispanic youths. Subgroup analyses indicated some differences between racial/ethnic groups, but impacts were not significant among Hispanic youths. Additional sensitivity analyses, in which we ran the full set of analyses on Hispanic youths only, indicated findings similar to those for the full sample. Thus, the lack of favorable behavioral impacts is unlikely to be a result of the inclusion of non-Hispanic youths.
This study was designed to address important research and policy questions about the effectiveness of an evidence-based program taken to scale and replicated with different populations and in different settings. This article provides important information on the early effectiveness of ¡Cuídate! However, the findings after 6 months are not intended to provide comprehensive evidence about behavioral outcomes. A final assessment of the program’s effectiveness will incorporate the findings from this study with the findings from the 18-month follow-up survey. However, these findings suggest that, after 6 months, positive program effects on some of the intermediate outcomes hypothesized to influence sexual behaviors did not translate into overall improvements in the sexual behavior of youths. CONTRIBUTORS M. Kelsey and J. Layzer conceptualized and designed the study, interpreted the data, and drafted and revised the content. C. Layzer and M. Blocklin drafted and revised the content. C. Price and R. Juras conceptualized and designed the study and analyzed and interpreted the data. J. Mendez drafted and revised the content.
ACKNOWLEDGMENTS This study was conducted by Abt Associates as part of the Teen Pregnancy Prevention Replication Study, a national replication study of 3 evidence-based teen pregnancy prevention reduction strategies funded by OAH and the Office of the Assistant Secretary for Planning and Evaluation in the US Department of Health and Human Services (HHS; contract no. HHSP23320095624WC, order no. HHSP23337011T, awarded in September 2011). This article is based on a larger report located on the HHS Web site at https://aspe.hhs.gov/teen-pregnancyprevention-tpp-replication-study. We thank each of the participating grantees for their guidance and assistance. Note. The views expressed in this report are those of the authors and do not necessarily represent the policies of HHS or the OAH.
HUMAN PARTICIPANT PROTECTION The study was approved by the institutional review board at Abt Associates for all sites and by additional local institutional review boards associated with the study sites as needed.
REFERENCES 1. Goesling B, Colman C, Trenholm C, Terzian M, Moore K. Programs to reduce teen pregnancy, sexually transmitted infections, and associated risk behaviors: a systematic review. J Adolesc Health. 2014;54(5):499–507. 2. Office of Adolescent Health, Office of Public Health and Science. Teenage Pregnancy Prevention: Replication of Evidence-Based Programs. Funding Opportunity Announcement and Application Instructions. Washington, DC: US Department of Health and Human Services; 2010. 3. Villarruel AM, Jemmott JB, Jemmott LS. A randomized controlled trial testing an HIV prevention intervention for
AJPH
Supplement 1, 2016, Vol 106, No. S1
AJPH RESEARCH
Latino youth. Arch Pediatr Adolesc Med. 2006;160(8): 772–777. 4. Kost K, Henshaw S. US teenage pregnancies, births and abortions, 2010: national and state trends by age, race, and ethnicity. Available at: http://www.guttmacher.org/ pubs/USTPtrends10.pdf. Accessed November 5, 2015. 5. Centers for Disease Control and Prevention. HIV Surveillance Report, 2003. Vol 15. Atlanta, GA: Centers for Disease Control and Prevention; 2004. 6. Jemmott LS, Jemmott JB III, McCaffree K. Be Proud! Be Responsible! Strategies to Empower Youth to Reduce Their Risk for AIDS. New York, NY: Select Media; 1995. 7. Kelsey M, Layzer J. Implementing three evidencebased program models: early lessons from the Teen Pregnancy Prevention Replication Study. J Adolesc Health. 2014;54(3, suppl):S45–S52. 8. Orr LL. Social Experiments: Evaluating Public Programs With Experimental Methods. Thousand Oaks, CA: Sage; 1999. 9. Puma MJ, Olsen RB, Bell SH, Price C. What to Do When Data Are Missing in Group Randomized Controlled Trials. Report No. NCEE 2009-0049. Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, US Department of Education; 2009. 10. What Works Clearinghouse. WWC Procedures and Standards Handbook Version 3.0. Washington, DC: What Works Clearinghouse, Institute of Education Sciences, US Department of Education; 2014. Available at: http:// ies.ed.gov/ncee/wwc/pdf/reference_resources/ wwc_procedures_v3_0_standards_handbook.pdf. Accessed March 7, 2016. 11. Schochet PZ. Technical Methods Report: Guidelines for Multiple Testing in Impact Evaluations. Report No. NCEE 2008-4018. Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, US Department of Education; 2008. 12. Layzer J, Layzer C, Kelsey M. Teen Pregnancy Prevention Replication Study: ¡Cuídate! Implementation Report. US Department of Health and Human Services, Office of Adolescent Health, Office of the Assistant Secretary for Planning and Evaluation. Available at: http://www.hhs. gov/ash/oah/oah-initiatives/evaluation/grantee-ledevaluation/grantees-2010-2014.html. Accessed September 30, 2016. 13. Kelsey M, Layzer J, Juras R. Teen Pregnancy Prevention Replication Study: Impact Study Design Report. US Department of Health and Human Services, Office of Adolescent Health, Office of the Assistant Secretary for Planning and Evaluation; 2012.
Supplement 1, 2016, Vol 106, No. S1
AJPH
Kelsey et al.
Peer Reviewed
Research
S77
