Journal of Evidence-Based Social Work, 11:423–436, 2014 Copyright © Taylor & Francis Group, LLC ISSN: 1543-3714 print/1543-3722 online DOI: 10.1080/15433714.2012.759465

Promoting Outcome Achievement in Child Welfare: Predictors of Evidence-Informed Practice Crystal Collins-Camargo School of Social Welfare, University of Louisville, Louisville, Kentucky, USA

Teri A. Garstka Center for Public Partnerships and Research, University of Kansas, Lawrence, Kansas, USA

The use of data and evidence to inform practice in child welfare is the subject of increased discussion in the literature as well as in agencies striving to achieve child safety, permanency, and well-being. Survey data was collected from workers and supervisors in private agencies providing out-of-home care case management and residential treatment services to children and youth across three states. Hierarchical linear modeling tested the role of goal-oriented teamwork and supervisory practice involving the use of data to assess practice effectiveness in predicting evidence-informed practice. The partially mediated relationship showed that a more goal-oriented approach combined with supervisory practice led to increased use of evidence-informed practice. Implications for promoting evidence-informed practice in child welfare are discussed. Keywords: Child welfare, data, outcomes, evidence-informed practice

Child welfare is a field in which it is of critical importance to assess evidence regarding the effectiveness of our practice with children and families. The safety and well-being of the children served can depend on it. Child welfare agencies are increasingly more focused on accountability and improving performance (Lindsey & Schlonsky, 2008; Weigensberg, 2009). Unfortunately, the field’s evidence base, particularly regarding effective practice techniques in core child welfare services, is at an early stage of development, although some treatment services, such as The Incredible Years parent training and multi-systemic therapy, are empirically supported (Barth, 2008). Meanwhile, frontline workers struggle to assist families in coping with complex social problems without sufficient evidence on how to do so or how to adapt evidence-based techniques to their own local context (Mullen, Bledsoe, & Bellamy, 2008). So how should child welfare approach using evidence in practice improvement? Angel (2003) argued that the field of child welfare is too complex to benefit from an emphasis on only empirically proven programs. Practitioners in this field need to consider an array of evidence types in their work, including but not limited to empirical research findings from randomized control trials that some see as the gold standard to evidence-based practice (EBP; Hall, 2008). This should include a multidimensional approach, including quantitative and qualitative data, internally focused practice-based research in which agency data is used in the practice environment and a valueAddress correspondence to Crystal Collins-Camargo, School of Social Welfare, University of Louisville, 307 Patterson Hall, Louisville, KY 40292, USA. E-mail: [email protected]

423

424

C. COLLINS-CAMARGO AND T. A. GARSTKA

critical analysis to identify best practices as the agency promotes performance achievement (Petr, 2009). Epstein (2009) referred to this as evidence-informed practice (EIP), in which a broad and inclusive approach to is taken to use of relevant data in day-to-day practice improvement efforts. Fielding, Crawford, Leitmann, and Anderson (2009) emphasized the importance of reflective practice and knowledge development across a continuum of forms of evidence. The federal government has promoted accountability in mandating the Child and Family Service Review (CFSR) process, which measures the achievement of safety, permanency, and well-being of children along with systemic factors believed to be associated with such outcomes (U.S. Governmental Accounting Office, 2004). Recent findings of the CFSR 2007 and 2008 demonstrate that all states are struggling to meet established standards, and must implement complex program improvement plans designed to promote improvement (Administration for Children and Families, n.d.). Since fiscal sanctions can be imposed on states that must provide services in the best interest of children and families, child welfare agencies are desperate to find ways to improve performance. Identifying strategies are available to promote EIP is a critical first step. EIP in Child Welfare The organizational and learning conditions that promote the development of evidence-informed practice must be examined (Franklin & Hopson, 2007). Chaffin (2006) suggested that an evaluation climate must be created in child welfare in which data is used to determine service effectiveness. Testa and Poertner (2010) advocated a results-oriented approach to accountability that includes outcomes monitoring, data analysis, review of research, evaluation, and quality improvement in the child welfare setting. These activities are integral to adopting an EIP approach: How may agencies achieve this? Cunningham and Duffee (2009) urged a “developmental” style of EBP in child welfare that begins with the practitioner’s desire to facilitate beneficial outcomes for clients. This style also requires the agency to provide supports that enable staff to use data to inform their practice. This approach is in contrast to an “adoption” style of applying empirical research findings in the field, or an externally imposed “compliance” style reporting of outcomes. One place for agencies to start is to assess frontline attitudes toward the use of evidence in evaluating practice. Frontline staff play a critical role in achieving organizational goals (Gioia, 2007). Agencies would then be required to make data and tools available for use by staff. Barratt (2003) found resources to support EIP inadequate in an array of child and family-serving agencies studied. Schoech, Basham, and Fluke (2006) described a project making data available to the frontline through the management information system to promote evidence-informed decision-making in child welfare, but they did not document whether the intervention resulted in a positive shift in outcomes. Carrilio, Packard, and Clapp (2003) found agencies did not consistently use their information systems’ data, despite training and technical assistance to staff. As a result, they endorsed promoting an organizational culture that values the use of data in promoting quality services and measuring outcomes. Manuel, Mullen, Fang, Bellamy, and Bledsoe (2009) found that a comprehensive array of supports on both the individual and organizational level is needed to promote use of EIP in social service agencies. Additional organizational supports might include teaching staff to use data (Wulczyn, 2005), and analyze the relationship between practice indicators and outcomes, so they can adjust practice techniques used with clients (Cozens, 1999). Child welfare service providers interviewed in one study indicated that organizational support and an understanding of the relationship between process and outcomes are critical for EIP implementation (Aarons & Palinkas, 2007). Frontline supervisors are in a crucial role related to helping workers understand and apply these concepts and for creating a culture in which EIP is valued. Relatively little is known about the extent to which workers in either the public or private sector use data in their practice (Usher & Wildfire, 2003). One cross-sectional study examined public

PROMOTING OUTCOME ACHIEVEMENT IN CHILD WELFARE

425

and private child welfare agencies’ staff perceptions of use of data in informing their practice and attitudes toward and array of EIPs in one state (Collins-Camargo, Sullivan, & Murphy, 2011). That study found that private agency staff, in general, reported more frequent use of data and more positive impressions of EIPs than public staff. Recently, Aarons, Hurlburt, and Horowitz (2011) proposed a conceptual model for implementing EBPs in the public sector, focusing primarily on child welfare, which recognized both the outer (e.g., the inter-organizational environment and consumer support) and inner contexts (e.g., intra-organizational characteristics and individual adopters characteristics) of this work across the exploration, adoption/preparation, implementation, and sustainment phases; however, frontline supervision was not expressly discussed. This model provides a platform to begin examining some of the inner context factors associated with using EIP in child welfare.

Child Welfare Supervision and Team Dynamics Child welfare supervision research has grown over the last decade (e.g., Clark et al., 2008; Dill & Bogo, 2009; Giddings, Cleveland, Smith, Collins-Carmargo, & Russell, 2007; Leitz, 2008). Over time, the empirical literature has documented the influence of social work supervision on organizational, worker, and client outcomes on many levels. This includes a number of aspects of worker practice, such as assessment and treatment of families (Young, 1994), the development of analytic skills (Berkman & Press, 1993), and even client outcome achievement (Collins-Camargo & Royse, 2010). Given that supervisors and workers often form cohesive work groups in practice, it is logical that this work unit—a team of practitioners answering to one frontline supervisor— is an appropriate place to promote EIP. One study (Collins-Camargo & Millar, 2010) tested the implementation of clinical supervision in child welfare agencies in four states. Qualitative findings showed that an increased supervisory emphasis on EIP as supervisors promoted selfreflective practice and outcome achievement in the staff they supervised (Collins-Camargo, 2007; Collins-Camargo & Millar, 2010). Fixsen, Naoom, Blase, Friedman, and Wallace (2005) conducted an extensive review of the literature to identify those components that are necessary for successful implementation of EBPs. One of these core implementation components was supervisory coaching; research has demonstrated its significant influence on organizational culture. For example, one statewide study found that perceptions of effective frontline supervision in child welfare explained 53% of the variance in ratings of professional organizational culture promoting EBP (Collins-Camargo & Royse, 2010). While bureaucratic settings have been found to negatively impact workers’ attitudes about the use of evidence in informing their practice (Aarons, 2004), frontline supervisors have the opportunity to influence the interpretation of organizational culture, potentially moderating this effect. Work place climate influences openness to change and innovation that are certainly important to an EIP approach (Anderson & West, 1998; Birleson, 1999). The other aspect of the work unit that may be relevant to the promotion of EIP is the team dynamic. A team of workers operates within an organizational climate supported by its frontline supervisor. In a review of the literature on EBP, a team approach was found to help practitioners develop the ability to use evidence to inform practice (Lawler & Bilson, 2004). Facilitated by the frontline supervisor, the team can discuss the relationship between evidence and practice together (Austin & Claassen, 2008). It seems reasonable that if practitioners work on a team in which emphasis is placed on the achievement of goals and outcomes for clients, and team meetings, for example, are devoted to reviewing data and evaluating practice effectiveness, EIP is reinforced. This suggests that when members of a work unit are collaboratively engaged in improving their practice and achieving outcomes, individual use of EIP would be supported. However, the nature of this conceptual relationship is an empirical question for child welfare researchers.

426

C. COLLINS-CAMARGO AND T. A. GARSTKA

The Purpose of the Current Study The current study examines the interrelationships among factors in the inner context (Aarons et al., 2011), including organizational culture and climate; leadership; worker attitudes toward EBP, with particular focus on frontline supervision; and a team-based focus on outcome and goal achievement. If promoting EIP in this field is important, then researchers and clinicians need to know more about how best to facilitate its development. Specifically, we tested the hypothesis that the use of EIP was a function of organizational supervision and team goals that focuses on using data and outcome to inform workers on how well they are doing. Importantly, we wanted to study these factors in context; that is, to examine the nature of these relationships within existing child welfare agencies that are held accountable for their outcomes. This current study involved secondary data analysis of a larger, federally funded study in three states evaluating the implementation of performance-based contracting and quality assurance systems within the context of a public/private partnership when out-of-home case management is provided by private contractors. The public child welfare agency and relevant private partners developed and implemented new contracting systems that included incentives or disincentives related to performance achievement, and then built processes in which data was used collaboratively by the system to promote practice and outcome improvement. The details of the intervention in each site and cross-site evaluation findings are beyond the scope of this article and are reported elsewhere (see Collins-Camargo et al., 2007; Garstka, Collins-Carmargo, Hall, Neal, & Ensign, 2012). The implementation of the three demonstration projects was, in part, driven by those implementation components described by Fixsen and colleagues (2005), so the cross-site evaluation included measurement of the level of activity in which data are used to promote performance improvement on work units and in supervision. For the current study, secondary analysis of data collected in the larger study related to frontline use of data to improve practice was conducted.

METHODS Participants Participants for this study included frontline case managers and supervisors employed in private contracted child welfare service agencies in Florida, Missouri, and Illinois. Overall, a total of 597 participants were included in this study. In Florida, this included 44 private agency staff providing case management services in one central judicial circuit serving multiple counties. In Missouri, respondents included 266 staff from multiple private agencies serving 3 large urban regions in the state providing case management services. In Illinois, 302 staff respondents employed by private residential treatment providers across the state were surveyed. Minimal descriptive demographic information was obtained from all respondents who completed surveys in each site, such as staff type, agency type, and characteristics related to their employment. Table 1 presents sample sizes, employment information, and education level of frontline case managers and supervisors by site. In general, the majority of staff surveyed held a bachelor’s degree or higher, had been in their current position approximately 3 years, and had been in the field of child welfare an average of 7 years.

Measures For this study, the variables of interest included those factors that may contribute to an organizational culture that promotes the use of data and outcomes to improve practice. As such, three key measures were included within a larger survey administered as a part of the larger evaluation

PROMOTING OUTCOME ACHIEVEMENT IN CHILD WELFARE

427

TABLE 1 Participant Characteristics of Private Provider Frontline and Supervisory Staff in Florida, Missouri, and Illinois Site

Total number of supervisors Total number of frontline case workers Years of education % High school diploma % Associate degree % Bachelor’s degree % Master’s degree % Postdoctoral degree Mean years employed in position Mean years employed in child welfare field

Florida (n = 44)

Illinois (n = 302)

Missouri (n = 266)

11 33

62 240

72 194

0% 0% 90.9% 9.1% 0% 3.00 7.63

25.2% 14.6% 44.7% 14.9% 0.7% 3.53 7.56

0% 0% 63% 38% 0% 2.96 7.04

study. These three measures were designed to capture the extent to which frontline case managers and supervisors (a) form a team with a goal-oriented approach to improving outcomes; (b) use data and evidence during supervision; and (c) use data and evidence to improve practice. Survey Development A 30-item survey was developed to measure the use of data, outcomes, and EIP by frontline and supervisor staff. In addition to the three measures for this study, other items were included in this full survey to address different topic areas in child welfare practice not directly related to this study. Individual survey items and composites were derived from three main sources and current work in the field of child welfare (Institute for Behavioral Research, 2002; Organizational Excellence Group, 2001; Research in Practice, 2006). The survey itself contained a mix of items that were scaled on 5-point Likert ratings, categorical responses, and a several open-ended questions. The Appendix contains the measures and items included in this analysis. Team Goal-Orientation Five items drawn primarily from the Survey of Organizational Excellence (Organizational Excellence Group, 2001) were used to measure the extent to which a team uses a goal-oriented approach. On a 5-point Likert scale (1 D strongly disagree to 5 D strongly agree), these items included the following statements: “We seem to be working toward the same goals” and “Decision making and control are given to employees doing the actual work.” Reliability analysis showed this measure had high reliability (Cronbach’s ˛ D 0.82). A mean score for this measure was calculated for each participant with higher numbers indicating that participants believe they are part of a group that is goal-oriented and actively involved in making work processes more effective. Supervisory Practice This five item measure, drawn primarily from Firm Foundations: A Practical Guide for Organisational Support for the Use of Research Evidence (Research in Practice, 2006), was used to ascertain the frequency with which supervision of both frontline and supervisory staff included discussions of data, outcomes, and evidence to support practice. On a 5-point Likert scale (1 D

428

C. COLLINS-CAMARGO AND T. A. GARSTKA

never to 5 D very often), participants rated the frequency of the following items: “How often do you and your supervisor discuss what your team’s performance data tells you that may help you improve your practice with clients?” and “How often do you and your supervisor discuss what success will look like (i.e., what measureable outcomes are we seeking for a child/youth/family)?” This measure showed high reliability (Cronbach’s ˛ D 0.91). A mean score for this measure was calculated for each participant with higher numbers indicating more frequent individual supervision that is data and evidence-oriented. EIP To measure the extent to which an agency or organization uses data, outcomes, and performance to improve practice, we developed an eight-item scale drawn primarily from Firm Foundations: A Practical Guide for Organisational Support for the Use of Research Evidence (Research in Practice, 2006). On a 5-point Likert scale (1 D never to 5 D very often), participants were given 8 areas to respond to the question: “How often does your team discuss the following in terms of what it might mean for your work with clients?” These eight areas included: (a) quality assurance reports; (b) state performance information/tables; (c) local performance information/tables; (d) research on what improves outcomes for children/families; (e) team performance in meeting practice standards; (f ) peer case reviews; (g) team performance in meeting client outcomes; and (h) how the team should work with children and/or families in order to achieve identified outcomes. Reliability for this measure was high (Cronbach’s ˛ D 0.89) and a mean score was calculated for each participant with higher numbers indicating more frequent use of data and evidence to inform practice. Procedure For each site, the local evaluators and project team identified private agency frontline case managers and supervisors in their respective sites who were involved in performing child welfare case management or residential youth care services. Specifically, staff that provided services under the performance-based contracts were selected by their agency to receive a survey. Hard copies of the surveys were either mailed to individual staff members or the survey was e-mailed to staff members in the form of a SurveyMonkey™ link. All responses were kept anonymous from agency administrators and participants’ names were never associated with their survey. As a part of the larger evaluation, the survey was administered three times during 2007–2009. We drew our data from the second administration of the survey in 2008, which occurred after performance-based contracts had been put in place. Response rates were not recorded. Data entry into SPSS (v.20) was standardized across all sites. Analysis Plan and Hypotheses Hierarchical linear regression (HLM) was chosen as the analysis method for this study because HLM allows us to simultaneously test whether the fixed factors of study site and staff position account for a significant amount of variance in the overall model and to test the hypothesized mediation model among the conceptual variables. In other words, HLM provides a robust test of the hypothesis that organizational factors, such as a team goal-oriented approach (team goals) and data-driven supervision (supervisory practice); predicts the use of EIP in child welfare service agencies; and predicts whether this relationship significantly differs across the study sites or the positions of the staff responding. There are two fixed factors in our model: site (3: Florida, Illinois, and Missouri) and staff position (2: supervisor and case Manager). While we do not specifically hypothesize that the relationship between organizational factors and EIP will be different based on the study site or

PROMOTING OUTCOME ACHIEVEMENT IN CHILD WELFARE

429

TABLE 2 Means and Correlations for Scores on the Team Goals, Supervisory Practice, and Evidence-Informed Practice (EIP) Measures Measure

1

2

3

1. Team goal (TG) 2. Supervisory practice (SUP) 3. Evidence-informed practice (EIP)

— 0.44* 0.32*

— 0.52*

—

M SD

3.51 0.78

3.48 0.85

3.45 0.76

*p < 0.05.

the staff member responding, using HLM to test if these differences exist will help inform the generalizability of this relationship across child welfare contexts and frontline practice. Thus, these two fixed factors may be entered into the HLM equation as the first block of predictors to test whether each accounts for a significant level of variance in the use of EIP. Additionally, HLM allows for a test of our predicted mediation model. Specifically, we hypothesize that two primary factors are key to the use of EIP: (a) team-oriented goals and collaboration toward shared outcomes and (b) data-driven supervisory practice that includes discussions of data and outcome for case management. More specifically, we hypothesize that use of EIP is driven by a team-oriented approach to setting goals when working with families. We further hypothesize that this relationship is mediated and reinforced by supervisory practice that includes discussion of data, evidence, and outcomes during individual supervision. Thus, our analyses are designed to test a null hypothesis that no relationship exists between our predictor variables (team goal and supervisory practice) and our primary dependent variable (EIP).

RESULTS Preliminary descriptive and correlational analyses were completed to obtain the sample means, standard deviations, and correlations of the composite variables of team goals, supervisory practice, and EIP. As seen in Table 2, all measures were significantly inter-correlated, suggesting complementary processes within the child welfare context. All means were slightly above the midpoint of the 5-point Likert scale, indicating moderate agreement with being part of a team that is goal-oriented and actively involved in making work processes more effective; more frequent discussions with supervisors about data on their performance and what it means in terms of outcomes for children and families; and more frequent use of data from a variety of sources to inform their work with clients. HLM As noted, several HLM regressions were run to test the null hypothesis regarding the relationship between key organizational factors and the use of EIP. To test the model, the HLM included the fixed factors of site and staff position in the first block of factors and the predictor variables of team goal and supervisory practice were included in the second block. The first model tests whether differences in the predicted models were a function of site (Florida, Illinois, and Missouri) or staff position (frontline caseworker and supervisor). The second model tests whether the team goal and supervisory practice account for a significant amount of variance in EIP when controlling

430

C. COLLINS-CAMARGO AND T. A. GARSTKA

TABLE 3 Hierarchical Multiple Regression Analyses Predicting Evidence-Informed Practice (EIP) From Supervisory Practice and Team Goal Predictor Step 1: Fixed factors of site and staff position predicting EIP (Constant) Site Agency staff position Step 2: Full mediation model predicting EIP (Constant) Site Agency staff position Supervisory practice Team goal

R

R2

0.04

0.002

0.54

b

SE b

B

3.46 0.03 0.04

0.12 0.03 0.07

0.03 0.02

1.58 0.06 0.09 0.42 0.12

0.17 0.03 0.06 0.03 0.04

0.08 0.05 0.47* 0.13*

0.29

Note. R2 D 0.29 for model 2 (p < 0.0005). *p < 0.001.

for the fixed factors. Table 3 presents HLM including all model summary statistics and coefficient information. As seen in Table 3, site and agency position did not account for a significant amount of variance in evidence-informed practice (R2 D 0.002, F(2,600) D 0.74, p D 0.48) in model 1. When team goal and supervisory practice were included in the second block as predictor variables, the R2 in model 2 D 0.29, and this was significant F(2, 592) D 119.7, p < 0.0005. Indeed, team goal (ˇ D 0.13) and supervisory practice (ˇ D 0.47) each have a significant impact on EIP, p < 0.001 and this relationship exists when controlling for site and agency position. This suggests that organizational factors that relate to how the work teams function and how supervision is done influence the use of EIP. Mediation Analysis Given the significant relationships between organizational factors and the use of EIP, further analyses were done to better understand the processes involved. Specifically, we hypothesized that before a child welfare staff learns to use data to inform practice, the organization in which they are employed must develop a goal-oriented approach to teamwork that includes common outcomes. Additionally, we believe that this approach is best inculcated into daily practice through individual supervision that includes discussing data, outcomes, and performance. Thus, we wanted to test a model in which we hypothesized that supervisory practice mediates the relationship between team goal and EIP. To test this mediation model (Baron & Kenny, 1986; Holmbeck, 1997), we conducted multiple regressions to assess whether the four conditions for mediation were met: (a) team goal is significantly associated with EIP; (b) team goal is significantly associated with supervisory practice; (c) supervisory practice is significantly associated with EIP after controlling for team goal; and (d) impact of team goal on EIP is significantly less after controlling for supervisory practice. We also included the fixed factors of site and agency position into the model to test whether this mediation relationship is independent of those factors. Two HLM regressions were conducted in which team goals was regressed on EIP (condition 1) and which team goal was regressed on supervisory practice (condition 2). The HLM initially run with the fixed factors also provided tests of conditions 3 and 4 of the mediation model by demonstrating the significant direct relationship between supervisory practice and EIP while

PROMOTING OUTCOME ACHIEVEMENT IN CHILD WELFARE

431

FIGURE 1 Mediation model predicting evidence-informed practice (EIP) from team goals and supervisory practice. *p < 0.001. **p < 0.0005.

controlling for team goal. Figure 1 depicts the mediation model with the ˇ included. As seen, all ˇ were significant and in the expected direction, though we found evidence of only partial mediation of supervisory practice. In other words, the direct relationship between team goals and EIP was reduced, but still significant. Thus, our data shows that the more likely a team of frontline and supervisory child welfare staffs is goal oriented, the more frequently they use data and evidence to inform their practice, and this is particularly the case when supervisory practice includes frequent discussions of data, outcomes, and evidence to improve work.

DISCUSSION AND IMPLICATIONS Child welfare agencies are being held more accountable for the achievement of positive outcomes, so determining mechanisms that support outcome achievement is of particular importance. One way of approaching this is for frontline staff and supervisors to engage in activities that involve the use of data and evidence in assessing the effectiveness of their practice with families. This is consistent with a current literature on the topic (e.g., Lindsey & Schlonsky, 2008; Testa & Poertner, 2010). This study lends evidence to the potential value of two different mechanisms for promoting EIP: frontline supervision and goal-oriented teamwork within the work unit. Emphasis on EIP and performance-focused activity in frontline supervision was a predictor of worker use of EIP. This fits with prior literature that links frontline supervision to attitudes toward EIP (Collins-Camargo, 2007) and demonstrates the role supervisors have in supporting and interpreting organizational culture that fosters the use of evidence to inform practice. Other studies have demonstrated the importance of organizational culture in an outcomes-focused approach to this field, as well as EIP (Chaffin, 2006; Cunningham & Durfee, 2009). Findings regarding team influence are consistent with those reported by Anderson and West (1998) who measured team climate as it relates to openness to practice improvement and openness to innovation. It is likely that the influence of supervision and the team dynamics are closely related, in that the extent to which the supervisor facilitates team discussion around peer consultation, use of data, and evaluation of practice effectiveness likely creates goal-oriented teamwork. This study does not elucidate the full mechanism of influence that would be an important area for future research. It must be acknowledged that this study is cross-sectional and relies on worker and supervisor self-report regarding their perceptions. The inclusion of data from three states, given that the

432

C. COLLINS-CAMARGO AND T. A. GARSTKA

findings did not differ significantly by site, does enhance the generalizability of the findings; however, the study is exploratory in nature. The sample itself presents some interesting interpretational challenges. Participants included frontline child welfare staff and residential treatment workers from private agencies only. The literature is very limited in terms of comparing the public and private child welfare work force and their practice. In one prior study, private child welfare workers tended to use and have more positive attitudes toward EIP than corresponding public agency workers in one state (Collins-Camargo et al., 2011). It is interesting, however, that there were no significant differences in perceptions between respondents who were frontline workers providing foster care case management and those working as youth treatment and care workers in residential facilities on the variables of interest. Practically, one might think that the nature of supervision and teamwork may present very differently in these two related but distinct settings. However, these findings suggest that the relationship among these variables as they contribute to the use of evidence to improve practice may be somewhat consistent across varied child welfare settings. Future research should examine these relationships within the public sector and across more varied settings. A limitation of this study is that all survey respondents were working in agencies implementing demonstration projects on performance-based contracting (PBC) and related quality assurance activities, to assess the relationship between these interventions and organizational, practice, and client outcomes. The emphasis on performance and participation in a national demonstration project could certainly have influenced the findings of the study. The cross-site evaluation suggests that in the short-term, all three states observed improved performance in their contract indicators and an array of promising lessons learned about enhanced public–private partnership (Garstka et al., 2012). All three projects had agreed on a sub-goal to improve EIP and supervisory emphasis on performance through their efforts at onset in an attempt to promote successful implementation. However, this stated goal did not yield predicted improvement, as the reported use of EIP and supervisory emphasis on performance did not significantly increase over time as project staff in each site had hoped. Qualitative findings related to process evaluation revealed that the expressed goal of promoting EIP did not translate into sufficient emphasis in the implementation, and that more emphasis was needed on driving performance and practice improvement strategies down to the frontline. Key informant interviews did reveal enhanced overall emphasis within agencies on practice improvement to achieve outcomes across sites however (National Quality Improvement Center on the Privatization of Child Welfare Services, 2011). These cross-site findings do not, however, influence the findings from the current study, illuminating the interrelationships among these variables. These findings may actually be of particular importance because of the connection to the use of PBC. Increasing numbers of child welfare agencies are implementing or considering the use of PBC (Collins-Camargo, McBeath, & Ensign, 2011). The literature has begun to weave a cautious tale related to the use of such contracts and outcome achievement. Some research has suggested that tying achievement of permanency outcomes to fiscal consequences in PBCs may be associated with workers overly focusing on specific child outcomes, such as adoption, rather than reunification, which may require more intensive services, and putting less effort into service provision (McBeath & Meezan, 2010). In the current study, the majority of performance standards across the three sites was proximal rather than outcome indicators. They included timely supervisory review, contact with biological parents, and meeting standards of treatment opportunity days within residential facilities (rather than the youth to be on run, hospitalized, etc.). One state (Missouri) did, however, focus on CFSR-level outcomes for its contract indicators. A recent study using data from the National Survey of Child and Adolescent Well-Being found that in agencies with PBCs families are less likely to receive necessary social and behavioral health services when frontline workers experience role overload (Chuang, Wells, Green & Reiter, 2011). In light of findings such as these, the type of contract indicators tied to fiscal incentives/

PROMOTING OUTCOME ACHIEVEMENT IN CHILD WELFARE

433

disincentives selected, as well as the way in which an agency’s other processes promote performance and outcome achievement (Garstka et al., 2012), may be of tremendous importance to frontline practice. Qualitative interviews with private agency administrators in the QICPCW projects disagreed on the extent to which actual contract indicators should be stressed on the frontline, with some feeling that this over-emphasizes fiscal matters and distracts from attention to best practice (Garstka et al., 2012). Our current study may suggest that if frontline supervision and team processes focus on achieving desired goals for children and families using evidence in assessing the effectiveness of their practice, then negative trends in service provision and the type of outcomes achieved could potentially be avoided. This is an important area for further research. Perhaps the most intriguing aspect of the hierarchical model tested in this study is the mechanism by which supervisory practice and goal-oriented teamwork both contribute to EIP. Given that frontline supervisors provide leadership to the work unit in which the teamwork takes place—and in many cases actually hired and helped train the workers on the unit—this is clearly a complex interaction. The mediating effect observed here deserves further study, particularly in terms of the actual behaviors involved in the interaction. It is important that the field of social welfare practice identify effective interventions agencies can use to promote the type of supervisory and team practice that is indicated here. As teams of workers and supervisors are goal-oriented in their focus on achievement of outcomes for children and families, they are more likely to use data and evidence to inform their practice, and this relationship is strongest when supervisory practice reinforces this behavior through focus on EIP within their practice with workers. Finally, it will be particularly important to investigate how goal-oriented teamwork and supervisory practice influence not only the occurrence of EIP, but also the actual achievement of desired client outcomes. This study represents one additional piece to the rather complex puzzle of outcome achievement in child welfare.

FUNDING This research was supported by funding from the Children’s Bureau, Administration for Children and Families as part of the National Quality Improvement Center on the Privatization of Child Welfare Services.

REFERENCES Aarons, G. A. (2004). Mental health providers’ attitudes toward adoption of evidence-based practice: The Evidence-Based Practice Attitude Scale (EBPAS). Mental Health Services Research, 6(2), 61–74. Aarons, G. A., Hurlburt, M., & Horwitz, S. M. (2011). Advancing a conceptual model of evidence-based practice implementation in public service sectors. Administrative Policy in Mental Health and Mental Health Services Research, 38, 4–23. doi: 10.1007/s10488-010-0327-7 Aarons, G. A., & Palinkas, L. A. (2007). Implementation of evidence-based practice in child welfare: Service provider perspectives. Administration and Policy in Mental Health, 34, 411–419. doi: 10.1007/s10488-007-0121-3 Administration for Children and Families. (n.d.). Results of the 2007 and 2008 Child and Family Services Reviews. Retrieved from http://www.acf.hhs.gov/programs/cb/cwmonitoring/results/agencies_courts.pdf Anderson, N. R., & West, M. A. (1998). Measuring climate for work group innovation: Development and validation of the team climate inventory. Journal of Organizational Behavior, 19, 235–258. Angel, B. O. (2003). Evidence-based programs: Forms of knowledge and outlooks on people in social work. Nordisk Sosialt Arbeid, 23(2), 66–72. Austin, M. J., & Claassen, J. (2008). Implementing evidence-based practice in human service organization: Preliminary lessons from the frontlines. Journal of Evidence-Based Social Work, 5(1–2), 271–293. doi: 10.1300/1394v05n01_10 Baron, R. M., & Kenny, D. A. (1986). The moderator-mediator variable distinction in social psychological research: Conceptual, strategic and statistical considerations. Journal of Personality and Social Psychology, 51, 1173–1182.

434

C. COLLINS-CAMARGO AND T. A. GARSTKA

Barratt, M. (2003). Organizational support for evidence-based practice within child and family social work: A collaborative study. Child & Family Social Work, 8, 143–151. Barth, R. P. (2008). The move to evidence-based practice: How well does it fit child welfare services. Journal of Public Child Welfare, 2(2), 145–171. Berkman, K., & Press, M. (1993). Process notes: Two candidates consider their training. Journal of Clinical Psychoanalysis, 2(3), 367–377. Birleson, P. (1999). Turning child and adolescent mental-health services into learning organizations. Clinical Child Psychology and Psychiatry, 4, 265–274. Carrilio, T. E., Packard, T., & Clapp, J. D. (2003). Nothing in—nothing out: Barriers to the use of performance data in social service programs. Administration in Social Work, 27(4), 61–76. Chaffin, M. (2006). Organizational culture and practice epistemologies. Clinical Psychology Science and Practice, 13(1), 90–93. Chuang, E., Wells, R., Green, S., & Reiter, K. (2011). Performance-based contracting and the moderating influence of ˝ caseworker role overload on service provision in child welfare. Administration in Social Work, 35, 453U474. Clark, S., Gilman, E., Jacquet, S., Johnson, B., Mathias, C., Paris, R., & Zeitler, L. (2008). Line worker, supervisor, and manager perceptions of supervisory practices and tasks in child welfare. Journal of Public Child Welfare, 2(1), 3–32. Cohen, B. J., & Austin, M. J. (1994). Organizational learning and change in a public child welfare agency. Administration in Social Work, 18(1), 1–19. Collins-Camargo, C. (2007). Administering research and demonstration projects aimed at promoting evidence-based practice in child welfare: Challenges and rewards. Journal of Evidence-Based Social Work, 4(3–4), 21–38. Collins-Camargo, C., Hall, J., Flaherty, C., Ensign, K., Garstka, T., Yoder, B., & Metz, A. (2007). Knowledge development and transfer on public/private partnerships in child welfare service provision: Using multi-site research to expand the evidence base. Professional Development: The International Journal of Continuing Social Work Education, 10(3), 14–31. Collins-Camargo, C., McBeath, B., & Ensign, K. (2011). Privatization and performance-based contracting in child welfare: Recent trends and implications for social service administrators. Administration in Social Work, 35, 494–516. Collins-Camargo, C. & Millar, K. (2010). The potential for a more clinical approach to child welfare supervision to promote practice and case outcomes: A qualitative study in four states. The Clinical Supervisor, 29, 164–187. Collins-Camargo, C., & Royse, D. (2010). A study of the relationships among effective supervision, organizational culture, and worker self-efficacy in public child welfare. Journal of Public Child Welfare, 4(1), 1–24. Collins-Camargo, C., Sullivan, D., & Murphy, A. (2011). Use of data to assess performance and promote outcome achievement by public and private child welfare agency staff. Children and Youth Services Review, 33, 330–339. Cozens, W. R. (1999). Data, data everywhere and not a program improvement in sight! Residential Treatment for Children and Youth, 17, 65–85. Cunningham, W. S., & Durfee, D. E. (2009). Styles of evidence-based practice in the child welfare system. Journal of Evidence-Based Social Work, 6, 176–197. Dill, K., & Bogo, M. (2009). Moving beyond the administrative: Supervisor’s perspectives on clinical supervision. Journal of Public Child Welfare, 3(1), 87–105. Epstein, I. (2009). Promoting harmony where there is commonly conflict: Evidence-informed practice as an integrative strategy. Social Work in Health Care, 48, 216–231. Fielding, A, Crawford, F, Leitmann, S., & Anderson, J. (2009). The interplay of evidence and knowledge for social work practice in a health setting. International Journal of Therapy and Rehabilitation, 16, 155–165. Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M., & Wallace, F. (2005). Implementation research: A synthesis of the literature. Tampa, FL: The University of South Florida. Franklin, C., & Hopson, L. M. (2007). Facilitating the use of evidence-based practice in community organizations. Journal of Social Work Education, 43, 377–404. Garstka, T. A., Collins-Camargo, C., Hall, J., Neal, M., & Ensign, K. (2012). Implementing performance based contracts and quality assurance systems in child welfare services: Preliminary results from a national cross-cite evaluation. Journal of Public Child Welfare, 6(10), 12–41. Giddings, M. M., Cleveland, P., Smith, C. H., Collins-Camargo, C. & Russell, R. D. (2008). Clinical supervision for MSWs in child welfare: A professional development model. Journal of Public Child Welfare, 2(3), 339–365. Gioia, D. (2007). Using an organizational change model to qualitatively understand practitioner adoption of evidence-based practice in community mental health. Best Practices in Mental Health, 3(1), 1–15. Hall, J.(2008). A practitioner’s application and deconstruction of evidence-based practice. Families in Society, 89, 385–393. Holmbeck, G. (1997). Toward terminological, conceptual, and statistical clarity in the study of mediators and moderators: Examples from the child-clinical and pediatric psychology literatures. Journal of Consulting and Clinical Psychology, 65, 599–610. Institute for Behavioral Research. (2002). Workshop Assessment Follow-up Survey. Retrieved from http://ibr.tcu.edu/pubs/ datacoll/tcutreatment.html

PROMOTING OUTCOME ACHIEVEMENT IN CHILD WELFARE

435

Lawler, L., & Bilson, A. (2004). Toward a more reflexive research-aware practice: The influence and potential of professional team culture. Social Work & Social Services Review, 11, 52–69. Leitz, C. A. (2008). Implementation of group supervision in child welfare: Findings from Arizona’s supervision circle project. Child Welfare, 87(6), 31–48. Lindsey, D., & Schlonsky, A. (Eds.). (2008). Child welfare research: Advances for practice and policy. New York, NY: Oxford University Press. Manual, J. I., Mullen, E. J., Fang, L., Bellamy, J. L. & Bledsoe, S. E. (2009). Preparing social work practitioners to use evidence-based practice: A comparison of experiences from an implementation project. Research on Social Work Practice, 19, 613–627. McBeath, B., & Meezan, W. (2010). Governance in motion: Service provision and child welfare outcomes in a performancebased, managed care contracting environment. Journal of Public Administration Research and Theory, 20, 101–123. Moore, T. D., Rapp, C. A., & Roberts, B (2000). Improving child welfare performance through supervisory use of client outcome data. Child Welfare, 79, 475–497. Mullen, E. J., Bledsoe, S. E., & Bellamy, J. L. (2008). Implementing evidence-based social work practice. Research on Social Work Practice, 18, 325–338. doi: 10.1177/1049731506297827 National Quality Improvement Center on the Privatization of Child Welfare Services. (2011). Performance-based contracts and quality assurance systems: Final report (Technical report). Lexington, KY: University of Kentucky. Organizational Excellence Group. (2001). Survey of Organizational Excellence. Retrieved from http://www.survey.utexas. edu Petr, C. G. (Ed.). (2009). Multidimensional evidence-based practice: Synthesizing knowledge, research and values. New York, NY: Routledge. Research in Practice Institute. (2006). Firm foundations: A practical guide to organisational support for the use of research evidence. Devon, UK: Author. Schoech, D., Basham, R., & Fluke, J. (2006). A technology enhanced EBP model. Journal of Evidence-Based Social Work, 3(3), 55–72. Testa, M., & Poertner, J., (Eds). (2010). Fostering accountability: Using evidence to guild and improve child welfare policy. New York, NY: Oxford University Press. U.S. General Accounting Office. (2004). Child and Family Service Reviews: Better use of data and improved guidance could enhance HHS’s oversight of state performance (Report No. GAO04-333). Washington, DC: United States General Accounting Office. Usher, C. L., & Wildfire, J. B. (2003). Evidence-based practice in community-based child welfare. Child Welfare, 72, 597–614 Weigensberg, E. C. (2009). Child welfare agency performance: How are child, agency, and county factors related to achieving timely permanency outcomes for children in foster care? (Unpublished doctoral dissertation). University of North Carolina at Chapel Hill, Chapel Hill, NC Wulczyn, F. (2005). Monitoring the performance of the child welfare system. In Rockefeller Institute of Government (Ed.), Performance management in state and local government (pp. 54–62). Albany, NY: Rockefeller Institute of Government. Young, T. (1994). Collaboration of a public child welfare agency and a school of social work: A clinical group supervision project. Child Welfare, 73, 659–671.

436

C. COLLINS-CAMARGO AND T. A. GARSTKA

APPENDIX Team Goal Measure As a Member of the Team, How Would You Rate the Following? Strongly Disagree

Disagree

Feel Neutral

Agree

Strongly Agree





















 

 

 

 

 

a. We have an opportunity to participate in the goal setting process. b. Decision making and control are given to employees doing the actual work. c. We seem to be working toward the same goals. d. Work groups are actively involved in making work processes more effective.

Supervisory Practice Measure How Often Do You and Your Supervisor Discuss:

a. What success will look like—i.e. what measurable outcomes are we seeking for a child/youth/family? b. What research tells us is most likely to work for someone in this situation? c. Which of the alternative courses of action is likely to be more effective, and how will you know? d. What evidence we have about what clients want or find helpful in these situations? e. What your team’s performance data tells you that may help you improve your practice with clients?

Very Often

Often

Sometimes

Rarely

Never



















































EIP Measure How Often Does Your Team Discuss the Following in Terms of What it Might Mean for Your Work with Clients?

a. b. c. d. e. f. g. h.

Quality assurance reports? Reports on the team’s performance in meeting practice standards? Reports on the team’s performance in meeting client outcomes? Peer case reviews? Local performance information/tables giving data for all teams? State performance information/tables giving data for all teams? Research on what improves outcomes for children and/or families? How we should work with children and/or families in order to achieve identified outcomes?

Very Often

Often

Sometimes

Rarely

Never

  

  

  

  

  

   

   

   

   

   

Copyright of Journal of Evidence-Based Social Work is the property of Taylor & Francis Ltd and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.