RESEARCH ARTICLE
Psychometric Support of the School Climate Measure in a Large, Diverse Sample of Adolescents: A Replication and Extension KEITH J. ZULLIG,a RANI COLLINS,b NADIA GHANI,c JON M. PATTON,d E. SCOTT HUEBNER,e JEAN AJAMIE,f
ABSTRACT BACKGROUND: The School Climate Measure (SCM) was developed and validated in 2010 in response to a dearth of psychometrically sound school climate instruments. This study sought to further validate the SCM on a large, diverse sample of Arizona public school adolescents (N = 20,953). METHODS: Four SCM domains (positive student-teacher relationships, academic support, order and discipline, and physical environment) were available for the analysis. Confirmatory factor analysis and structural equation modeling were established to construct validity, and criterion-related validity was assessed via selected Youth Risk Behavior Survey (YRBS) school safety items and self-reported grade (GPA) point average. RESULTS: Analyses confirmed the 4 SCM school climate domains explained approximately 63% of the variance (factor loading range .45-.92). Structural equation models fit the data well χ 2 = 14,325 (df = 293, p < .001), comparative fit index (CFI) = .951, Tuker-Lewis index (TLI) = .952, root mean square error of approximation (RMSEA) = .05). The goodness-of-fit index was .940. Coefficient alphas ranged from .82 to .93. Analyses of variance with post hoc comparisons suggested the SCM domains related in hypothesized directions with the school safety items and GPA. CONCLUSIONS: Additional evidence supports the validity and reliability of the SCM. Measures, such as the SCM, can facilitate data-driven decisions and may be incorporated into evidenced-based processes designed to improve student outcomes. Keywords: school climate; adolescents; psychometrics; measurement. Citation: Zullig KJ, Collins R, Ghani N, Patton JM, Scott Huebner E, Ajamie J. Psychometric support of the school climate measure in a large, diverse sample of adolescents: a replication and extension. J Sch Health. 2014; 84: 82-90. Received on September 28, 2012 Accepted on May 12, 2013
A
dolescents spend much of their time in school. Thus, it is important that their school experience be positive and learning environment be supportive. To determine how a student is influenced by his or her school environment, researchers have examined a construct known as school climate. A positive school climate ‘‘ . . . fosters youth development and learning necessary for a productive, contributive, and satisfying life in a democratic society.’’1(p182) Thus, school climate
is an important construct to define, examine, and measure in students. Although researchers have examined the construct of school climate for over 100 years,2 discrepancies in this construct’s definition have been observed. For example, Zullig et al3 note that definitions range from objective to subjective as well as measuring effect and context. These researchers also highlighted additional difficulties when US Healthy People 2020
a Associate Professor, ([email protected]), Department of Social and Behavioral Sciences, West Virginia University, PO Box 9190, Morgantown, WV 26506. bProject Director, ([email protected]), Arizona Department of Education, School Safety and Prevention, 1535 West Jefferson Street, Phoenix, AZ 85007. c Evaluation/Research Consultant, ([email protected]), Arizona Department of Education, School Safety and Prevention, 1535 West Jefferson Street, Phoenix, AZ 85007. dSenior Research Computing Specialist, ([email protected]), Department of Computer Sciences and Software Engineering, Miami University, 310E Laws Hall, Oxford, OH 45056. e Professor, ([email protected]), Department of Psychology, School Psychology, University of South Carolina, Columbia, SC 29208. f Director, ([email protected]), Arizona Department of Education, School Safety and Prevention, 1535 West Jefferson Street, Phoenix, AZ 85007.
Address correspondence to: Keith J. Zullig, Associate Professor, ([email protected]), Department of Social and Behavioral Sciences, West Virginia University, PO Box 9190, Morgantown, WV 26506. This research was supported by a US Department of Education Office of Safe and Healthy Students Safe and Supportive Schools grant to the Arizona Department of Education.
82
•
Journal of School Health
•
February 2014, Vol. 84, No. 2
•
© 2014, American School Health Association
goals4 are examined for a healthy school environment. A healthy school environment, as defined by the Healthy People 2020 goals, is related to the actual physical environment of a school (including mold and ventilation issues, as well as indoor air quality, etc). Although broad consensus of a definition of school climate is difficult, what is clear is that researchers have moved away from a strictly physical environment focus5 to viewing school climate as a measure of subjective school experience,6 including feelings of safety (eg, order and rules, social and emotional safety).1 Specifically, school climate is ‘‘ . . . based on patterns of people’s experiences of school life and reflects norms, goals, values, interpersonal relationships, teaching and learning practices, and organizational structures.’’1(p182) School climate has also been associated with important school outcomes. Although educational policy for the past decade has been driven primarily by the measures of reading and mathematical skill as dictated by No Child Left Behind (NCLB), growing evidence suggests that school climate can affect students’ social, behavioral, and learning outcomes and that by addressing organizational processes and social relationships, positive behavioral change can occur.7-9 For instance, Hoy and Hannum10 found that the most important school climate variables influencing student achievement were a serious and orderly learning environment (academic emphasis), teachers displaying a commitment to their students (teacher affiliation), and adequate supply and material support for teaching (resource support), even after controlling for socioeconomic status (SES). Moreover, a recent systematic review of interventions targeting school climate factors also suggested that school relationships and teacher support can positively impact student emotional health.11 Thus, many states have policies that directly or indirectly address school climate. For example, the National School Climate Center provides a useful database where individual state policies can be searched at http://www.schoolclimate.org/climate/database.php. Consequently, the School Climate Measure (SCM)3 was developed to assess students’ perceptions of their school climate. The goals of the original study3 were to perform a historical analysis of the extant school climate literature to (1) define school climate, (2) determine the historically common school climate domains, and (3) identify the most widely historically cited school climate measurement tools to develop a psychometrically sound measure of school climate. These researchers identified 5 primary, historically common school climate domains, order, safety, and discipline; academic outcomes; social relationships; school facilities; and school connectedness, and 5 widely cited school climate measurement tools. Journal of School Health
•
When these instruments were combined, preliminary analyses supported an 8-factor structure of the scale, including but extending beyond the 5 hypothesized domains. To explore the psychometric properties of the SCM, a sample of over 2000 public school students was randomly split into exploratory and confirmatory samples and subjected to factor analytic and structural equation modeling (SEM) techniques.3 Factor analysis confirmed an 8-factor solution with demonstrated construct validity and acceptable internal consistency estimates. In addition, structural equation models revealed that the final models fit the data well in both the exploratory and confirmatory samples. The resulting SCM instrument contains 39 items spread among 8 scales measuring dimensions of school climate: positive student-teacher relationships (9 items), school connectedness (6 items), academic support (6 items), order and discipline (7 items), school physical environment (4 items), school social environment (2 items), perceived exclusion/privilege (3 items), and academic satisfaction (2 items). All items use the same five response option formats: (Strongly Disagree = 1); (Disagree = 2); (Neither = 3); (Agree = 4); and (Strongly Agree = 5). The SCM’s robust psychometrics, ease and cost of administration, and broad applicability when compared with the many available tools gave rise to its inclusion as the only suitable measure of school climate in the PhenXToolkit12 that was funded by the National Human Genome Research Institute to compose a core set of high-quality, well-established, low-burden measures intended for use in large-scale genomic studies. However, the preliminary study by Zullig et al3 testing the validity and reliability of SCM was hampered by several limitations. First, the sample was composed mainly of Caucasian students from 3 school districts in a Midwestern state, suggesting the need for additional research with more diverse populations from different geographical locations. Second, although Zullig et al3 established some evidence for the construct validity of the SCM, these researchers were unable to test the performance of the SCM against established measures of adolescent behaviors potentially impacted by the school environment. This replication and extension study sought to address these identified limitations by exploring the reliability and validity of the SCM on a large, diverse sample of public high school students from Arizona. The first aim of this study was to provide generalizability data for the SCM. The second aim was to assess the SCM against established items from the Centers for Disease Control and Prevention’s (CDC) Youth Risk Behavior Survey (YRBS), while others were developed specifically for the Arizona YRBS. For this second aim, items that query students on perception and prevalence of school safety (carrying
February 2014, Vol. 84, No. 2
•
© 2014, American School Health Association • 83
a weapon on school property, feeling safe at school, fighting at school, etc) were chosen for the analysis, given the evidence that a positive school climate has been shown to reduce delinquency behavior.13,14 It was hypothesized that the SCM would (1) replicate acceptable validity and reliability psychometric properties in this sample and (2) operate in predictable ways against established YRBS school safety items in that lower perceptions of school climate would be significantly associated with lower perceptions of school safety.
Table 1. Sample Characteristics Variable Sex Male Female Age 13 years 14 years 15 years 16 years 17 years ≥18 years Grade 9th 10th 11th 12th GPA Mostly A’s Mostly B’s Mostly C’s Mostly D’s Mostly F’s None of these Not sure
METHODS Participants As part of emerging federal priorities to improve conditions for learning in public schools, the Arizona Department of Education (ADE) was awarded a competitive Safe and Supportive Schools (S3) Grant. Local Education Agencies (LEAs) with public high schools in Arizona were informed of the grant through e-mail during summer 2009-2010 and invited to partner with the ADE by submitting an application. The criteria for selection included (1) LEAs with at least one public high school with safety or academic concerns, (2) LEA and school support for the grant, (3) schools within LEAs expressing an interest in improving school climate, and (4) schools demonstrating a promise for data management and collection. Fourteen partnering LEAs were selected by ADE to be recipients of this funding. As a part of mandatory grant requirements, student surveys which incorporated several climate measures were administered in each of the 61 participating high schools. From an original sampling frame of 25,638 9th to 12th grade students, 21,082 students were surveyed in the 2010-2011 school year through a census (N = 13 schools), or random sampling procedures if the enrollment exceeded 300 students (N = 48 schools). For these 48 schools, classes that met during second period were selected using random starts with systematic, equal probability sampling. Enrollments in these schools ranged between 418 and 2685 with schools sampling between 214 and 787 students. The total sample consisted of 10,253 boys (48.93%) and 10,700 girls (51.07%). Grade level was fairly evenly distributed, but there were several students in the sample who reported being 14 years or younger (7.38%) or 18 years or older (15.3%). Race/ethnicity was calculated from 2 questions: (1) ‘‘Are you Hispanic or Latino?’’ (yes/no) and (2) ‘‘What is your race?’’ For the second question, students could select more than one response option. Students reported being nonwhite Hispanic (10,184, 48.6%), white/non-Hispanic (7564, 36.1%), American Indian or Alaskan Native (1027, 4.9 %), Native Hawaiian or Other Pacific Islander (293, 1.4%), black or African American (1299, 84
•
Journal of School Health
•
February 2014, Vol. 84, No. 2
N
%
10,253 10,700
48.93 51.07
54 1505 5039 5836 5289 3230
0.3 7.2 24.1 27.8 25.2 15.4
5108 5916 5615 4314
24.4 28.2 26.8 20.6
5226 7298 5001 1444 603 127 1254
24.9 34.8 23.9 6.9 2.9 0.6 6.0
6.2%), or Asian (586, 2.8%) descent. Eleven LEAs were classified as urban centers and 3 as rural in conjunction with definitions from the US Census. The LEA’s represented a wide range of socioeconomic strata with the percent of students eligible for free and reduced-price lunch ranging from 4% to 91%; (mean 55%) in the selected schools. Table 1 provides additional sample characteristics. Procedure Parent and student consent was gained prior to administering the survey. However, consent was active or passive depending on LEA policy. In Arizona, parents are informed in their school policy documents that students will be occasionally requested to participate in surveys. In this study, three LEAs and corresponding 11 participating schools employed active consent policies. The remaining LEAs (ie, 11 LEAs and corresponding 50 participating schools) employed passive consent policies. For passive consent, forms were sent home with students, and parents were asked to respond only if they did not wish for their child to participate. Active consent required parents to send in signed consent forms if they consented to their child’s participation; students without signed consent forms could not participate. Trained data collectors were placed in each of the selected schools to ensure consistency in instruction and overall data collection. Because student absenteeism may be related to a poor perception of school climate and engagement in high-risk behaviors, students who were absent were given opportunities •
© 2014, American School Health Association
to take the survey at a later date. This procedure also allowed for a more robust response rate. Data cleaning followed a multistage process similar to the YRBS. Responses that showed a conflict in logic were removed (N = 67), as well as questionnaires with less than 20 valid responses prior to conducting any analysis (N = 129), yielding a final sample of 20,953 for an overall student response rate of approximately 82%. Instruments The school climate measure (SCM). The 4 SCM subscales available for analysis in this study were positive student-teacher relationships, academic support, order and discipline, and physical environment. These 4 domains were selected because they (1) aligned most closely with Arizona’s S3 Grant priorities and (2) explained 36% out of 45.7% of the variance in the Zullig et al3 study. Specific item wording for each of the available SCM constructs in this study can be found in Table 2 and on the PhenXToolkit website: https://www.phenxtoolkit.org/index.php?pageLink= browse.protocoldetails&id=211001. Similar to previous research, all items use the same 5 response option formats: (Strongly Disagree = 1); (Disagree = 2); (Neither = 3); (Agree = 4); and (Strongly Agree = 5). School safety. Items composing school safety were focused specifically on behaviors taking place on school property. Specific school safety items from the YRBS that have been shown to display adequate testretest reliability15 were ‘‘During the past 30 days, on how many days did you carry a weapon such as a gun, knife, or club on school property?’’; ‘‘During the past 12 months, how many times has someone threatened or injured you with a weapon such as a gun, knife, or club on school property?’’; and ‘‘During the past 12 months, how many times were you in a physical fight on school property?’’ Arizona YRBS school safety items were ‘‘How often do you feel safe and secure at school?’’; ‘‘During the past 12 months, how frequently have you been harassed or bullied on school property?’’; ‘‘During the past 12 months, how frequently have you harassed or bullied someone else on school property?’’; and ‘‘During the past 12 months, how frequently has someone stolen or deliberately damaged your property such as your car, clothing, or books on school property?’’ In addition, self-reported grades (GPAs) was selected as a demographic variable and worded as ‘‘During the past 12 months, how would you describe your grades in school?’’ Data Analysis All analyses were conducted using SAS version 9.2 (SAS Institute Inc., Cary, NC). Because reliability is a prerequisite for validity analysis, Cronbach’s alpha16 Journal of School Health
•
Table 2. SCM Items, Alpha Coefficients, and Factor Loadings Item (% Variance Explained) Factor 1: Positive Student-Teacher Relationships (21.0%) Teachers understand my problems Teachers and staff seemto take a real interest in my future Teachers are available when I need to talk with them It is easy to talk with teachers Students get along well with teachers At my school, there is a teacher or some other adult who notices when I’m not there Teachers at my school help us children with our problems My teachers care about me My teacher makes me feel good about myself Factor 2: Academic Support (15.5%) I usually understand my homework assignments Teachers make it clear what work needs to be done to get the grade I want I believe that teachers expect all students to learn I feel that I can do well in this school My teachers believe that I can do well in my school work I try hard to succeed in my classes Factor 3: Order and Discipline (13.4%) Classroom rules are applied equally Problems in this school are solved by students and staff Students get in trouble if they do not follow school rules The rules of the school are fair School rules are enforced consistently and fairly My teachers make it clear to me when I have misbehaved in class Discipline is fair Factor 4: School Physical Environment (12.7%) The school grounds are kept clean My school is neat and clean My school buildings are generally pleasant and well maintained My school is usually clean and tidy
Factor Loading .91 .75 .75 .75 .76 .63 .57 .80 .83 .81
.82 .60 .68 .69 .75 .80 .45
.88 .75 .72 .61 .75 .80 .66 .76
.93 .88 .92 .82 .91
SCM, School Climate Measure. Cronbach’s alpha is reported in the bold.
determined scale internal consistency estimates and provided evidence for items that might be suppressors. Construct validity. Confirmatory factor analysis was performed on the 4 available SCM constructs to establish construct validity. Items were retained on factors if they had high loadings (absolute values >.40), were not complex (ie, did not load on 2 or more different factors with a difference of .2), and contained eigenvalues >1, according to Kaiser’s rule17 on a scree plot.18 After the factor structure was confirmed, relationships between the latent and manifest variables were explored utilizing SEM the ‘‘Proc Calis’’ procedure. Structural equation modeling (SEM) serves 2 purposes in measurement development. First, the structure developed through the confirmatory factor analysis process can confirm results provided by the preliminary study.3 Second, latent factor structure intercorrelations were examined. Confirmatory factor analysis did not allow for the examination of highly
February 2014, Vol. 84, No. 2
•
© 2014, American School Health Association • 85
correlated structures or hierarchical structures in scale development (ie, an underlying theme or guiding concept to the whole measure). Structural equation models are deemed good representations of the theory based on parsimony, chi-square goodness of fit, descriptive fit indices (comparative fit index: CFI & Tuker-Lewis index: TLI), and alternative fit indices (root mean squared error of approximation: root mean square error of approximation [RMSEA] and residuals). With respect to parsimony, models with the fewest parameters to explain the relationship were retained. For the descriptive indices, fits of >.9 (preferably >.95) indicated a well-fitting model (CFI, TLI).19 For RMSEA, a fit of
Psychometric Support of the School Climate Measure in a Large, Diverse Sample of Adolescents: A Replication and Extension KEITH J. ZULLIG,a RANI COLLINS,b NADIA GHANI,c JON M. PATTON,d E. SCOTT HUEBNER,e JEAN AJAMIE,f
ABSTRACT BACKGROUND: The School Climate Measure (SCM) was developed and validated in 2010 in response to a dearth of psychometrically sound school climate instruments. This study sought to further validate the SCM on a large, diverse sample of Arizona public school adolescents (N = 20,953). METHODS: Four SCM domains (positive student-teacher relationships, academic support, order and discipline, and physical environment) were available for the analysis. Confirmatory factor analysis and structural equation modeling were established to construct validity, and criterion-related validity was assessed via selected Youth Risk Behavior Survey (YRBS) school safety items and self-reported grade (GPA) point average. RESULTS: Analyses confirmed the 4 SCM school climate domains explained approximately 63% of the variance (factor loading range .45-.92). Structural equation models fit the data well χ 2 = 14,325 (df = 293, p < .001), comparative fit index (CFI) = .951, Tuker-Lewis index (TLI) = .952, root mean square error of approximation (RMSEA) = .05). The goodness-of-fit index was .940. Coefficient alphas ranged from .82 to .93. Analyses of variance with post hoc comparisons suggested the SCM domains related in hypothesized directions with the school safety items and GPA. CONCLUSIONS: Additional evidence supports the validity and reliability of the SCM. Measures, such as the SCM, can facilitate data-driven decisions and may be incorporated into evidenced-based processes designed to improve student outcomes. Keywords: school climate; adolescents; psychometrics; measurement. Citation: Zullig KJ, Collins R, Ghani N, Patton JM, Scott Huebner E, Ajamie J. Psychometric support of the school climate measure in a large, diverse sample of adolescents: a replication and extension. J Sch Health. 2014; 84: 82-90. Received on September 28, 2012 Accepted on May 12, 2013
A
dolescents spend much of their time in school. Thus, it is important that their school experience be positive and learning environment be supportive. To determine how a student is influenced by his or her school environment, researchers have examined a construct known as school climate. A positive school climate ‘‘ . . . fosters youth development and learning necessary for a productive, contributive, and satisfying life in a democratic society.’’1(p182) Thus, school climate
is an important construct to define, examine, and measure in students. Although researchers have examined the construct of school climate for over 100 years,2 discrepancies in this construct’s definition have been observed. For example, Zullig et al3 note that definitions range from objective to subjective as well as measuring effect and context. These researchers also highlighted additional difficulties when US Healthy People 2020
a Associate Professor, ([email protected]), Department of Social and Behavioral Sciences, West Virginia University, PO Box 9190, Morgantown, WV 26506. bProject Director, ([email protected]), Arizona Department of Education, School Safety and Prevention, 1535 West Jefferson Street, Phoenix, AZ 85007. c Evaluation/Research Consultant, ([email protected]), Arizona Department of Education, School Safety and Prevention, 1535 West Jefferson Street, Phoenix, AZ 85007. dSenior Research Computing Specialist, ([email protected]), Department of Computer Sciences and Software Engineering, Miami University, 310E Laws Hall, Oxford, OH 45056. e Professor, ([email protected]), Department of Psychology, School Psychology, University of South Carolina, Columbia, SC 29208. f Director, ([email protected]), Arizona Department of Education, School Safety and Prevention, 1535 West Jefferson Street, Phoenix, AZ 85007.
Address correspondence to: Keith J. Zullig, Associate Professor, ([email protected]), Department of Social and Behavioral Sciences, West Virginia University, PO Box 9190, Morgantown, WV 26506. This research was supported by a US Department of Education Office of Safe and Healthy Students Safe and Supportive Schools grant to the Arizona Department of Education.
82
•
Journal of School Health
•
February 2014, Vol. 84, No. 2
•
© 2014, American School Health Association
goals4 are examined for a healthy school environment. A healthy school environment, as defined by the Healthy People 2020 goals, is related to the actual physical environment of a school (including mold and ventilation issues, as well as indoor air quality, etc). Although broad consensus of a definition of school climate is difficult, what is clear is that researchers have moved away from a strictly physical environment focus5 to viewing school climate as a measure of subjective school experience,6 including feelings of safety (eg, order and rules, social and emotional safety).1 Specifically, school climate is ‘‘ . . . based on patterns of people’s experiences of school life and reflects norms, goals, values, interpersonal relationships, teaching and learning practices, and organizational structures.’’1(p182) School climate has also been associated with important school outcomes. Although educational policy for the past decade has been driven primarily by the measures of reading and mathematical skill as dictated by No Child Left Behind (NCLB), growing evidence suggests that school climate can affect students’ social, behavioral, and learning outcomes and that by addressing organizational processes and social relationships, positive behavioral change can occur.7-9 For instance, Hoy and Hannum10 found that the most important school climate variables influencing student achievement were a serious and orderly learning environment (academic emphasis), teachers displaying a commitment to their students (teacher affiliation), and adequate supply and material support for teaching (resource support), even after controlling for socioeconomic status (SES). Moreover, a recent systematic review of interventions targeting school climate factors also suggested that school relationships and teacher support can positively impact student emotional health.11 Thus, many states have policies that directly or indirectly address school climate. For example, the National School Climate Center provides a useful database where individual state policies can be searched at http://www.schoolclimate.org/climate/database.php. Consequently, the School Climate Measure (SCM)3 was developed to assess students’ perceptions of their school climate. The goals of the original study3 were to perform a historical analysis of the extant school climate literature to (1) define school climate, (2) determine the historically common school climate domains, and (3) identify the most widely historically cited school climate measurement tools to develop a psychometrically sound measure of school climate. These researchers identified 5 primary, historically common school climate domains, order, safety, and discipline; academic outcomes; social relationships; school facilities; and school connectedness, and 5 widely cited school climate measurement tools. Journal of School Health
•
When these instruments were combined, preliminary analyses supported an 8-factor structure of the scale, including but extending beyond the 5 hypothesized domains. To explore the psychometric properties of the SCM, a sample of over 2000 public school students was randomly split into exploratory and confirmatory samples and subjected to factor analytic and structural equation modeling (SEM) techniques.3 Factor analysis confirmed an 8-factor solution with demonstrated construct validity and acceptable internal consistency estimates. In addition, structural equation models revealed that the final models fit the data well in both the exploratory and confirmatory samples. The resulting SCM instrument contains 39 items spread among 8 scales measuring dimensions of school climate: positive student-teacher relationships (9 items), school connectedness (6 items), academic support (6 items), order and discipline (7 items), school physical environment (4 items), school social environment (2 items), perceived exclusion/privilege (3 items), and academic satisfaction (2 items). All items use the same five response option formats: (Strongly Disagree = 1); (Disagree = 2); (Neither = 3); (Agree = 4); and (Strongly Agree = 5). The SCM’s robust psychometrics, ease and cost of administration, and broad applicability when compared with the many available tools gave rise to its inclusion as the only suitable measure of school climate in the PhenXToolkit12 that was funded by the National Human Genome Research Institute to compose a core set of high-quality, well-established, low-burden measures intended for use in large-scale genomic studies. However, the preliminary study by Zullig et al3 testing the validity and reliability of SCM was hampered by several limitations. First, the sample was composed mainly of Caucasian students from 3 school districts in a Midwestern state, suggesting the need for additional research with more diverse populations from different geographical locations. Second, although Zullig et al3 established some evidence for the construct validity of the SCM, these researchers were unable to test the performance of the SCM against established measures of adolescent behaviors potentially impacted by the school environment. This replication and extension study sought to address these identified limitations by exploring the reliability and validity of the SCM on a large, diverse sample of public high school students from Arizona. The first aim of this study was to provide generalizability data for the SCM. The second aim was to assess the SCM against established items from the Centers for Disease Control and Prevention’s (CDC) Youth Risk Behavior Survey (YRBS), while others were developed specifically for the Arizona YRBS. For this second aim, items that query students on perception and prevalence of school safety (carrying
February 2014, Vol. 84, No. 2
•
© 2014, American School Health Association • 83
a weapon on school property, feeling safe at school, fighting at school, etc) were chosen for the analysis, given the evidence that a positive school climate has been shown to reduce delinquency behavior.13,14 It was hypothesized that the SCM would (1) replicate acceptable validity and reliability psychometric properties in this sample and (2) operate in predictable ways against established YRBS school safety items in that lower perceptions of school climate would be significantly associated with lower perceptions of school safety.
Table 1. Sample Characteristics Variable Sex Male Female Age 13 years 14 years 15 years 16 years 17 years ≥18 years Grade 9th 10th 11th 12th GPA Mostly A’s Mostly B’s Mostly C’s Mostly D’s Mostly F’s None of these Not sure
METHODS Participants As part of emerging federal priorities to improve conditions for learning in public schools, the Arizona Department of Education (ADE) was awarded a competitive Safe and Supportive Schools (S3) Grant. Local Education Agencies (LEAs) with public high schools in Arizona were informed of the grant through e-mail during summer 2009-2010 and invited to partner with the ADE by submitting an application. The criteria for selection included (1) LEAs with at least one public high school with safety or academic concerns, (2) LEA and school support for the grant, (3) schools within LEAs expressing an interest in improving school climate, and (4) schools demonstrating a promise for data management and collection. Fourteen partnering LEAs were selected by ADE to be recipients of this funding. As a part of mandatory grant requirements, student surveys which incorporated several climate measures were administered in each of the 61 participating high schools. From an original sampling frame of 25,638 9th to 12th grade students, 21,082 students were surveyed in the 2010-2011 school year through a census (N = 13 schools), or random sampling procedures if the enrollment exceeded 300 students (N = 48 schools). For these 48 schools, classes that met during second period were selected using random starts with systematic, equal probability sampling. Enrollments in these schools ranged between 418 and 2685 with schools sampling between 214 and 787 students. The total sample consisted of 10,253 boys (48.93%) and 10,700 girls (51.07%). Grade level was fairly evenly distributed, but there were several students in the sample who reported being 14 years or younger (7.38%) or 18 years or older (15.3%). Race/ethnicity was calculated from 2 questions: (1) ‘‘Are you Hispanic or Latino?’’ (yes/no) and (2) ‘‘What is your race?’’ For the second question, students could select more than one response option. Students reported being nonwhite Hispanic (10,184, 48.6%), white/non-Hispanic (7564, 36.1%), American Indian or Alaskan Native (1027, 4.9 %), Native Hawaiian or Other Pacific Islander (293, 1.4%), black or African American (1299, 84
•
Journal of School Health
•
February 2014, Vol. 84, No. 2
N
%
10,253 10,700
48.93 51.07
54 1505 5039 5836 5289 3230
0.3 7.2 24.1 27.8 25.2 15.4
5108 5916 5615 4314
24.4 28.2 26.8 20.6
5226 7298 5001 1444 603 127 1254
24.9 34.8 23.9 6.9 2.9 0.6 6.0
6.2%), or Asian (586, 2.8%) descent. Eleven LEAs were classified as urban centers and 3 as rural in conjunction with definitions from the US Census. The LEA’s represented a wide range of socioeconomic strata with the percent of students eligible for free and reduced-price lunch ranging from 4% to 91%; (mean 55%) in the selected schools. Table 1 provides additional sample characteristics. Procedure Parent and student consent was gained prior to administering the survey. However, consent was active or passive depending on LEA policy. In Arizona, parents are informed in their school policy documents that students will be occasionally requested to participate in surveys. In this study, three LEAs and corresponding 11 participating schools employed active consent policies. The remaining LEAs (ie, 11 LEAs and corresponding 50 participating schools) employed passive consent policies. For passive consent, forms were sent home with students, and parents were asked to respond only if they did not wish for their child to participate. Active consent required parents to send in signed consent forms if they consented to their child’s participation; students without signed consent forms could not participate. Trained data collectors were placed in each of the selected schools to ensure consistency in instruction and overall data collection. Because student absenteeism may be related to a poor perception of school climate and engagement in high-risk behaviors, students who were absent were given opportunities •
© 2014, American School Health Association
to take the survey at a later date. This procedure also allowed for a more robust response rate. Data cleaning followed a multistage process similar to the YRBS. Responses that showed a conflict in logic were removed (N = 67), as well as questionnaires with less than 20 valid responses prior to conducting any analysis (N = 129), yielding a final sample of 20,953 for an overall student response rate of approximately 82%. Instruments The school climate measure (SCM). The 4 SCM subscales available for analysis in this study were positive student-teacher relationships, academic support, order and discipline, and physical environment. These 4 domains were selected because they (1) aligned most closely with Arizona’s S3 Grant priorities and (2) explained 36% out of 45.7% of the variance in the Zullig et al3 study. Specific item wording for each of the available SCM constructs in this study can be found in Table 2 and on the PhenXToolkit website: https://www.phenxtoolkit.org/index.php?pageLink= browse.protocoldetails&id=211001. Similar to previous research, all items use the same 5 response option formats: (Strongly Disagree = 1); (Disagree = 2); (Neither = 3); (Agree = 4); and (Strongly Agree = 5). School safety. Items composing school safety were focused specifically on behaviors taking place on school property. Specific school safety items from the YRBS that have been shown to display adequate testretest reliability15 were ‘‘During the past 30 days, on how many days did you carry a weapon such as a gun, knife, or club on school property?’’; ‘‘During the past 12 months, how many times has someone threatened or injured you with a weapon such as a gun, knife, or club on school property?’’; and ‘‘During the past 12 months, how many times were you in a physical fight on school property?’’ Arizona YRBS school safety items were ‘‘How often do you feel safe and secure at school?’’; ‘‘During the past 12 months, how frequently have you been harassed or bullied on school property?’’; ‘‘During the past 12 months, how frequently have you harassed or bullied someone else on school property?’’; and ‘‘During the past 12 months, how frequently has someone stolen or deliberately damaged your property such as your car, clothing, or books on school property?’’ In addition, self-reported grades (GPAs) was selected as a demographic variable and worded as ‘‘During the past 12 months, how would you describe your grades in school?’’ Data Analysis All analyses were conducted using SAS version 9.2 (SAS Institute Inc., Cary, NC). Because reliability is a prerequisite for validity analysis, Cronbach’s alpha16 Journal of School Health
•
Table 2. SCM Items, Alpha Coefficients, and Factor Loadings Item (% Variance Explained) Factor 1: Positive Student-Teacher Relationships (21.0%) Teachers understand my problems Teachers and staff seemto take a real interest in my future Teachers are available when I need to talk with them It is easy to talk with teachers Students get along well with teachers At my school, there is a teacher or some other adult who notices when I’m not there Teachers at my school help us children with our problems My teachers care about me My teacher makes me feel good about myself Factor 2: Academic Support (15.5%) I usually understand my homework assignments Teachers make it clear what work needs to be done to get the grade I want I believe that teachers expect all students to learn I feel that I can do well in this school My teachers believe that I can do well in my school work I try hard to succeed in my classes Factor 3: Order and Discipline (13.4%) Classroom rules are applied equally Problems in this school are solved by students and staff Students get in trouble if they do not follow school rules The rules of the school are fair School rules are enforced consistently and fairly My teachers make it clear to me when I have misbehaved in class Discipline is fair Factor 4: School Physical Environment (12.7%) The school grounds are kept clean My school is neat and clean My school buildings are generally pleasant and well maintained My school is usually clean and tidy
Factor Loading .91 .75 .75 .75 .76 .63 .57 .80 .83 .81
.82 .60 .68 .69 .75 .80 .45
.88 .75 .72 .61 .75 .80 .66 .76
.93 .88 .92 .82 .91
SCM, School Climate Measure. Cronbach’s alpha is reported in the bold.
determined scale internal consistency estimates and provided evidence for items that might be suppressors. Construct validity. Confirmatory factor analysis was performed on the 4 available SCM constructs to establish construct validity. Items were retained on factors if they had high loadings (absolute values >.40), were not complex (ie, did not load on 2 or more different factors with a difference of .2), and contained eigenvalues >1, according to Kaiser’s rule17 on a scree plot.18 After the factor structure was confirmed, relationships between the latent and manifest variables were explored utilizing SEM the ‘‘Proc Calis’’ procedure. Structural equation modeling (SEM) serves 2 purposes in measurement development. First, the structure developed through the confirmatory factor analysis process can confirm results provided by the preliminary study.3 Second, latent factor structure intercorrelations were examined. Confirmatory factor analysis did not allow for the examination of highly
February 2014, Vol. 84, No. 2
•
© 2014, American School Health Association • 85
correlated structures or hierarchical structures in scale development (ie, an underlying theme or guiding concept to the whole measure). Structural equation models are deemed good representations of the theory based on parsimony, chi-square goodness of fit, descriptive fit indices (comparative fit index: CFI & Tuker-Lewis index: TLI), and alternative fit indices (root mean squared error of approximation: root mean square error of approximation [RMSEA] and residuals). With respect to parsimony, models with the fewest parameters to explain the relationship were retained. For the descriptive indices, fits of >.9 (preferably >.95) indicated a well-fitting model (CFI, TLI).19 For RMSEA, a fit of
