Research Quarterly for Exercise and Sport

ISSN: 0270-1367 (Print) 2168-3824 (Online) Journal homepage: http://www.tandfonline.com/loi/urqe20

Reliability and Validity Issues concerning LargeScale Surveillance of Physical Activity Richard A. Washburn, Gregory W. Heath & Allen W. Jackson To cite this article: Richard A. Washburn, Gregory W. Heath & Allen W. Jackson (2000) Reliability and Validity Issues concerning Large-Scale Surveillance of Physical Activity, Research Quarterly for Exercise and Sport, 71:sup2, 104-113, DOI: 10.1080/02701367.2000.11082793 To link to this article: http://dx.doi.org/10.1080/02701367.2000.11082793

Published online: 13 Feb 2015.

Submit your article to this journal

Article views: 13

View related articles

Citing articles: 29 View citing articles

Full Terms & Conditions of access and use can be found at http://www.tandfonline.com/action/journalInformation?journalCode=urqe20 Download by: [York University Libraries]

Date: 05 November 2015, At: 17:27

Washburn, Heath andJackson

Research Quarterly for Exercise andSport ©2000 bythe American Alliancefor Health, Physical Education, Recreation and Dance Vol.71, No.2, pp. 104-113

Reliability and Validity Issues Concerning Large-scale SurveiIlance of Physical Activity

Downloaded by [York University Libraries] at 17:27 05 November 2015

Richard A. Washburn, Gregory W Heath, Allen W Jackson

Keywords: physical activity, surveillance, reliability, validity

A

s Drs. Macera and Pratt (2000) indicated, large-scale surveillance of physical activity behaviors in the U. S. population is a necessity for planning, implementing, and evaluating health promotion and chronic disease prevention programs. An example of this type of surveillance is the Behavioral Risk Factor Surveillance System (BRFSS). The physical activity data generated from the system is based on telephone survey responses to questions about physical activity behaviors. Establishing the reliability and validity of physical activity questionnaires are difficult tasks without totally satisfactory solutions. The most difficult problem in establishing the validity of a physical activity survey is the lack of an accepted criterion measure of physical activity with which questionnaire results can be compared. Questionnaire validity studies must rely on criterion measures such as activity estimated by activity diaries, motion sensors, or energy expenditure estimated from diet records or doubly labeled water (Paffenbarger, Blair, Lee, & Hyde, 1993; Jacobs, Ainsworth, Hartman, & Leon, 1993). Alternatively, the association of questionnaire assessed physical activity with physiological parameters such as aerobic capacity, body fat, muscular strength, and high density lipoprotein cholesterol, variables thought to be associ-

Richard A. Washburn is with the Department of Kinesiology at the University of Illinois, Gregory W Heath is with the Cardiovascular Health BranchDivision of Adult and Community Health Centers for Disease Control andPrevention andAllen W Jacksonis with the Department of Kinesiology, Health Promotion, andRecreation at the University of North Texas.

104

ated with physical activity, are often used as indirect measures of validity (Paffenbarger et aI., 1993; Jacobs et aI., 1993). The association of questionnaire assessed physical activity with the risk for chronic disease, or with activity assessed by previously validated or widely used questionnaires, is also used as indirect evidence of validity. These approaches, while not optimal, are the only currently available practical alternatives. None of these alternatives provide a criterion measure of physical activity and hence the magnitude of the associations between questionnaire assessed physical activity and these criterion variables are not expected to be strong. Therefore, when evaluating the literature regarding the validity of a physical activity questionnaire both the magnitude of the associations and the consistency of results across studies should be considered. A construct validity approach to validity determination for physical activity questionnaires may be an alternative to concurrent validity approaches and contribute additional information to the truthfulness oflarge-scale surveillance data. Demonstrating convergent evidence and discriminant evidence from a multi-method approach could enhance the confidence in physical activity data. Using external criteria such as type of employment and membership in fitness programs to establish known physically active and inactive groups and then demonstrating that data gathered from physical activity questionnaires successfully discriminate between these groups would provide further evidence of the construct validity of the questionnaires. Physical activity assessment is compounded by the multivariate nature of physical activity behavior. Frequency, intensity, duration, and mode of activity are necessary to provide an accurate picture of physical activity behaviors. Conducting experimental research, manipulating frequency, intensity, duration, and mode, and determining the ability of physical activity questions to detect these known changes would be extremely valuable.

ROES: June 2000

Downloaded by [York University Libraries] at 17:27 05 November 2015

Washburn, Heath andJackson

The assessment of questionnaire reliability (test- retest) is more straight forward, however, reliability may be affected by both unreliability associated with the physical activity assessment instrument itself and true changes in physical activity that occur over time. It is therefore important to consider reliability over different time frames and consistency across study samples when evaluating the literature relative to questionnaire reliability. Another potential reliability issue relates to the concept of a physical activity scale. If a scale is developed to measure the total construct of physical activity behavior then the internal consistency of the scale should be determined and reported. The issues of developing better physical activity assessment instruments and establishing their validity and reliability have long been at the forefront of physical activity epidemiology and have been revisited and reawakened periodically over the last 10 years (Haskell et a!., 1992; LaPorte, Montoye, & Caspersen, 1985). Considerable work on physical activity assessment has been completed to date. However, in total we still have limited information on the validity of physical activity assessment instruments for use in general population samples either to assess the association of physical activity with chronic disease risk or for use in physical activity surveillance (Montoye, Kemper, Saris, & Washburn, 1995). As Dishman (1994) has indicated, the task of determining effective interventions for promoting adoption and maintenance of healthful physical activity behaviors will be impossible without reliable and valid measurements of the dependent variable, physical activity. The primary purpose of this paper is to present a review of the primary scientific literature available on the reliability and validity of physical activity questions useful for large-scale public health surveillance. The questions examined are from the Behavioral Risk Factor Surveillance System and questions from two similar surveys, the Minnesota Leisure Time Physical Activity Survey and the Harvard Alumni Physical Activity Survey. A secondary purpose of this paper is to present recommendations to guide further scientific research into the reliability and validity of questions and methods for surveillance of physical activityto answer important public health questions.

The Behavioral Risk Factor Surveillance Physical Activity Questions The Behavioral Risk Factor Surveillance System (BRFSS) is a telephone administered survey and was designed to provide state-specific estimates of the percentage of individuals with risk behaviors associated with the 10 leading causes of premature death among adults

RUES: June2000

in the United States. All information is self-reported. Questions focus on smoking practices, alcohol use, hypertension detection, height and weight, seat-belt use, sociodemographic variables, and special topics such as AIDs awareness. The present 10 questions addressing leisure-time physical activity have been used consistently since 1984. Information about the quality and quantity of a respondent's leisure-time physical activity is obtained from these 10 questions. The lead question asks whether a respondent has participated in any physical activities in the past month such as running, calisthenics, golf, yard work, or walking. For those respondents who answer yes to this lead question, they have the opportunity to answer a series of questions, which ask about the activity they spent the most time doing in the past month. The frequency in days and duration in hours and minutes are asked. In addition, for the activities ofwalking, running/ jogging, and swimming, information about the distance covered per session is asked. A second series of identical questions are asked about the physical activity they spent the second most time doing in the past month. The compilation of these responses allows for respondents to be scored for their amount of physical activity relative to the type, frequency, duration, and relative intensity.

Validity and Reliability ofSurveys Similar to the BRFSS The best estimate of the likely validity and reliability of the BRFSS physical activity questions comes from examining these qualities in physical activity surveys with characteristics similar to the BRFSS. The following section describes the available literature on the reliability and validity of two such physical activity assessment instruments: the Minnesota Leisure Time Physical Activity Survey (MLTPAS) (Taylor et a!., 1978) and the Harvard Alumni Physical Activity Survey (HAAS) (Paffenbarger, Hyde et a!., 1993). These two instruments were selected for comparison because they both assess physical activity during leisure time; request information regarding frequency, intensity and duration of activity; have comparatively well established validity and reliability; and their face validity has been established by studies showing associations between physical activity assessed by these instruments and the risk of chronic disease (Leon, Connett, Jacobs, & Rauramaa, 1987; Leon & Connett, 1991; Paffenbarger et a!., 1993). However, there are two important differences, in addition to the actual wording of the questions, between these instruments and the BRFSS questionnaire that need to be considered. First, the BRFSS requests recall of activity over the past month while the MLTPAS asks about activity over the past year

105

Downloaded by [York University Libraries] at 17:27 05 November 2015

Washburn, Heath andJackson

and the HAAS asks about usual activity. Second, BRFSS information is collected during a telephone interview, while the MLTPAS uses an in-person interview and the HAAS is self-administered. A comparison of the characteristics of the MLTPAS, HAAS and the BRFSS instruments is presented in Table 1. A summary of the literature on the validity and reliability the MLTPAS and the HASS is presented in Tables 2 - 5. Although the validity and reliability of HAAS and the MLTPAS are as well studied as any currently available physical activity surveys, the total number of studies and subjects evaluated is still quite small. Most study samples include predominately white, middle age, educated persons. There is limited data to support the reliability and validity of either instrument in younger, older, less educated, or minority people. Because the available criterion measures provide only estimates of the true physical activity level, the magnitudes of the correlations between questionnaire assessed physical activity and criterion measures tend to be low; generally in the range .30 to .50. The strength of these associations varies depending on the criterion measure used. Associations tend to be stronger for more direct (and potentially more accurate) criterion measures, such as those obtained from a detailed activity diary, when compared with indirect criteria, such as percent body fat or high density lipoprotein cholesterol level. Additionally, arguments can be made for and against the use of cardiovascular fitness as assessed by measurement of maximal oxygen uptake as a criterion measure for assessing the validity of physical activity in-

struments (Paffenbarger et aI., 1993). However, the associations between maximal oxygen uptake and both the MLTPAS and the HAAS are consistent and suggest that these instruments capture participation in more intense physical activities of the type likely to elicit changes in cardiovascular fitness. One could speculate that the BRFSS physical activity questions, with their focus on leisure time activity, may show a similar association with cardiovascular fitness, considering the hypothesized likelihood of more accurate recall of more intense, conditioning physical activity. To the extent that the BRFSS is similar to the MLTPAS and the HAAS, the validity results presented in Tables 2 and 3 offer indirect support for the validity of the BRFSS physical activity questions. However, it is imperative that additional work be completed to directly assess the validity of the BRFSS physical activity questions in samples of individuals representative ofthose in which this survey will be used. The validity of the BRFSS would be supported if the magnitude of the associations with similar criterion measures were comparable or better than those reported in Tables 2 and 3. The test-retest reliability of both the MLTPAS (Table 4) and the HAAS (Table 5) are acceptable and, as expected, are generally higher when the time interval between tests is reduced. When these results are considered in conjunction with the results reported later for the reliability of percent sedentary lifestyle and participation in regular aerobic activity as assessed by the BRFSS (see next section on BRFSS Questions), it is reasonable to speculate that the overall test-retest reliability of the

Table 1:Characteristics ofthe BRFSS, Minnesota Leisure lime Physical ActivitySurvey (MLTPAS), and Harvard Alumni ActivitySurvey (HAAS) Survey

Administration Mode

lime Frame and Measures

Information Requested

Outcome Variables

BRFSS

Telephone Interview

PastMonth lime/month

Activity Type (2 mostprevalent) Frequency

Activity categories: No activity, regularintense, regularsustained

MLTPAS

In-person interview

PastYear Months/year limes/month Minutes/time

Activity Type (Check list)

METS/day Kcal/day

HAAS

Self-administered

Usual/year limes/year Minutes/time

Blockswalked Stairs climbed Sport & Recreation

Kcal/week

The physical activity questions used in the BRFSS can be found at this Internet address: http://www.cdc.gov/nccdphp/brfss/brfsques-questionnaires.htm.

106

ROES: June 2000

Washburn, Heath andJackson

BRFSS is acceptable. However the reliability of more specific aspects of the BRFSS (distance walk/run/jog/swim, time of participation, frequency of participation) and indices calculated from this information should be evaluated in samples of varying age, ethnicity, and socioeconomic status.

Downloaded by [York University Libraries] at 17:27 05 November 2015

Validity ofthe BRFSS Physical Activity Questions Although the BRFSS physical activity survey is widely used to assess the prevalence of physical activity behavior and to track changes in activity behavior over time, there are no reports describing the validity of this survey (Caspersen & Merritt, 1995). Given the history and current state of knowledge regarding physical activity assessment, the lack ofvalidity information on the BRFSS physical activity questionnaire is not surprising or unique.

Reliability ofthe BRFSS Physical Activity Questions Three reports have addressed the reliability of the BRFSS physical activity questions. Stein, Lederman, and Shea (1993) studied the reliability of the BRFSS physical activity questions in two samples, (mean age ± SD, 40.7 ± 16.7 yrs.); respondents to the state of Massachusetts BRFSS in March 1992 (response rate 65%), and a separately recruited sample of Black and Hispanic Massachusetts residents (response rate 39%). Respondents ranged in age from the early 30s in Hispanics to the middle 40s in white non-Hispanics. Re-interviews were conducted with 86 of 122 respondents to the state survey (70%) and 124 of200 respondents to the minority survey (62%). Of the 210 completed interviews 95% were completed within 44 days, all re-interviews were completed within 64 days. The reliability of the percentage reporting a sedentary lifestyle (answered no to BRFSS Q. 10 - During the

Table 2: Minnesota Leisure Time Physical Activity Survey ValidityStudies

Comparison Method

Sample

Correlation Motion Sensors

Reference

Caltrac accelerometer

LSI motion monitor 4-week activity histories Physical activity records Energy intake Energy intake Doubly labeled water Treadmill endurance Treadmill endurance Submaximal treadmill (heart rate) Submaximal treadmill (heart rate) Peak oxygen uptake Peak oxygen uptake Percentbodyfat Harvard Alumni Stanford 7 DayRecall Framingham Activity Survey Baeckelndex Lipid Research Clinics Survey Health Insurance Plan of NY * p < .05

ROES: June 2000

28men 28men 49women 77 men and women 42men 28men and 50 women

.40*

Gretebeck et al., 1993

.18 .21 .45*

Jacobs et al., 1990 Richardson et al., 1994 LaPorte et al.,1982

.74* .47* 21 men .17 28men -.06 28men .26 Health-Related Physical Fitness Parameters 175 men .45* 175 men .41* 1513 men .10* 64men and women .45* 75men and women .47* 64men and women .45* -.24 75men and women Other Physical Activity Surveys .54* 21 men .48* .33* .36* .63* .52*

77 men and women

Jacobs et al., 1993 Jacobs et al., 1993 Albanes et al., 1990 Gretebeck et al., 1993 Gretebeck et al., 1993 Taylor et aI., 1978 Leon et al., 1981 Debacker et al., 1981 Jacobs et al., 1993 Richardson et al., 1994 Jacobs et al., 1993 Richardson et al., 1994 Albanes et al., 1990

107

Washburn, Heath andJackson

past month did you participate in any physical activities or exercise such as ... ) and percent reporting regular aerobic activity (BRFSS Q. 10/15 =yes; Q. 11/16 - What type of physical activity or exercise did you spend the most time doing during the past month? = aerobic activity, and Q. 14/19 - ...how many minutes or hours did you

usually keep at it = > 20 minutes) was assessed. The test-retest reliability of the activity categories was acceptable with kappa coefficients for sedentary behavior above 0.50 and kappa for participation in regular exercise above 0.45. There was not an appreciable difference in reliability across ethnic groups.

Downloaded by [York University Libraries] at 17:27 05 November 2015

Table 3: Harvard Alumni Activity Survey ValidityStudies Comparison Method

Sample

CaltracAccelerometer METS (7-day) Kcals (14 2-dayadministrations) METS Kcals/day METS/day LSI Motion Monitor

28 men 28 men,49women 28 men,49women 28men, 50 women 28men, 50 women

.70* -.03 .30* .17 .29*

Gretebeck et aI.,1993 Jacobs et aI.,1993 Jacobs et al., 1993 Ainsworth et aI.,1993 Ainsworth et aI., 1993

76women 225 postmenopausal women 130 postmenopausal women

.23*

.n.33*

Laporte et aI.,1982 Cauley et al., 1986 Cauley et al., 1987

28 men,50 women 28 men

.42* .44*

Ainsworth et aI.,1993 Gretebeck et aI., 1993

21 men

.32

Albanes et aI.,1990

28men

.39*

Gretebeck et aI.,1993

6-48 hr. activity records Energy intake (METS-7 davs) Energy intake (Activity - rest) Doubly labeled water (METS - 7 davs) Maximaltreadmill time

Maximal oxygen uptake

HDL cholesterol Bodymassindex Percent bodyfat Stanford 7-day recall Framingham Activity Survey Minnesota Leisure lime Activity Survey Baecke Physical Activity Index Health Insurance Plan of

Correlation Motion Sensors

Health and Health-related Physical Fitness Measures 56 Adventist men .38* 59 Non-Adventistmen .27* 56 Adventistwomen .07 34 Non-Adventist women .15 .29* 36 men 32women .46* 64men andwomen .52* 28 men and 50 women .60* 241 men .18* 332 women .23* 271 men -. 378 women -.17* -.44* 28 men and 50 women Other Physical Activity Surveys 21 men .09 .36*

"

Reference

Singh et al., 1996

Siconolfi et aI.,1985 Jacobs et aI.,1993 Ainsworth et aI.,1993 Washburn et aI.,1991 Washburn et aI., 1991 Ainsworth et al., 1993 Albanes et al., 1990

.54* .56* .53*

NY Lipid Research Clinic Baecke Physical Activity Index Minnesota Leisure lime Activity Survey *p < .05

108

130 post menopausal women 28men and 50 women

.81* .20*

Cauley et aI.,1987

.31*

Jacobs et al., 1993

RaES: June 2000

Downloaded by [York University Libraries] at 17:27 05 November 2015

Washburn, Heath andJackson

Shea, Stein, Lantigua, and Basch (1991) assessed the test-retest reliability (1O-21day interval) of the percent participation in physical activity during the past month (answered yes to BRFSS Q.lO) in 145 randomly selected subjects from 3 ethnic groups: white (N = 49, mean age = 38 yrs., completion rate 56.2%), blacks (N = 43, mean age = 41 yrs., completion rate 59.5%), and Hispanics (N = 53, mean age 34 yrs., completion rate 52%). The results indicated kappa coefficients about .57, with little difference across ethnic groups. Kappa for the entire sample was .59. Brownson et al. (1999) assessed the test-retest reliability of the BRFSS physical activity question over a 7 to 90 day interval in women age 40 years and over. The sample included 43 White, 60 Black, 51 American Indian/Alaska native, and 45 Hispanic participants in the U.S. Women's Determinants Study. Results showed kappa coefficients ranged from .26 for regular physical activity (participation in leisure-time physical activity ~ 5 time per week for ~30 minutes per session) to .51 for occupational activity (~ 300 minutes per week of vigorous job tasks, including walking, lifting/carrying, and other activities of similar exertion). For 3 of the 5 physical activity indices (no leisure activity, regular activity, and occupational activity) kappa coefficients were lower among black women than in any of the other racial/ethnic groups. Although ethnically diverse, these studies represent a small sample (157 whites, 167 blacks, 143 Hispanics, 51 American Indian/Alaskan native) with average or high levels of socioeconomic status as assessed by education and income. Neither of these reports addressed the reliability of Q. 12/17 (How far did you usually walk/ run/jog/swim?) orQ.13/18 (Howmanytimesperweek or per month did you take part in this activity during the

past month?), or the reliability of any physical activity indices that may be calculated using these questions. These issues, as well as the impact of respondent age should be considered in future studies on the reliability of the BRFSS physical activity questions.

Discussion and Recommendations The BRFSS physical activity questions have been used for over 15 years and have provided useful information regarding the physical activity habits of the U.S. population. However, there is an urgent need to directly measure the validity of the current BRFSS physical activity questions. Despite the lack of direct validity studies there appears to be no compelling reason to make major modifications in the basic BRFSS format in terms of assessing life-time physical activity. This conclusion is based on information reported earlier using indirect methods suggesting acceptable reliability and validity of similar surveys. These findings provide a high probability that the results of proposed studies will show the current BRFSS to have acceptable reliability and validity. There are obvious benefits to maintaining the integrity of the BRFSS questionnaire for continued use as a physical activity surveillance instrument. This section will briefly summarize the Healthy People (HP) 2000, Centers for Disease Control and Prevention (CDC), American College of Sports Medicine (ACSM), and the Surgeon General's physical activity and health objectives. It will estimate the likelihood that the current BRFSS physical activity questions will provide information sufficient to assess and track those objectives. In those instances where the current

Table 4: Minnesota Leisure Time Physical Activity Survey Reliability Studies Sample 150 men 140 men/women 129 men

322 women

78 men/women 78men/women

ROES: June 2000

Time Interval 2 weeks 35-37 days 10years 8-9years 5-7years 3-4years 1-2 years 10years 8-9years 5-7 years 3-4years 1-2years 12months 1 month 1 month 1year

Test-retest correlation

.79 .88 .09 -.22 .14 .52 .47 .30 -.04 .29 .46 .36 .69 .92

.72

Reference Folsom et aI., 1986 Blair et al., 1991

Blair et al., 1991

Jacobs et aI., 1993 Richardson et al.,1994

.50

109

Washburn, Heath andJackson

BRFSS questions appear inadequate, suggestions for modification and assessment of the validity and reliability of those modifications will be offered.

Downloaded by [York University Libraries] at 17:27 05 November 2015

Healthy People 2000 objective 1.4 and 1.5 The current BRFSS physical activity questions obtain information that is sufficient to evaluate progress toward both HP 2000 objectives 1.4 ( ...who engage in vigorous physical activity that promotes the development and maintenance of cardiorespiratory fitness 3 or more days per week for 20 or more minutes per occasion) and 1.5 ( ... reduce the proportion of people who engage in no leisure time physical activity). These categories are currently assessed by the BRFSS with the classifications "regularly active - intensive" or "no leisure time activity". However, as previously discussed, there is no information on the validity, and only limited information on the reliability of the current BRFSS. This deficiency should be rectified by conducting studies of both reliability and validity of the BRFSS in population samples with adequate representation of age, income, and ethnicity to ensure that the BRFSS adequately assesses physical activity in the specific population groups targeted by the objectives. It is important to consider not only the reliability and validity of the BRFSS, as currently assessed, but also the reliability and validity of alternative scoring procedures and outcome variables.

Healthy People 2000 objective 1.3, and theACSM/CDC and Surgeon General's Recommendations The current BRFSS physical activity questions are not suitable for effectively assessing progress toward and HP

2000 objective 1.3 ( ... people who engage regularly, preferably daily, in light to moderate physical activity for at least 30 minutes per day) or the ACSM/CDC recommendation (Every adult should accumulate 30 minutes or more of moderate intensity activity on most, preferably all, days of the week.). Both these recommendations focus on participation in light to moderate intensity physical activity. In the context of these recommendations light to moderate activity is considered to be at an intensity of 3 - 6 METS, equivalent to sustained brisk walking for most adults. While many light to moderate activities such as walking, swimming, cycling, dancing etc. are generally considered in the context of physical activity or exercise, most people also perform considerable light to moderate intensity activity during various domestic and or occupational activities. Participation in these types of activity likely varies as a function of both age and socioeconomic status. It is evident that some light to moderate activities, for example calisthenics, golf, and gardening, which are suggested in question 10 ( ... did you participate in physical activities or exercises such as.. .? ), will be assessed by the BRFSS. However, the current BRFSS ignores most light to moderate household or occupational activity. Given the potential health benefits of light to moderate activity, as suggested by the current ACSM/CDC recommendation, an argument can be made that questionnaire items should be developed or modified to more adequately assess activity of this type. Potentially, a light-moderate activity module should be developed and used in the BRFSS. However, the assessment oflight and moderate physical activity is problematic, particularly over longer time periods, where recall of such activities is poor. In addition, the number and diversity of physi-

Table 5: Harvard Alumni Activity Survey Reliability Studies Sample 14post-menopausal women 59 post-menopausal women 28 Adventistmen 30 Non-Adventist men 41 Adventistwomen 12Non-Adventist women 59men/women 261 men 272 women Total sample with no change in activity pattern

110

Time Interval 4 weeks

Test-retest Correlation .76

Reference Cauley et al., 1987

52weeks

.59

Cauley et al., 1987

6 weeks 6 weeks

.56 .80

Singh et al., 1996

6 weeks 6 weeks

.76 .76

8 months 1 month 7-12 weeks 7-12weeks 7-12weeks

.50 .72 .56 .58 .69

Jacobs et al., 1993 Washburn et al.,1991

ROES: June 2000

Downloaded by [York University Libraries] at 17:27 05 November 2015

Washburn, Heath andJackson

cal activities that can be considered light to moderate is enormous (Ainsworth, Haskell et aI., 1993). Results from previous attempts to validate assessments oflight to moderate activities and to develop instruments specific to household activity are not encouraging. For example, Richardson, Leon,Jacobs, Ainsworth, and Serfass (1994) reported correlations of.1O and -.06 between moderate activity (> 4.0 and < 5.0 METS) estimated from a 48-hour physical activity record and moderate and light « 4.0 METS) activity assessed with the Minnesota Leisure Time Physical Activity Survey. Richardson et al. (1994) also reported correlations of - .03 and .25 between moderate and light activity estimated from a 48-hour physical activity record and household activity over the past year assessed by a household activity supplement that used the same basic format of the MLTPAS. Initial efforts to develop reliable and valid questions to measure moderate and strength training physical activities with the BRFSS have been reported (Ainsworth et al., 1999; Jones et aI., 1999). For proposed walking questions, test-retest reliability estimates ranged from .42 to .61 and concurrent validity estimates from .18 to .43 (Jones et aI., 1999). In evaluating a new strength question, the test-retest reliability estimate was .69 and a concurrent validity coefficient with a l-week physical activity log was .58 (Ainsworth et aI., 1999). Adopting these new questions would allow the BRFSS to assess more of the national physical activity goals.

Recommendations for Assessing the Validity andReliability of the Current BRFSS Concurrent validity could most efficiently be evaluated by comparing BRFSS-assessed activity with cardiovascular fitness and with physical activity assessed by an activity diary. These two criterion measures provide both a physiologic (fitness) as well as an estimate of actual participation in physical activity (diary), are relatively easy to assess, and consistently show the strongest associations with questionnaire-assessed physical activity (Tables 2 and 3). The use of additional criterion measures such as doubly-labeled water, motion sensors, or other physiologic parameters such as HDL cholesterol, or body composition indices may be informative, but would not contribute sufficient useful information to warrant the additional time and expense required to complete these measurements. A validity study may be conducted most cost-efficiently by forming cooperative ventures between state health departments that currently administer the BRFSS, and university or hospital-based investigators. Respondents to a regular cycle of the BRFSS who are located near a cooperating testing facility could be solicited to participate in either a validity or a reliability study or both. Most likely, multiple study sites would be needed to obtain adequate coverage of the general population and to

ROES: June 2000

ensure sufficient samples of the major population subgroups of interest: older individuals, ethnic minorities (Blacks, Hispanics), and low socioeconomic groups. Sample size estimates should be based on the ability to detect validity and reliability coefficients with reasonable confidence intervals similar to those reported in the literature for the association of questionnaire assessed physical activity and similar criterion measures. Although the HP2000 objectives target other population subgroups, including children and adolescents, people with disabilities, and American Indian/Alaska Natives, given the likely size and difficulty in sampling these groups it is probably neither cost efficient or necessary to assess validity and reliability of the BRFSS in these SUbgroups. Other available surveys, including the Youth Risk Behavior Survey and potentially the re-administration of the National Children and Youth Fitness Survey may be useful in monitoring physical activity in children and adolescents. As we mentioned earlier in this report, construct validity approaches to evaluating the truthfulness of BRFSS assessments of physical activity could be enlightening. These would be especially valuable in experimental studies where physical activity behaviors are manipulated in terms of frequency, intensity, duration, and type and the BRFSS questions can be demonstrated to detect those predictable changes. We currently have limited data to support the reliability of the BRFSS. Additional reliability testing (test/ retest interval 2-4 weeks) should be conducted in the target populations of interest and include evaluation of specific BRFSS components as well as physical activity indices calculated from BRFSS data. A test-retest interval of 2 to 4 weeks should be considered. This in terval is short enough to minimize actual changes in behavior but long enough to minimize the influence of the initial administration. The importance of reliability can not be underestimated. The standard errors of measurement (SEMs) of physical activity, which are a function of the reliability of the measurement of physical activity and the standard deviation of the physical activity data, demonstrate this point. Jacobs et al. (1993) evaluated the reliability and validity of 10 physical activity questionnaires. For the MLTPAS, they reported a 12-month test-retest reliability of .69 and a l-month test-retest reliability of .92. For the 50 females in the study, the average total physical activity was 396.5 (MET*min*dayl) and the standard deviation was 161. The SEM for the 12-month reliability estimate would have been 90 and for the l-month reliability estimate, 45. Ninety-five percent confidence intervals for individual physical activity scores would have been approximately ± 180 and ± 90, respectively. It is obvious that in large-scale surveillance studies that even moderately reliable assessments of physical activity could lead to large numbers of misclassifications of individuals into various health-related categories of physical activity behavior.

111

Washburn, Heath andJackson

Thus, research trying to relate physical activity to various health outcomes would be subject not only to sampling errors but also to measurement errors of physical activity in trying to identity those relations. The issue of potential misclassifications is further demonstrated by the observation that obese inactive individuals, a group that a surveillance system should identity, reported physical activity levels 51 % higher than they actually achieved (Lichtman et al., 1992).

Downloaded by [York University Libraries] at 17:27 05 November 2015

Summary The BRFSS is our major source of population-based information on physical activity in the United States. The top priority for the immediate future should be to make sure that the information obtained from the BRFSS in its current form is both reliable and valid. As outlined in this report, this task could be completed in relatively short time frame and at reasonable cost in terms of both financial and human resources. The development of an additional module to assess light-to-moderate activity and strength is currently underway with support from the CDC. As suggested in this report, this task may prove to be extremely difficult, or it may be difficult to develop an instrument of this type given the constraints of a population based telephone survey. The current focus on the health benefits of moderate activity and the interest in tracking this type of activity behavior on a population basis as it relates to national physical activity objectives both argue in favor of developing a physical activity assessment instrument that provides a valid and reliable measurement of moderate activity. Given the complexity and uncertainty of this task it is important to develop a procedure that will allow for periodic evaluation of progress and the potential for success to minimize the risk of making a considerable financial investment in an instrument that may be oflimited utility.

References Ainsworth, B. E., Haskell, W. L., Leon, A S.,jacobs, D. R,jr., Montoye, H.]., Sallis,]. F., & Paffenbarger, R. S.,jr. (1993). Compendium of physical activities: classification of energy costs of human physical activities. Medicine and Science in Sports and Exercise, 25, 71-80. Ainsworth, B. E., Leon, A S., Richardson, M. T.,jacobs, D. R., & Paffenbarger, R S., jr. (1993). Accuracy of the college alumnus physical activity questionnaire. Journal of Clinical Epidemiology, 46, 1403-1411. Ainsworth, B. E.,jones, D. A, Macera, C. A., Pratt, M., Kinsey, C. D., & Thompson, R. (1999). Evaluation of a strength question in the behavioral risk factor surveillance system (BRFSS) (Abstract). Medicine and Science in Sports and Exer-

112

cise, 31, S143. Albanes, D., Conway, ]. M., Taylor, P. R., Moe, P. w, &judd,]. (1990). Validation and comparison of eight physical activity questionnaires. Epidemiology, 1,65-71. Bassett, D. R.,jr., Ainsworth, B. E., Leggett, S. R., Mathien, C. A, Main,]. A, Hunter, D. C., & Duncan, G. E. (1996). Accuracy of five electronic pedometers for measuring distance walked. Medicine and Science in Sports and Exercise, 28, 1071-1077. Blair, S., N., Dowda, M., Pate, R. R., Kroenfeld,]., Howe H.G., j., Parker, G., Blair, A, & Fridinger, F. (1991). Reliability of long-term recall of participation in physical activity by middle-aged men and women. Americanjournal ofEpidemiology, 133, 266-275. Brownson, R. C., Eyler, A A, King, A. C., Yuh-Ling, S., Brown, D. R., & Homan, S. M. (1999). Reliability of information on physical activity and other chronic disease risk factors among US women aged 40 years or older. American Journal ofEpidemiology, 149, 379-391. Caspersen, C.]., & Merritt, R. K. (1995). Physical activity trends among 26 states, 1986-1990. Medicine and Science in Sports and Exercise, 27,713-720. Cauley, j. A, LaPorte, R. E., Black-Sandler, R., Orchard, T.]., Slemenda, C. w., & Petrini, A. M. (1986). The relationship of physical activity to high density lipoprotein cholesterol in postmenopausal women. Journal of Chronic Disease, 39, 687-697. Cauley, j. A., LaPorte, R E., Black-Sandler, R., Schramm, M., M., & Kriska, A. M. (1987). Comparison of methods to measure physical activity in postmenopausal women. American Journal of Clinical Nutrition, 45, 14-22. DeBacker, G., Kornitzer, M., Sobolski,]., Dramaix, M., Degre, S., DeMarneffe, M., & Denolin, H. (1981). Physical activity and physical fitness levels of Belgian males aged 40-55 years. Cardiology, 67, 110-128. Dishman, R. K. (1994). The measurement conundrum in exercise adherence research. Medicine and Science in Sports and Exercise, 26, 1382-1390. Folsom, A R., jacobs, D. R, Jr., C.]., C, Gomez-Marin, 0., & Knudsen,]. (1986). Test-retest reliability of the Minnesota leisure time physical activity questionnaire. Journal of Chronic Disease, 39, 505-511. Haskell, W. L., Leon, A S., Caspersen, C. J. Froelicher, V. F., Hagberg,]. M., Harlan, W., Holloszv, ], 0., Regensteiner, ]. G., Thompson, P. D., Washburn, R. A, & Wilson, P. W. F. (1992). Cardiovascular benefits and assessment of physical activity and physical fitness in adults. Medicine and Science in Sports and Exercise, 24(Suppl.), S20l-S220. jacobs, D. R.,jr., Ainsworth, B. E., Hartman, rj., & Leon, AS. (1993). A simultaneous evaluation of 10 commonly used physical activity questionnaires. Medicine and Science in Sport and Exercise, 25, 81-91. jones, D. A, Ainsworth, B. E., Macera, C. A, Pratt, M., Kinsey, C. D., & Morgan, A (1999). Reliability and validity ofwalking questions in the behavioral risk factor surveillance system (BRFSS) (Abstract). Medicine and Science in Sport and Exercise, 31, S232. LaPorte, R E., Cauley, ]. A., Kinsey, C. M., Corbett, W., Robert, R, Black-Sandler, R., Kuller, L. H., & Falkel,j. (1982). The epidemiology of physical activity in children, college students, middle aged men, menopausal females and monkeys. Journal of Chronic Disease, 35, 787-795.

ROES: June 2000

Downloaded by [York University Libraries] at 17:27 05 November 2015

Washburn, Heath andJackson

LaPorte, R E., Montoye, H.J., & Caspersen, c.j. (1985). Assessment of physical activity in epidemiologic research: Problems and prospects. Public Health Reports, 100, 131-146. Leon, A. S., & Connett,J. (1991). Physical activity and 10.5 year mortality in the multiple risk factor intervention trial. InternationalJournal ofEpidemiology, 20, 690-697. Leon, A. S., Connett,J.,Jacobs, D. RJ., & Rauramaa, R. (1987). Leisure-time physical activity levels and risk of coronary heart disease and death: The Multiple RiskFactor Intervention Trial..Joumal ofthe American Medical Association, 258, 2388-2395. Leon, A. S., Jacobs, D. R., Jr., DeBacker, G., & Taylor, H. L. (1981). Relationship of physical characteristics and life habits to treadmill exercise capacity. American Journal of Epidemiology, 113, 653-660. Lichtman, S. W., Pisarska, K., Berman, E. R, Pestone, M., Dowling, H., Offenbacher E., WeIsel, H., Heshka, S., Matthews, D. E., & Heymsfield, S. B. (1992). Discrepancy between self-reported and actual calorie intake and exercise in obese subjects. New EnglandJoumal of Medicine, 327, 1893-1898. Macera, C. A., Pratt, M. (2000). Public health surveillance of physical activity. Research Quarterlyfor Exercise and Sport, 70, (in press). Montoye, H.J., Kemper, H. C. G., Saris, W. H. M., & Washburn, R. A. (1995). Measuring physical activity and energy expenditure. Champaign, IL: Human Kinetics. Paffenbarger, R. S., Jr., Blair, S. N., Lee, I.-M., & Hyde, R. T. (1993). Measurement of physical activity to assess health effects in free-living populations. Medicine and Science in Sports and Exercise, 25, 60-70. Paffenbarger, R. S.J., Hyde, R. T.,Wing, A w, Lee, l.-M.,Jung, D. L., & Kampert,J. B. (1993). The association of changes in physi-

RDES: June 2000

cal activityleveland other lifestylecharacteristics with mortality among men. New EnglandJoumal ofMedicine, 328, 538-545. Richardson, M. T., Leon, A. S.,Jacobs, D. R, Ainsworth, B. E., & Serfass, R (1994). Comprehensive evaluation of the Minnesota Leisure Time Physical Activity Questionnaire. Journal of Clinical Epidemiology, 47, 271-281. Shea, S., Stein, A. D., Lantigua, R, & Basch, C. E. (1991). Reliability of the Behavioral Risk Factor Survey in a triethnic population. AmericanJournal ofEpidemiology, 133, 489-500. Siconolfi, S. F., Lasater, T. M., Snow, R C. K., & Carleton, R A. (1985). Self-reported physical activity compared with maximal oxygen uptake. American Journal of Epidemiology, 122, 101-105. Singh, P. N., Tonstad, S., Abbey, D. E., & Fraser, G. E. (1996). Validity of selected physical activity questions in white Seventh-day Adventists and non-Adventists. Medicine and Science in Sports and Exercise, 28, 1026-1037. Stein, A. D., Lederman, R I., & Shea, S. (1993). The Behavioral Risk Factor Surveillance System Questionnaire: Its reliability in a statewide sample. AmericanJournal ofPublic Health, 83, 1768-1772. Taylor, H. L.,Jacobs, D. R.,Jr., Schucker, B., Knudsen,]., Leon, A. S., & Debacker, G. (1978). A questionnaire for the assessment ofleisure time physical activities. Journal ofChronic Disease, 31,741-755. Washburn, R. A., Smith, K. W., Goldfield, S. R. W., & McKinlay,]. B. (1991). Reliability and physiologic correlates of the Harvard Alumni Activity Survey in a general population. Journal of Clinical Epidemiology, 44, 1123-1129.

173