IP Online First, published on January 23, 2015 as 10.1136/injuryprev-2014-041399 Brief report
Assessing the reliability and validity of direct observation and traffic camera streams to measure helmet and motorcycle use Heather N Zaccaro,1 Emily C Carbone,2 Nishita Dsouza,2 Michelle R Xu,3 Mary C Byrne,2 John D Kraemer3 ▸ Additional material is published online only. To view please visit the journal online (http://dx.doi.org/10.1136/ injuryprev-2014-041399). 1
Department of International Health, Georgetown University, Washington, DC, USA 2 Department of Human Science, Georgetown University, Washington, DC, USA 3 Department of Health Systems Administration, Georgetown University, Washington, DC, USA Correspondence to Professor John D Kraemer, Department of Health Systems Administration, Georgetown University, 231 St Mary’s Hall, 3700 Reservoir Road, Washington, DC 20057, USA; [email protected] Received 12 August 2014 Revised 9 December 2014 Accepted 7 January 2015
To cite: Zaccaro HN, Carbone EC, Dsouza N, et al. Inj Prev Published Online First: [ please include Day Month Year] doi:10.1136/injuryprev2014-041399
ABSTRACT There is a need to develop motorcycle helmet surveillance approaches that are less labour intensive than direct observation (DO), which is the commonly recommended but never formally validated approach, particularly in developing settings. This study sought to assess public traffic camera feeds as an alternative to DO, in addition to the reliability of DO under field conditions. DO had high inter-rater reliability, κ=0.88 and 0.84, respectively, for cycle type and helmet type, which reinforces its use as a gold standard. However, traffic camera-based data collection was found to be unreliable, with κ=0.46 and 0.53 for cycle type and helmet type. When bicycles, motorcycles and scooters were classified based on traffic camera streams, only 68.4% of classifications concurred with those made via DO. Given the current technology, helmet surveillance via traffic camera streams is infeasible, and there remains a need for innovative traffic safety surveillance approaches in low-income urban settings.
BACKGROUND Motorcyclists constitute an increasing share of road traffic injuries (RTIs), particularly in developing countries where self-reported and observational data consistently indicate low helmet use rates.1–8 Direct observation (DO) is the preferred methodology for measuring helmet use as it is not subject to error introduced by response bias in selfreported data, though no published study has assessed the reliability of motorcycle helmet DO under field conditions in high-income countries or low-income countries (HICs or LICs). In LICs, use of DO is constrained by high resource costs, including time, personnel and transportation, limiting the availability of helmet surveillance data. To conduct DO more efficiently, studies in HICs have set up high-resolution cameras to facilitate data collection;9 10 however, in LICs and nonresearch settings in HICs, this solution is often considered impractical as the up-front expense of high-quality cameras is likely to limit or disincentivise their use. In lieu of setting up new cameras, the authors of this study sought to test the feasibility of using pre-existent traffic cameras, which have proliferated in urban areas in HICs and LICs, to collect data on motorcyclists. This study had three objectives: 1. To empirically verify the inter-rater reliability of direct field observation 2. To assess the reliability of data collection using public traffic cameras
3. To validate traffic camera-based data collection against the presumed gold standard of DO.
METHODS Study design To assess the reliability of DO and validate the use of traffic cameras, four trained team members simultaneously collected data on motor/bicyclists at a high-traffic intersection in northwest Washington, DC, USA: two via DO and two via online traffic camera streams. Data collection occurred on weekday mornings between 07:45 and 08:45 (a high-volume period to increase sampling efficiency) from October to December 2013. The observation site was selected because it was monitored by a highquality, publicly available traffic camera; there existed a safe, unobstructed vantage point for DO of the roadway; and it was located conveniently for the study team. Data were collected on persons riding bicycles, motorcycles or scooters as they passed through a forked, merging intersection onto a higher volume road (see online supplementary appendix 1). The real-time traffic camera feed was accessed from the District of Columbia Department of Transportation website (app.ddot.dc.gov) where image quality was comparable with that available online from LICs. The feed consisted of a series of static, colour images refreshing every 2 s. The camera used in this study provided an obstructionfree overhead vantage of the intersection and was located lower to the ground than most other DC cameras. The observation instrument was designed to enable rapid recording of observed motor/bicyclists and their characteristics. It was pilot tested through DO and online for 90 min each then revised accordingly (see online supplementary appendix 2). Data collectors received training on the protocol prior to initiating the study, including classroom training and 90 min of supervised data collection by DO and traffic camera. Additionally, they received a reference sheet to assist with the categorisation of cycle and helmet types. The importance of independent observation was emphasised during this training session to avoid biased estimates of reliability. Field and online observers synchronised starting times of each data collection session by phone. Collectors also recorded the time observed and colour of each motorbike to ensure observations were matched correctly. Technical problems such as camera misalignment or failure of the video stream were noted on the instrument and affected observations were excluded from analysis. Although most
Zaccaro HN, et al. Inj Prev 2015;0:1–3. doi:10.1136/injuryprev-2014-041399
Copyright Article author (or their employer) 2015. Produced by BMJ Publishing Group Ltd under licence.
1
Brief report data collection sessions involved four observers, during four sessions, data were collected either via DO or traffic cameras due to inclement weather or scheduling conflicts. Georgetown University’s IRB determined that human subjects’ committee oversight was not required for this study.
Variables The following variables were collected: cycle type, helmet usage and type, and whether the cyclist weaved between lanes. Cycle type was categorised as bicycle, motorcycle or scooter. Motorbikes, which include motorcycles and scooters, were differentiated based on the placement of the rider’s feet (out to the side on a motorcycle versus forward on a scooter) and wheel size. Helmet type included bicycle helmets, full-head helmets, half-head helmets and other. Consistent with the approach used by the National Highway Traffic Safety Administration, full-head and half-head motorcycle helmets were distinguished based on whether or not they covered the user’s ears.11
Analysis Data were entered into Microsoft Excel, then exported to Stata V.13.1 (StataCorp: College Station, Texas, USA) for statistical analyses. Every 10th record was visually inspected for quality assurance prior to analysis. Inter-rater reliability was calculated separately for field-based and camera-based observation using κ statistics. Percent agreement between DO and online data collection was assessed for variables with κ>0.6 with DO as the assumed gold standard.10 If DO observers disagreed about an observation, it was excluded from cross-methodological comparison.
RESULTS A total of 530 observations of 185 motor/bicyclists were recorded during 56 person-hours of data collection (table 1). In all, 58.5% of observations occurred in the field (N=310), and 41.5% were viewed online (N=220). Motorcycles comprised the majority of observations (64.3%), followed by scooters (18.8%), and bicycles (16.8%). Full-head helmets were most common (78.1%), then bike helmets (16.5%) and half-
Table 1 Distribution of selected characteristics of observations (N=530) Indicator Location Field Traffic camera Vehicle type Bicycle Motorcycle Scooter Wearing helmet Yes No Helmet type Bike helmet Full-head helmet Half-head helmet Other Between lanes Yes No
2
N
Percent
310/530 220/530
58.5 41.5
92/516 329/516 95/516
16.8 64.3 18.8
512/519 7/519
98.7 1.3
83/503 393/503 26/503 1/503
16.5 78.1 5.2 0.2
71/514 443/514
13.8 86.2
head helmets (5.2%). Most cyclists (86.2%) remained in one lane, while 13.8% wove between lanes. DO’s inter-rater reliability (table 2) was high: the κ statistic for differentiating bicycles from motorbikes was 0.98 (95% CI 0.82 to 1.00) and decreased marginally when differentiating motorcycles from scooters (κ=0.88, 95% CI 0.76 to 0.99). Helmet type had a κ statistic of 0.84 (95% CI 0.71 to 0.98); however, inter-rater reliability for helmet use, defined as the presence or absence of a helmet, could not be properly assessed because near universal helmet use caused very high expected agreement for this variable. The reliability of DO for whether or not cyclists wove between lanes was low (κ=0.31, 95% CI of 0.15 to 0.47). Traffic camera-based observation was comparatively less reliable. Agreement was high when differentiating bicycles from motorbikes (κ=0.79, 95% CI 0.59 to 0.98), but not when distinguishing among bicycles, motorcycles and scooters (κ=0.46, 95% CI 0.32 to 0.61). Helmet type had a κ of 0.53 (95% CI 0.36 to 0.69), and whether or not motor/bicyclists wove between lanes had stronger agreement than via DO (κ=0.48, 95% CI 0.29 to 0.67). Due to the aforementioned lack of variation in helmet use, camera-based observation exhibited less than chance agreement for this variable (κ=−0.03, 95% CI −0.21 to 0.15). Agreement for cycle type variable was assessed between DO and camera-based observation using the former as the goldstandard classification (table 3). Bicycles were missed relatively often in the traffic camera feeds: 84.4% (95% CI 67.2% to 94.7%) of bikes seen in the field were identified by online observers compared with 97.8% (95% CI 93.7% to 99.5%) of motorbikes, meaning approximately 15.6% of bikes and 2.2% of motorbikes seen in the field were missed on camera. Furthermore, 96.3% of cycles identified as motorbikes in the field were classified correctly on camera (95% CI 91.6% to 98.8%) compared with only 75.0% of bikes (95% CI 56.5% to 88.5%). Error increased substantially when differentiating motorcycles from scooters: although 96.7% (95% CI 90.6% to 99.3%) of motorcycles and 100.0% (95% CI 90.3% to 100.0%) of scooters observed in the field were detected on camera, only 80.0% (95% CI 70.2% to 87.7%) and 33.3% (95% CI 18.6% to 51.0%), respectively, were classified as the correct type of motorbike. Overall, 94.9% (95% CI 90.3 to 97.8%) of cyclists observed in the field were seen on camera, and 68.4% (95% CI 60.5% to 75.5%) were categorised correctly.
DISCUSSION This study empirically verifies the high inter-rater reliability of DO: κ statistics exceeded 0.8 for most variables, including both cycle and helmet type, which is generally considered to be very good agreement.12 Despite DO’s general acceptance,11 it has, to Table 2 Inter-rater reliability of the direct observation of cyclists in the field and via traffic camera Field-based observation
Traffic camera-based observation
Variable
κ Statistic
95% CI
κ Statistic
Vehicle type (bike vs motorbike) Vehicle type (bike vs motorcycle vs scooter) Helmet type Between lanes Wearing helmet
0.98
0.82 to 1.00
0.79
0.59 to 0.98
0.88
0.76 to 0.99
0.46
0.32 to 0.61
0.84 0.31 0.00
0.71 to 0.98 0.15 to 0.47 0.00 to 0.00
0.53 0.48 −0.03
0.36 to 0.69 0.29 to 0.67 −0.21 to 0.15
95% CI
Zaccaro HN, et al. Inj Prev 2015;0:1–3. doi:10.1136/injuryprev-2014-041399
Brief report Table 3 Agreement between traffic camera-based data collection and direct field observation
Bicycles Motorbikes* Motorcycles Scooters Total
Percent observed on camera (95% CI)
Percent correctly identified on camera (95% CI)
84.4 97.8 96.7 100.0 94.9
75.0 96.3 80.0 33.3 68.4
(67.2 to (93.7 to (90.6 to (90.3 to (90.3 to
94.7) 99.5) 99.3) 100.0) 97.8)
(56.6 to (91.6 to (70.2 to (18.6 to (60.5 to
88.5) 98.8) 87.7) 51.0) 75.5)
*Motorbikes is a broad category including both motorcycles and scooters.
our knowledge, never been formally validated in the literature. While strong reliability does not inherently establish validity, it is plausible to interpret high agreement as field observers both observing the same, accurate characteristics. On the other hand, traffic camera-based observation had substantial to poor agreement, with κ statistics ranging from 0.46 to 0.79.12 Due to high compliance with local helmet laws, the inter-rater reliability of helmet use versus non-use could not be properly assessed either in the field or via traffic camera. While this study’s results indicate the inability to use traffic cameras to categorise helmet type, the plausibility of using this methodology for the purpose of evaluating helmet use prevalence could be reassessed in locations such as LICs where helmet use rates are typically low;2–8 however, considering the low reliability of traffic camera-based observation, this may not be worthwhile. This study provides empirical support for the presumed validity of DO, which is valuable in settings with the ability to commit adequate personnel and time to safety surveillance. However, surveillance is challenging in LICs where resources are scarce and the burden of RTIs is high.13 Though publicly available traffic camera feeds appear not to be a labour-saving approach of adequate quality, there remains a need to develop more feasible safety intervention surveillance approaches. When higher resolution camera feeds become more widely available in LICs, this method should be reassessed. Data collection to inform and evaluate road safety interventions in LICs will necessarily require innovative, efficient approaches.14
What this study adds ▸ The inter-rater reliability of direct observation was high, which supports the use of this method as the gold standard for helmet surveillance. ▸ Traffic camera-based data collection was not found to be reliable and is not currently a viable alternative for jurisdictions lacking the resources for direct observation. There remains a need to develop innovative approaches for validly assessing helmet use in low-resource settings.
Contributors HNZ and JDK designed the study. All authors participated in data collection. Data entry and analysis were performed by HNZ while JDK performed quality assessment of the data. HNZ and JDK drafted the manuscript and all other authors revised it critically. All authors approved the final manuscript for publication. Competing interests None. Provenance and peer review Not commissioned; externally peer reviewed. Data sharing statement Original data are available from the corresponding author and will be provided upon request.
REFERENCES 1 2
3
4
5
6
7
8
What is already known on the subject ▸ Motorcyclists account for an increasing share of road traffic injuries, particularly in low-income countries where self-reported and observed rates of helmet usage are consistently low. ▸ There is a scarcity of data in low-income settings to inform and assess road safety interventions, partially due to the excessive resource cost of direct observation and a lack of innovative, more efficient methodologies. ▸ Though it is usually considered the gold standard, direct observation has never been formally validated for motorcycle helmet use.
Zaccaro HN, et al. Inj Prev 2015;0:1–3. doi:10.1136/injuryprev-2014-041399
9 10
11
12 13
14
Bachani AM, Koradia P, Herbert HK, et al. Road traffic injuries in Kenya: the health burden and risk factors in two districts. Traffic Inj Prev 2012;13(Suppl 1):24–30. Arosanyin GT, Olowosulu AT, Oyeyemi GM. An examination of some safety issues among commercial motorcyclists in Nigeria: a case study. Int J Inj Contr Saf Promot 2013;20:103–10. Chalya PL, Mabula JB, Ngayomela IH, et al. Motorcycle injuries as an emerging public health problem in Mwanza City, north-western Tanzania. Tanzan J Health Res 2010;12:214–21. Nzegwu MA, Aligbe JU, Banjo AAF, et al. Patterns of morbidity and mortality amongst motorcycle riders and their passengers in Benin-City Nigeria: one-year review. Ann Afr Med 2008;7:82–5. Ichikawa M, Nakahara S, Phommachanh S, et al. Roadside observation of secondary school students’ commuting to school in Vientiane, Laos. Int J Inj Contr Saf Promot Published Online First: 18 Oct 2013. Ackaah W, Afukaar F, Agyemang W, et al.; Road Traffic Injuries Research Network Multicenter Collaborators. The use of non-standard motorcycle helmets in low- and middle-income countries: a multicentre study. Inj Prev 2013;19:158–63. Bachani AM, Branching C, Ear C, et al. Trends in prevalence, knowledge, attitudes, and practices of helmet use in Cambodia: results from a two year study. Injury 2013;44(Suppl 4):S31–37. Ackaah W, Afukaar FK. Prevalence of helmet use among motorcycle users in Tamale Metropolis, Ghana: an observational study. Traffic Inj Prev 2010;11:522–5. Basch CH, Zagnit EA, Rajan S, et al. Helmet Use Among Cyclists in New York City. J Community Health 2014;39:956–8. Constant A, Messiah A, Felonneau M-L, et al. Investigating helmet promotion for cyclists: results from a randomised study with observation of behaviour, using a semi-automatic video system. PLoS ONE 2012;7:e31651. Pickrell TM, Ye TJ. Motorcylce Helmet Use in 2012—Overall Results. Report No. DOT HS 811 759. Washington, DC: National Highway Traffic Safety Administration, 2013:1–5. Landis JR, Koch GG. The Measurement of observer agreement for categorical data. Biometrics 1977;33:159–74. Global Road Safety Facility, The World Bank; Institute for Health Metrics and Evaluation. Transport for Health: The Global Burden of Disease from Motorized Road Transport. Seattle, WA: IHME; Washington, DC: The World Bank, 2014. Bhalla K, Adofo K, Mock CN, et al. Non-traditional data sources for injury control: an agenda for action in Ghana. Inj Prev 2012;18:277.
3
Assessing the reliability and validity of direct observation and traffic camera streams to measure helmet and motorcycle use Heather N Zaccaro,1 Emily C Carbone,2 Nishita Dsouza,2 Michelle R Xu,3 Mary C Byrne,2 John D Kraemer3 ▸ Additional material is published online only. To view please visit the journal online (http://dx.doi.org/10.1136/ injuryprev-2014-041399). 1
Department of International Health, Georgetown University, Washington, DC, USA 2 Department of Human Science, Georgetown University, Washington, DC, USA 3 Department of Health Systems Administration, Georgetown University, Washington, DC, USA Correspondence to Professor John D Kraemer, Department of Health Systems Administration, Georgetown University, 231 St Mary’s Hall, 3700 Reservoir Road, Washington, DC 20057, USA; [email protected] Received 12 August 2014 Revised 9 December 2014 Accepted 7 January 2015
To cite: Zaccaro HN, Carbone EC, Dsouza N, et al. Inj Prev Published Online First: [ please include Day Month Year] doi:10.1136/injuryprev2014-041399
ABSTRACT There is a need to develop motorcycle helmet surveillance approaches that are less labour intensive than direct observation (DO), which is the commonly recommended but never formally validated approach, particularly in developing settings. This study sought to assess public traffic camera feeds as an alternative to DO, in addition to the reliability of DO under field conditions. DO had high inter-rater reliability, κ=0.88 and 0.84, respectively, for cycle type and helmet type, which reinforces its use as a gold standard. However, traffic camera-based data collection was found to be unreliable, with κ=0.46 and 0.53 for cycle type and helmet type. When bicycles, motorcycles and scooters were classified based on traffic camera streams, only 68.4% of classifications concurred with those made via DO. Given the current technology, helmet surveillance via traffic camera streams is infeasible, and there remains a need for innovative traffic safety surveillance approaches in low-income urban settings.
BACKGROUND Motorcyclists constitute an increasing share of road traffic injuries (RTIs), particularly in developing countries where self-reported and observational data consistently indicate low helmet use rates.1–8 Direct observation (DO) is the preferred methodology for measuring helmet use as it is not subject to error introduced by response bias in selfreported data, though no published study has assessed the reliability of motorcycle helmet DO under field conditions in high-income countries or low-income countries (HICs or LICs). In LICs, use of DO is constrained by high resource costs, including time, personnel and transportation, limiting the availability of helmet surveillance data. To conduct DO more efficiently, studies in HICs have set up high-resolution cameras to facilitate data collection;9 10 however, in LICs and nonresearch settings in HICs, this solution is often considered impractical as the up-front expense of high-quality cameras is likely to limit or disincentivise their use. In lieu of setting up new cameras, the authors of this study sought to test the feasibility of using pre-existent traffic cameras, which have proliferated in urban areas in HICs and LICs, to collect data on motorcyclists. This study had three objectives: 1. To empirically verify the inter-rater reliability of direct field observation 2. To assess the reliability of data collection using public traffic cameras
3. To validate traffic camera-based data collection against the presumed gold standard of DO.
METHODS Study design To assess the reliability of DO and validate the use of traffic cameras, four trained team members simultaneously collected data on motor/bicyclists at a high-traffic intersection in northwest Washington, DC, USA: two via DO and two via online traffic camera streams. Data collection occurred on weekday mornings between 07:45 and 08:45 (a high-volume period to increase sampling efficiency) from October to December 2013. The observation site was selected because it was monitored by a highquality, publicly available traffic camera; there existed a safe, unobstructed vantage point for DO of the roadway; and it was located conveniently for the study team. Data were collected on persons riding bicycles, motorcycles or scooters as they passed through a forked, merging intersection onto a higher volume road (see online supplementary appendix 1). The real-time traffic camera feed was accessed from the District of Columbia Department of Transportation website (app.ddot.dc.gov) where image quality was comparable with that available online from LICs. The feed consisted of a series of static, colour images refreshing every 2 s. The camera used in this study provided an obstructionfree overhead vantage of the intersection and was located lower to the ground than most other DC cameras. The observation instrument was designed to enable rapid recording of observed motor/bicyclists and their characteristics. It was pilot tested through DO and online for 90 min each then revised accordingly (see online supplementary appendix 2). Data collectors received training on the protocol prior to initiating the study, including classroom training and 90 min of supervised data collection by DO and traffic camera. Additionally, they received a reference sheet to assist with the categorisation of cycle and helmet types. The importance of independent observation was emphasised during this training session to avoid biased estimates of reliability. Field and online observers synchronised starting times of each data collection session by phone. Collectors also recorded the time observed and colour of each motorbike to ensure observations were matched correctly. Technical problems such as camera misalignment or failure of the video stream were noted on the instrument and affected observations were excluded from analysis. Although most
Zaccaro HN, et al. Inj Prev 2015;0:1–3. doi:10.1136/injuryprev-2014-041399
Copyright Article author (or their employer) 2015. Produced by BMJ Publishing Group Ltd under licence.
1
Brief report data collection sessions involved four observers, during four sessions, data were collected either via DO or traffic cameras due to inclement weather or scheduling conflicts. Georgetown University’s IRB determined that human subjects’ committee oversight was not required for this study.
Variables The following variables were collected: cycle type, helmet usage and type, and whether the cyclist weaved between lanes. Cycle type was categorised as bicycle, motorcycle or scooter. Motorbikes, which include motorcycles and scooters, were differentiated based on the placement of the rider’s feet (out to the side on a motorcycle versus forward on a scooter) and wheel size. Helmet type included bicycle helmets, full-head helmets, half-head helmets and other. Consistent with the approach used by the National Highway Traffic Safety Administration, full-head and half-head motorcycle helmets were distinguished based on whether or not they covered the user’s ears.11
Analysis Data were entered into Microsoft Excel, then exported to Stata V.13.1 (StataCorp: College Station, Texas, USA) for statistical analyses. Every 10th record was visually inspected for quality assurance prior to analysis. Inter-rater reliability was calculated separately for field-based and camera-based observation using κ statistics. Percent agreement between DO and online data collection was assessed for variables with κ>0.6 with DO as the assumed gold standard.10 If DO observers disagreed about an observation, it was excluded from cross-methodological comparison.
RESULTS A total of 530 observations of 185 motor/bicyclists were recorded during 56 person-hours of data collection (table 1). In all, 58.5% of observations occurred in the field (N=310), and 41.5% were viewed online (N=220). Motorcycles comprised the majority of observations (64.3%), followed by scooters (18.8%), and bicycles (16.8%). Full-head helmets were most common (78.1%), then bike helmets (16.5%) and half-
Table 1 Distribution of selected characteristics of observations (N=530) Indicator Location Field Traffic camera Vehicle type Bicycle Motorcycle Scooter Wearing helmet Yes No Helmet type Bike helmet Full-head helmet Half-head helmet Other Between lanes Yes No
2
N
Percent
310/530 220/530
58.5 41.5
92/516 329/516 95/516
16.8 64.3 18.8
512/519 7/519
98.7 1.3
83/503 393/503 26/503 1/503
16.5 78.1 5.2 0.2
71/514 443/514
13.8 86.2
head helmets (5.2%). Most cyclists (86.2%) remained in one lane, while 13.8% wove between lanes. DO’s inter-rater reliability (table 2) was high: the κ statistic for differentiating bicycles from motorbikes was 0.98 (95% CI 0.82 to 1.00) and decreased marginally when differentiating motorcycles from scooters (κ=0.88, 95% CI 0.76 to 0.99). Helmet type had a κ statistic of 0.84 (95% CI 0.71 to 0.98); however, inter-rater reliability for helmet use, defined as the presence or absence of a helmet, could not be properly assessed because near universal helmet use caused very high expected agreement for this variable. The reliability of DO for whether or not cyclists wove between lanes was low (κ=0.31, 95% CI of 0.15 to 0.47). Traffic camera-based observation was comparatively less reliable. Agreement was high when differentiating bicycles from motorbikes (κ=0.79, 95% CI 0.59 to 0.98), but not when distinguishing among bicycles, motorcycles and scooters (κ=0.46, 95% CI 0.32 to 0.61). Helmet type had a κ of 0.53 (95% CI 0.36 to 0.69), and whether or not motor/bicyclists wove between lanes had stronger agreement than via DO (κ=0.48, 95% CI 0.29 to 0.67). Due to the aforementioned lack of variation in helmet use, camera-based observation exhibited less than chance agreement for this variable (κ=−0.03, 95% CI −0.21 to 0.15). Agreement for cycle type variable was assessed between DO and camera-based observation using the former as the goldstandard classification (table 3). Bicycles were missed relatively often in the traffic camera feeds: 84.4% (95% CI 67.2% to 94.7%) of bikes seen in the field were identified by online observers compared with 97.8% (95% CI 93.7% to 99.5%) of motorbikes, meaning approximately 15.6% of bikes and 2.2% of motorbikes seen in the field were missed on camera. Furthermore, 96.3% of cycles identified as motorbikes in the field were classified correctly on camera (95% CI 91.6% to 98.8%) compared with only 75.0% of bikes (95% CI 56.5% to 88.5%). Error increased substantially when differentiating motorcycles from scooters: although 96.7% (95% CI 90.6% to 99.3%) of motorcycles and 100.0% (95% CI 90.3% to 100.0%) of scooters observed in the field were detected on camera, only 80.0% (95% CI 70.2% to 87.7%) and 33.3% (95% CI 18.6% to 51.0%), respectively, were classified as the correct type of motorbike. Overall, 94.9% (95% CI 90.3 to 97.8%) of cyclists observed in the field were seen on camera, and 68.4% (95% CI 60.5% to 75.5%) were categorised correctly.
DISCUSSION This study empirically verifies the high inter-rater reliability of DO: κ statistics exceeded 0.8 for most variables, including both cycle and helmet type, which is generally considered to be very good agreement.12 Despite DO’s general acceptance,11 it has, to Table 2 Inter-rater reliability of the direct observation of cyclists in the field and via traffic camera Field-based observation
Traffic camera-based observation
Variable
κ Statistic
95% CI
κ Statistic
Vehicle type (bike vs motorbike) Vehicle type (bike vs motorcycle vs scooter) Helmet type Between lanes Wearing helmet
0.98
0.82 to 1.00
0.79
0.59 to 0.98
0.88
0.76 to 0.99
0.46
0.32 to 0.61
0.84 0.31 0.00
0.71 to 0.98 0.15 to 0.47 0.00 to 0.00
0.53 0.48 −0.03
0.36 to 0.69 0.29 to 0.67 −0.21 to 0.15
95% CI
Zaccaro HN, et al. Inj Prev 2015;0:1–3. doi:10.1136/injuryprev-2014-041399
Brief report Table 3 Agreement between traffic camera-based data collection and direct field observation
Bicycles Motorbikes* Motorcycles Scooters Total
Percent observed on camera (95% CI)
Percent correctly identified on camera (95% CI)
84.4 97.8 96.7 100.0 94.9
75.0 96.3 80.0 33.3 68.4
(67.2 to (93.7 to (90.6 to (90.3 to (90.3 to
94.7) 99.5) 99.3) 100.0) 97.8)
(56.6 to (91.6 to (70.2 to (18.6 to (60.5 to
88.5) 98.8) 87.7) 51.0) 75.5)
*Motorbikes is a broad category including both motorcycles and scooters.
our knowledge, never been formally validated in the literature. While strong reliability does not inherently establish validity, it is plausible to interpret high agreement as field observers both observing the same, accurate characteristics. On the other hand, traffic camera-based observation had substantial to poor agreement, with κ statistics ranging from 0.46 to 0.79.12 Due to high compliance with local helmet laws, the inter-rater reliability of helmet use versus non-use could not be properly assessed either in the field or via traffic camera. While this study’s results indicate the inability to use traffic cameras to categorise helmet type, the plausibility of using this methodology for the purpose of evaluating helmet use prevalence could be reassessed in locations such as LICs where helmet use rates are typically low;2–8 however, considering the low reliability of traffic camera-based observation, this may not be worthwhile. This study provides empirical support for the presumed validity of DO, which is valuable in settings with the ability to commit adequate personnel and time to safety surveillance. However, surveillance is challenging in LICs where resources are scarce and the burden of RTIs is high.13 Though publicly available traffic camera feeds appear not to be a labour-saving approach of adequate quality, there remains a need to develop more feasible safety intervention surveillance approaches. When higher resolution camera feeds become more widely available in LICs, this method should be reassessed. Data collection to inform and evaluate road safety interventions in LICs will necessarily require innovative, efficient approaches.14
What this study adds ▸ The inter-rater reliability of direct observation was high, which supports the use of this method as the gold standard for helmet surveillance. ▸ Traffic camera-based data collection was not found to be reliable and is not currently a viable alternative for jurisdictions lacking the resources for direct observation. There remains a need to develop innovative approaches for validly assessing helmet use in low-resource settings.
Contributors HNZ and JDK designed the study. All authors participated in data collection. Data entry and analysis were performed by HNZ while JDK performed quality assessment of the data. HNZ and JDK drafted the manuscript and all other authors revised it critically. All authors approved the final manuscript for publication. Competing interests None. Provenance and peer review Not commissioned; externally peer reviewed. Data sharing statement Original data are available from the corresponding author and will be provided upon request.
REFERENCES 1 2
3
4
5
6
7
8
What is already known on the subject ▸ Motorcyclists account for an increasing share of road traffic injuries, particularly in low-income countries where self-reported and observed rates of helmet usage are consistently low. ▸ There is a scarcity of data in low-income settings to inform and assess road safety interventions, partially due to the excessive resource cost of direct observation and a lack of innovative, more efficient methodologies. ▸ Though it is usually considered the gold standard, direct observation has never been formally validated for motorcycle helmet use.
Zaccaro HN, et al. Inj Prev 2015;0:1–3. doi:10.1136/injuryprev-2014-041399
9 10
11
12 13
14
Bachani AM, Koradia P, Herbert HK, et al. Road traffic injuries in Kenya: the health burden and risk factors in two districts. Traffic Inj Prev 2012;13(Suppl 1):24–30. Arosanyin GT, Olowosulu AT, Oyeyemi GM. An examination of some safety issues among commercial motorcyclists in Nigeria: a case study. Int J Inj Contr Saf Promot 2013;20:103–10. Chalya PL, Mabula JB, Ngayomela IH, et al. Motorcycle injuries as an emerging public health problem in Mwanza City, north-western Tanzania. Tanzan J Health Res 2010;12:214–21. Nzegwu MA, Aligbe JU, Banjo AAF, et al. Patterns of morbidity and mortality amongst motorcycle riders and their passengers in Benin-City Nigeria: one-year review. Ann Afr Med 2008;7:82–5. Ichikawa M, Nakahara S, Phommachanh S, et al. Roadside observation of secondary school students’ commuting to school in Vientiane, Laos. Int J Inj Contr Saf Promot Published Online First: 18 Oct 2013. Ackaah W, Afukaar F, Agyemang W, et al.; Road Traffic Injuries Research Network Multicenter Collaborators. The use of non-standard motorcycle helmets in low- and middle-income countries: a multicentre study. Inj Prev 2013;19:158–63. Bachani AM, Branching C, Ear C, et al. Trends in prevalence, knowledge, attitudes, and practices of helmet use in Cambodia: results from a two year study. Injury 2013;44(Suppl 4):S31–37. Ackaah W, Afukaar FK. Prevalence of helmet use among motorcycle users in Tamale Metropolis, Ghana: an observational study. Traffic Inj Prev 2010;11:522–5. Basch CH, Zagnit EA, Rajan S, et al. Helmet Use Among Cyclists in New York City. J Community Health 2014;39:956–8. Constant A, Messiah A, Felonneau M-L, et al. Investigating helmet promotion for cyclists: results from a randomised study with observation of behaviour, using a semi-automatic video system. PLoS ONE 2012;7:e31651. Pickrell TM, Ye TJ. Motorcylce Helmet Use in 2012—Overall Results. Report No. DOT HS 811 759. Washington, DC: National Highway Traffic Safety Administration, 2013:1–5. Landis JR, Koch GG. The Measurement of observer agreement for categorical data. Biometrics 1977;33:159–74. Global Road Safety Facility, The World Bank; Institute for Health Metrics and Evaluation. Transport for Health: The Global Burden of Disease from Motorized Road Transport. Seattle, WA: IHME; Washington, DC: The World Bank, 2014. Bhalla K, Adofo K, Mock CN, et al. Non-traditional data sources for injury control: an agenda for action in Ghana. Inj Prev 2012;18:277.
3
