Evaluating New York City’s Abortion Reporting System: Insights for Public Health Data Collection Systems Amita Toprani, MD, MPH; Ann Madsen, PhD; Tara Das, PhD; Melissa Gambatese, MPH; Carolyn Greene, MD; Elizabeth Begier, MD, MPH rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr
Context: New York City (NYC) mandates reporting of all abortion procedures. These reports enable tracking of abortion incidence and underpin programs, policy, and research. Since January 2011, the majority of abortion facilities must report electronically. Objectives: We conducted an evaluation of NYC’s abortion reporting system and its transition to electronic reporting. We summarize the evaluation methodology and results and draw lessons relevant to other vital statistics and public health reporting systems. Design: The evaluation followed Centers for Disease Control and Prevention guidelines for evaluating public health surveillance systems. We interviewed key stakeholders and conducted a data provider survey. In addition, we compared the system’s abortion counts with external estimates and calculated the proportion of missing and invalid values for each variable on the report form. Finally, we assessed the process for changing the report form and estimated system costs. Setting: NYC Health Department’s Bureau of Vital Statistics. Main Outcome Measures: Usefulness, simplicity, flexibility, data quality, acceptability, sensitivity, timeliness, and stability of the abortion reporting system. Results: Ninety-five percent of abortion data providers considered abortion reporting important; 52% requested training regarding the report form. Thirty percent reported problems with electronic biometric fingerprint certification, and 18% reported problems with the electronic system’s stability. Estimated system sensitivity was 88%. Of 17 variables, education and ancestry had more than 5% missing values in 2010. Changing the electronic reporting module was costly and time-consuming. System operating costs were estimated at $80 136 to $89 057 annually. Conclusions: The J Public Health Management Practice, 2014, 20(4), 392–400 C 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins Copyright
NYC abortion reporting system is sensitive and provides high-quality data, but opportunities for improvement include facilitating biometric certification, increasing electronic platform stability, and conducting ongoing outreach and training for data providers. This evaluation will help data users determine the degree of confidence that should be placed on abortion data. In addition, the evaluation results are applicable to other vital statistics reporting and surveillance systems. KEY WORDS: abortion, induced, pregnancy, pregnancy outcome,
unplanned
Nearly all abortions in the United States result from unintended pregnancy.1 In New York City (NYC), approximately 40% of all pregnancies and 60% of pregnancies among teens end in abortion.2 Abortion data are used to calculate population health indicators, including the total pregnancy rate and rates of unintended and teen pregnancy, and to establish
Author Affiliations: Epidemic Intelligence Service, Centers for Disease Control and Prevention, Atlanta, Georgia (Dr Toprani); and New York City Department of Health and Mental Hygiene, New York (Drs Toprani, Madsen, Das, Greene, and Begier and Ms Gambatese). The findings and conclusions in this report are those of the author(s) and do not necessarily represent the official position of the Centers for Disease Control and Prevention. The authors thank the following persons for their invaluable assistance: Rachel Jones, Julie Magri, Karen Pazol, Judith Sackoff, Steven Schwartz, C. Kay Smith, Cristina Yunzal-Butler, and Regina Zimmerman. No financial disclosures were reported by the authors of this article. Correspondence: Amita Toprani, MD, MPH, New York City Department of Health and Mental Hygiene, 125 Worth St, Room 205, New York, NY 10013 ([email protected]). DOI: 10.1097/PHH.0b013e31829c88b8
392 Copyright © 2014 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
New York City’s Abortion Reporting System
and evaluate reproductive health policies, programs, and services. Accurate and timely abortion reporting is challenging because of social stigma attached to the procedure and history of harassment and violence toward abortion providers. Despite mechanisms for protecting provider and patient confidentiality, widespread underreporting is suspected.3 NYC’s health code mandates reporting of all abortions.4 Forty-six states also mandate abortion reporting. New Hampshire, New Jersey, and the District of Columbia have voluntary reporting, and California and Maryland do not collect abortion data.5 In the past decade, NYC has accounted for 10% of the approximately 800 000 total abortions reported to the Centers for Disease Control and Prevention (CDC) nationally each year.6 No systematic evaluation of an abortion reporting system has been published. Such an evaluation is important because the information it generates is unavailable from other sources. It can document the degree of abortion underreporting and reasons for underreporting. Furthermore, assessments of system sensitivity, data quality, and timeliness can help data users determine the degree of confidence that should be placed on abortion data. Moreover, assessment of usefulness, simplicity, flexibility, acceptability, and stability will help determine whether the costs of maintaining an abortion reporting system are justified. Finally, evaluation results can be applied to improve other vital statistics reporting systems and surveillance systems. We evaluated the NYC abortion reporting system by using CDC’s Guidelines for Evaluating Public Health Surveillance Systems7 and conducted a preliminary evaluation of the impact of a recent transition to electronic reporting.
● Methods Abortion reporting requirements NYC’s health code defines abortion as “a purposeful interruption of an intrauterine pregnancy with the intention other than to produce a live-born infant and that does not result in a live birth.” All health care facilities performing abortions in NYC, including hospitals, ambulatory surgery centers, women’s health or abortion clinics, and physicians’ offices, must report each abortion performed. Each abortion procedure must be certified by a physician or nurse practitioner and be reported to the NYC Health Department within 5 business days of its occurrence.4 During 2000-2010, an average of 90 000 abortions were reported annually in NYC. As of January 1, 2011, all facilities performing 100 or more abortions yearly must use the Electronic Vi-
❘ 393
tal Events Registration System (EVERS) for reporting. Electronic reporting is optional for smaller-volume facilities, which together report approximately 1% of total abortions. One data provider from each facility was required to participate in EVERS training. The training comprised a classroom- or Internet-based session wherein the electronic system was explained in detail and participants practiced entering sample data into the EVERS. Bureau of Vital Statistics (BVS) registration and quality assurance staff were present at the sessions to answer questions.
Reporting system structure and operation The abortion reporting system is managed by NYC’s Department of Health and Mental Hygiene within the BVS. The system’s purpose is to document the number of legal abortions occurring in NYC and to describe characteristics of patients undergoing abortion and abortion procedures. In the first step of the reporting process, health care providers or clinic administrative staff complete a report for each abortion performed. The data collected on the report form include patient initials, date of birth, state and country of birth, duration of residence in the United States, place of usual residence (eg, zip code, borough or county, city, state, and country), race/ethnicity, ancestry, education level, marital status, previous pregnancy outcomes, gestational age, reporting facility name and address, date and type of procedure, and primary financial coverage. Providers reporting in paper format transcribe the data onto a paper form, certify the abortion by signing the form, and mail it to the BVS. Paper reports are entered into the EVERS by BVS staff. To ensure confidentiality, paper submissions are shredded after being stored for 1 year. Since mandatory electronic reporting took effect in January 2011, approximately 99% of events have been reported electronically. Providers certify electronically reported events by using a biometric fingerprint recognition device, and the EVERS allows providers to certify multiple events simultaneously. Electronic records are retained indefinitely on a secure server. BVS staff compile, analyze, and disseminate the data on an annual basis. At calendar year’s end, reporting facilities are asked to review their billing records and inform the BVS of the number of procedures performed. This number is compared with the number of abortions they have reported to the BVS. If a discrepancy is identified, facilities are contacted and encouraged to report any previously unreported procedures. Next, BVS staff identify missing and invalid values in the data; for variables routinely recorded on medical or billing records (eg, zip code), the BVS contacts reporting facilities to obtain corrected values. For all other variables,
Copyright © 2014 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
394 ❘ Journal of Public Health Management and Practice missing and invalid values are coded as unknown. Data are disseminated in 2 forms. Event-level data are released to the NYC Health Department’s Bureau of Maternal, Infant and Reproductive Health. Aggregate data are published in the NYC Annual Summary of Vital Statistics and are voluntarily reported to the New York State Health Department and CDC annually. In addition, aggregate data are available to the public by request. All requests for abortion data are reviewed by the BVS data use officer before fulfillment to preclude release of personally identifiable information. In recognition of the stigma surrounding abortion, the BVS does not release facility names or data that can be linked to a specific facility or provider. Because of the frequent appearance of new providers and discontinuation of services by other providers, BVS staff periodically review telephone book listings and newspaper and online advertisements to identify new providers. Staff contact new providers by telephone to inform them of abortion reporting requirements and then follow up to ensure they begin reporting.
System evaluation Our evaluation was based on CDC guidelines for evaluating public health surveillance systems.6 The attributes of usefulness, simplicity, flexibility, data quality, acceptability, sensitivity, timeliness, and stability were evaluated for the entire system. Definitions and evaluation criteria for each attribute are described in the CDC guidance document. We focused the evaluation on events occurring during 2010, the latest year for which a complete data set was available. Data and processes during 2011 were assessed where possible. We assessed the electronic platform’s flexibility, data quality, acceptability, and timeliness. Analyses were performed by using SAS (version 9.2; SAS Institute, Inc, Cary, North Carolina). Usefulness was assessed by conducting telephone interviews with persons who had requested abortion data from the BVS during 2010. Representatives from the NYC Health Department’s Bureau of Maternal, Infant and Reproductive Health were interviewed. In addition, we interviewed a representative from the Bronx District Public Health Office, a satellite office of the health department based in a community with high unintended pregnancy rates. Interviews were also conducted with representatives from the New York State Department of Health BVS, CDC Division of Reproductive Health, Guttmacher Institute, Chiaroscuro Foundation, and Planned Parenthood of NYC. Usefulness was further assessed by searching the MEDLINE literature database for English language publications using NYC abortion data.
To assess simplicity, we interviewed 9 BVS staff members from the registration, registrar’s, quality improvement, statistical analysis, and records management departments. We documented system operations, data flow, and integration with other NYC vital statistics reporting systems. We evaluated flexibility by comparing the process for changing or adding questions to the abortion report form during 1996 and 2012. Data quality was assessed by tabulating the number and percentage of missing, invalid, and unknown values for each variable during 2010. To assess data quality during the electronic transition, we compared missing, invalid, and unknown values for 2010 and 2011. We assessed acceptability by conducting an Internetbased survey of data providers during September to December 2011. A survey link was e-mailed to all facilities that report abortions to the BVS. Administrative personnel are responsible for abortion reporting at the majority of abortion facilities, but this task is performed by health care providers at some facilities. In this context, “data provider” refers to the person with primary responsibility for reporting, irrespective of their involvement in performing the abortion procedure. The survey included questions about abortion reporting in general and data providers’ experience with the electronic platform. To assess sensitivity, we compared data from NYC’s abortion reporting system with data reported by Guttmacher Institute, a nonprofit organization dedicated to advancing sexual and reproductive health through research, policy analysis, and education.8 We defined sensitivity as the number of abortions reported to the BVS divided by the number of abortions reported to Guttmacher Institute. As has been done in similar analyses,9,10 we chose Guttmacher Institute data as the standard for comparison because of its rigorous data collection methodology. Guttmacher Institute conducted national abortion provider censuses to estimate abortion incidence during 2004, 2005, 2007, and 2008. Their process included 3 mailings of a paper survey, with telephone follow-up to nonresponders. For providers not reached by mail or telephone, Guttmacher Institute generated estimates by using data from prior years or asking other providers from the same geographic area to provide approximate figures.11 In addition, we assessed sensitivity specifically for medication-induced abortions by including a question assessing completeness of reporting of medicationinduced abortions in the data provider survey described previously. Facilities whose survey responses indicated that they perform but do not report such abortions were contacted for estimates of the number of unreported medication-induced abortions during 2010. We generated an estimate of timeliness by calculating the proportion of 2010 events reported during
Copyright © 2014 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
New York City’s Abortion Reporting System
the 2011 calendar year because the report date was not retained for events reported before 2011. We assessed stability by asking BVS staff and data providers to estimate the frequency of system malfunctions. To create an estimate of system costs, the average number of hours per month each BVS staff member spent on abortion reporting activities was used to calculate the corresponding salary expenditures. Additional costs related to the electronic transition included development and deployment of the EVERS abortion module, purchase of biometric fingerprint recognition devices, and routine maintenance. Costs incurred by abortion providers were not assessed.
● Results With respect to usefulness, we determined that the abortion reporting system is useful in several regards. The health department uses NYC abortion data to determine agency priorities, guide resource allocation, and develop programs and policies. During 2002, the Bureau of Maternal, Infant and Reproductive Health used abortion and birth data to develop a teen pregnancy prevention program in the South Bronx, a neighborhood with high teen abortion rates. In addition, the NYC health department included teen pregnancy prevention as one of 10 priority areas in its 2009 strategic plan, setting a goal of reducing the teen pregnancy rate from 83.2 per 1000 to 70.0 per 1000 by 2012.12 Abortion statistics are used to evaluate the South Bronx teen pregnancy prevention program and to monitor the health department’s progress toward reducing the citywide teen pregnancy rate. These data are also used for epidemiologic research: scholarly publications based on NYC abortion data have explored socioeconomic, racial/ethnic, and geographic factors related to abortion.13-16 Furthermore, abortion data are frequently requested from the health department by media outlets and advocacy groups. Finally, the data are published in the Annual Summary of Vital Statistics of New York City,2 Vital Statistics Tables of New York State,17 and CDC’s abortion surveillance summaries.5 Regarding simplicity, abortion reporting is integrated with reporting of births, deaths, miscarriages, and stillbirths in the EVERS. However, abortion differs from other vital events because new reporting facilities appear frequently, creating a need for the BVS to identify and train new providers. Data providers need special training to use EVERS, and despite the fact that training was provided by the health department before mandatory electronic reporting took effect, 38% of data provider survey respondents requested additional EVERS training. Moreover, 52% of data providers requested training on how to answer questions on the
❘ 395
report form correctly and 29% noted that the report form has too many questions. The system’s flexibility can be assessed by comparing the process of adding a question to the abortion report form before and after the electronic transition. In 1996, the report form was updated to include medication abortion in the list of options for the type of procedure performed. This change required NYC Board of Health approval and an update to the paper form. The process was initiated during October 1996; Board of Health approval was granted in December 1996; and the new paper form was used in January 1997. More recently, a question about postabortion contraception dispensation was added to the form. This addition required Board of Health approval and updating the EVERS abortion module. The process was initiated during March 2011, and Board of Health approval was granted in June 2011. EVERS software changes were made in July 2011, at a cost of approximately $8000. The new module’s testing and implementation was delayed because of competing BVS priorities. The revised form was activated in April 2012. Data quality was generally high before the electronic transition. Table 1 indicates variables for which more than 5% of values were missing, invalid, or unknown during either 2010 or 2011. The influence of electronic reporting on data quality varied. The education variable improved; missing, invalid, or unknown items declined from 21.2% to 6.4%. In contrast, missing marital status increased from 3.9% to 16.4%. Further investigation revealed that 82% of records with unusable data for marital status were from 2 high-volume facilities that had been marking marital status as unknown for all patients. Missing ancestry increased from 7.8% to 17%, and missing birthplace increased from 4.9% to 17.7%. Fifty-six of 65 facilities (86%) completed the data provider survey assessing acceptability (Table 2). Ninety-five percent of respondents considered abortion reporting “very important” or “somewhat important,” and 86% reported that electronic reporting was “easier” or “neither easier nor more difficult” than paper TABLE 1 ● New York City Abortion Reporting System
Data Quality Analysis: Percentage of Missing, Invalid, and Unknown Values for Selected Data Points, 2010 and 2011 qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq 2010 Variable Education Ancestry Race/ethnicity Birthplace Marital status
2011
n
%
n
%
17 725 6 519 4 074 4 072 3 253
21.2 7.8 4.9 4.9 3.9
5 151 13 657 4 234 14 214 13 196
6.4 17.0 5.3 17.7 16.4
Copyright © 2014 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
396 ❘ Journal of Public Health Management and Practice TABLE 2 ● Results of the New York City Abortion Reporting Data Provider Survey, 2011 (N = 56)
qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq
Provider characteristics Facility type Hospital Doctor’s office Clinic Other Title or role of respondent Administrator, supervisor Registrar, clerk Clinician Health educator Knowledge Respondent answered the following correctly: Data are used to make it harder to obtain an abortion in NYC Data are used to understand reproductive health problems At what gestational age is abortion reporting required? Abortion data are used to calculate pregnancy rates Data are used to prioritize public health programs and services The federal government mandates collection of abortion data Attitudes How important is the collection and reporting of abortion data? Very important Somewhat important Of minor importance Not at all important How does the process of reporting electronically compare with paper reporting?a (n = 48) Electronic reporting is easier Neither easier nor more difficult More difficult How does the time required to report electronically compare with paper reporting?a (n = 48) Electronic reporting takes less time Takes the same amount of time Takes more time Behaviors Does your facility file abortions electronically? (Yes responses) At your facility, are medication abortions reported to the NYC health department? (Yes responses)
False True All ages True True False
n
%
25 21 8 2
44.6 37.5 14.3 3.6
21 20 14 1
37.5 35.7 25.0 1.8
53 49 48 48 47 23
94.6 87.5 85.7 85.7 83.9 41.1
32 21 3 0
57.1 37.5 5.4 0
21 20 7
43.8 41.7 14.6
21 14 13
43.8 29.2 27.1
48 38
85.7 67.9
Abbreviation: NYC, New York City a Only facilities that report electronically answered this question.
reporting. Median time required to report an abortion was 10 minutes, and 73% of respondents reported that electronic reporting took “less time” or “the same amount of time” as paper reporting. The majority of respondents correctly answered questions about abortion reporting requirements and uses of abortion data. Table 3 summarizes perceived barriers and potential facilitators of abortion reporting. Difficulty in certifying events was not listed among the survey multiple-choice options but was independently entered as a free-text comment by 30% of respondents, making it the most frequently cited barrier. Survey respondents reported
that the biometric fingerprint certification devices require multiple attempts to recognize a fingerprint at each use. They expressed frustration at the time required to certify each report individually and requested that nonphysician providers be allowed to certify abortions. Downtime or slowness of the EVERS was cited as a barrier by 18% of respondents—this barrier was also independently entered as free-text comment. The most commonly identified facilitator was a direct link between electronic medical records systems and the EVERS (54%). Eighteen percent of respondents indicated that there were no significant barriers to reporting.
Copyright © 2014 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
New York City’s Abortion Reporting System
TABLE 3 ● Barriers and Facilitators to Abortion
Reporting: Results From New York City Abortion Reporting Data Provider Survey, 2011 (N = 56) qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq Facilities, % Barriers Difficulties with the biometric device for physician certification Too many questions to answer for each patient Downtime or slowness of the Internet-based reporting interface The information I need to answer the questions is not available from the medical record The electronic reporting system is difficult to use It is not clear where to find the information I need to answer the questions The questions about ancestry, race/ethnicity are confusing No significant barriers to accurate and timely reporting Facilitators A direct link between my electronic medical records system and the EVERS to allow automated filing Training related to correctly completing questions on the report form Additional trainings related to the use of the EVERS Increased operating hours of the EVERS Help Desk
30.4 28.6 17.9 10.7 5.4 5.4 5.4 17.9
53.6
51.8 37.5 17.9
Abbreviation: EVERS, Electronic Vital Events Registration System.
System sensitivity for all abortion procedure types combined was estimated at 88% compared with Guttmacher Institute data (Table 4). The health department received 11 154 reports of medication-induced abortions performed during 2010. Our data provider survey revealed that 7 of the 43 providers (16%) that performed medication abortions during 2010 did not
❘ 397
report them because they were unaware that medication abortions are reportable events. The 7 nonreporting providers were low-volume facilities, and the number of medication abortions they collectively performed during 2010 was estimated at 213 procedures, less than 2% of the difference between the health department and Guttmacher Institute abortion incidence estimates. Regarding timeliness, 21% of 83 730 abortions performed in 2010 were reported to the health department during the 2011 calendar year, after BVS staff began outreach to providers who were late in reporting. With respect to stability, BVS staff estimated that electronic system outages occurred 2 to 3 times per week but were resolved within 30 to 60 minutes of being reported to the EVERS Help Desk. Data providers gave similar estimates of the frequency of system outages, but they experienced slowness and downtime of the Internetbased interface to be a substantial barrier. With respect to resources and costs, abortion reporting was partially funded by CDC’s National Center for Health Statistics during 1978-1993. Presently, the system has no dedicated financial resources. System activities are supported by BVS’s general operating budget, but birth and death reporting are prioritized. Estimated personnel expenditures for abortion reporting were $19 057 during 2010 and decreased to $10 136 during 2011, primarily because BVS staff were no longer entering data from paper forms into the EVERS. Development of the abortion software module entailed a 1-time expense of $195 600. Biometric fingerprint recognition devices were provided to 50 facilities at a cost of $104 per device. We estimate this system’s share of the annual EVERS maintenance costs at approximately $70 000.
● Discussion TABLE 4 ● Comparison of Abortion Incidence Estimates:
New York City Abortion Reporting System and the Guttmacher Institute qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq
Year 2004 2005 2007 2008 Overall
DOHMH Incidence, n
Guttmacher Institute Incidence,a n
Sensitivity: DOHMH/ Guttmacher Institute, %
91 673 88 891 90 870 89 469 360 903
104 536 103 565 100 046 101 756 409 903
88 86 91 88 88
Abbreviation: DOHMH, New York City Department of Health and Mental Hygiene. a Rachel Jones, PhD (electronic communication, Guttmacher Institute, September 2, 2011).
This is the first published systematic evaluation of a health department’s abortion reporting system. Specifically, we conducted a comprehensive evaluation of the NYC abortion reporting system and a preliminary evaluation of the impact of the transition on mandatory electronic reporting. We documented the system’s sensitivity, which was previously unknown, and demonstrated that NYC’s abortion reporting system provides high-quality data that are used to guide public health practice and research at the local and national levels. Our analysis of the electronic reporting platform provided insight into data providers’ experiences during the early transition period. The evaluation provided valuable information including insights on provider training, system downtime, and biometric certification
Copyright © 2014 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
398 ❘ Journal of Public Health Management and Practice that can be generalized to other vital statistics reporting systems and surveillance systems in general.
Surveillance system The system captures approximately 88% of abortions performed in NYC in comparison with Guttmacher Institute estimates. Sensitivity remained stable over the years for which comparison data were available, ranging from 86% to 91%. Analogous comparisons for the 46 states with mandatory abortion reporting demonstrate sensitivities ranging from 52% to more than 100% of Guttmacher Institute estimates, with 21 states achieving more than 90% sensitivity.18 A 2010 analysis of abortion reporting at the Vermont Department of Health estimated that approximately 99% of Vermont abortions are captured. This high sensitivity is partly the result of quarterly reconciliation of abortion counts with the state’s principal abortion provider and to partnerships between the health department and Vermont medical societies facilitating provider outreach.19 However, sensitivity estimates from other jurisdictions are not directly comparable with NYC’s large urban system because many more abortion facilities are operating in NYC and the opening of new providers and discontinuation of services on the part of other providers occur frequently. The results of our data provider survey indicate that the majority of respondents considered abortion reporting important and are knowledgeable about reporting requirements. More than half of respondents requested training on how to correctly complete questions on the report form, indicating that they recognize the value of accurate data but consider themselves unequipped to furnish it. Seven providers were unaware of the requirement to report medication-induced abortions, but this accounted for a small proportion of unreported events. This finding is consistent with a national comparison of reported abortion incidence to mifepristone sales data, indicating that underreporting of medication-induced abortions is infrequent.20 Delayed reporting by abortion facilities, which previously had been identified by BVS quality assurance staff, was confirmed. Widespread disregard of the 5-day reporting requirement occurs, likely because no repercussions for late reporting have been implemented. Delayed reporting adversely affects data quality because details that are not recorded in the medical record become difficult to recall and BVS staff are less likely to successfully obtain clarification of invalid or missing data points.
Electronic platform The majority of users reported that electronic reporting was no more difficult or time-consuming than paper
reporting. However, system stability, flexibility, and acceptability worsened during the electronic transition in unexpected ways. The required biometric fingerprint certification devices increased system complexity and caused provider frustration. Many respondents were unaware that the EVERS allows certification of multiple events simultaneously or that nonphysician providers can certify abortions. Frequent outages and slowness of the EVERS were among the most common barriers identified, but these problems were general to all NYC vital statistics reporting systems. The problems with certification and the Internet interface were not anticipated by BVS staff and were not included in the survey’s multiple-choice answer options. Instead, written comments were added by data providers. Electronic reporting compromised system flexibility because the time and expense involved in adding a question to the software module in 2012 were greater than were required to make similar changes to the paper form in 1996. Finally, the majority of users requested additional EVERS training despite having received training before the electronic transition.
Lessons learned and actions taken This evaluation generated insights that are broadly applicable to vital statistics reporting systems and public health surveillance systems. Our survey results demonstrated that reporting barriers anticipated by BVS staff did not coincide with those experienced by data providers. In response to the evaluation’s findings, providers were instructed on how to simultaneously certify multiple events and were notified that nonphysician providers can certify abortions. If feedback had been solicited from data providers sooner, much of their frustration could have been prevented. Ongoing 2-way communication with data providers (eg, through periodic conference calls, refresher trainings or webinars) might improve system sensitivity, acceptability, and data quality. To improve system sensitivity, the BVS has increased the frequency of Internet and other media searches for new abortion facilities to at least once a year. In addition, the BVS plans to explore the option of conducting periodic searches of outpatient Medicaid billing databases to identify new abortion facilities. Collaboration with state and local medical societies and professional organizations provides another opportunity to communicate information to new and existing providers. Through this mechanism, important information (eg, the requirement to report medication abortions) can be circulated through multiple channels and reinforced by periodic reminders. Reconciling data with providers on a more frequent basis (ie, quarterly rather than annually) might improve reporting timeliness and provide opportunities to seek
Copyright © 2014 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
New York City’s Abortion Reporting System
feedback. In addition, the BVS is considering implementing formal citations for late reporting. Other jurisdictions transitioning from paper to electronic reporting should be aware that the operational impacts of downtime of Internet-based reporting systems are substantial. Sensitivity to downtime is likely heightened for systems in which the volume of events necessitates full-time reporting personnel. For abortion facilities in which physicians were present only during specific days or times, considerable frustration occurred when system downtime coincided with times that physicians were available to certify abortions. Since the evaluation occurred, the electronic abortion reporting interface has undergone intensive troubleshooting by the health department’s Division of Informatics and Information Technology staff, resulting in improved system stability. System outages now occur no more than 2 times per week. Furthermore, the BVS has made upgrading the biometric fingerprint recognition devices a priority. Finally, our evaluation indicated that data providers want additional training that addresses both the electronic interface and the content of the questions on the report form, both of which can be addressed by including periodic refresher trainings in regular interactions with data providers. In addition, the BVS plans to develop an Internet-based training module explaining the content of questions on the report form.
Limitations Our evaluation is subject to certain limitations. Because the evaluation occurred as data providers were transitioning to electronic reporting, our findings regarding acceptability, stability, and data quality might not apply after the transition period ends. In addition, our sensitivity assessment was limited to years for which Guttmacher Institute data were available, whereas the remainder of the evaluation focused on 2010 and 2011 data and processes. Moreover, Guttmacher Institute estimates might have under- or overestimated true abortion incidence because Guttmacher Institute collects aggregate numbers rather than individual event reports and accepts estimates by nearby providers for census nonresponders. Because our data provider survey was not piloted with data providers, 2 of the major barriers identified were written in as comments, raising the possibility that difficulties with biometric certification and system downtime were underreported. Our estimate of timeliness was crude because report date was unavailable for procedures reported before 2011. For patient confidentiality reasons, we were unable to conduct medical record reviews to assess the prevalence of misclassification or recording errors or to assess the impact of late reporting on data quality. A 1979 analysis
❘ 399
of an apparent increase in third trimester abortions in Georgia indicates that misclassification of other pregnancy outcomes as abortions does occur and that inaccurate recording of gestational age is common.21,22 Finally, our assessment of system costs did not include expenditures made by reporting facilities.
● Conclusions Our evaluation of NYC’s abortion reporting system underscores the fact that ongoing outreach to data providers is essential for maintaining high-quality vital statistics reporting systems. Developing relationships with data providers and community partners can enhance a health department’s ability to identify and communicate with data providers, facilitating timely recognition and management of barriers to reporting. Our analysis of the transition to electronic reporting revealed adverse influences on certain attributes during the early transition period. Although these findings are expected during a transition, they emphasize the importance of the user interface to acceptability of an electronic reporting system and afforded us the opportunity to swiftly address serious concerns. Agencies implementing electronic reporting of public health data should anticipate some decrease in system flexibility, acceptability, and stability in the short term. In addition, agencies should thoroughly train data providers on both the content and process of reporting. Ideally, periodic refresher trainings should occur to reinforce or clarify important messages. Finally, ongoing 2-way communication should be maintained to address unexpected problems as they arise.
REFERENCES 1. Finer LB, Kost K. Unintended pregnancy rates at the state level. Perspect Sex Reprod Health. 2011;43:78-87. 2. New York City Department of Health and Mental Hygiene. Annual summary of vital statistics. http://www.nyc.gov/ html/doh/html/vs/vs.shtml. Accessed January 30, 2013. 3. Henshaw SK. Birth and abortion data. In: Besharov DJ, ed. Birth and Abortion Data. Data Needs for Measuring Family and Fertility Change After Welfare Reform. College Park, MD: University of Maryland School of Public Policy Welfare Reform Academy; 2001:43-63. http://www.welfareacademy. org/pubs/dataneeds/dataneeds-henshaw.pdf. Accessed January 30, 2013. 4. Rules of the City of New York, Title 24, Article 203. http:// 72-0-151-116.tvc-ip.com/nyc/rcny/Title24 203 01.asp. Accessed January 30, 2013. 5. Guttmacher Institute. State Policies in Brief: abortion reporting requirements. http://www.guttmacher.org/ statecenter/spibs/spib ARR.pdf. Accessed January 30, 2013.
Copyright © 2014 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
400 ❘ Journal of Public Health Management and Practice 6. Centers for Disease Control and Prevention. Abortion surveillance—United States, 2008. MMWR Surveill Summ. 2011;60(SS-15):1-41. 7. Centers for Disease Control and Prevention. Updated guidelines for evaluating public health surveillance systems: recommendations from the Guidelines Working Group. MMWR Recommend Rep. 2001;50(RR-13):1-35. 8. Guttmacher Institute. Unintended pregnancies and resulting births remain common among young U.S. women. http:// www.guttmacher.org/. Accessed January 30, 2013. 9. Pazol K Abortion Surveillance in the United States: future directions and challenges. Paper presented at: the annual meeting of the National Association for Public Health Statistics and Information Systems; June 6-10, 2010; St Louis, MO. 10. Atrash HA, Allen DT, Rochat RW Legal abortions in Tennessee 1974-1978. J Tenn Med Assoc. 1980;73:855-863. 11. Jones RK, Kooistra K Abortion incidence and access to services in the United States, 2008. Perspect Sex Reprod Health. 2011;43:41-50. 12. Summers C, Cohen L, Havusha A, Sliger F, Farley T Take Care New York 2012: A Policy for a Healthier New York City. New York, NY: New York City Department of Health and Mental Hygiene; 2009. http://www.nyc.gov/html/doh/ downloads/pdf/tcny/tcny-2012.pdf. Accessed January 30, 2013. 13. Daily EF, Nicholas N, Nelson F, Pakter J Repeat abortions in New York City: 1970-1972. Family Plann Perspect. 1973;5: 89-93. 14. Joyce T. The social and economic correlates of pregnancy resolution among adolescents in New York City, by race
15. 16.
17.
18.
19.
20.
21.
22.
and ethnicity: a multivariate analysis. Am J Public Health. 1988;78:626-631. Potter RG, Ford K Repeat abortion. Demography. 1976;13: 65-82. Yunzal-Butler C, Sackoff J, Li W Medication abortions among New York City residents, 2001-2008. Perspect Sex Reprod Health. 2011;43:218-223. New York State Department of Public Health. Vital statistics of New York State. http://www.health.ny.gov/statistics/ vital statistics. Published 2013. Accessed January 30, 2013. Guttmacher Institute. State Data Center, Trend Data, Abortions by S tate of O ccurrence. New York, NY: Guttmacher Institute. http://www.guttmacher.org/datacenter/trend. jsp#. Accessed January 30, 2013. McCoy R. Induced termination of pregnancy: auditing and reporting activities in Vermont. Paper presented at: the annual meeting of the National Association for Public Health Statistics and Information Systems; June 6-10, 2010; St Louis, MO. Pazol K, Creanga AA, Zane SB. Trends in use of medical abortion in the United States: reanalysis of surveillance data from the Centers for Disease Control and Prevention, 20012008. Contraception. 2012;86(6):746-751. Spitz AM, Lee NC, Grimes DA, Schoenbucher AK, Lavoie M Third-trimester induced abortion in Georgia, 1979 and 1980. Am J Public Health. 1983;73:594-595. Centers for Disease Control and Prevention. Current trends third-trimester induced abortion—Georgia, 1979 and 1980. MMWR Morb Mortal Wkly Rep. 1983;32: 246-247.
Copyright © 2014 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
Context: New York City (NYC) mandates reporting of all abortion procedures. These reports enable tracking of abortion incidence and underpin programs, policy, and research. Since January 2011, the majority of abortion facilities must report electronically. Objectives: We conducted an evaluation of NYC’s abortion reporting system and its transition to electronic reporting. We summarize the evaluation methodology and results and draw lessons relevant to other vital statistics and public health reporting systems. Design: The evaluation followed Centers for Disease Control and Prevention guidelines for evaluating public health surveillance systems. We interviewed key stakeholders and conducted a data provider survey. In addition, we compared the system’s abortion counts with external estimates and calculated the proportion of missing and invalid values for each variable on the report form. Finally, we assessed the process for changing the report form and estimated system costs. Setting: NYC Health Department’s Bureau of Vital Statistics. Main Outcome Measures: Usefulness, simplicity, flexibility, data quality, acceptability, sensitivity, timeliness, and stability of the abortion reporting system. Results: Ninety-five percent of abortion data providers considered abortion reporting important; 52% requested training regarding the report form. Thirty percent reported problems with electronic biometric fingerprint certification, and 18% reported problems with the electronic system’s stability. Estimated system sensitivity was 88%. Of 17 variables, education and ancestry had more than 5% missing values in 2010. Changing the electronic reporting module was costly and time-consuming. System operating costs were estimated at $80 136 to $89 057 annually. Conclusions: The J Public Health Management Practice, 2014, 20(4), 392–400 C 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins Copyright
NYC abortion reporting system is sensitive and provides high-quality data, but opportunities for improvement include facilitating biometric certification, increasing electronic platform stability, and conducting ongoing outreach and training for data providers. This evaluation will help data users determine the degree of confidence that should be placed on abortion data. In addition, the evaluation results are applicable to other vital statistics reporting and surveillance systems. KEY WORDS: abortion, induced, pregnancy, pregnancy outcome,
unplanned
Nearly all abortions in the United States result from unintended pregnancy.1 In New York City (NYC), approximately 40% of all pregnancies and 60% of pregnancies among teens end in abortion.2 Abortion data are used to calculate population health indicators, including the total pregnancy rate and rates of unintended and teen pregnancy, and to establish
Author Affiliations: Epidemic Intelligence Service, Centers for Disease Control and Prevention, Atlanta, Georgia (Dr Toprani); and New York City Department of Health and Mental Hygiene, New York (Drs Toprani, Madsen, Das, Greene, and Begier and Ms Gambatese). The findings and conclusions in this report are those of the author(s) and do not necessarily represent the official position of the Centers for Disease Control and Prevention. The authors thank the following persons for their invaluable assistance: Rachel Jones, Julie Magri, Karen Pazol, Judith Sackoff, Steven Schwartz, C. Kay Smith, Cristina Yunzal-Butler, and Regina Zimmerman. No financial disclosures were reported by the authors of this article. Correspondence: Amita Toprani, MD, MPH, New York City Department of Health and Mental Hygiene, 125 Worth St, Room 205, New York, NY 10013 ([email protected]). DOI: 10.1097/PHH.0b013e31829c88b8
392 Copyright © 2014 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
New York City’s Abortion Reporting System
and evaluate reproductive health policies, programs, and services. Accurate and timely abortion reporting is challenging because of social stigma attached to the procedure and history of harassment and violence toward abortion providers. Despite mechanisms for protecting provider and patient confidentiality, widespread underreporting is suspected.3 NYC’s health code mandates reporting of all abortions.4 Forty-six states also mandate abortion reporting. New Hampshire, New Jersey, and the District of Columbia have voluntary reporting, and California and Maryland do not collect abortion data.5 In the past decade, NYC has accounted for 10% of the approximately 800 000 total abortions reported to the Centers for Disease Control and Prevention (CDC) nationally each year.6 No systematic evaluation of an abortion reporting system has been published. Such an evaluation is important because the information it generates is unavailable from other sources. It can document the degree of abortion underreporting and reasons for underreporting. Furthermore, assessments of system sensitivity, data quality, and timeliness can help data users determine the degree of confidence that should be placed on abortion data. Moreover, assessment of usefulness, simplicity, flexibility, acceptability, and stability will help determine whether the costs of maintaining an abortion reporting system are justified. Finally, evaluation results can be applied to improve other vital statistics reporting systems and surveillance systems. We evaluated the NYC abortion reporting system by using CDC’s Guidelines for Evaluating Public Health Surveillance Systems7 and conducted a preliminary evaluation of the impact of a recent transition to electronic reporting.
● Methods Abortion reporting requirements NYC’s health code defines abortion as “a purposeful interruption of an intrauterine pregnancy with the intention other than to produce a live-born infant and that does not result in a live birth.” All health care facilities performing abortions in NYC, including hospitals, ambulatory surgery centers, women’s health or abortion clinics, and physicians’ offices, must report each abortion performed. Each abortion procedure must be certified by a physician or nurse practitioner and be reported to the NYC Health Department within 5 business days of its occurrence.4 During 2000-2010, an average of 90 000 abortions were reported annually in NYC. As of January 1, 2011, all facilities performing 100 or more abortions yearly must use the Electronic Vi-
❘ 393
tal Events Registration System (EVERS) for reporting. Electronic reporting is optional for smaller-volume facilities, which together report approximately 1% of total abortions. One data provider from each facility was required to participate in EVERS training. The training comprised a classroom- or Internet-based session wherein the electronic system was explained in detail and participants practiced entering sample data into the EVERS. Bureau of Vital Statistics (BVS) registration and quality assurance staff were present at the sessions to answer questions.
Reporting system structure and operation The abortion reporting system is managed by NYC’s Department of Health and Mental Hygiene within the BVS. The system’s purpose is to document the number of legal abortions occurring in NYC and to describe characteristics of patients undergoing abortion and abortion procedures. In the first step of the reporting process, health care providers or clinic administrative staff complete a report for each abortion performed. The data collected on the report form include patient initials, date of birth, state and country of birth, duration of residence in the United States, place of usual residence (eg, zip code, borough or county, city, state, and country), race/ethnicity, ancestry, education level, marital status, previous pregnancy outcomes, gestational age, reporting facility name and address, date and type of procedure, and primary financial coverage. Providers reporting in paper format transcribe the data onto a paper form, certify the abortion by signing the form, and mail it to the BVS. Paper reports are entered into the EVERS by BVS staff. To ensure confidentiality, paper submissions are shredded after being stored for 1 year. Since mandatory electronic reporting took effect in January 2011, approximately 99% of events have been reported electronically. Providers certify electronically reported events by using a biometric fingerprint recognition device, and the EVERS allows providers to certify multiple events simultaneously. Electronic records are retained indefinitely on a secure server. BVS staff compile, analyze, and disseminate the data on an annual basis. At calendar year’s end, reporting facilities are asked to review their billing records and inform the BVS of the number of procedures performed. This number is compared with the number of abortions they have reported to the BVS. If a discrepancy is identified, facilities are contacted and encouraged to report any previously unreported procedures. Next, BVS staff identify missing and invalid values in the data; for variables routinely recorded on medical or billing records (eg, zip code), the BVS contacts reporting facilities to obtain corrected values. For all other variables,
Copyright © 2014 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
394 ❘ Journal of Public Health Management and Practice missing and invalid values are coded as unknown. Data are disseminated in 2 forms. Event-level data are released to the NYC Health Department’s Bureau of Maternal, Infant and Reproductive Health. Aggregate data are published in the NYC Annual Summary of Vital Statistics and are voluntarily reported to the New York State Health Department and CDC annually. In addition, aggregate data are available to the public by request. All requests for abortion data are reviewed by the BVS data use officer before fulfillment to preclude release of personally identifiable information. In recognition of the stigma surrounding abortion, the BVS does not release facility names or data that can be linked to a specific facility or provider. Because of the frequent appearance of new providers and discontinuation of services by other providers, BVS staff periodically review telephone book listings and newspaper and online advertisements to identify new providers. Staff contact new providers by telephone to inform them of abortion reporting requirements and then follow up to ensure they begin reporting.
System evaluation Our evaluation was based on CDC guidelines for evaluating public health surveillance systems.6 The attributes of usefulness, simplicity, flexibility, data quality, acceptability, sensitivity, timeliness, and stability were evaluated for the entire system. Definitions and evaluation criteria for each attribute are described in the CDC guidance document. We focused the evaluation on events occurring during 2010, the latest year for which a complete data set was available. Data and processes during 2011 were assessed where possible. We assessed the electronic platform’s flexibility, data quality, acceptability, and timeliness. Analyses were performed by using SAS (version 9.2; SAS Institute, Inc, Cary, North Carolina). Usefulness was assessed by conducting telephone interviews with persons who had requested abortion data from the BVS during 2010. Representatives from the NYC Health Department’s Bureau of Maternal, Infant and Reproductive Health were interviewed. In addition, we interviewed a representative from the Bronx District Public Health Office, a satellite office of the health department based in a community with high unintended pregnancy rates. Interviews were also conducted with representatives from the New York State Department of Health BVS, CDC Division of Reproductive Health, Guttmacher Institute, Chiaroscuro Foundation, and Planned Parenthood of NYC. Usefulness was further assessed by searching the MEDLINE literature database for English language publications using NYC abortion data.
To assess simplicity, we interviewed 9 BVS staff members from the registration, registrar’s, quality improvement, statistical analysis, and records management departments. We documented system operations, data flow, and integration with other NYC vital statistics reporting systems. We evaluated flexibility by comparing the process for changing or adding questions to the abortion report form during 1996 and 2012. Data quality was assessed by tabulating the number and percentage of missing, invalid, and unknown values for each variable during 2010. To assess data quality during the electronic transition, we compared missing, invalid, and unknown values for 2010 and 2011. We assessed acceptability by conducting an Internetbased survey of data providers during September to December 2011. A survey link was e-mailed to all facilities that report abortions to the BVS. Administrative personnel are responsible for abortion reporting at the majority of abortion facilities, but this task is performed by health care providers at some facilities. In this context, “data provider” refers to the person with primary responsibility for reporting, irrespective of their involvement in performing the abortion procedure. The survey included questions about abortion reporting in general and data providers’ experience with the electronic platform. To assess sensitivity, we compared data from NYC’s abortion reporting system with data reported by Guttmacher Institute, a nonprofit organization dedicated to advancing sexual and reproductive health through research, policy analysis, and education.8 We defined sensitivity as the number of abortions reported to the BVS divided by the number of abortions reported to Guttmacher Institute. As has been done in similar analyses,9,10 we chose Guttmacher Institute data as the standard for comparison because of its rigorous data collection methodology. Guttmacher Institute conducted national abortion provider censuses to estimate abortion incidence during 2004, 2005, 2007, and 2008. Their process included 3 mailings of a paper survey, with telephone follow-up to nonresponders. For providers not reached by mail or telephone, Guttmacher Institute generated estimates by using data from prior years or asking other providers from the same geographic area to provide approximate figures.11 In addition, we assessed sensitivity specifically for medication-induced abortions by including a question assessing completeness of reporting of medicationinduced abortions in the data provider survey described previously. Facilities whose survey responses indicated that they perform but do not report such abortions were contacted for estimates of the number of unreported medication-induced abortions during 2010. We generated an estimate of timeliness by calculating the proportion of 2010 events reported during
Copyright © 2014 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
New York City’s Abortion Reporting System
the 2011 calendar year because the report date was not retained for events reported before 2011. We assessed stability by asking BVS staff and data providers to estimate the frequency of system malfunctions. To create an estimate of system costs, the average number of hours per month each BVS staff member spent on abortion reporting activities was used to calculate the corresponding salary expenditures. Additional costs related to the electronic transition included development and deployment of the EVERS abortion module, purchase of biometric fingerprint recognition devices, and routine maintenance. Costs incurred by abortion providers were not assessed.
● Results With respect to usefulness, we determined that the abortion reporting system is useful in several regards. The health department uses NYC abortion data to determine agency priorities, guide resource allocation, and develop programs and policies. During 2002, the Bureau of Maternal, Infant and Reproductive Health used abortion and birth data to develop a teen pregnancy prevention program in the South Bronx, a neighborhood with high teen abortion rates. In addition, the NYC health department included teen pregnancy prevention as one of 10 priority areas in its 2009 strategic plan, setting a goal of reducing the teen pregnancy rate from 83.2 per 1000 to 70.0 per 1000 by 2012.12 Abortion statistics are used to evaluate the South Bronx teen pregnancy prevention program and to monitor the health department’s progress toward reducing the citywide teen pregnancy rate. These data are also used for epidemiologic research: scholarly publications based on NYC abortion data have explored socioeconomic, racial/ethnic, and geographic factors related to abortion.13-16 Furthermore, abortion data are frequently requested from the health department by media outlets and advocacy groups. Finally, the data are published in the Annual Summary of Vital Statistics of New York City,2 Vital Statistics Tables of New York State,17 and CDC’s abortion surveillance summaries.5 Regarding simplicity, abortion reporting is integrated with reporting of births, deaths, miscarriages, and stillbirths in the EVERS. However, abortion differs from other vital events because new reporting facilities appear frequently, creating a need for the BVS to identify and train new providers. Data providers need special training to use EVERS, and despite the fact that training was provided by the health department before mandatory electronic reporting took effect, 38% of data provider survey respondents requested additional EVERS training. Moreover, 52% of data providers requested training on how to answer questions on the
❘ 395
report form correctly and 29% noted that the report form has too many questions. The system’s flexibility can be assessed by comparing the process of adding a question to the abortion report form before and after the electronic transition. In 1996, the report form was updated to include medication abortion in the list of options for the type of procedure performed. This change required NYC Board of Health approval and an update to the paper form. The process was initiated during October 1996; Board of Health approval was granted in December 1996; and the new paper form was used in January 1997. More recently, a question about postabortion contraception dispensation was added to the form. This addition required Board of Health approval and updating the EVERS abortion module. The process was initiated during March 2011, and Board of Health approval was granted in June 2011. EVERS software changes were made in July 2011, at a cost of approximately $8000. The new module’s testing and implementation was delayed because of competing BVS priorities. The revised form was activated in April 2012. Data quality was generally high before the electronic transition. Table 1 indicates variables for which more than 5% of values were missing, invalid, or unknown during either 2010 or 2011. The influence of electronic reporting on data quality varied. The education variable improved; missing, invalid, or unknown items declined from 21.2% to 6.4%. In contrast, missing marital status increased from 3.9% to 16.4%. Further investigation revealed that 82% of records with unusable data for marital status were from 2 high-volume facilities that had been marking marital status as unknown for all patients. Missing ancestry increased from 7.8% to 17%, and missing birthplace increased from 4.9% to 17.7%. Fifty-six of 65 facilities (86%) completed the data provider survey assessing acceptability (Table 2). Ninety-five percent of respondents considered abortion reporting “very important” or “somewhat important,” and 86% reported that electronic reporting was “easier” or “neither easier nor more difficult” than paper TABLE 1 ● New York City Abortion Reporting System
Data Quality Analysis: Percentage of Missing, Invalid, and Unknown Values for Selected Data Points, 2010 and 2011 qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq 2010 Variable Education Ancestry Race/ethnicity Birthplace Marital status
2011
n
%
n
%
17 725 6 519 4 074 4 072 3 253
21.2 7.8 4.9 4.9 3.9
5 151 13 657 4 234 14 214 13 196
6.4 17.0 5.3 17.7 16.4
Copyright © 2014 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
396 ❘ Journal of Public Health Management and Practice TABLE 2 ● Results of the New York City Abortion Reporting Data Provider Survey, 2011 (N = 56)
qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq
Provider characteristics Facility type Hospital Doctor’s office Clinic Other Title or role of respondent Administrator, supervisor Registrar, clerk Clinician Health educator Knowledge Respondent answered the following correctly: Data are used to make it harder to obtain an abortion in NYC Data are used to understand reproductive health problems At what gestational age is abortion reporting required? Abortion data are used to calculate pregnancy rates Data are used to prioritize public health programs and services The federal government mandates collection of abortion data Attitudes How important is the collection and reporting of abortion data? Very important Somewhat important Of minor importance Not at all important How does the process of reporting electronically compare with paper reporting?a (n = 48) Electronic reporting is easier Neither easier nor more difficult More difficult How does the time required to report electronically compare with paper reporting?a (n = 48) Electronic reporting takes less time Takes the same amount of time Takes more time Behaviors Does your facility file abortions electronically? (Yes responses) At your facility, are medication abortions reported to the NYC health department? (Yes responses)
False True All ages True True False
n
%
25 21 8 2
44.6 37.5 14.3 3.6
21 20 14 1
37.5 35.7 25.0 1.8
53 49 48 48 47 23
94.6 87.5 85.7 85.7 83.9 41.1
32 21 3 0
57.1 37.5 5.4 0
21 20 7
43.8 41.7 14.6
21 14 13
43.8 29.2 27.1
48 38
85.7 67.9
Abbreviation: NYC, New York City a Only facilities that report electronically answered this question.
reporting. Median time required to report an abortion was 10 minutes, and 73% of respondents reported that electronic reporting took “less time” or “the same amount of time” as paper reporting. The majority of respondents correctly answered questions about abortion reporting requirements and uses of abortion data. Table 3 summarizes perceived barriers and potential facilitators of abortion reporting. Difficulty in certifying events was not listed among the survey multiple-choice options but was independently entered as a free-text comment by 30% of respondents, making it the most frequently cited barrier. Survey respondents reported
that the biometric fingerprint certification devices require multiple attempts to recognize a fingerprint at each use. They expressed frustration at the time required to certify each report individually and requested that nonphysician providers be allowed to certify abortions. Downtime or slowness of the EVERS was cited as a barrier by 18% of respondents—this barrier was also independently entered as free-text comment. The most commonly identified facilitator was a direct link between electronic medical records systems and the EVERS (54%). Eighteen percent of respondents indicated that there were no significant barriers to reporting.
Copyright © 2014 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
New York City’s Abortion Reporting System
TABLE 3 ● Barriers and Facilitators to Abortion
Reporting: Results From New York City Abortion Reporting Data Provider Survey, 2011 (N = 56) qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq Facilities, % Barriers Difficulties with the biometric device for physician certification Too many questions to answer for each patient Downtime or slowness of the Internet-based reporting interface The information I need to answer the questions is not available from the medical record The electronic reporting system is difficult to use It is not clear where to find the information I need to answer the questions The questions about ancestry, race/ethnicity are confusing No significant barriers to accurate and timely reporting Facilitators A direct link between my electronic medical records system and the EVERS to allow automated filing Training related to correctly completing questions on the report form Additional trainings related to the use of the EVERS Increased operating hours of the EVERS Help Desk
30.4 28.6 17.9 10.7 5.4 5.4 5.4 17.9
53.6
51.8 37.5 17.9
Abbreviation: EVERS, Electronic Vital Events Registration System.
System sensitivity for all abortion procedure types combined was estimated at 88% compared with Guttmacher Institute data (Table 4). The health department received 11 154 reports of medication-induced abortions performed during 2010. Our data provider survey revealed that 7 of the 43 providers (16%) that performed medication abortions during 2010 did not
❘ 397
report them because they were unaware that medication abortions are reportable events. The 7 nonreporting providers were low-volume facilities, and the number of medication abortions they collectively performed during 2010 was estimated at 213 procedures, less than 2% of the difference between the health department and Guttmacher Institute abortion incidence estimates. Regarding timeliness, 21% of 83 730 abortions performed in 2010 were reported to the health department during the 2011 calendar year, after BVS staff began outreach to providers who were late in reporting. With respect to stability, BVS staff estimated that electronic system outages occurred 2 to 3 times per week but were resolved within 30 to 60 minutes of being reported to the EVERS Help Desk. Data providers gave similar estimates of the frequency of system outages, but they experienced slowness and downtime of the Internetbased interface to be a substantial barrier. With respect to resources and costs, abortion reporting was partially funded by CDC’s National Center for Health Statistics during 1978-1993. Presently, the system has no dedicated financial resources. System activities are supported by BVS’s general operating budget, but birth and death reporting are prioritized. Estimated personnel expenditures for abortion reporting were $19 057 during 2010 and decreased to $10 136 during 2011, primarily because BVS staff were no longer entering data from paper forms into the EVERS. Development of the abortion software module entailed a 1-time expense of $195 600. Biometric fingerprint recognition devices were provided to 50 facilities at a cost of $104 per device. We estimate this system’s share of the annual EVERS maintenance costs at approximately $70 000.
● Discussion TABLE 4 ● Comparison of Abortion Incidence Estimates:
New York City Abortion Reporting System and the Guttmacher Institute qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq
Year 2004 2005 2007 2008 Overall
DOHMH Incidence, n
Guttmacher Institute Incidence,a n
Sensitivity: DOHMH/ Guttmacher Institute, %
91 673 88 891 90 870 89 469 360 903
104 536 103 565 100 046 101 756 409 903
88 86 91 88 88
Abbreviation: DOHMH, New York City Department of Health and Mental Hygiene. a Rachel Jones, PhD (electronic communication, Guttmacher Institute, September 2, 2011).
This is the first published systematic evaluation of a health department’s abortion reporting system. Specifically, we conducted a comprehensive evaluation of the NYC abortion reporting system and a preliminary evaluation of the impact of the transition on mandatory electronic reporting. We documented the system’s sensitivity, which was previously unknown, and demonstrated that NYC’s abortion reporting system provides high-quality data that are used to guide public health practice and research at the local and national levels. Our analysis of the electronic reporting platform provided insight into data providers’ experiences during the early transition period. The evaluation provided valuable information including insights on provider training, system downtime, and biometric certification
Copyright © 2014 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
398 ❘ Journal of Public Health Management and Practice that can be generalized to other vital statistics reporting systems and surveillance systems in general.
Surveillance system The system captures approximately 88% of abortions performed in NYC in comparison with Guttmacher Institute estimates. Sensitivity remained stable over the years for which comparison data were available, ranging from 86% to 91%. Analogous comparisons for the 46 states with mandatory abortion reporting demonstrate sensitivities ranging from 52% to more than 100% of Guttmacher Institute estimates, with 21 states achieving more than 90% sensitivity.18 A 2010 analysis of abortion reporting at the Vermont Department of Health estimated that approximately 99% of Vermont abortions are captured. This high sensitivity is partly the result of quarterly reconciliation of abortion counts with the state’s principal abortion provider and to partnerships between the health department and Vermont medical societies facilitating provider outreach.19 However, sensitivity estimates from other jurisdictions are not directly comparable with NYC’s large urban system because many more abortion facilities are operating in NYC and the opening of new providers and discontinuation of services on the part of other providers occur frequently. The results of our data provider survey indicate that the majority of respondents considered abortion reporting important and are knowledgeable about reporting requirements. More than half of respondents requested training on how to correctly complete questions on the report form, indicating that they recognize the value of accurate data but consider themselves unequipped to furnish it. Seven providers were unaware of the requirement to report medication-induced abortions, but this accounted for a small proportion of unreported events. This finding is consistent with a national comparison of reported abortion incidence to mifepristone sales data, indicating that underreporting of medication-induced abortions is infrequent.20 Delayed reporting by abortion facilities, which previously had been identified by BVS quality assurance staff, was confirmed. Widespread disregard of the 5-day reporting requirement occurs, likely because no repercussions for late reporting have been implemented. Delayed reporting adversely affects data quality because details that are not recorded in the medical record become difficult to recall and BVS staff are less likely to successfully obtain clarification of invalid or missing data points.
Electronic platform The majority of users reported that electronic reporting was no more difficult or time-consuming than paper
reporting. However, system stability, flexibility, and acceptability worsened during the electronic transition in unexpected ways. The required biometric fingerprint certification devices increased system complexity and caused provider frustration. Many respondents were unaware that the EVERS allows certification of multiple events simultaneously or that nonphysician providers can certify abortions. Frequent outages and slowness of the EVERS were among the most common barriers identified, but these problems were general to all NYC vital statistics reporting systems. The problems with certification and the Internet interface were not anticipated by BVS staff and were not included in the survey’s multiple-choice answer options. Instead, written comments were added by data providers. Electronic reporting compromised system flexibility because the time and expense involved in adding a question to the software module in 2012 were greater than were required to make similar changes to the paper form in 1996. Finally, the majority of users requested additional EVERS training despite having received training before the electronic transition.
Lessons learned and actions taken This evaluation generated insights that are broadly applicable to vital statistics reporting systems and public health surveillance systems. Our survey results demonstrated that reporting barriers anticipated by BVS staff did not coincide with those experienced by data providers. In response to the evaluation’s findings, providers were instructed on how to simultaneously certify multiple events and were notified that nonphysician providers can certify abortions. If feedback had been solicited from data providers sooner, much of their frustration could have been prevented. Ongoing 2-way communication with data providers (eg, through periodic conference calls, refresher trainings or webinars) might improve system sensitivity, acceptability, and data quality. To improve system sensitivity, the BVS has increased the frequency of Internet and other media searches for new abortion facilities to at least once a year. In addition, the BVS plans to explore the option of conducting periodic searches of outpatient Medicaid billing databases to identify new abortion facilities. Collaboration with state and local medical societies and professional organizations provides another opportunity to communicate information to new and existing providers. Through this mechanism, important information (eg, the requirement to report medication abortions) can be circulated through multiple channels and reinforced by periodic reminders. Reconciling data with providers on a more frequent basis (ie, quarterly rather than annually) might improve reporting timeliness and provide opportunities to seek
Copyright © 2014 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
New York City’s Abortion Reporting System
feedback. In addition, the BVS is considering implementing formal citations for late reporting. Other jurisdictions transitioning from paper to electronic reporting should be aware that the operational impacts of downtime of Internet-based reporting systems are substantial. Sensitivity to downtime is likely heightened for systems in which the volume of events necessitates full-time reporting personnel. For abortion facilities in which physicians were present only during specific days or times, considerable frustration occurred when system downtime coincided with times that physicians were available to certify abortions. Since the evaluation occurred, the electronic abortion reporting interface has undergone intensive troubleshooting by the health department’s Division of Informatics and Information Technology staff, resulting in improved system stability. System outages now occur no more than 2 times per week. Furthermore, the BVS has made upgrading the biometric fingerprint recognition devices a priority. Finally, our evaluation indicated that data providers want additional training that addresses both the electronic interface and the content of the questions on the report form, both of which can be addressed by including periodic refresher trainings in regular interactions with data providers. In addition, the BVS plans to develop an Internet-based training module explaining the content of questions on the report form.
Limitations Our evaluation is subject to certain limitations. Because the evaluation occurred as data providers were transitioning to electronic reporting, our findings regarding acceptability, stability, and data quality might not apply after the transition period ends. In addition, our sensitivity assessment was limited to years for which Guttmacher Institute data were available, whereas the remainder of the evaluation focused on 2010 and 2011 data and processes. Moreover, Guttmacher Institute estimates might have under- or overestimated true abortion incidence because Guttmacher Institute collects aggregate numbers rather than individual event reports and accepts estimates by nearby providers for census nonresponders. Because our data provider survey was not piloted with data providers, 2 of the major barriers identified were written in as comments, raising the possibility that difficulties with biometric certification and system downtime were underreported. Our estimate of timeliness was crude because report date was unavailable for procedures reported before 2011. For patient confidentiality reasons, we were unable to conduct medical record reviews to assess the prevalence of misclassification or recording errors or to assess the impact of late reporting on data quality. A 1979 analysis
❘ 399
of an apparent increase in third trimester abortions in Georgia indicates that misclassification of other pregnancy outcomes as abortions does occur and that inaccurate recording of gestational age is common.21,22 Finally, our assessment of system costs did not include expenditures made by reporting facilities.
● Conclusions Our evaluation of NYC’s abortion reporting system underscores the fact that ongoing outreach to data providers is essential for maintaining high-quality vital statistics reporting systems. Developing relationships with data providers and community partners can enhance a health department’s ability to identify and communicate with data providers, facilitating timely recognition and management of barriers to reporting. Our analysis of the transition to electronic reporting revealed adverse influences on certain attributes during the early transition period. Although these findings are expected during a transition, they emphasize the importance of the user interface to acceptability of an electronic reporting system and afforded us the opportunity to swiftly address serious concerns. Agencies implementing electronic reporting of public health data should anticipate some decrease in system flexibility, acceptability, and stability in the short term. In addition, agencies should thoroughly train data providers on both the content and process of reporting. Ideally, periodic refresher trainings should occur to reinforce or clarify important messages. Finally, ongoing 2-way communication should be maintained to address unexpected problems as they arise.
REFERENCES 1. Finer LB, Kost K. Unintended pregnancy rates at the state level. Perspect Sex Reprod Health. 2011;43:78-87. 2. New York City Department of Health and Mental Hygiene. Annual summary of vital statistics. http://www.nyc.gov/ html/doh/html/vs/vs.shtml. Accessed January 30, 2013. 3. Henshaw SK. Birth and abortion data. In: Besharov DJ, ed. Birth and Abortion Data. Data Needs for Measuring Family and Fertility Change After Welfare Reform. College Park, MD: University of Maryland School of Public Policy Welfare Reform Academy; 2001:43-63. http://www.welfareacademy. org/pubs/dataneeds/dataneeds-henshaw.pdf. Accessed January 30, 2013. 4. Rules of the City of New York, Title 24, Article 203. http:// 72-0-151-116.tvc-ip.com/nyc/rcny/Title24 203 01.asp. Accessed January 30, 2013. 5. Guttmacher Institute. State Policies in Brief: abortion reporting requirements. http://www.guttmacher.org/ statecenter/spibs/spib ARR.pdf. Accessed January 30, 2013.
Copyright © 2014 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
400 ❘ Journal of Public Health Management and Practice 6. Centers for Disease Control and Prevention. Abortion surveillance—United States, 2008. MMWR Surveill Summ. 2011;60(SS-15):1-41. 7. Centers for Disease Control and Prevention. Updated guidelines for evaluating public health surveillance systems: recommendations from the Guidelines Working Group. MMWR Recommend Rep. 2001;50(RR-13):1-35. 8. Guttmacher Institute. Unintended pregnancies and resulting births remain common among young U.S. women. http:// www.guttmacher.org/. Accessed January 30, 2013. 9. Pazol K Abortion Surveillance in the United States: future directions and challenges. Paper presented at: the annual meeting of the National Association for Public Health Statistics and Information Systems; June 6-10, 2010; St Louis, MO. 10. Atrash HA, Allen DT, Rochat RW Legal abortions in Tennessee 1974-1978. J Tenn Med Assoc. 1980;73:855-863. 11. Jones RK, Kooistra K Abortion incidence and access to services in the United States, 2008. Perspect Sex Reprod Health. 2011;43:41-50. 12. Summers C, Cohen L, Havusha A, Sliger F, Farley T Take Care New York 2012: A Policy for a Healthier New York City. New York, NY: New York City Department of Health and Mental Hygiene; 2009. http://www.nyc.gov/html/doh/ downloads/pdf/tcny/tcny-2012.pdf. Accessed January 30, 2013. 13. Daily EF, Nicholas N, Nelson F, Pakter J Repeat abortions in New York City: 1970-1972. Family Plann Perspect. 1973;5: 89-93. 14. Joyce T. The social and economic correlates of pregnancy resolution among adolescents in New York City, by race
15. 16.
17.
18.
19.
20.
21.
22.
and ethnicity: a multivariate analysis. Am J Public Health. 1988;78:626-631. Potter RG, Ford K Repeat abortion. Demography. 1976;13: 65-82. Yunzal-Butler C, Sackoff J, Li W Medication abortions among New York City residents, 2001-2008. Perspect Sex Reprod Health. 2011;43:218-223. New York State Department of Public Health. Vital statistics of New York State. http://www.health.ny.gov/statistics/ vital statistics. Published 2013. Accessed January 30, 2013. Guttmacher Institute. State Data Center, Trend Data, Abortions by S tate of O ccurrence. New York, NY: Guttmacher Institute. http://www.guttmacher.org/datacenter/trend. jsp#. Accessed January 30, 2013. McCoy R. Induced termination of pregnancy: auditing and reporting activities in Vermont. Paper presented at: the annual meeting of the National Association for Public Health Statistics and Information Systems; June 6-10, 2010; St Louis, MO. Pazol K, Creanga AA, Zane SB. Trends in use of medical abortion in the United States: reanalysis of surveillance data from the Centers for Disease Control and Prevention, 20012008. Contraception. 2012;86(6):746-751. Spitz AM, Lee NC, Grimes DA, Schoenbucher AK, Lavoie M Third-trimester induced abortion in Georgia, 1979 and 1980. Am J Public Health. 1983;73:594-595. Centers for Disease Control and Prevention. Current trends third-trimester induced abortion—Georgia, 1979 and 1980. MMWR Morb Mortal Wkly Rep. 1983;32: 246-247.
Copyright © 2014 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
