Use of Death Records to Augment Notifiable Conditions Reporting in Washington State Hanna N. Oltean, MPH; Charla A. DeBolt, MPH, RN; Marcia J. Goldoft, MD, MPH, MS; Kathryn H. Lofy, MD rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr
Context: Health care providers are required to report newly diagnosed notifiable conditions including the case’s vital status according to state regulations, but it is uncertain how many cases remain unreported. Death certificates could potentially serve as a data source for detecting unreported deaths due to notifiable conditions. Objective: We sought to evaluate the usefulness of electronic death records to augment notifiable conditions reporting in Washington State. Design: Cross-sectional study. Setting: All residents of Washington State. Participants: Decedents from 2010-2012. Main Outcome Measures: Total number of fatal cases of acute infectious notifiable conditions in Washington residents estimated by capture-recapture analysis, proportion of estimated fatal cases reported to Washington’s notifiable conditions database (Public Health Issue Management System [PHIMS]), and the proportion of estimated fatal cases identified solely from the death records. Information was obtained by searching multiple cause-of death fields on 2010-2012 death records for keywords for acute infectious notifiable conditions. Results: Capture-recapture analysis estimated 317 fatal cases of these conditions could be expected over the 3 years studied (95% CI: 276,358). Public Health Issue Management System alone identified 38% of total estimated cases; using PHIMS and death record data increased identification to 71%. Electronic filing of death records was very timely, with a median of 4 days to visibility. Death record data were highly complete. Conclusions: Use of death records will augment the notifiable condition reporting system and potentially improve mortality estimates and disease control. KEY WORDS: epidemiologic surveillance, public health
surveillance, death certificates
J Public Health Management Practice, 2016, 22(3), 283–289 C 2016 Wolters Kluwer Health, Inc. All rights reserved. Copyright
Efficient, complete, and timely surveillance systems are essential for monitoring public health trends and guiding public health activities.1-3 For many conditions and health problems, surveillance data are used to estimate the scope and magnitude of a problem, detect changes in health practices, monitor changes in infectious and environmental agents, evaluate control measures, and describe the natural history of a health event in a community.3 However, many infectious diseases are unique in that, unlike noncommunicable disease, there is potentially an immediate opportunity to limit further spread of the disease to other members of the population. In these instances, timely knowledge of each case is important to public health practice regardless of the effect on estimated incidence or prevalence in the community. A degree of underestimation and uncertainty remains around the true incidence of diseases in most surveillance systems.1 Infectious diseases are particularly prone to underestimation due to characteristics such as asymptomatic carriage or self-limiting disease course.1 There are 2 distinct contributors to underestimation: underascertainment and underreporting. Underascertainment occurs when individuals do not seek health care and therefore are never captured by the
Author Affiliation: Office of Communicable Disease Epidemiology, Washington State Department of Health, Shoreline. The authors thank Amy Poel from the Washington Department of Health Center for Health Statistics for her help in obtaining death certificate data, and for her explanation of the current vital record systems in Washington. This report was supported in part by an appointment to the Applied Epidemiology Fellowship Program administered by the Council of State and Territorial Epidemiologists (CSTE) and funded by the Centers for Disease Control and Prevention (CDC) Cooperative Agreement Number 5U38HM000414-5. No conflicts of interest are declared. Supplemental digital content is available for this article. Direct URL citation appears in the printed text and is provided in the HTML and PDF versions of this article on the journal’s Web site (http://www.JPHMP.com). Correspondence: Hanna N. Oltean, MPH, Office of Communicable Disease Epidemiology, Washington State Department of Health, Shoreline, WA ([email protected]). DOI: 10.1097/PHH.0000000000000245
283 Copyright © 2016 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
284 ❘ Journal of Public Health Management and Practice surveillance systems. Underreporting occurs when individuals do seek health care, but the event is not captured by a surveillance system.1 Underestimation varies greatly by notifiable condition; one review of the literature indicated a range in completeness of disease reporting from 9% to 99% by condition.4 The use of electronic health information systems, such as electronic laboratory reporting, should help reduce dependence on individual providers and improve completeness of disease reporting.4-6 However, the completeness of disease reporting for some conditions remains low.6,7 This suggests that the identification of novel ways to improve notifiable conditions surveillance systems remains necessary. The notifiable conditions list in Washington State for 2010-2012 includes 63 specific reportable conditions, as well as 10 rare diseases of public health significance and unexplained critical illness or death in persons aged 1 to 49 years.8,9 Death certificate data include multiple cause-of-death fields and could potentially serve as a data source for detecting unreported deaths due to notifiable conditions and unexplained deaths potentially due to infectious diseases.10 Use of death certificates to augment surveillance systems has previously been reported for a range of conditions, including influenza and tuberculosis.11-14 Death certificates in Washington State are filed either on paper or electronically, through the Department of Health (DOH) Electronic Death Registry System (EDRS).15 Electronic Death Registry System searches keywords related to cause of death, including deaths due to some notifiable conditions. If identified, the system automatically sends an alert message to the death certifier. The alert message identifies the condition as a “rare cause term” and asks the certifier to “double check and verify” the cause of death. This automatic keyword search and alert system might improve the positive predictive value of reporting on deaths due to notifiable conditions. During the study period, the proportion of records filed electronically rapidly increased, growing from 1.2% in 2010 to 29.5% in 2012. This increase is expected to continue as Washington moves to a completely electronic system. Death certificates filed on paper can be held by the county for up to 60 days before submission to DOH. Once sent to the state, paper records are hand-entered in the electronic database by DOH employees, a process that may take up to 1 month, depending on current capacity for data entry. In general, there is a delay up to 3 months from date of death to electronic record creation for records filed and submitted on paper.15 There is no system in place to alert death certifiers when death records filed on paper contain keywords related to rare causes.
Once per week, DOH sends all newly received death certificates to the National Center for Health Statistics (NCHS) for ICD-10 (International Classification of Diseases, Tenth Revision) coding. The death certificate cause of death fields are coded and returned to DOH at the convenience of NCHS, adding additional time to the process required for completion of death certificates. The average time from delivery to NCHS to return to DOH is approximately 2 weeks, with a range from 2 days to 4 weeks. The objective of this project is threefold. First, we evaluate the usefulness of electronic death records to augment notifiable conditions reporting in Washington State. Specifically, we assess the proportion of deaths due to selected acute notifiable conditions missing from our notifiable conditions surveillance system for each condition, the timeliness of receipt of death record data, the completeness of death record data, and the simplicity of augmenting current surveillance by using this system. Second, we conduct a capture-recapture analysis to estimate the total number of fatal notifiable conditions in the population and therefore estimate underreporting. Last, we evaluate differences between records filed electronically and those filed by paper to determine if delays in reporting and lack of an alert system for paper records significantly affect notifiable conditions reporting.
● Methods Three years of retrospective death records data (20102012) were obtained from the DOH Center for Health Statistics. Data included both ICD-10 coded as well as text “literal” cause of death fields; text fields were searched instead of ICD-10 codes to allow for immediate use of death records after receipt by DOH in a prospective system. SAS 9.3 (SAS Institute Inc, Cary, North Carolina) was used to search the data for keywords related to selected acute notifiable conditions (Supplemental Digital Content, Appendix A, available at http://links.lww.com/JPHMP/A144). Files containing deaths due to notifiable conditions were then matched to deaths in the Public Health Issue Management System (PHIMS), the current electronic notifiable condition surveillance database for Washington State. Several infectious notifiable conditions were excluded from this search, as they were not captured in PHIMS over the study period and therefore could not be matched. These include chancroid, chlamydia, gonorrhea, granuloma inguinale, herpes simplex, HIV, lymphogranuloma venereum, prion disease, syphilis, tuberculosis, chronic hepatitis B, and chronic hepatitis C. Although many notifiable conditions do not regularly lead to death, some conditions may contribute to
Copyright © 2016 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
Use of Death Records to Augment Notifiable Conditions
death or be included as underlying causes on death certificates. We included all of the acute notifiable conditions tracked in PHIMS to test the usefulness of this surveillance system in identifying such deaths. Matching for cases of fatal notifiable conditions in the death records and deaths in PHIMS was completed using both Link Plus, a probabilistic record linkage program, and searching ad hoc, using first and last names, as well as date of birth and date of death. Phonetic spellings and switching of first and last names was allowed. If deaths due to notifiable conditions were found in the death certificates but not among deaths in PHIMS, all cases of the notifiable condition in PHIMS were searched for a match to the death record. Likewise, if fatal cases were found in PHIMS but not found through the keyword search of the death records, all death records were searched using available identifying information. Capture-recapture analysis was used to estimate the total number of fatal cases of the notifiable conditions included in this search in the population of Washington State. The 2 original data sets (all death records containing notifiable condition terms and all PHIMS records noting fatalities) were used as the 2 samples in this calculation. Several conditions with relatively high prevalence over the study period were analyzed individually; these included influenza, legionellosis, listeriosis, and varicella. In addition, the overall number of cases of fatal notifiable conditions was estimated. The Seber adjustment to the Peterson estimator was used for individual conditions due to small sample sizes. The completeness of both databases was also estimated on the basis of capture-recapture, as well as 95% confidence intervals of the estimates. To assess timeliness of the system, data from all death records for 2010-2012 were used. Timeliness was defined as the number of days between day of death and visibility to DOH of the death record. Overall timeliness of all death certificates was calculated, as well as timeliness of receipt of paper records compared to electronic records or mixed records (those completed partially on paper and partially electronically). The proportion of death records filed electronically versus on paper was calculated across each year of the study. Data completeness was assessed using all death records containing a notifiable condition search term selected for this project. Variables assessed for completeness included: last name, date of birth, date of death, race, ethnicity, sex, zip code, educational level, date the record was received by DOH, and method by which the record was filed. An evaluation of differences between records filed electronically and by paper was conducted to determine if delays in reporting and lack of an alert system
❘ 285
in the paper-based process affected notifiable conditions reporting. Records were categorized according to the method by which they were filed (paper, electronic, or mixed methods), and analysis of variance was conducted to determine if differences exist in notifiable condition reporting to PHIMS for the 3 filing methods.
● Results Death records for 2010-2012 contained 167 deaths due to the selected notifiable conditions (Figure 1, SDC Table 1, available at http://links.lww.com/JPHMP/ A145). A total of 122 deaths due to these notifiable conditions were reported in PHIMS during the same time period. In total, 64 deaths were detected in both PHIMS and the death certificates, 58 were detected in PHIMS but not initially in the death certificates, and 103 were detected in the death certificates but not initially in PHIMS (Figure 1). Of the 58 deaths in PHIMS that were not identified through the keyword search of the death certificates, 42 were matched by linking records using identifying information (last name, first name, and/or date of birth). For 41 of these deaths, the cause of death on the certificate was not specific enough to yield results in the keyword search. For one match, the cause of death listed both influenza A and haemophilus influenza, which was an exclusionary term for influenza deaths. Therefore, this death was originally automatically deleted from the data set. The remaining 16 deaths in PHIMS could not be matched to a death record. Of these, 4 records listed no date of death in PHIMS and no record of autopsy and may have been mistakenly marked as “died from illness.” Three others are known to have died outside of Washington State (2 in California and 1 in Canada); therefore, Washington does not have jurisdiction over the death and names are not included on the death certificate. The final 9 people had a date of death listed and addresses in Washington; therefore, they may have been missed due to undeclared death out-of-state or a mismatch of identifiers between databases such as a changed name. Of the 167 notifiable condition deaths identified through the death records, a total of 103 (61.7%) were detected through the death certificate keyword search but were not immediately matched to PHIMS. Five of these fatal cases of notifiable conditions found in the death records had also been reported to PHIMS as cases but not marked in PHIMS as “died from illness.” These were later matched by searching all cases of the condition in PHIMS by identifying information. The remaining 98 cases of fatal notifiable conditions identified in the death records were never reported in PHIMS.
Copyright © 2016 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
286 ❘ Journal of Public Health Management and Practice FIGURE 1 ● Deaths Reported to PHIMS and Deaths Found Through the Notifiable Condition Keyword Search for
2010-2012 Death Records qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq
Capture-recapture analysis was used to compare data obtained through both case-finding methods: death records and fatal cases reported to PHIMS (Table). The PHIMS showed a maximum conditionspecific sensitivity of 100% (listeriosis), as did the death records (for varicella). The PHIMS showed a minimum condition-specific sensitivity of 9.09% (varicella), whereas the minimum for the death records was 31.3% (legionellosis). Overall estimations indicate that 317 cases of fatal notifiable conditions could be expected over the 3-year time period studied (95% CI: 276 358). The PHIMS alone identified 38% of these total estimated cases; using PHIMS with death record data increased identification to 71%. Timeliness of death records varied greatly depending on the method of filing (Figure 2). Overall, the average time from date of death to visibility of the death
certificate at DOH was 65 days (median = 75 days). If a record was filed via paper, this average increased to 89.5 days, with a median of 84 days. Electronic filing of the record was found to be much timelier, with an average of 9.9 days to visibility and a median of only 4 days. Likewise, if a record was considered “mixed” (completed partly via paper and partly electronically), the average time to visibility was 6.6 days with a median of 4 days. This shorter time to visibility for mixed compared to electronic records was assessed and determined to be due to external cause deaths (accidents, suicides, homicides, etc). External cause deaths are certified by a medical examiner, coroner, or prosecuting attorney; these groups are required to agree to file electronically before EDRS is implemented in a county. Thus, external cause deaths are filed either on paper or electronically, never by mixed methods. Because
TABLE ● Capture-Recapture Estimation of Total Number of Fatal Cases of Notifiable Conditions in Washington State
qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq
Condition Influenza Legionella Listeria Varicella All notifiable conditions
Total Estimated No. of Fatal Cases
95% CI of Estimate
Estimated Completeness of PHIMS Reporting, %
Estimated Completeness of Death Records, %
175 16 8 11 317
147-204 1120 ... ... 276-358
37.7 81.3 100 9.09 38.5
56.6 31.3 75.0 100 52.7
Abbreviations: CI, confidence interval; PHIMS, Public Health Issue Management System.
Copyright © 2016 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
Use of Death Records to Augment Notifiable Conditions
FIGURE 2 ● Number of Death Certificates Visible to DOH
by Days After Death, All Death Records 2010-2012 qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq
these death certificates are often held much longer than deaths due to natural causes, they contribute to a longer mean time to visibility for electronic records compared to mixed records. When external cause deaths are excluded from the timeliness calculations, the mean timeliness for electronic records decreases to 6.7 days. Although the majority of records are currently filed via paper, the proportion filed electronically increased over the years studied. In 2010, only 1.2% of death records were filed electronically; by 2012, the percentage had increased to 29.5%. Data from the 167 death records for notifiable conditions were very complete. All variables analyzed, including last name, first name, date of birth, date of death, date received, and educational level were each more than 98% complete. To determine if the electronic keyword alert system changed reporting patterns among clinicians, medical examiners, or coroners filling out death records, we looked at the percentage of deaths due to notifiable conditions that were reported in PHIMS on the basis of the method by which they were filed. Approximately 43% of paper death records including keywords for notifiable conditions had been reported as a case in PHIMS. Of the records filed electronically, 42% had been reported as a case in PHIMS; of mixed files, only 29% were listed in PHIMS. Analysis of variance did not show a significant difference in percent reported to PHIMS by method of filing, indicating that the keyword alert system available in EDRS has yet to significantly impact reporting of notifiable conditions (data not shown).
❘ 287
● Discussion The results of this analysis suggest that the use of death records for prospective surveillance may provide valuable additional information to augment the traditional notifiable condition surveillance system. We have shown that use of the electronic death files has the potential to almost double the number of cases of fatal notifiable conditions currently identified by public health. This knowledge could potentially improve our understanding of the impact of notifiable conditions on the population and increase our ability to respond to communicable diseases. As more records are received electronically, timeliness of data will continue to improve and data on acute infections will become increasingly useful. Surprisingly, mixed records were found to have the same median timeliness as electronic records and a shorter mean time to visibility. The main difference between mixed and electronic records is the method death certifiers use to fill in the record. For mixed records, certifiers use a paper form, whereas for electronic records, it is filled out electronically. It is conceivable that certifiers take the same amount of time to fill in information using either method, explaining the same median timeliness between the 2 record types. All other steps in the filing process are handled electronically by the funeral home and the county registrar. This was an unexpected finding of this study, and it shows that any use of EDRS in a county will increase timeliness. There are several potential limitations to this study and the methodology of identifying deaths due to notifiable conditions using death record data. First, the capture-recapture analysis assumes that the cases of fatal notifiable conditions detected in the death certificates and in PHIMS meet case definition for the respective conditions. In PHIMS, the case classification can be used to determine the number of true cases; in EDRS, the number of cases meeting case definition cannot be determined without further investigation. There are likely some false-positive deaths that were captured in our keyword search of EDRS; therefore, the number of true cases of fatal notifiable conditions in this sample is likely lower than the number identified and used in the capture-recapture calculation. Conducting follow-up investigations to clarify which cases meet case definition would improve our estimate of cases of fatal notifiable conditions captured by EDRS and thus improve our estimate of the number of cases missed by both systems. Prospectively, this system will require additional follow-up to determine if fatalities detected through death records using the keyword search terms meet surveillance case definitions for notifiable conditions. We expect that the positive predictive value of the
Copyright © 2016 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
288 ❘ Journal of Public Health Management and Practice death records will vary by condition. For example, we have determined with prospective follow-up that the keyword search terms for varicella generally indicate shingles rather than primary varicella infection, leading to a low positive predictive value for deaths due to primary varicella infection. Second, the study did not include data on infectious notifiable conditions not captured by PHIMS. Prospectively, deaths due to these notifiable conditions will be investigated by department staff. Personal communication from these staff to-date indicates that death records are similarly useful for these conditions, as demonstrated previously for tuberculosis.11 Third, this analysis was largely driven by influenza deaths, which accounted for more than half of deaths due to notifiable conditions identified in the death records and more than half of fatal cases of notifiable conditions estimated through the capture-recapture analysis. Fourth, there are discrepancies between the death records and the PHIMS records. Discrepant residencies or mismatched identifying information between PHIMS and the death records could lead to a nonmatched case, as well as an overestimation of the true number of fatal cases in the population. Finally, this system will only improve capture of cases of fatal notifiable conditions—any nonfatal cases will likely continue to be missed in similar numbers to present without further improvements in reporting. Strengths of this system include the use of the population-wide data set of death records, data completeness, simplicity, and timeliness. Although capturerecapture analysis indicates that fatal cases of notifiable conditions are still being missed, we found that among death records that contained notifiable condition keywords, data completeness was high. The system is fairly simple and straightforward, and SAS code for conducting electronic searches has now been written and can be used on a continuing basis to search newly uploaded death records. Electronic or mixed data was found to be very timely from date of death; this will be useful for communicable disease surveillance. However, it should be noted that timeliness was assessed from date of death, not from the date of disease onset. Future investigations into the true timeliness (time from disease onset to reporting) of the system are needed. As more records are received electronically, overall timeliness of data should improve. An assessment of death records filed during the first 6 months of 2014 indicated that more than 57% of records were filed using electronic or mixed methods, and overall timeliness decreased to an average of 28 days, with a median time of 9 days. Use of the death records to augment current notifiable condition reporting can become a stable system to increase sensitivity for fatal notifiable
conditions and improve knowledge of communicable disease in the population of Washington State. The completeness of reporting deaths due to some notifiable conditions through PHIMS remains low. Reasons for the large percentage of missing cases should be explored in-depth to determine if any systemic problems with capturing cases exist. It is likely that underreporting contributes to missing cases; some providers and laboratories may be unaware of their responsibility to report. In addition, not all laboratories in Washington currently report electronically, and this most likely contributes to the number of missing cases. Continuing education on reporting should be provided to laboratories and providers, as well as continued and expeditious transition to electronic data reporting. Novel ways of improving notifiable conditions surveillance systems are also needed. One way to improve reporting could be tied to electronic death files. For these electronic files, keywords related to cause of death, including some notifiable conditions, automatically alert the coroner or medical examiner in the county. This automatic keyword search may improve reporting on deaths due to notifiable conditions; however, we found no difference between reporting for death records filed electronically and those filed on paper. There are potentially several factors influencing this. First, counties generally report either through the paper system or through the electronic system, so differences in county reporting practices could be confounding these results. Second, the list of “rare cause terms” that trigger an alert is not comprehensive and should be updated to include common terms for all notifiable conditions. Last, the messages that death certifiers currently receive through the system do not specify the need to report and some death certifiers may not be aware of their responsibility. Death certifiers should be educated on their responsibility to report all notifiable conditions diagnosed at or around death.
● Conclusion Use of electronic death records is a novel way to increase notifiable condition reporting and improve mortality estimates for notifiable conditions. States should consider exploring the use of electronic death records and other electronic sources for notifiable condition surveillance. REFERENCES 1. Gibbons CL, Mangen MJJ, Plass D, et al. Measuring underreporting and under-ascertainment in infectious disease datasets: a comparison of methods. BMC Public Health. 2014;14:147.
Copyright © 2016 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
Use of Death Records to Augment Notifiable Conditions
2. German RR, Lee LM, Horan JM, et al. Updated guidelines for evaluating public health surveillance systems. MMWR. 2001;50(RR13):1-35. 3. Thacker SB, Qualters JR, Lee LM. Public health surveillance in the United States: evolution and challenges. MMWR. 2012;61:3-9. 4. Doyle TJ, Glynn MK, Groseclose SL. Completeness of notifiable infectious disease reporting in the United States: an analytical literature review. Am J Epidemiol. 2002;155(9):866874. 5. Overhage JM, Grannis S, McDonald CJ. A comparison of the completeness and timeliness of automated electronic laboratory reporting and spontaneous reporting of notifiable conditions. Am J Public Health. 2008;98(2):344350. 6. Silk BJ, Berkelman RL. A review of strategies for enhancing the completeness of notifiable disease reporting. J Public Health Manag Pract. 2005;11(3):191-200. 7. Sickbert-Bennett EE, Weber DJ, Poole C, MacDonald PDM, Maillard JM. Completeness of communicable disease reporting, North Carolina, USA, 1995-1997 and 2000-2006. Emerg Infect Dis. 2011;17(1):23-29. 8. Washington State Department of Health. List of notifiable conditions. http://www.doh.wa.gov/ForPublicHealthand HealthcareProviders/NotifiableConditions/ListofNotifiable Conditions.aspx. Accessed September 13, 2013.
❘ 289
9. Washington State Legislature. Washington administrative code: chapter 246-101, notifiable conditions. http://apps.leg. wa.gov/wac/default.aspx?cite=246-101. Accessed September 13, 2013. 10. Lees CH, Avery C, Asherin R, et al. Pandemic (H1N1) 2009associated deaths detected by unexplained death and medical examiner surveillance. Emerg Infect Dis. 2011;17(8):14791483. 11. Gallivan MD, Lofy KH, Goldbaum GM. Use of death certificates to identify tuberculosis-related deaths in Washington State . J Public Health Manag Pract. http://www.ncbi.nlm.nih. gov/pubmed/24149649. Published October 2013. Accessed August 1, 2014. 12. Curtis AB, McCray E, McKenna M, Onorato IM. Completeness and timeliness of tuberculosis case reporting. A multistate study. Am J Prev Med. 2001;20(2):108-112. 13. Hoefer D, Cherry B, Kacica M, McClamroch K, Kilby K. Pediatric influenza-associated deaths in New York State: death certificate coding and comparison to laboratoryconfirmed deaths. Influenza Res Treat. 2012;2012:397890. doi: 10.1155/2012/397890. 14. Bancroft E, Lee S. Use of electronic death certificates for influenza death surveillance. Emerg Infect Dis. 2014;20(1):78-82. 15. Washington State Department of Health. EDRS information. https://fortress.wa.gov/doh/edrsinformation/EDRS Information/Info.aspx. Accessed September 13, 2013.
Copyright © 2016 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
Context: Health care providers are required to report newly diagnosed notifiable conditions including the case’s vital status according to state regulations, but it is uncertain how many cases remain unreported. Death certificates could potentially serve as a data source for detecting unreported deaths due to notifiable conditions. Objective: We sought to evaluate the usefulness of electronic death records to augment notifiable conditions reporting in Washington State. Design: Cross-sectional study. Setting: All residents of Washington State. Participants: Decedents from 2010-2012. Main Outcome Measures: Total number of fatal cases of acute infectious notifiable conditions in Washington residents estimated by capture-recapture analysis, proportion of estimated fatal cases reported to Washington’s notifiable conditions database (Public Health Issue Management System [PHIMS]), and the proportion of estimated fatal cases identified solely from the death records. Information was obtained by searching multiple cause-of death fields on 2010-2012 death records for keywords for acute infectious notifiable conditions. Results: Capture-recapture analysis estimated 317 fatal cases of these conditions could be expected over the 3 years studied (95% CI: 276,358). Public Health Issue Management System alone identified 38% of total estimated cases; using PHIMS and death record data increased identification to 71%. Electronic filing of death records was very timely, with a median of 4 days to visibility. Death record data were highly complete. Conclusions: Use of death records will augment the notifiable condition reporting system and potentially improve mortality estimates and disease control. KEY WORDS: epidemiologic surveillance, public health
surveillance, death certificates
J Public Health Management Practice, 2016, 22(3), 283–289 C 2016 Wolters Kluwer Health, Inc. All rights reserved. Copyright
Efficient, complete, and timely surveillance systems are essential for monitoring public health trends and guiding public health activities.1-3 For many conditions and health problems, surveillance data are used to estimate the scope and magnitude of a problem, detect changes in health practices, monitor changes in infectious and environmental agents, evaluate control measures, and describe the natural history of a health event in a community.3 However, many infectious diseases are unique in that, unlike noncommunicable disease, there is potentially an immediate opportunity to limit further spread of the disease to other members of the population. In these instances, timely knowledge of each case is important to public health practice regardless of the effect on estimated incidence or prevalence in the community. A degree of underestimation and uncertainty remains around the true incidence of diseases in most surveillance systems.1 Infectious diseases are particularly prone to underestimation due to characteristics such as asymptomatic carriage or self-limiting disease course.1 There are 2 distinct contributors to underestimation: underascertainment and underreporting. Underascertainment occurs when individuals do not seek health care and therefore are never captured by the
Author Affiliation: Office of Communicable Disease Epidemiology, Washington State Department of Health, Shoreline. The authors thank Amy Poel from the Washington Department of Health Center for Health Statistics for her help in obtaining death certificate data, and for her explanation of the current vital record systems in Washington. This report was supported in part by an appointment to the Applied Epidemiology Fellowship Program administered by the Council of State and Territorial Epidemiologists (CSTE) and funded by the Centers for Disease Control and Prevention (CDC) Cooperative Agreement Number 5U38HM000414-5. No conflicts of interest are declared. Supplemental digital content is available for this article. Direct URL citation appears in the printed text and is provided in the HTML and PDF versions of this article on the journal’s Web site (http://www.JPHMP.com). Correspondence: Hanna N. Oltean, MPH, Office of Communicable Disease Epidemiology, Washington State Department of Health, Shoreline, WA ([email protected]). DOI: 10.1097/PHH.0000000000000245
283 Copyright © 2016 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
284 ❘ Journal of Public Health Management and Practice surveillance systems. Underreporting occurs when individuals do seek health care, but the event is not captured by a surveillance system.1 Underestimation varies greatly by notifiable condition; one review of the literature indicated a range in completeness of disease reporting from 9% to 99% by condition.4 The use of electronic health information systems, such as electronic laboratory reporting, should help reduce dependence on individual providers and improve completeness of disease reporting.4-6 However, the completeness of disease reporting for some conditions remains low.6,7 This suggests that the identification of novel ways to improve notifiable conditions surveillance systems remains necessary. The notifiable conditions list in Washington State for 2010-2012 includes 63 specific reportable conditions, as well as 10 rare diseases of public health significance and unexplained critical illness or death in persons aged 1 to 49 years.8,9 Death certificate data include multiple cause-of-death fields and could potentially serve as a data source for detecting unreported deaths due to notifiable conditions and unexplained deaths potentially due to infectious diseases.10 Use of death certificates to augment surveillance systems has previously been reported for a range of conditions, including influenza and tuberculosis.11-14 Death certificates in Washington State are filed either on paper or electronically, through the Department of Health (DOH) Electronic Death Registry System (EDRS).15 Electronic Death Registry System searches keywords related to cause of death, including deaths due to some notifiable conditions. If identified, the system automatically sends an alert message to the death certifier. The alert message identifies the condition as a “rare cause term” and asks the certifier to “double check and verify” the cause of death. This automatic keyword search and alert system might improve the positive predictive value of reporting on deaths due to notifiable conditions. During the study period, the proportion of records filed electronically rapidly increased, growing from 1.2% in 2010 to 29.5% in 2012. This increase is expected to continue as Washington moves to a completely electronic system. Death certificates filed on paper can be held by the county for up to 60 days before submission to DOH. Once sent to the state, paper records are hand-entered in the electronic database by DOH employees, a process that may take up to 1 month, depending on current capacity for data entry. In general, there is a delay up to 3 months from date of death to electronic record creation for records filed and submitted on paper.15 There is no system in place to alert death certifiers when death records filed on paper contain keywords related to rare causes.
Once per week, DOH sends all newly received death certificates to the National Center for Health Statistics (NCHS) for ICD-10 (International Classification of Diseases, Tenth Revision) coding. The death certificate cause of death fields are coded and returned to DOH at the convenience of NCHS, adding additional time to the process required for completion of death certificates. The average time from delivery to NCHS to return to DOH is approximately 2 weeks, with a range from 2 days to 4 weeks. The objective of this project is threefold. First, we evaluate the usefulness of electronic death records to augment notifiable conditions reporting in Washington State. Specifically, we assess the proportion of deaths due to selected acute notifiable conditions missing from our notifiable conditions surveillance system for each condition, the timeliness of receipt of death record data, the completeness of death record data, and the simplicity of augmenting current surveillance by using this system. Second, we conduct a capture-recapture analysis to estimate the total number of fatal notifiable conditions in the population and therefore estimate underreporting. Last, we evaluate differences between records filed electronically and those filed by paper to determine if delays in reporting and lack of an alert system for paper records significantly affect notifiable conditions reporting.
● Methods Three years of retrospective death records data (20102012) were obtained from the DOH Center for Health Statistics. Data included both ICD-10 coded as well as text “literal” cause of death fields; text fields were searched instead of ICD-10 codes to allow for immediate use of death records after receipt by DOH in a prospective system. SAS 9.3 (SAS Institute Inc, Cary, North Carolina) was used to search the data for keywords related to selected acute notifiable conditions (Supplemental Digital Content, Appendix A, available at http://links.lww.com/JPHMP/A144). Files containing deaths due to notifiable conditions were then matched to deaths in the Public Health Issue Management System (PHIMS), the current electronic notifiable condition surveillance database for Washington State. Several infectious notifiable conditions were excluded from this search, as they were not captured in PHIMS over the study period and therefore could not be matched. These include chancroid, chlamydia, gonorrhea, granuloma inguinale, herpes simplex, HIV, lymphogranuloma venereum, prion disease, syphilis, tuberculosis, chronic hepatitis B, and chronic hepatitis C. Although many notifiable conditions do not regularly lead to death, some conditions may contribute to
Copyright © 2016 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
Use of Death Records to Augment Notifiable Conditions
death or be included as underlying causes on death certificates. We included all of the acute notifiable conditions tracked in PHIMS to test the usefulness of this surveillance system in identifying such deaths. Matching for cases of fatal notifiable conditions in the death records and deaths in PHIMS was completed using both Link Plus, a probabilistic record linkage program, and searching ad hoc, using first and last names, as well as date of birth and date of death. Phonetic spellings and switching of first and last names was allowed. If deaths due to notifiable conditions were found in the death certificates but not among deaths in PHIMS, all cases of the notifiable condition in PHIMS were searched for a match to the death record. Likewise, if fatal cases were found in PHIMS but not found through the keyword search of the death records, all death records were searched using available identifying information. Capture-recapture analysis was used to estimate the total number of fatal cases of the notifiable conditions included in this search in the population of Washington State. The 2 original data sets (all death records containing notifiable condition terms and all PHIMS records noting fatalities) were used as the 2 samples in this calculation. Several conditions with relatively high prevalence over the study period were analyzed individually; these included influenza, legionellosis, listeriosis, and varicella. In addition, the overall number of cases of fatal notifiable conditions was estimated. The Seber adjustment to the Peterson estimator was used for individual conditions due to small sample sizes. The completeness of both databases was also estimated on the basis of capture-recapture, as well as 95% confidence intervals of the estimates. To assess timeliness of the system, data from all death records for 2010-2012 were used. Timeliness was defined as the number of days between day of death and visibility to DOH of the death record. Overall timeliness of all death certificates was calculated, as well as timeliness of receipt of paper records compared to electronic records or mixed records (those completed partially on paper and partially electronically). The proportion of death records filed electronically versus on paper was calculated across each year of the study. Data completeness was assessed using all death records containing a notifiable condition search term selected for this project. Variables assessed for completeness included: last name, date of birth, date of death, race, ethnicity, sex, zip code, educational level, date the record was received by DOH, and method by which the record was filed. An evaluation of differences between records filed electronically and by paper was conducted to determine if delays in reporting and lack of an alert system
❘ 285
in the paper-based process affected notifiable conditions reporting. Records were categorized according to the method by which they were filed (paper, electronic, or mixed methods), and analysis of variance was conducted to determine if differences exist in notifiable condition reporting to PHIMS for the 3 filing methods.
● Results Death records for 2010-2012 contained 167 deaths due to the selected notifiable conditions (Figure 1, SDC Table 1, available at http://links.lww.com/JPHMP/ A145). A total of 122 deaths due to these notifiable conditions were reported in PHIMS during the same time period. In total, 64 deaths were detected in both PHIMS and the death certificates, 58 were detected in PHIMS but not initially in the death certificates, and 103 were detected in the death certificates but not initially in PHIMS (Figure 1). Of the 58 deaths in PHIMS that were not identified through the keyword search of the death certificates, 42 were matched by linking records using identifying information (last name, first name, and/or date of birth). For 41 of these deaths, the cause of death on the certificate was not specific enough to yield results in the keyword search. For one match, the cause of death listed both influenza A and haemophilus influenza, which was an exclusionary term for influenza deaths. Therefore, this death was originally automatically deleted from the data set. The remaining 16 deaths in PHIMS could not be matched to a death record. Of these, 4 records listed no date of death in PHIMS and no record of autopsy and may have been mistakenly marked as “died from illness.” Three others are known to have died outside of Washington State (2 in California and 1 in Canada); therefore, Washington does not have jurisdiction over the death and names are not included on the death certificate. The final 9 people had a date of death listed and addresses in Washington; therefore, they may have been missed due to undeclared death out-of-state or a mismatch of identifiers between databases such as a changed name. Of the 167 notifiable condition deaths identified through the death records, a total of 103 (61.7%) were detected through the death certificate keyword search but were not immediately matched to PHIMS. Five of these fatal cases of notifiable conditions found in the death records had also been reported to PHIMS as cases but not marked in PHIMS as “died from illness.” These were later matched by searching all cases of the condition in PHIMS by identifying information. The remaining 98 cases of fatal notifiable conditions identified in the death records were never reported in PHIMS.
Copyright © 2016 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
286 ❘ Journal of Public Health Management and Practice FIGURE 1 ● Deaths Reported to PHIMS and Deaths Found Through the Notifiable Condition Keyword Search for
2010-2012 Death Records qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq
Capture-recapture analysis was used to compare data obtained through both case-finding methods: death records and fatal cases reported to PHIMS (Table). The PHIMS showed a maximum conditionspecific sensitivity of 100% (listeriosis), as did the death records (for varicella). The PHIMS showed a minimum condition-specific sensitivity of 9.09% (varicella), whereas the minimum for the death records was 31.3% (legionellosis). Overall estimations indicate that 317 cases of fatal notifiable conditions could be expected over the 3-year time period studied (95% CI: 276 358). The PHIMS alone identified 38% of these total estimated cases; using PHIMS with death record data increased identification to 71%. Timeliness of death records varied greatly depending on the method of filing (Figure 2). Overall, the average time from date of death to visibility of the death
certificate at DOH was 65 days (median = 75 days). If a record was filed via paper, this average increased to 89.5 days, with a median of 84 days. Electronic filing of the record was found to be much timelier, with an average of 9.9 days to visibility and a median of only 4 days. Likewise, if a record was considered “mixed” (completed partly via paper and partly electronically), the average time to visibility was 6.6 days with a median of 4 days. This shorter time to visibility for mixed compared to electronic records was assessed and determined to be due to external cause deaths (accidents, suicides, homicides, etc). External cause deaths are certified by a medical examiner, coroner, or prosecuting attorney; these groups are required to agree to file electronically before EDRS is implemented in a county. Thus, external cause deaths are filed either on paper or electronically, never by mixed methods. Because
TABLE ● Capture-Recapture Estimation of Total Number of Fatal Cases of Notifiable Conditions in Washington State
qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq
Condition Influenza Legionella Listeria Varicella All notifiable conditions
Total Estimated No. of Fatal Cases
95% CI of Estimate
Estimated Completeness of PHIMS Reporting, %
Estimated Completeness of Death Records, %
175 16 8 11 317
147-204 1120 ... ... 276-358
37.7 81.3 100 9.09 38.5
56.6 31.3 75.0 100 52.7
Abbreviations: CI, confidence interval; PHIMS, Public Health Issue Management System.
Copyright © 2016 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
Use of Death Records to Augment Notifiable Conditions
FIGURE 2 ● Number of Death Certificates Visible to DOH
by Days After Death, All Death Records 2010-2012 qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq
these death certificates are often held much longer than deaths due to natural causes, they contribute to a longer mean time to visibility for electronic records compared to mixed records. When external cause deaths are excluded from the timeliness calculations, the mean timeliness for electronic records decreases to 6.7 days. Although the majority of records are currently filed via paper, the proportion filed electronically increased over the years studied. In 2010, only 1.2% of death records were filed electronically; by 2012, the percentage had increased to 29.5%. Data from the 167 death records for notifiable conditions were very complete. All variables analyzed, including last name, first name, date of birth, date of death, date received, and educational level were each more than 98% complete. To determine if the electronic keyword alert system changed reporting patterns among clinicians, medical examiners, or coroners filling out death records, we looked at the percentage of deaths due to notifiable conditions that were reported in PHIMS on the basis of the method by which they were filed. Approximately 43% of paper death records including keywords for notifiable conditions had been reported as a case in PHIMS. Of the records filed electronically, 42% had been reported as a case in PHIMS; of mixed files, only 29% were listed in PHIMS. Analysis of variance did not show a significant difference in percent reported to PHIMS by method of filing, indicating that the keyword alert system available in EDRS has yet to significantly impact reporting of notifiable conditions (data not shown).
❘ 287
● Discussion The results of this analysis suggest that the use of death records for prospective surveillance may provide valuable additional information to augment the traditional notifiable condition surveillance system. We have shown that use of the electronic death files has the potential to almost double the number of cases of fatal notifiable conditions currently identified by public health. This knowledge could potentially improve our understanding of the impact of notifiable conditions on the population and increase our ability to respond to communicable diseases. As more records are received electronically, timeliness of data will continue to improve and data on acute infections will become increasingly useful. Surprisingly, mixed records were found to have the same median timeliness as electronic records and a shorter mean time to visibility. The main difference between mixed and electronic records is the method death certifiers use to fill in the record. For mixed records, certifiers use a paper form, whereas for electronic records, it is filled out electronically. It is conceivable that certifiers take the same amount of time to fill in information using either method, explaining the same median timeliness between the 2 record types. All other steps in the filing process are handled electronically by the funeral home and the county registrar. This was an unexpected finding of this study, and it shows that any use of EDRS in a county will increase timeliness. There are several potential limitations to this study and the methodology of identifying deaths due to notifiable conditions using death record data. First, the capture-recapture analysis assumes that the cases of fatal notifiable conditions detected in the death certificates and in PHIMS meet case definition for the respective conditions. In PHIMS, the case classification can be used to determine the number of true cases; in EDRS, the number of cases meeting case definition cannot be determined without further investigation. There are likely some false-positive deaths that were captured in our keyword search of EDRS; therefore, the number of true cases of fatal notifiable conditions in this sample is likely lower than the number identified and used in the capture-recapture calculation. Conducting follow-up investigations to clarify which cases meet case definition would improve our estimate of cases of fatal notifiable conditions captured by EDRS and thus improve our estimate of the number of cases missed by both systems. Prospectively, this system will require additional follow-up to determine if fatalities detected through death records using the keyword search terms meet surveillance case definitions for notifiable conditions. We expect that the positive predictive value of the
Copyright © 2016 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
288 ❘ Journal of Public Health Management and Practice death records will vary by condition. For example, we have determined with prospective follow-up that the keyword search terms for varicella generally indicate shingles rather than primary varicella infection, leading to a low positive predictive value for deaths due to primary varicella infection. Second, the study did not include data on infectious notifiable conditions not captured by PHIMS. Prospectively, deaths due to these notifiable conditions will be investigated by department staff. Personal communication from these staff to-date indicates that death records are similarly useful for these conditions, as demonstrated previously for tuberculosis.11 Third, this analysis was largely driven by influenza deaths, which accounted for more than half of deaths due to notifiable conditions identified in the death records and more than half of fatal cases of notifiable conditions estimated through the capture-recapture analysis. Fourth, there are discrepancies between the death records and the PHIMS records. Discrepant residencies or mismatched identifying information between PHIMS and the death records could lead to a nonmatched case, as well as an overestimation of the true number of fatal cases in the population. Finally, this system will only improve capture of cases of fatal notifiable conditions—any nonfatal cases will likely continue to be missed in similar numbers to present without further improvements in reporting. Strengths of this system include the use of the population-wide data set of death records, data completeness, simplicity, and timeliness. Although capturerecapture analysis indicates that fatal cases of notifiable conditions are still being missed, we found that among death records that contained notifiable condition keywords, data completeness was high. The system is fairly simple and straightforward, and SAS code for conducting electronic searches has now been written and can be used on a continuing basis to search newly uploaded death records. Electronic or mixed data was found to be very timely from date of death; this will be useful for communicable disease surveillance. However, it should be noted that timeliness was assessed from date of death, not from the date of disease onset. Future investigations into the true timeliness (time from disease onset to reporting) of the system are needed. As more records are received electronically, overall timeliness of data should improve. An assessment of death records filed during the first 6 months of 2014 indicated that more than 57% of records were filed using electronic or mixed methods, and overall timeliness decreased to an average of 28 days, with a median time of 9 days. Use of the death records to augment current notifiable condition reporting can become a stable system to increase sensitivity for fatal notifiable
conditions and improve knowledge of communicable disease in the population of Washington State. The completeness of reporting deaths due to some notifiable conditions through PHIMS remains low. Reasons for the large percentage of missing cases should be explored in-depth to determine if any systemic problems with capturing cases exist. It is likely that underreporting contributes to missing cases; some providers and laboratories may be unaware of their responsibility to report. In addition, not all laboratories in Washington currently report electronically, and this most likely contributes to the number of missing cases. Continuing education on reporting should be provided to laboratories and providers, as well as continued and expeditious transition to electronic data reporting. Novel ways of improving notifiable conditions surveillance systems are also needed. One way to improve reporting could be tied to electronic death files. For these electronic files, keywords related to cause of death, including some notifiable conditions, automatically alert the coroner or medical examiner in the county. This automatic keyword search may improve reporting on deaths due to notifiable conditions; however, we found no difference between reporting for death records filed electronically and those filed on paper. There are potentially several factors influencing this. First, counties generally report either through the paper system or through the electronic system, so differences in county reporting practices could be confounding these results. Second, the list of “rare cause terms” that trigger an alert is not comprehensive and should be updated to include common terms for all notifiable conditions. Last, the messages that death certifiers currently receive through the system do not specify the need to report and some death certifiers may not be aware of their responsibility. Death certifiers should be educated on their responsibility to report all notifiable conditions diagnosed at or around death.
● Conclusion Use of electronic death records is a novel way to increase notifiable condition reporting and improve mortality estimates for notifiable conditions. States should consider exploring the use of electronic death records and other electronic sources for notifiable condition surveillance. REFERENCES 1. Gibbons CL, Mangen MJJ, Plass D, et al. Measuring underreporting and under-ascertainment in infectious disease datasets: a comparison of methods. BMC Public Health. 2014;14:147.
Copyright © 2016 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
Use of Death Records to Augment Notifiable Conditions
2. German RR, Lee LM, Horan JM, et al. Updated guidelines for evaluating public health surveillance systems. MMWR. 2001;50(RR13):1-35. 3. Thacker SB, Qualters JR, Lee LM. Public health surveillance in the United States: evolution and challenges. MMWR. 2012;61:3-9. 4. Doyle TJ, Glynn MK, Groseclose SL. Completeness of notifiable infectious disease reporting in the United States: an analytical literature review. Am J Epidemiol. 2002;155(9):866874. 5. Overhage JM, Grannis S, McDonald CJ. A comparison of the completeness and timeliness of automated electronic laboratory reporting and spontaneous reporting of notifiable conditions. Am J Public Health. 2008;98(2):344350. 6. Silk BJ, Berkelman RL. A review of strategies for enhancing the completeness of notifiable disease reporting. J Public Health Manag Pract. 2005;11(3):191-200. 7. Sickbert-Bennett EE, Weber DJ, Poole C, MacDonald PDM, Maillard JM. Completeness of communicable disease reporting, North Carolina, USA, 1995-1997 and 2000-2006. Emerg Infect Dis. 2011;17(1):23-29. 8. Washington State Department of Health. List of notifiable conditions. http://www.doh.wa.gov/ForPublicHealthand HealthcareProviders/NotifiableConditions/ListofNotifiable Conditions.aspx. Accessed September 13, 2013.
❘ 289
9. Washington State Legislature. Washington administrative code: chapter 246-101, notifiable conditions. http://apps.leg. wa.gov/wac/default.aspx?cite=246-101. Accessed September 13, 2013. 10. Lees CH, Avery C, Asherin R, et al. Pandemic (H1N1) 2009associated deaths detected by unexplained death and medical examiner surveillance. Emerg Infect Dis. 2011;17(8):14791483. 11. Gallivan MD, Lofy KH, Goldbaum GM. Use of death certificates to identify tuberculosis-related deaths in Washington State . J Public Health Manag Pract. http://www.ncbi.nlm.nih. gov/pubmed/24149649. Published October 2013. Accessed August 1, 2014. 12. Curtis AB, McCray E, McKenna M, Onorato IM. Completeness and timeliness of tuberculosis case reporting. A multistate study. Am J Prev Med. 2001;20(2):108-112. 13. Hoefer D, Cherry B, Kacica M, McClamroch K, Kilby K. Pediatric influenza-associated deaths in New York State: death certificate coding and comparison to laboratoryconfirmed deaths. Influenza Res Treat. 2012;2012:397890. doi: 10.1155/2012/397890. 14. Bancroft E, Lee S. Use of electronic death certificates for influenza death surveillance. Emerg Infect Dis. 2014;20(1):78-82. 15. Washington State Department of Health. EDRS information. https://fortress.wa.gov/doh/edrsinformation/EDRS Information/Info.aspx. Accessed September 13, 2013.
Copyright © 2016 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
