Evaluation of Immunization Data Completeness Within a Large Community Health Care System Exchanging Data With a State Immunization Information System Bryan K. Hendrickson, MS, MHSM; Sarada S. Panchanathan, MD, MS; Diana Petitti, MD, MPH rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr
Context: Information systems are used by most states to maintain registries of immunization data both for monitoring population-level adherence and for use in clinical practice and research. Direct data exchange between such systems and electronic health record systems presents an opportunity to improve the completeness and quality of information available. Objective: Our goals were to describe and compare the completeness of the Arizona State Immunization System, the electronic health record at a large community health provider in Arizona exchanging electronic data with the Arizona system, and personal immunization records in an effort to contribute to the discussion on the completeness of state-run immunization registries and data exchange with these registries. Design: Immunization histories from these sources were collected and reviewed sequentially. Unique dates of vaccination administrations were counted for each patient and tagged on the basis of comparisons across sources. Results: We quantified completeness by combining information from all 3 sources and comparing each source with the complete set. We determined that the state registry was 71.8% complete, the hospital electronic health record was 81.9% complete, and personal records were 87.8% complete. Of the 2017 unique vaccination administrations, 65% were present in all 3 sources, 24.6% in 2 of the 3 sources, and 10.4% in only 1 source. Only 11% of patients had records in complete agreement across the 3 sources. Conclusion: This study highlights issues related to data completeness, exchange, and reporting of immunization information to state registries and suggests that there is some
J Public Health Management Practice, 2015, 21(3), 288–295 C 2015 Wolters Kluwer Health, Inc. All rights reserved. Copyright
degree of deficiency in completeness of immunization registries and other sources. This study indicates that there is a need to strengthen links between electronic data sources with immunization information and describes potential improvements in completeness that such efforts could provide, enabling providers to better rely on state immunization registries and to improve research utilization of immunization information systems. KEY WORDS: electronic health records, immunization,
information systems, registries, vaccinations
Most states now maintain an immunization information system (IIS) to facilitate documentation and review of vaccination status for patients across providers.1 Arizona uses the Arizona State Immunization Information System (ASIIS) for this purpose and has required providers to report immunization data since the Author Affiliation: College of Medicine–Phoenix, The University of Arizona, Phoenix (Mr Hendrickson and Dr Panchanathan); Department of Biomedical Informatics, Arizona State University, Scottsdale (Drs Panchanathan and Petitti); and Department of Pediatrics, Maricopa Integrated Health System, Phoenix, Arizona (Dr Panchanathan). The authors thank the staff of the Pediatric Clinic at the Maricopa Integrated Health System Comprehensive Healthcare Center for assistance with the collection of and access to the data used for this study. The authors also thank Drs Mathew Scotch and Robert Greenes, Biomedical Informatics Department, Arizona State University, for additional review and guidance on the writing of the manuscript. None of the authors have any financial or nonfinancial conflicts of interest. There are no sources of funding to declare. Correspondence: Bryan K. Hendrickson, MS, MHSM, College of Medicine– Phoenix, The University of Arizona, 550 E. Van Buren St, Phoenix, AZ 85004 ([email protected]). DOI: 10.1097/PHH.0000000000000045
288 Copyright © 2015 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
Immunization Data Completeness and Exchange With an IIS
creation of the system in 1998, tracking information for the now 1.6 million children in the state.2,3 Arizona Department of Health Services goals for ASIIS include full capture of vaccines administered in the state for children, providing information readily to providers, and the ability to monitor immunization levels in the state.4 Electronic health record (EHR) connectivity with an IIS is 1 of the 3 means by which providers can meet public health requirements related to “meaningful use” of EHR incentives in the American Reinvestment & Recovery Act of 2009, with sending information to an IIS as the first stage and bidirectional exchange as a likely future requirement.5 The Arizona Department of Health Services was recently able to establish HL7 interfaces for transmission of data directly from provider EHRs to ASIIS and is working toward achieving EHRASIIS bidirectional exchange. The completeness and validity of IISs for research purposes is an active area of study.6,7 A review of Minnesota’s IIS in 2007, as an example, showed that despite charted documentation of the administration of hepatitis B vaccinations for nearly 90% for all newborns at 3 major hospitals in the state, the state IIS showed rates ranging from about 2% to 40%.8 A study conducted in Philadelphia also identified a discrepancy between chart documentation and registry capture of immunizations, showing that direct electronic data transfer from health care organizations resulted in increased completeness of registry data.9 It has been suggested that high enrolment and activity of providers entering information into a state immunization registry aids in improving completeness of these registries; however, there remains a need to improve provider reporting.10,11 The Arizona Department of Health Services provides a Web portal for providers to review and report immunization information in Arizona and also allows large organizations, such as Maricopa Integrated Health System (MIHS), to send large sets of electronic data.12 Electronic batch processes such as this enable large systems to report information for their many providers, thus improving completeness of registries and also helping to ensure that children receive the correct number of doses of various vaccine series. However, additional challenges arise in matching patients in the data uploaded from individual sites to the patients in the IIS databases, which can result in duplicate records within immunization registries and contribute to the apparent incompleteness of the individual records. Although this has been addressed partially through quality assurance guidelines, continued improvement and vigilance are required.13,14 Data do not currently flow out of the ASIIS to EHR systems. There is evidence that consulting state IISs and personal records in combination provide an improved view of a patient’s immunization history.15 With these
❘ 289
discussions about the completeness of state immunization registries, issues in matching, and evidence that combinations of sources are better than any individually, we decided to conduct an evaluation of the completeness of ASIIS in comparison with other immunization information sources available to physicians at the MIHS. We investigated ASIIS, the MIHS EHR, and copies of patient personal vaccine records. The objective of this study was to evaluate the completeness of the 3 sources and to explore the effectiveness of improved data exchange between the EHR and ASIIS. Rather than the often-used approach of using medical records as the primary reference point to evaluate other sources, we determined a measure of completeness by comparing the information between the 3 available record sources in comparison with the combined set of information, considering all sources together. We also evaluated steps in data sharing across sources, such as clinical workflows to update and review medical records at the time of vaccine administration and data exchange between the MIHS EHR and ASIIS. This study highlights potential issues with the completeness of these sources and gaps in the processes in place to share and reconcile immunization information in Arizona. Our results may inform discussions about the use of immunization registries in clinical practice and research and efforts to improve data exchange between electronic health records and IISs.
● Methods Study site This study was conducted at a pediatric primary care site for an integrated community health system, the Pediatric Clinic at the Comprehensive Healthcare Center for MIHS in Phoenix, Arizona. The study included a convenience sample of patients whose records were copied at the time of a visit to the Comprehensive Healthcare Center Pediatrics Clinic, during August and September of 2012. Records were reviewed in consecutive order, based on the day of visit over a period of 1 month. For each of these patients, immunization information was retrieved both from the MIHS EHR system and from the ASIIS and compared with copies of personal records. Only records for patients who had brought personal records with immunization information at the time of the visit were included in this study. Pediatricians at MIHS rely heavily on the information within the ASIIS database to ensure that their patients are up to date on their immunizations, particularly in ambulatory clinics. When MIHS converted to an integrated EHR system, Epic, for its outpatient clinics
Copyright © 2015 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
290 ❘ Journal of Public Health Management and Practice in 2009, physicians requested that data exchange be enabled directly between the integrated EHR system and the ASIIS database, which was implemented in 2011, although still generated indirectly through billing data.
Data collection For each patient, 3 sources of immunization, ASIIS, the MIHS EHR, and personal records, were reviewed by counting the number of vaccinations in each source for all patients. Any unique date recorded across all 3 sources for a vaccination administration of a specific type of vaccine was considered as 1 administration. All vaccination administrations recorded for the patients were tagged according to whether they appeared in all 3, 2 of the 3, or only 1 of the sources (eg, a specific tag indicated that the same date was recorded for the same vaccination type across all 3 records), and tagged according to which sources they were in. Reviews and tagging were completed by a single reviewer, with counts for the 3 sources checked against counts for the various tags to validate counts for each patient. The age of patients was also recorded. Whether the most recent date of an administration for the 3 sources was in agreement, or not, was also noted. Vaccinations given on the day of record collection or after were excluded from this study. If across any of the 3 sources, an administration date listed for a specific vaccine type was not present in another source, it was considered as a separate unique administration of the vaccination. We did not attempt to reconcile dates to determine whether they were true administrations of a vaccine or errors, as this would require an assumption of 1 record as the true and complete record to use as a reference. The other potential way of reconciling would be to compare the timing of administrations to determine whether a date listed fell outside the expected vaccination schedule. However, again, there would be no way to tell whether a date is erroneous or potentially an inappropriate administration of a vaccination without a true and complete reference. If records appeared to contain a likely transcription error, they were tagged but still included in the original counts. Administrations recorded in any source on the day of data collection were excluded. The following vaccination types were included in the counts: BCG (bacille Calmette-Guerin for tuberculosis), DTaP (diphtheria and tetanus toxoids and acellular pertussis), Hep A (hepatitis A), Hep B (hepatitis B), Hib (Haemophilus influenzae type B), HPV (human papilloma virus), H1N1 (specific influenza type), influenza (any other type), IPV or OPV (inactivated or oral polio), MCV (meningococcal), (measles, mumps, rubella), PCV (pneumococcal conjugated vaccine), PPSV (pneumococcal polysaccharide vaccine),
Rotavirus (rotavirus), Tdap (tetanus, diphtheria and acellular pertussis with reduced diphtheria and pertussis doses), Td (tetanus and diphtheria booster), and varicella (chicken pox). Administrations recorded as part of a combination, other than those listed here, were considered separately (eg, Pediarix was considered as separate DTaP, Hep B, and IPV administrations). PCV7 and PCV13 were considered as the same type of vaccination. OPV and IPV were also considered as the same type of polio vaccinations. Influenza vaccinations were often found only in one source, so administrations for these vaccinations were also tagged as such. Because vaccinations are often included as part of a routine “well-child check,” and possibly not as commonly at other visits, the type of visit at the time of data collection was recorded as either “well-child” or as “other.” The “well-child” category was further divided to separate out “newborn” visits (from about 5 days to 2 weeks old). Patients were also categorized as either “established” patients at the clinic or “new” patients at the clinic.
Human participant compliance This study, its utilization of medical records, and the measures taken to protect the privacy and security of records used in this study were reviewed and approved by MIHS institutional review board (IRB), IRB registration: IRB00002620, FWA# 00003087.
● Results This study included 100 patients in total, 12 newborn patients, 16 new patients, and 72 established patients, with 60 of the established patients visiting the clinic for a well-child check (83%) and 12 visiting for other reasons (17%). Patients comprised from newborn to 14 years old.
General status of immunization records For the 100 patients with personal records brought to the Comprehensive Healthcare Center Pediatrics Clinic during the study, there were 1771 vaccination administrations in personal records, 1651 administrations in the MIHS EHR, and 1713 administrations in the ASIIS database. There was complete agreement across all 3 record sources for only 11% of patients (11/100); conversely, 89% of patients had at least 1 vaccination administration missing from one of the sources. These gaps cannot be explained by 1 record not having been updated recently. While 42% of patients (42/100) had the same most recent vaccination date listed for all
Copyright © 2015 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
Immunization Data Completeness and Exchange With an IIS
3 sources, 73.8% of these patients (31/42) had at least 1 prior immunization missing from 1 of the 3 sources. Of the 100 patients in the study, only 26 (26%) had records in ASIIS that captured all of the vaccination administrations found in the MIHS EHR or personal records. There were 17 patients (17%) not found in the ASIIS database at all. However, 14 of these were seen at an age of 1 month or less and it is likely that these visits occurred before ASIIS received an update for vaccinations administered at MIHS and included in this analysis. If we exclude newborns, a high proportion of patients (96%; 80/83) do have records in the ASIIS database. During the data collection process, we noted 45 administration entries (2.2% of 2017) where a transcription error may have occurred (Table 1). These were identified only if when tagging vaccine administrations, 2 of the 3 sources agreed on a specific date and the third had a slight difference in a date, or if it appeared that 2 vaccination types with similar names were confused on one of the records but the other 2 agreed (PPSV/PCV). There may be other sources of error, but for the purposes of this study, we counted any unique date of administration of a vaccine type as a separate administration. Four patients had more than 1 ASIIS record. We included all unique dates of administrations from these records. One additional patient beyond the 100 was excluded because of what appeared to have been an inappropriate merger of records in the ASIIS database.
Completeness We determined the completeness of each source by determining a union set of vaccination administrations. The union set was identified by counting the number of vaccination administrations in all 3 sources, 2 of the 3 sources, or only 1 source for any type of vaccine with a unique date for all patients (Figure 1). This assumes that the 3 sources together contain all administrations for TABLE 1 ● Potential Transcription Errors
qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq
Potential Errors ASIIS error (compared against EHR and PR) EHR error (compared against ASIIS and PR) PR error (compared against ASIIS and EHR) Total potential errors
Date Typo
PCV7/PPSV Interchange
All
22
7
29
14
14
2
2
38
7
45
Abbreviations: ASIIS, Arizona State Immunization Information System; EHR, electronic health record; PCV, pneumococcal conjugated vaccine; PPSV, pneumococcal polysaccharide vaccine; PR, personal records.
❘ 291
FIGURE 1 ● Union Set of Vaccination Administrationsa
qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq 1713 ASIIS 84.9% complete (1713/2017)
ASIIS 82 (4.1%)
116 (5.8%) MIHS EHR MIHS EHR 81.9% complete (1651/2017)
48 (2.4%)
1156
204 (10.1%) 1,311 (65.0%) 176 (8.7%)
Personal records 80 (4.0%)
Personal records 87.8% complete (1771/2017) 1177
Abbreviations: ASIIS, Arizona State Immunization Information System; EHR, electronic health record; MIHS, Maricopa Integrated Health System. a The union set of the 3 information sources, with 2017 unique vaccine administrations in total, as determined by identifying which administrations were recorded in all sources, 1311 (65.0%), 2 of the 3 sources 496 (24.6%), or only 1 of the 3 sources 210 (10.4%). Completeness values are given, as calculated by dividing the number of administrations in each source individually by the total unique administrations in the union set.
any patient. The total count for the union set was 2017 unique administrations. We found that 65.0% of administrations (1311/2017) were present in all 3 sources, 24.6% (496/2017) were present in 2 of the 3 sources, and 10.4% (210/2017) were present in only 1 source. The average number of administrations missing from at least 1 source for each patient was 7. The completeness of each data source was calculated by comparing the number of vaccination administrations recorded in each source independently with this union set, with ASIIS completeness described as 84.9% (1713/2017), MIHS EHR completeness as 81.9% (1651/2017), and personal record completeness as 87.8% (1771/2017). A second, unweighted measure of completeness was determined by comparing the 3 sources for each patient with the union set of the sources for that specific patient and then averaged across all patients. This approach yielded similar estimates of completeness to the aggregate measures of completeness, with personal records being, on average, 89.7% complete, MIHS EHR records 84.8% complete, and ASIIS records 71.8% complete. ASIIS seems to be less complete using this method; however, when patients younger than 2 months were excluded, to account for potential lag time between administration of the first vaccination at birth, the finalizing of the birth record in the EHR and transmission to ASIIS for newborns, the ASIIS records were, on average, 84.0% complete, similar to the aggregate measure. To illustrate how gaps in completeness varied from patient to patient, we calculated the standard deviation of individually calculated completeness values for each patient. This showed a 16% standard deviation in
Copyright © 2015 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
292 ❘ Journal of Public Health Management and Practice completeness across personal records, 30% across EHR records, and 38% across ASIIS records. We did not see any strong patterns in completeness of the immunization sources when comparing groups of patients based on age, visit types (established or new, well-child check, or other), or the number of administrations per record (Table 2). Results also did not change when influenza vaccinations were excluded. We noted a few potential patterns; however, the only groups that had any noticeable difference were newborn visits or those 2 months or younger, likely due to the extra lag time involved with newborn records from hospitals.
Influenza vaccinations There were 202 seasonal or H1N1 influenza vaccination administrations in the union set, with 64.4% of them recorded in a personal record, 71.3% recorded in the MIHS EHR, and 81.7% recorded in ASIIS. These rates were reduced from the overall completeness values. The decreased completeness for influenza immunization information could be due to vaccinations often being obtained at sites other than with established providers or at mobile or school clinics where separate cards for personal records are often handed out and potentially lost.
Investigating steps in the data exchange process: 2-Source comparisons We continued our analysis to investigate a few assumptions along expected steps in sharing of information between these data sources. The first assumption tested was that ASIIS should contain all vaccination administrations. If this were the case, ASIIS would be 100% complete; however, as stated, we calculated completeness as only 81.6% or 84.0%. Information regarding location of administrations was not available for most records reviewed in this study. As such, it is possible that some of the gaps noted could be due to patients receiving vaccinations outside of Arizona, The next assumption investigated was that if MIHS has been consistently reporting vaccination administrations directly to ASIIS, then all information in the MIHS EHR should be in ASIIS. However, we found ASIIS contains all of the information in the EHR for only 25% of patients (25/100) and only 13% of patients (13/100) had complete agreement between ASIIS and the EHR. So, in many cases, ASIIS is missing information stored in the EHR. Currently, only vaccinations administered at MIHS are reported to the ASIIS database, as such, some of this missing information could be historical information entered into the EHR from other sources. Considering the set of ASIIS and the EHR alone, excluding personal records, we determined that ASIIS was
missing 11.6% of administrations (224/1937) and the EHR was missing 14.8% of administrations (286/1937) (Figure 2A). Notably, 4.0% of administrations (80/2017) were recorded only in personal records. Thus, 96% of administrations were captured in ASIIS, the MIHS EHR, or both. The next scenario tested the assumption that any patient seen at MIHS should have all vaccination administrations listed in personal records included in the EHR system, since updating of the EHR is an expected part of the normal care process. However, we determined that 14.6% of administrations (284/1935) were missing from the EHR and 8.5% (164/1935) were missing from personal records, in the combined set of the MIHS EHR and personal records, excluding ASIIS (Figure 2B). In addition, 8% of patients (8/100) had an EHR record with no vaccinations recorded in the EHR at all, despite a personal record brought to a visit at MIHS, indicating that their immunizations were not updated in the EHR as expected. The last scenario we investigated was a comparison of personal records with ASIIS. When looking at ASIIS and personal records alone, 13.0% of vaccinations (256/1969) were missing from ASIIS (Figure 2C) and 10.1% were missing from personal records.
● Discussion This study describes the current status of immunization information sources used by providers at a large community health system, MIHS. The system uses an EHR and routinely enters information from personal records into its EHR. It has established an electronic interface with the ASIIS and regularly uploads data. With these processes in place, it is perhaps an ideal environment to assess completeness of immunization information across the sources available to providers in Arizona. Notwithstanding, we found evidence that neither the immunization information in the MIHS EHR nor the ASIIS is complete. Personal records are also not complete. Our study is small and isolated to what is possibly a unique patient population; however, it depicts a situation where there are gaps in information across a state IIS, an EHR at a large provider within that state, and personal records, even when automated data exchange processes are in place between the state IIS and this EHR. This emphasizes the need to better understand the potential incompleteness of these data and the possible effects of incompleteness on research and clinical care using state immunization registries, even when providers are registered and actively sending information to a state database. The Centers for Disease Control MIROW (Modeling of Immunization Registry
Copyright © 2015 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
82.4
5.9
24.6
10.4
12.0
15.6
72.4
11.8
65.0
60 3.1
6.6
22.8
70.6
13.7
31.5
54.7
4.0
68.0
28.0
13.3
23.4
63.3
7.5
31.7
60.9
11.7
18.7
69.6
11.2
18.5
70.4
90.2% 83.0% 36.0%a 79.9%f 87.1%f 86.7%f 84.3%
697 27.9
25 6.4
11.8%a 81.8%
379 15.2
25 5.8
84.9%
25 1.0
25 0.8
86.5% 88.5%
707 32.1
25 (0.9)
100.0%a 86.6% 88.1% 88.0% 96.0%a 86.8% 89.2% 86.9% 94.1%a 92.0%d 85.8%d 70.0%d 92.0%a 85.5% 77.2% 83.6%
793 33.0
22 10.0
Visit Type
5.4
32.6
62.1
85.7%
87.9% 83.0%
224 18.7
12 3.4 12 1.0
12 0.0
NB
10.8
37.9
51.3
89.0%e
8.3
91.7
0.0
0.0%a,b
91.6% 100.0%a,b 60.0%e 91.7%a,b
427 26.7
16 6.1
25-31 32-45 WC-EST Other-EST WC-New
87.8% 81.9%
500 13.5
24 4.0
5-24
1354 22.6
17 1.0
2017 20.2
37 (8.9)
1
916 36.6
17 (0.2)
100 3.3
7-14 y
No. Vaccination Administrations per Patient
Abbreviations: ASIIS, Arizona State Immunization Information System; EHR, electronic health record; EST, established; NB, newborn; PR, personal record; WC, well child. a Values influenced by small vaccination administration counts for this group. b Many empty or nonexistent ASIIS records in this group. c Completeness appears potentially higher for all sources for patients for whom the last date recorded in each source is in agreement. d The EHR appears potentially less complete with older patients. e New patients appear to have potentially less complete EHR records than other patients. f ASIIS appears potentially less complete for patients with increasing numbers of vaccination administrations.
No. patients Average age, y (or mo) No. vaccinations Average no. vaccinations PR completeness EHR completeness ASIIS completeness % Shots in all 3 sources % Shots in 2 of 3 sources % Shots in only 1 source
All Patients
Context: Information systems are used by most states to maintain registries of immunization data both for monitoring population-level adherence and for use in clinical practice and research. Direct data exchange between such systems and electronic health record systems presents an opportunity to improve the completeness and quality of information available. Objective: Our goals were to describe and compare the completeness of the Arizona State Immunization System, the electronic health record at a large community health provider in Arizona exchanging electronic data with the Arizona system, and personal immunization records in an effort to contribute to the discussion on the completeness of state-run immunization registries and data exchange with these registries. Design: Immunization histories from these sources were collected and reviewed sequentially. Unique dates of vaccination administrations were counted for each patient and tagged on the basis of comparisons across sources. Results: We quantified completeness by combining information from all 3 sources and comparing each source with the complete set. We determined that the state registry was 71.8% complete, the hospital electronic health record was 81.9% complete, and personal records were 87.8% complete. Of the 2017 unique vaccination administrations, 65% were present in all 3 sources, 24.6% in 2 of the 3 sources, and 10.4% in only 1 source. Only 11% of patients had records in complete agreement across the 3 sources. Conclusion: This study highlights issues related to data completeness, exchange, and reporting of immunization information to state registries and suggests that there is some
J Public Health Management Practice, 2015, 21(3), 288–295 C 2015 Wolters Kluwer Health, Inc. All rights reserved. Copyright
degree of deficiency in completeness of immunization registries and other sources. This study indicates that there is a need to strengthen links between electronic data sources with immunization information and describes potential improvements in completeness that such efforts could provide, enabling providers to better rely on state immunization registries and to improve research utilization of immunization information systems. KEY WORDS: electronic health records, immunization,
information systems, registries, vaccinations
Most states now maintain an immunization information system (IIS) to facilitate documentation and review of vaccination status for patients across providers.1 Arizona uses the Arizona State Immunization Information System (ASIIS) for this purpose and has required providers to report immunization data since the Author Affiliation: College of Medicine–Phoenix, The University of Arizona, Phoenix (Mr Hendrickson and Dr Panchanathan); Department of Biomedical Informatics, Arizona State University, Scottsdale (Drs Panchanathan and Petitti); and Department of Pediatrics, Maricopa Integrated Health System, Phoenix, Arizona (Dr Panchanathan). The authors thank the staff of the Pediatric Clinic at the Maricopa Integrated Health System Comprehensive Healthcare Center for assistance with the collection of and access to the data used for this study. The authors also thank Drs Mathew Scotch and Robert Greenes, Biomedical Informatics Department, Arizona State University, for additional review and guidance on the writing of the manuscript. None of the authors have any financial or nonfinancial conflicts of interest. There are no sources of funding to declare. Correspondence: Bryan K. Hendrickson, MS, MHSM, College of Medicine– Phoenix, The University of Arizona, 550 E. Van Buren St, Phoenix, AZ 85004 ([email protected]). DOI: 10.1097/PHH.0000000000000045
288 Copyright © 2015 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
Immunization Data Completeness and Exchange With an IIS
creation of the system in 1998, tracking information for the now 1.6 million children in the state.2,3 Arizona Department of Health Services goals for ASIIS include full capture of vaccines administered in the state for children, providing information readily to providers, and the ability to monitor immunization levels in the state.4 Electronic health record (EHR) connectivity with an IIS is 1 of the 3 means by which providers can meet public health requirements related to “meaningful use” of EHR incentives in the American Reinvestment & Recovery Act of 2009, with sending information to an IIS as the first stage and bidirectional exchange as a likely future requirement.5 The Arizona Department of Health Services was recently able to establish HL7 interfaces for transmission of data directly from provider EHRs to ASIIS and is working toward achieving EHRASIIS bidirectional exchange. The completeness and validity of IISs for research purposes is an active area of study.6,7 A review of Minnesota’s IIS in 2007, as an example, showed that despite charted documentation of the administration of hepatitis B vaccinations for nearly 90% for all newborns at 3 major hospitals in the state, the state IIS showed rates ranging from about 2% to 40%.8 A study conducted in Philadelphia also identified a discrepancy between chart documentation and registry capture of immunizations, showing that direct electronic data transfer from health care organizations resulted in increased completeness of registry data.9 It has been suggested that high enrolment and activity of providers entering information into a state immunization registry aids in improving completeness of these registries; however, there remains a need to improve provider reporting.10,11 The Arizona Department of Health Services provides a Web portal for providers to review and report immunization information in Arizona and also allows large organizations, such as Maricopa Integrated Health System (MIHS), to send large sets of electronic data.12 Electronic batch processes such as this enable large systems to report information for their many providers, thus improving completeness of registries and also helping to ensure that children receive the correct number of doses of various vaccine series. However, additional challenges arise in matching patients in the data uploaded from individual sites to the patients in the IIS databases, which can result in duplicate records within immunization registries and contribute to the apparent incompleteness of the individual records. Although this has been addressed partially through quality assurance guidelines, continued improvement and vigilance are required.13,14 Data do not currently flow out of the ASIIS to EHR systems. There is evidence that consulting state IISs and personal records in combination provide an improved view of a patient’s immunization history.15 With these
❘ 289
discussions about the completeness of state immunization registries, issues in matching, and evidence that combinations of sources are better than any individually, we decided to conduct an evaluation of the completeness of ASIIS in comparison with other immunization information sources available to physicians at the MIHS. We investigated ASIIS, the MIHS EHR, and copies of patient personal vaccine records. The objective of this study was to evaluate the completeness of the 3 sources and to explore the effectiveness of improved data exchange between the EHR and ASIIS. Rather than the often-used approach of using medical records as the primary reference point to evaluate other sources, we determined a measure of completeness by comparing the information between the 3 available record sources in comparison with the combined set of information, considering all sources together. We also evaluated steps in data sharing across sources, such as clinical workflows to update and review medical records at the time of vaccine administration and data exchange between the MIHS EHR and ASIIS. This study highlights potential issues with the completeness of these sources and gaps in the processes in place to share and reconcile immunization information in Arizona. Our results may inform discussions about the use of immunization registries in clinical practice and research and efforts to improve data exchange between electronic health records and IISs.
● Methods Study site This study was conducted at a pediatric primary care site for an integrated community health system, the Pediatric Clinic at the Comprehensive Healthcare Center for MIHS in Phoenix, Arizona. The study included a convenience sample of patients whose records were copied at the time of a visit to the Comprehensive Healthcare Center Pediatrics Clinic, during August and September of 2012. Records were reviewed in consecutive order, based on the day of visit over a period of 1 month. For each of these patients, immunization information was retrieved both from the MIHS EHR system and from the ASIIS and compared with copies of personal records. Only records for patients who had brought personal records with immunization information at the time of the visit were included in this study. Pediatricians at MIHS rely heavily on the information within the ASIIS database to ensure that their patients are up to date on their immunizations, particularly in ambulatory clinics. When MIHS converted to an integrated EHR system, Epic, for its outpatient clinics
Copyright © 2015 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
290 ❘ Journal of Public Health Management and Practice in 2009, physicians requested that data exchange be enabled directly between the integrated EHR system and the ASIIS database, which was implemented in 2011, although still generated indirectly through billing data.
Data collection For each patient, 3 sources of immunization, ASIIS, the MIHS EHR, and personal records, were reviewed by counting the number of vaccinations in each source for all patients. Any unique date recorded across all 3 sources for a vaccination administration of a specific type of vaccine was considered as 1 administration. All vaccination administrations recorded for the patients were tagged according to whether they appeared in all 3, 2 of the 3, or only 1 of the sources (eg, a specific tag indicated that the same date was recorded for the same vaccination type across all 3 records), and tagged according to which sources they were in. Reviews and tagging were completed by a single reviewer, with counts for the 3 sources checked against counts for the various tags to validate counts for each patient. The age of patients was also recorded. Whether the most recent date of an administration for the 3 sources was in agreement, or not, was also noted. Vaccinations given on the day of record collection or after were excluded from this study. If across any of the 3 sources, an administration date listed for a specific vaccine type was not present in another source, it was considered as a separate unique administration of the vaccination. We did not attempt to reconcile dates to determine whether they were true administrations of a vaccine or errors, as this would require an assumption of 1 record as the true and complete record to use as a reference. The other potential way of reconciling would be to compare the timing of administrations to determine whether a date listed fell outside the expected vaccination schedule. However, again, there would be no way to tell whether a date is erroneous or potentially an inappropriate administration of a vaccination without a true and complete reference. If records appeared to contain a likely transcription error, they were tagged but still included in the original counts. Administrations recorded in any source on the day of data collection were excluded. The following vaccination types were included in the counts: BCG (bacille Calmette-Guerin for tuberculosis), DTaP (diphtheria and tetanus toxoids and acellular pertussis), Hep A (hepatitis A), Hep B (hepatitis B), Hib (Haemophilus influenzae type B), HPV (human papilloma virus), H1N1 (specific influenza type), influenza (any other type), IPV or OPV (inactivated or oral polio), MCV (meningococcal), (measles, mumps, rubella), PCV (pneumococcal conjugated vaccine), PPSV (pneumococcal polysaccharide vaccine),
Rotavirus (rotavirus), Tdap (tetanus, diphtheria and acellular pertussis with reduced diphtheria and pertussis doses), Td (tetanus and diphtheria booster), and varicella (chicken pox). Administrations recorded as part of a combination, other than those listed here, were considered separately (eg, Pediarix was considered as separate DTaP, Hep B, and IPV administrations). PCV7 and PCV13 were considered as the same type of vaccination. OPV and IPV were also considered as the same type of polio vaccinations. Influenza vaccinations were often found only in one source, so administrations for these vaccinations were also tagged as such. Because vaccinations are often included as part of a routine “well-child check,” and possibly not as commonly at other visits, the type of visit at the time of data collection was recorded as either “well-child” or as “other.” The “well-child” category was further divided to separate out “newborn” visits (from about 5 days to 2 weeks old). Patients were also categorized as either “established” patients at the clinic or “new” patients at the clinic.
Human participant compliance This study, its utilization of medical records, and the measures taken to protect the privacy and security of records used in this study were reviewed and approved by MIHS institutional review board (IRB), IRB registration: IRB00002620, FWA# 00003087.
● Results This study included 100 patients in total, 12 newborn patients, 16 new patients, and 72 established patients, with 60 of the established patients visiting the clinic for a well-child check (83%) and 12 visiting for other reasons (17%). Patients comprised from newborn to 14 years old.
General status of immunization records For the 100 patients with personal records brought to the Comprehensive Healthcare Center Pediatrics Clinic during the study, there were 1771 vaccination administrations in personal records, 1651 administrations in the MIHS EHR, and 1713 administrations in the ASIIS database. There was complete agreement across all 3 record sources for only 11% of patients (11/100); conversely, 89% of patients had at least 1 vaccination administration missing from one of the sources. These gaps cannot be explained by 1 record not having been updated recently. While 42% of patients (42/100) had the same most recent vaccination date listed for all
Copyright © 2015 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
Immunization Data Completeness and Exchange With an IIS
3 sources, 73.8% of these patients (31/42) had at least 1 prior immunization missing from 1 of the 3 sources. Of the 100 patients in the study, only 26 (26%) had records in ASIIS that captured all of the vaccination administrations found in the MIHS EHR or personal records. There were 17 patients (17%) not found in the ASIIS database at all. However, 14 of these were seen at an age of 1 month or less and it is likely that these visits occurred before ASIIS received an update for vaccinations administered at MIHS and included in this analysis. If we exclude newborns, a high proportion of patients (96%; 80/83) do have records in the ASIIS database. During the data collection process, we noted 45 administration entries (2.2% of 2017) where a transcription error may have occurred (Table 1). These were identified only if when tagging vaccine administrations, 2 of the 3 sources agreed on a specific date and the third had a slight difference in a date, or if it appeared that 2 vaccination types with similar names were confused on one of the records but the other 2 agreed (PPSV/PCV). There may be other sources of error, but for the purposes of this study, we counted any unique date of administration of a vaccine type as a separate administration. Four patients had more than 1 ASIIS record. We included all unique dates of administrations from these records. One additional patient beyond the 100 was excluded because of what appeared to have been an inappropriate merger of records in the ASIIS database.
Completeness We determined the completeness of each source by determining a union set of vaccination administrations. The union set was identified by counting the number of vaccination administrations in all 3 sources, 2 of the 3 sources, or only 1 source for any type of vaccine with a unique date for all patients (Figure 1). This assumes that the 3 sources together contain all administrations for TABLE 1 ● Potential Transcription Errors
qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq
Potential Errors ASIIS error (compared against EHR and PR) EHR error (compared against ASIIS and PR) PR error (compared against ASIIS and EHR) Total potential errors
Date Typo
PCV7/PPSV Interchange
All
22
7
29
14
14
2
2
38
7
45
Abbreviations: ASIIS, Arizona State Immunization Information System; EHR, electronic health record; PCV, pneumococcal conjugated vaccine; PPSV, pneumococcal polysaccharide vaccine; PR, personal records.
❘ 291
FIGURE 1 ● Union Set of Vaccination Administrationsa
qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq 1713 ASIIS 84.9% complete (1713/2017)
ASIIS 82 (4.1%)
116 (5.8%) MIHS EHR MIHS EHR 81.9% complete (1651/2017)
48 (2.4%)
1156
204 (10.1%) 1,311 (65.0%) 176 (8.7%)
Personal records 80 (4.0%)
Personal records 87.8% complete (1771/2017) 1177
Abbreviations: ASIIS, Arizona State Immunization Information System; EHR, electronic health record; MIHS, Maricopa Integrated Health System. a The union set of the 3 information sources, with 2017 unique vaccine administrations in total, as determined by identifying which administrations were recorded in all sources, 1311 (65.0%), 2 of the 3 sources 496 (24.6%), or only 1 of the 3 sources 210 (10.4%). Completeness values are given, as calculated by dividing the number of administrations in each source individually by the total unique administrations in the union set.
any patient. The total count for the union set was 2017 unique administrations. We found that 65.0% of administrations (1311/2017) were present in all 3 sources, 24.6% (496/2017) were present in 2 of the 3 sources, and 10.4% (210/2017) were present in only 1 source. The average number of administrations missing from at least 1 source for each patient was 7. The completeness of each data source was calculated by comparing the number of vaccination administrations recorded in each source independently with this union set, with ASIIS completeness described as 84.9% (1713/2017), MIHS EHR completeness as 81.9% (1651/2017), and personal record completeness as 87.8% (1771/2017). A second, unweighted measure of completeness was determined by comparing the 3 sources for each patient with the union set of the sources for that specific patient and then averaged across all patients. This approach yielded similar estimates of completeness to the aggregate measures of completeness, with personal records being, on average, 89.7% complete, MIHS EHR records 84.8% complete, and ASIIS records 71.8% complete. ASIIS seems to be less complete using this method; however, when patients younger than 2 months were excluded, to account for potential lag time between administration of the first vaccination at birth, the finalizing of the birth record in the EHR and transmission to ASIIS for newborns, the ASIIS records were, on average, 84.0% complete, similar to the aggregate measure. To illustrate how gaps in completeness varied from patient to patient, we calculated the standard deviation of individually calculated completeness values for each patient. This showed a 16% standard deviation in
Copyright © 2015 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
292 ❘ Journal of Public Health Management and Practice completeness across personal records, 30% across EHR records, and 38% across ASIIS records. We did not see any strong patterns in completeness of the immunization sources when comparing groups of patients based on age, visit types (established or new, well-child check, or other), or the number of administrations per record (Table 2). Results also did not change when influenza vaccinations were excluded. We noted a few potential patterns; however, the only groups that had any noticeable difference were newborn visits or those 2 months or younger, likely due to the extra lag time involved with newborn records from hospitals.
Influenza vaccinations There were 202 seasonal or H1N1 influenza vaccination administrations in the union set, with 64.4% of them recorded in a personal record, 71.3% recorded in the MIHS EHR, and 81.7% recorded in ASIIS. These rates were reduced from the overall completeness values. The decreased completeness for influenza immunization information could be due to vaccinations often being obtained at sites other than with established providers or at mobile or school clinics where separate cards for personal records are often handed out and potentially lost.
Investigating steps in the data exchange process: 2-Source comparisons We continued our analysis to investigate a few assumptions along expected steps in sharing of information between these data sources. The first assumption tested was that ASIIS should contain all vaccination administrations. If this were the case, ASIIS would be 100% complete; however, as stated, we calculated completeness as only 81.6% or 84.0%. Information regarding location of administrations was not available for most records reviewed in this study. As such, it is possible that some of the gaps noted could be due to patients receiving vaccinations outside of Arizona, The next assumption investigated was that if MIHS has been consistently reporting vaccination administrations directly to ASIIS, then all information in the MIHS EHR should be in ASIIS. However, we found ASIIS contains all of the information in the EHR for only 25% of patients (25/100) and only 13% of patients (13/100) had complete agreement between ASIIS and the EHR. So, in many cases, ASIIS is missing information stored in the EHR. Currently, only vaccinations administered at MIHS are reported to the ASIIS database, as such, some of this missing information could be historical information entered into the EHR from other sources. Considering the set of ASIIS and the EHR alone, excluding personal records, we determined that ASIIS was
missing 11.6% of administrations (224/1937) and the EHR was missing 14.8% of administrations (286/1937) (Figure 2A). Notably, 4.0% of administrations (80/2017) were recorded only in personal records. Thus, 96% of administrations were captured in ASIIS, the MIHS EHR, or both. The next scenario tested the assumption that any patient seen at MIHS should have all vaccination administrations listed in personal records included in the EHR system, since updating of the EHR is an expected part of the normal care process. However, we determined that 14.6% of administrations (284/1935) were missing from the EHR and 8.5% (164/1935) were missing from personal records, in the combined set of the MIHS EHR and personal records, excluding ASIIS (Figure 2B). In addition, 8% of patients (8/100) had an EHR record with no vaccinations recorded in the EHR at all, despite a personal record brought to a visit at MIHS, indicating that their immunizations were not updated in the EHR as expected. The last scenario we investigated was a comparison of personal records with ASIIS. When looking at ASIIS and personal records alone, 13.0% of vaccinations (256/1969) were missing from ASIIS (Figure 2C) and 10.1% were missing from personal records.
● Discussion This study describes the current status of immunization information sources used by providers at a large community health system, MIHS. The system uses an EHR and routinely enters information from personal records into its EHR. It has established an electronic interface with the ASIIS and regularly uploads data. With these processes in place, it is perhaps an ideal environment to assess completeness of immunization information across the sources available to providers in Arizona. Notwithstanding, we found evidence that neither the immunization information in the MIHS EHR nor the ASIIS is complete. Personal records are also not complete. Our study is small and isolated to what is possibly a unique patient population; however, it depicts a situation where there are gaps in information across a state IIS, an EHR at a large provider within that state, and personal records, even when automated data exchange processes are in place between the state IIS and this EHR. This emphasizes the need to better understand the potential incompleteness of these data and the possible effects of incompleteness on research and clinical care using state immunization registries, even when providers are registered and actively sending information to a state database. The Centers for Disease Control MIROW (Modeling of Immunization Registry
Copyright © 2015 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
82.4
5.9
24.6
10.4
12.0
15.6
72.4
11.8
65.0
60 3.1
6.6
22.8
70.6
13.7
31.5
54.7
4.0
68.0
28.0
13.3
23.4
63.3
7.5
31.7
60.9
11.7
18.7
69.6
11.2
18.5
70.4
90.2% 83.0% 36.0%a 79.9%f 87.1%f 86.7%f 84.3%
697 27.9
25 6.4
11.8%a 81.8%
379 15.2
25 5.8
84.9%
25 1.0
25 0.8
86.5% 88.5%
707 32.1
25 (0.9)
100.0%a 86.6% 88.1% 88.0% 96.0%a 86.8% 89.2% 86.9% 94.1%a 92.0%d 85.8%d 70.0%d 92.0%a 85.5% 77.2% 83.6%
793 33.0
22 10.0
Visit Type
5.4
32.6
62.1
85.7%
87.9% 83.0%
224 18.7
12 3.4 12 1.0
12 0.0
NB
10.8
37.9
51.3
89.0%e
8.3
91.7
0.0
0.0%a,b
91.6% 100.0%a,b 60.0%e 91.7%a,b
427 26.7
16 6.1
25-31 32-45 WC-EST Other-EST WC-New
87.8% 81.9%
500 13.5
24 4.0
5-24
1354 22.6
17 1.0
2017 20.2
37 (8.9)
1
916 36.6
17 (0.2)
100 3.3
7-14 y
No. Vaccination Administrations per Patient
Abbreviations: ASIIS, Arizona State Immunization Information System; EHR, electronic health record; EST, established; NB, newborn; PR, personal record; WC, well child. a Values influenced by small vaccination administration counts for this group. b Many empty or nonexistent ASIIS records in this group. c Completeness appears potentially higher for all sources for patients for whom the last date recorded in each source is in agreement. d The EHR appears potentially less complete with older patients. e New patients appear to have potentially less complete EHR records than other patients. f ASIIS appears potentially less complete for patients with increasing numbers of vaccination administrations.
No. patients Average age, y (or mo) No. vaccinations Average no. vaccinations PR completeness EHR completeness ASIIS completeness % Shots in all 3 sources % Shots in 2 of 3 sources % Shots in only 1 source
All Patients
