J Med Syst (2015) 39: 44 DOI 10.1007/s10916-015-0232-4
SYSTEMS-LEVEL QUALITY IMPROVEMENT
Metadata from Data: Identifying Holidays from Anesthesia Data Joseph R. Starnes & Jonathan P. Wanderer & Jesse M. Ehrenfeld
Received: 26 December 2014 / Accepted: 11 February 2015 / Published online: 3 March 2015 # Springer Science+Business Media New York 2015
Abstract The increasingly large databases available to researchers necessitate high-quality metadata that is not always available. We describe a method for generating this metadata independently. Cluster analysis and expectation-maximization were used to separate days into holidays/weekends and regular workdays using anesthesia data from Vanderbilt University Medical Center from 2004 to 2014. This classification was then used to describe differences between the two sets of days over time. We evaluated 3802 days and correctly categorized 3797 based on anesthesia case time (representing an error rate of 0.13 %). Use of other metrics for categorization, such as billed anesthesia hours and number of anesthesia cases per day, led to similar results. Analysis of the two categories showed that surgical volume increased more quickly with time for non-holidays than holidays (p<0.001). We were able to successfully generate metadata from data by distinguishing holidays based on anesthesia data. This data can then be used for economic analysis and scheduling purposes. It is possible that the method can be expanded to similar bimodal and multimodal variables.
Keywords Cluster analysis . Information science . Medical informatics . Database management systems
This article is part of the Topical Collection on Systems-Level Quality Improvement. J. R. Starnes (*) : J. P. Wanderer : J. M. Ehrenfeld Department of Anesthesiology, Vanderbilt University Medical Center, 1301 Medical Center Dr., Nashville, TN 37232, USA e-mail: [email protected] J. P. Wanderer e-mail: [email protected] J. M. Ehrenfeld e-mail: [email protected]
Introduction The implementation of electronic health records has led to an explosion of data about patients, providers, and ultimately how healthcare is delivered. This has created a growing need to manage large stores of patient data, resulting in the creation of data warehouses, which necessitate not only warehouse administration and maintenance but also the management of metadata . Metadata are information about how data were collected, when they were collected, the tools that were used, and other aspects that might be necessary for proper interpretation. Even if a researcher is able to find a dataset that fits their needs, metadata is often necessary to use it appropriately. Unfortunately, these data require significant effort to generate and primarily benefit another party . This, along with technological barriers and the general reluctance to share data, makes it unlikely that complete and effective metadata will be consistently created . These problems have been illustrated by numerous attempts to facilitate data sharing that depend on user-generated metadata [4, 5]. If metadata could be generated by the person benefiting from it or a third party database manager, this would shift the workload and allow for better data sharing. The needed metadata could be created without burdening the original creator of the data. For example, patients are generally receptive to generating contextual resources about their health conditions that are not normally generated by providers . Such metadata could also help inform ongoing efforts to track and utilize operating room data for practice improvement [7, 8]. In performing operations management research, it is often important to take into consideration processes that occur on weekdays compared to weekends and holidays. For example, staffing on weekends and holidays can be difficult because there are fewer data points on which to base projections . Additionally, the care received by patients has been shown to
44 Page 2 of 5
be different during these days compared to typical working days . Being able to separate these two categories (weekdays vs. weekends / holidays) in large databases would enable more effective data analysis for a variety of use cases. However, differing hospital schedules and lack of reporting make this challenging and typically subject to manual review. For example, the Vanderbilt University Medical Center (VUMC) recognizes both Christmas and Christmas Eve while many hospitals do not. As a proof of concept, we set out to develop a methodology to separate holidays and weekends from regular workdays in a large dataset based on anesthesia data, effectively creating metadata from data. By removing periodic and linear effects from surgical data, it is possible to identify holidays in the effort to model operation room demand . By using similar methodology and utilizing cluster analysis, we identified holidays and weekends from anesthesia data from VUMC between 2004 and 2014.
J Med Syst (2015) 39: 44
Classification, and Density Estimation (mclust) , and Hartigan’s dip test statistic for unimodality (diptest)  packages—was used for all data processing and analysis. Data were aggregated by day for three separate variables: case time (from entry into room to exit), number of cases, and anesthesia time. These aggregate data were then modeled using cluster analysis. Bayes Information Criterion (BIC) were calculated for each set of aggregate data, and expectation maximization for a two-component Gaussian mixture with unequal variance was used to assign individual days to one of two groups. Whichever group the day was more likely to belong to (p>0.5) was assumed to be correct without consideration of how high the calculated probability was. The cluster having the smaller values according to each metric was deemed to be the holiday group. For some analyses, linear trend was removed from the dataset. To do this, a least squares linear regression was used to create a fit for aggregated daily data. The residual of this regression was then used in place of the raw aggregates. To compare regressions between non-holidays and holidays/ weekends, analysis of covariance (ANCOVA) was used.
Data source For analysis we used a de-identified dataset containing all anesthesia cases at VUMC between January 1st, 2004 and May 29th, 2014. This set included 725,144 individual cases covering 3802 days. The average number of cases per day was 190.7, and the average length of a case was 1.83 h. We excluded cases with a duration greater than 24 h, as they were most likely caused by data entry errors. Cases that were missing times necessary for calculation of case length, representing 7.66 % of cases, were coded as lasting zero minutes. This missing data did not significantly affect the ability of the method to classify days. Data analysis R—supplemented with the eXtensible Time Series (xts) , Normal Mixture Modeling for Model-Based Clustering,
Results All three metrics—case time, number of cases, and billed anesthesia time—aggregated by day created histograms with two easily distinguishable groups (Fig. 1). At least bimodality was shown for each using Hartigan’s dip statistic (D≈ 1, p<0.001 in all cases). The peak on the left of each graph represents, with very few exceptions, weekends and holidays recognized by VUMC. Using cluster analysis and expectation-maximization, each day was designated as either a holiday/weekend or a normal workday based on these metrics. These assignments were then compared to a list of weekends and holidays to check for accuracy. Of 3802 days, 1158 were either weekends or recognized holidays. Categorization based on both case time and number of cases misidentified just 5 days giving an error rate of 0.13 %. Days missed were either the Friday following
Fig. 1 Histograms of aggregated daily metrics show at least two distinct groups of days. The peak on the left of each graph represents low-throughput days during which fewer cases occurred. The vast majority of these days are holidays and weekends
J Med Syst (2015) 39: 44
Page 3 of 5 44
Fig. 2 Removal of linear trend from data. a Case time for each day since 1/1/2004. The fit line shows the least squares regression line with a slope of 3.897 min/day and intercept of 1352 min. b Residual of regression for
each day since 1/1/2004. The fit line shows that the least squares regression of the data is now a line with a slope of zero
Thanksgiving or high-volume weekend days falling near the separating line based on the variable used. Both variables showed approximately equal miscategorizations of holidays as non-holidays and non-holidays as holidays. Errors were not more common early or late in the dataset with errors occurring as early as 2004 and as late as 2013 without more than one error in any year for either variable. Categorization based on billed anesthesia time yielded 19 miscategorizations for an error rate of 0.50 %. In choosing between case time and number of cases as the preferred metric, lost days were considered. Lost days were defined as data points that were miscategorized because they
actually fell within the wrong group according to the metric. For example, a holiday that has more case time than at least one non-holiday is lost to the algorithm because a day will be miscategorized no matter where the delineation between categories is drawn. It was found that both metrics shared two lost days (11/26/04 and 11/25/05) and that number of cases yielded a third at 12/30/11. Based on this, further analysis was conducted using case time as the metric. It has been previously observed that surgery volume generally increases with time . We observed a similar trend in our dataset (Fig. 2a). The effect of time on case time was significant (p<0.001). By using the residual of a least squares
Table 1 Results for various metrics and categorization methods. Importantly, using three clusters for case time did not change the results as one of the clusters split in two without changing how the clusters were divided at the true intersection point. Miscategorizations are days that were put in the wrong category by the algorithm. Lost days are days that fall in the wrong category based on the chosen metric
No Trend Removal Case Time Number of Cases Anesthesia Time Case Time (Elective Only) Case Time (3 Clusters) Trend Removal Case Time Number of Cases Case Time (Elective Only) Case Time (3 Clusters)
5 5 19 10 5
2 3 12 6 2
15 14 18 13
6 8 7 6
Fig. 3 Linear trend for non-holidays (top) and holidays/weekends (bottom). From ANCOVA it is evident that the regressions have different slopes (p<0.001). The calculated slope for non-holidays is 5.373 min/day while for holidays/weekends it is 0.4677 min/day
44 Page 4 of 5
linear fit, we were able to successfully remove this trend from the data (Fig. 2b). Interestingly, performing the same cluster analysis and expectation-maximization on the adjusted data yielded worse results with 15 miscategorizations and six lost days. Similarly, other attempts to improve results—like using only elective cases and modeling with more than two clusters—failed to improve results. The outcomes of each scenario tested are summarized in Table 1. To determine why correcting for linear trend had this effect, separate least squares regressions were performed on holidays/weekends and non-holidays (Fig. 3). While time had a significant effect on both categories (p<0.001 in both cases), this effect was different between categories. The calculated slope for non-holidays was 5.373 min/day while for holidays and weekends it was just 0.4677 min/day. ANCOVA showed a significant interaction between category and time indicating that the slopes were different between categories (p<0.001). This difference has significant implications for operating room scheduling and hospital management.
Conclusion We have developed a methodology to automatically identify holidays and weekends in a dataset from anesthesia data with an error rate of 0.13 %. Because this method does not require the creation of parameters and cutoffs, it can be generalized to other datasets. While it performs better on large datasets, the same method used on a single year misses 1 day in each of the years between 2010 and 2013 for an error rate of 0.27 % (always the Friday following Thanksgiving). This could allow for a wide range of applications depending on whether the researcher is trying to make long-term forecasts or analyze a smaller subset of hospital data. Additionally, the method of determination is date-independent so complications will not arise from multi-hospital datasets in which facilities recognize different holidays. Data can be segregated by facility, and the holidays can be determined independently before remerging. It is important to note, however, that both the whole dataset and data for individual years reflect surgeries performed at a single high-volume academic center. Other centers hoping to use this method may not share similar factors such as proportions of emergent and elective procedures or ratios of procedures on holidays and non-holidays. Application of this method to other datasets will require taking this into account, and the method will need to be validated systematically for a given center. Although these days were identified using anesthesia data, once holidays are identified they can be recorded and used to analyze any aspect of the hospital’s function on those days compared to regular days. One problem constantly faced by hospital administration is a relatively small sample size to use when determining holiday scheduling . This algorithm can
J Med Syst (2015) 39: 44
be used to quickly and accurately identify these days in large, multi-center datasets for analysis. Although it was an incidental finding in our work, it was observed that the increase in case volume over time was significantly less for holidays compared to non-holidays. This is not necessarily intuitive and provides insight that would impact scheduling and management choices. Further analysis could elucidate additional differences between the sets allowing for more accurate scheduling algorithms. While this proof of concept focused on identifying holidays, the algorithm is not limited to these variables. It is possible that it could be expanded to similar bimodal and multimodal datasets for both economic and clinical variables. Further research is needed to identify potential candidate variables and determine the validity of this method in other use cases. The algorithm would be most applicable in situations where metadata is missing for many data points but has been tracked for some. The data points that have the metadata of interest could be used to identify a variable that corresponds to clustering of the desired metadata, just as we used case time to correspond to the type of day. This variable could then be used to fill in the missing metadata for other data points. Several aspects of the analysis were unexpected. First, the removal of linear trend from the data caused the algorithm to perform worse. This is counterintuitive because it would seem that this trend would cause high-throughput days from earlier in the set to be miscategorized as lowthroughput days because of the upward linear trend. This was not observed, however, and more lost days occurred after removing trend. Worsened performance arose from the differential change in surgical volume between the categories with respect to time. This led to the important realization that the two sets follow different trends but also underlines the fact that metadata must be present for a small subset of the data to validate the categorization if the algorithm is to be expanded to other variables. The second unexpected aspect of the analysis was that almost no cleaning of the data was necessary for the algorithm to work. This suggests that missing data, mainly when a missing time recording prevented the calculation of case time, was equally likely to occur on a holiday or non-holiday. Because the dataset is so large (even when considering a single year), these gaps did not need to be corrected. This makes the tool much more robust and applicable without significant time input on the part of the researcher. In conclusion, we demonstrate that metadata about holidays can be generated quickly from anesthesia data and can then be used for additional analysis. This cluster analysis method can potentially be applied to other datasets and variables, and future research will focus on expanding this proof of concept to other categorizations.
J Med Syst (2015) 39: 44
Page 5 of 5 44
Acknowledgments Maxim Terekhov for his help in conducting and compiling the statistical analysis. 5. Compliance with ethical standards Competing interests None. Funding This work was supported by Department of Anesthesiology funds. Additionally, Dr. Wanderer is funded by the Foundation for Anesthesia Education and Research (FAER) Health Service Research Mentored Research Training Grant (HSR-MRTG).
Ethical approval This research was exempt from IRB approval because it does not meet the requirements for human subjects research.
1. Hu, H., Correll, M., Kvecher, L., et al., DW4TR: A Data Warehouse for Translational Research. J Biomed Inform 44:1004–1019, 2011. doi:10.1016/j.jbi.2011.08.003. 2. Birnholtz, J. P., and Bietz, M. J., Data at Work: Supporting Sharing in Science and Engineering. Proc. 2003 Int. ACM SIGGROUP Conf. Support. Group Work. New York, NY, USA, New York, pp. 339– 348, 2003. 3. Orlikowski, W. J., Learning from Notes: Organizational Issues in Groupware Implementation. Proc. 1992 ACM Conf. Comput.Support. Coop. Work. ACM, New York, NY, USA, pp. 362–369, 1992. 4. Chin, G., Jr., and Lansing, C. S., Capturing and Supporting Contexts for Scientific Data Sharing via the Biological Sciences
Collaboratory. Proc. 2004 ACM Conf. Comput. Support. ACM, New York, NY, USA, pp. 409–418, 2004. Wang, F., Vergara-Niedermayr, C., and Liu, P., Metadata based management and sharing of distributed biomedical data. Int J Metadata Semant Ontol 9:42–57, 2014. doi:10.1504/ IJMSO.2014.059126. Lee, L. H., Chou, Y. T., Huan, E. W., and Liou, D. M., Design of a Personal Health Record and Health Knowledge Sharing System using IHE-XDS and OWL. J Med Syst 37(2):9921, 2013. doi:10. 1007/s10916-012-9921-4. Peccora, C. D., Gimlich, R., Cornell, R. P., et al., Anesthesia report card - a customizable tool for performance improvement. J Med Syst 38:105, 2014. doi:10.1007/s10916-014-0105-2. Gabriel, R. A., Gimlich, R., Ehrenfeld, J. M., and Urman, R. D., Operating room metrics score card-creating a prototype for individualized feedback. J Med Syst 38:144, 2014. doi:10.1007/s10916014-0144-8. Dexter, F., and Epstein, R. H., Holiday and weekend operating room on-call staffing requirements. Anesth Analg 103:1494–1498, 2006. doi:10.1213/01.ane.0000237176.10774.17. Blomqvist, P., Ljung, H., Nyrén, O., and Ekbom, A., Appendectomy in Sweden 1989–1993 assessed by the Inpatient Registry. J Clin Epidemiol 51:859–865, 1998. Moore, I. C., Strum, D. P., Vargas, L. G., and Thomson, D. J., Observations on surgical demand time series: detection and resolution of holiday variance. Anesthesiology 109:408–416, 2008. doi:10. 1097/ALN.0b013e318182a955. Ryan JA, Ulrich JM (2014) xts: eXtensible Time Series. Fraley C, Raftery A, Scrucca L (2014) mclust: Normal Mixture Modeling for Model-Based Clustering, Classification, and Density Estimation. Ringach MM (originally from F and S -plu. by D, NYU.edu) (2013) diptest: Hartigan’s dip test statistic for unimodality - corrected code.
Insertional mutagenesis using engineered transposons is a potent forward genetic screening technique used to identify cancer genes in mouse model systems. In the analysis of these screens, transposon insertion sites are typically identified by target
Methods for inferring signaling networks using single gene knockdown RNAi experiments and reference networks have been proposed in recent years. These methods assume that RNAi information is available for all the genes in the signal transduction path
The design and use of a metadata-driven data repository for research data management is described. Metadata is collected automatically during the submission process whenever possible and is registered with DataCite in accordance with their current me
Data fully utilized by the community resources promote progress rather than repetition. Effective data sharing can accelerate the transition from data to actionable knowledge, yet barriers to data sharing remain, both technological and procedural. Th
Biomedical research data sharing is becoming increasingly important for researchers to reuse experiments, pool expertise and validate approaches. However, there are many hurdles for data sharing, including the unwillingness to share, lack of flexible
Systematic data management and controlled data sharing aim at increasing reproducibility, reducing redundancy in work, and providing a way to efficiently locate complementing or contradicting information. One method of achieving this is collecting da
The large diversity and volume of extracellular RNA (exRNA) data that will form the basis of the exRNA Atlas generated by the Extracellular RNA Communication Consortium pose a substantial data integration challenge. We here present the strategy that
The increased availability of genomic and metagenomic data poses challenges at multiple analysis levels, including visualization of very large-scale microbial and microbial community data paired with rich metadata. We developed GraPhlAn (Graphical Ph
The core mission of ELIXIR is to build a stable and sustainable infrastructure for biological information across Europe. At the heart of this are the data resources, tools and services that ELIXIR offers to the life-sciences community, providing stab