ARTICLE IN PRESS
Identifying children at risk of death within 30 days of surgery at an NSQIP pediatric hospital Max R. Langham, Jr, MD, FACS,a Arianne Walter, BA,a Timothy C. Boswell, BS,a Robert Beck, RRT,b and Tamekia L. Jones, PhD,c Memphis, TN
Background. Informed consent for operative procedures performed on children relies on the ability of the surgeon to estimate and describe accurately the risks and benefits of the planned operation to the parents. Understanding patient-specific risks is also an important prerequisite for surgeons and hospital administrators who wish to change hospital processes and improve patient safety. This study tests the feasibility of estimating the risk of death within 30 days of surgery using National Surgical Quality Improvement Program (NSQIP)-Pediatric data from a single children’s hospital. Study design. Patient data submitted to NSQIP-Pediatric from our hospital were analyzed to identify variables predictive of death within 30 days of operation. A multiple logistic regression model was constructed using 3 years of data and validated using data submitted the following year. The model was then tested using the participant use file provided by NSQIP-Pediatric for 2012. Results. The model identified 7 variables predictive of death: neonatal status, respiratory support, inotropic support, having a blood disorder, cerebrovascular injury, previous cardiac intervention, and the work relative value unit for the procedure. The resulting final model had a c statistic = 0.97. Conclusion. It is possible for a participating children’s hospital to use NSQIP-Pediatric data to develop risk models for patient mortality occurring within 30 days of operation at their institution. The model presented may be generalizable to other institutions, but needs further testing and refining. (Surgery 2015;j:j-j.) From the Department of Surgery,a University of Tennessee Health Science Center, and Le Bonheur Children’s Hospital; the Department of Decision Support,b Le Bonheur Children’s Hospital; and the Departments of Pediatrics and Preventive Medicine,c University of Tennessee Health Science Center, Children’s Foundation Research Institute at Le Bonheur Children’s Hospital, Memphis, TN
THE NATIONAL SURGICAL QUALITY IMPROVEMENT PROGRAMPEDIATRIC (NSQIP-Pediatric) was developed through a collaboration of the American College of Surgeons and the American Pediatric Surgical Association. After demonstration of feasibility in a pilot program involving four hospitals,1 the project was opened in 2010 to 29 children’s hospitals for beta testing to demonstrate benefit.2 Successful implementation has resulted in rapid growth with 59 hospitals reporting data included in the most recent NSQIP-Pediatric semiannual report. NSQIP-Pediatric provides a robust dataset that is abstracted by trained clinical reviewers, and based Accepted for publication April 22, 2015. Reprint requests: Max R. Langham, Jr, MD, FACS, Department of Surgery, University of Tennessee Health Science Center, and Le Bonheur Children’s Hospital, 51 North Dunlap, Suite P230, Memphis, TN 38105. E-mail: [email protected]. 0039-6060/$ - see front matter Ó 2015 Elsevier Inc. All rights reserved. http://dx.doi.org/10.1016/j.surg.2015.04.026
on rigorously defined variables and a published, consistent sampling methodology. These strengths are unique and support the program’s goal of allowing surgeons and their hospitals to collect actionable clinical outcomes data specific to children’s surgery and act on clinical data to improve care in children’s surgery. A growing body of literature has been published comparing hospitals providing surgical care using analytic methods similar to those originally used for the adult NSQIP program.3 Identifying significant differences in event rates between institutions and adjusting for differences in procedural volume, case mix, and patient characteristics among institutions has been challenging,4–6 even in adult hospitals. Event rates such as death within 30 days of operation at children’s hospitals have been much lower than those noted in adult hospitals.7 The semiannual reports published by NSQIPPediatric have consistently shown no differences in mortality at 30 days among institutions participating in the program. This has led to the SURGERY 1
ARTICLE IN PRESS 2 Langham et al
conclusion that mortality has a low utility as a performance indicator for children’s hospitals reporting to NSQIP-Pediatric.8 Although uncommon, the death of a child after elective surgery is a sentinel event for the hospital.9 Parents and children often are stressed by the idea of surgery10 and anesthesia,11 and may seek comfort in the informed consent process. Surgeons wishing to provide sensitive but transparent information to parents during the informed consent process currently cannot use children’s preoperative data to estimate accurately the risk for that individual patient. The ability to estimate accurately the risk of death within 30 days of surgery based on specific patient characteristics would inform this process significantly, while simultaneously augmenting data used in hospitals’ quality improvement programs. This study tests the feasibility of using data from a single children’s hospital submitted to NSQIPPediatric to estimate the risk of death within 30 days of operation from information available preoperatively. The goals of this effort are to (1) provide individual risk estimates for parents whose children need operations, thus improving informed consent and (2) create a threshold for risk of death within 30 days of surgery, above which our hospital might change its clinical care processes. METHODS Data source and subject eligibility. The study was determined to be exempt by the University of Tennessee Health Science Center Institutional Review Board. NSQIP-Pediatric methodology was used to generate a sample of operations from the operative procedures performed at Le Bonheur Children’s Hospital, Memphis, Tennessee, between January 1, 2010, and December 31, 2013. Briefly, a trained Surgical Clinical Reviewer reviewed the operative case log for specific operations. The Current Procedural Terminology (CPT) codes for operations included in the NSQIPPediatric sample during the time frame of the study are listed in Appendix I. Eight-day cycles were used to prevent systematic oversampling based on surgical block times. For the period of study, the first 35 eligible operations in any 8-day block were abstracted by our Surgical Clinical Reviewer and the data submitted to NSQIPPediatric. Operations without complete 30-day follow-up were excluded. For patients with >1 procedure captured within the data, a single operation was selected randomly selected for use in this study. Data abstracted from Le Bonheur
Surgery j 2015
Children’s Hospital between January 1, 2010, and December 31, 2012, were used to build our logistic regression model, and data collected from our institution between January 1, 2013, and December 31, 2013, were used to internally validate it. The NSQIP-Pediatrics Participant Use Data File (PUF) data from January 1, 2012, to December 31, 2012, were used to assess generalizability of the model. Variable definitions. NSQIP-Pediatric variable definitions12 current on July 1, 2012, were chosen as the reference definitions for this study. Changes made by the NSQIP Data Definitions Committee during the period of the study were mapped. Variables with major changes in definition were evaluated and altered to reflect core information that was consistent throughout the time period of the study. Variable criteria included in analysis. Levels of preoperative categorical variables that had few positive responses were collapsed and related NSQIP-Pediatric variables were combined to create comprehensive composite variables. Frequency tables of original variables were compared with composite variables to ensure the logic used in creating the composite variables was valid. Any observations from our hospital identified as being inconsistent were verified and corrected where possible using the patient’s electronic medical record. If data could not be verified, the observation was considered missing. Variables with >2% missing data were excluded from the model. For variables with
Identifying children at risk of death within 30 days of surgery at an NSQIP pediatric hospital Max R. Langham, Jr, MD, FACS,a Arianne Walter, BA,a Timothy C. Boswell, BS,a Robert Beck, RRT,b and Tamekia L. Jones, PhD,c Memphis, TN
Background. Informed consent for operative procedures performed on children relies on the ability of the surgeon to estimate and describe accurately the risks and benefits of the planned operation to the parents. Understanding patient-specific risks is also an important prerequisite for surgeons and hospital administrators who wish to change hospital processes and improve patient safety. This study tests the feasibility of estimating the risk of death within 30 days of surgery using National Surgical Quality Improvement Program (NSQIP)-Pediatric data from a single children’s hospital. Study design. Patient data submitted to NSQIP-Pediatric from our hospital were analyzed to identify variables predictive of death within 30 days of operation. A multiple logistic regression model was constructed using 3 years of data and validated using data submitted the following year. The model was then tested using the participant use file provided by NSQIP-Pediatric for 2012. Results. The model identified 7 variables predictive of death: neonatal status, respiratory support, inotropic support, having a blood disorder, cerebrovascular injury, previous cardiac intervention, and the work relative value unit for the procedure. The resulting final model had a c statistic = 0.97. Conclusion. It is possible for a participating children’s hospital to use NSQIP-Pediatric data to develop risk models for patient mortality occurring within 30 days of operation at their institution. The model presented may be generalizable to other institutions, but needs further testing and refining. (Surgery 2015;j:j-j.) From the Department of Surgery,a University of Tennessee Health Science Center, and Le Bonheur Children’s Hospital; the Department of Decision Support,b Le Bonheur Children’s Hospital; and the Departments of Pediatrics and Preventive Medicine,c University of Tennessee Health Science Center, Children’s Foundation Research Institute at Le Bonheur Children’s Hospital, Memphis, TN
THE NATIONAL SURGICAL QUALITY IMPROVEMENT PROGRAMPEDIATRIC (NSQIP-Pediatric) was developed through a collaboration of the American College of Surgeons and the American Pediatric Surgical Association. After demonstration of feasibility in a pilot program involving four hospitals,1 the project was opened in 2010 to 29 children’s hospitals for beta testing to demonstrate benefit.2 Successful implementation has resulted in rapid growth with 59 hospitals reporting data included in the most recent NSQIP-Pediatric semiannual report. NSQIP-Pediatric provides a robust dataset that is abstracted by trained clinical reviewers, and based Accepted for publication April 22, 2015. Reprint requests: Max R. Langham, Jr, MD, FACS, Department of Surgery, University of Tennessee Health Science Center, and Le Bonheur Children’s Hospital, 51 North Dunlap, Suite P230, Memphis, TN 38105. E-mail: [email protected]. 0039-6060/$ - see front matter Ó 2015 Elsevier Inc. All rights reserved. http://dx.doi.org/10.1016/j.surg.2015.04.026
on rigorously defined variables and a published, consistent sampling methodology. These strengths are unique and support the program’s goal of allowing surgeons and their hospitals to collect actionable clinical outcomes data specific to children’s surgery and act on clinical data to improve care in children’s surgery. A growing body of literature has been published comparing hospitals providing surgical care using analytic methods similar to those originally used for the adult NSQIP program.3 Identifying significant differences in event rates between institutions and adjusting for differences in procedural volume, case mix, and patient characteristics among institutions has been challenging,4–6 even in adult hospitals. Event rates such as death within 30 days of operation at children’s hospitals have been much lower than those noted in adult hospitals.7 The semiannual reports published by NSQIPPediatric have consistently shown no differences in mortality at 30 days among institutions participating in the program. This has led to the SURGERY 1
ARTICLE IN PRESS 2 Langham et al
conclusion that mortality has a low utility as a performance indicator for children’s hospitals reporting to NSQIP-Pediatric.8 Although uncommon, the death of a child after elective surgery is a sentinel event for the hospital.9 Parents and children often are stressed by the idea of surgery10 and anesthesia,11 and may seek comfort in the informed consent process. Surgeons wishing to provide sensitive but transparent information to parents during the informed consent process currently cannot use children’s preoperative data to estimate accurately the risk for that individual patient. The ability to estimate accurately the risk of death within 30 days of surgery based on specific patient characteristics would inform this process significantly, while simultaneously augmenting data used in hospitals’ quality improvement programs. This study tests the feasibility of using data from a single children’s hospital submitted to NSQIPPediatric to estimate the risk of death within 30 days of operation from information available preoperatively. The goals of this effort are to (1) provide individual risk estimates for parents whose children need operations, thus improving informed consent and (2) create a threshold for risk of death within 30 days of surgery, above which our hospital might change its clinical care processes. METHODS Data source and subject eligibility. The study was determined to be exempt by the University of Tennessee Health Science Center Institutional Review Board. NSQIP-Pediatric methodology was used to generate a sample of operations from the operative procedures performed at Le Bonheur Children’s Hospital, Memphis, Tennessee, between January 1, 2010, and December 31, 2013. Briefly, a trained Surgical Clinical Reviewer reviewed the operative case log for specific operations. The Current Procedural Terminology (CPT) codes for operations included in the NSQIPPediatric sample during the time frame of the study are listed in Appendix I. Eight-day cycles were used to prevent systematic oversampling based on surgical block times. For the period of study, the first 35 eligible operations in any 8-day block were abstracted by our Surgical Clinical Reviewer and the data submitted to NSQIPPediatric. Operations without complete 30-day follow-up were excluded. For patients with >1 procedure captured within the data, a single operation was selected randomly selected for use in this study. Data abstracted from Le Bonheur
Surgery j 2015
Children’s Hospital between January 1, 2010, and December 31, 2012, were used to build our logistic regression model, and data collected from our institution between January 1, 2013, and December 31, 2013, were used to internally validate it. The NSQIP-Pediatrics Participant Use Data File (PUF) data from January 1, 2012, to December 31, 2012, were used to assess generalizability of the model. Variable definitions. NSQIP-Pediatric variable definitions12 current on July 1, 2012, were chosen as the reference definitions for this study. Changes made by the NSQIP Data Definitions Committee during the period of the study were mapped. Variables with major changes in definition were evaluated and altered to reflect core information that was consistent throughout the time period of the study. Variable criteria included in analysis. Levels of preoperative categorical variables that had few positive responses were collapsed and related NSQIP-Pediatric variables were combined to create comprehensive composite variables. Frequency tables of original variables were compared with composite variables to ensure the logic used in creating the composite variables was valid. Any observations from our hospital identified as being inconsistent were verified and corrected where possible using the patient’s electronic medical record. If data could not be verified, the observation was considered missing. Variables with >2% missing data were excluded from the model. For variables with
