How a COVID-19 mortality prediction model created by CU data scientists could provide insights for the next pandemic

Overflowing intensive care units. A shortage of personal protective equipment. A scramble for hospital beds and ventilators. Health care workers pushed to the brink. The COVID-19 pandemic laid bare many well-documented vulnerabilities of health care systems. The need for accurate and early clinical assessment of severity related to COVID-19 was vital to developing crisis standards of care to meet the growing pandemic. These standards of care are informed by mortality prediction models, which assess the risk of imminent death in patients.

Shortly after the first case of COVID-19 was officially diagnosed in Colorado, UCHealth University of Colorado Hospital (UCH) approached a team led by Tell Bennett, MD, MS, associate professor of biomedical informatics at the CU School of Medicine, to create a mortality prediction model specific to COVID-19 patients.

“COVID forced us to crystallize our thinking around mortality prediction models,” Bennett recalls. “The pieces were out there. People thought about crisis standards of care, but they were never implemented. We never tested whether existing predictors would meet our needs. They did not.”

Bennett was already focused on using informatics and data science in health care settings, including to assist with clinical decision making in ICUs. His experience as an attending physician in the pediatric ICU at Children’s Hospital Colorado and as informatics director of the Colorado Clinical and Translational Sciences Institute laid the groundwork for a COVID-19 mortality prediction model.

Bennett worked against the clock with UCHealth IT personnel, CU Anschutz Medical Campus data scientists, front-line clinicians, and ethicists to develop a tool that would extract data from patients’ electronic health records (EHRs) in real time.

“We deployed it in the UCHealth EHR,” says Bennett, who also serves as one of the leads in the National COVID Cohort Collaborative (N3C) Data Enclave, which collects COVID-19 data from institutions around the country. “It runs in the background. We applied a relatively innovative modeling strategy to make it accurate and interpretable.”

A primer on mortality prediction models

Before COVID-19, most crisis standards of care protocols relied on the sequential organ failure assessment (SOFA), a scoring system that assesses several organ systems (neurologic, blood, liver, kidney and blood pressure) and assigns a score based on the data obtained in each category. The higher the SOFA score, the higher the risk of death. However, the SOFA score proved inadequate during the 2009 swine flu pandemic, particularly among patients experiencing respiratory failure.

Bennett’s team added four other predictive models to SOFA, including a pneumonia score and the Charlson Comorbidity Index, which measures risk of death for hospitalized patients with multiple co-occurring conditions (such as diabetes and heart failure). Then they integrated COVID-specific predictors: lactate dehydrogenase (LDH; sometimes associated with viral infection) and ferritin (an indicator of inflammation) levels. Within six weeks, they launched the predictor and began collecting data across UCHealth’s 12 hospitals.

“This was early in the pandemic and clinicians were working incredibly hard in very difficult conditions," says Bennett. "Our team of data scientists felt a sense of purpose building this tool because it was a way that we could contribute during a very uncertain time.”

Over the next 10 weeks, the model amassed data from thousands of inpatients, with and without COVID-19. Bennett’s team compared these scores with those of patients hospitalized before the pandemic and found that the new model was more accurate than all other individual models in predicting death from COVID-19.

Lessons learned

Although the model constantly gathers data and updates the score every 15 minutes, Bennett cautions that the tool is not meant to be used in isolation to make treatment decisions for individual patients.

The goal of the new model has always been to provide information to multidisciplinary decision-making teams at large health care systems that may be responsible for the difficult task of triaging patients in a crisis scenario. With the evolution of COVID-19 and emerging new variants, adjustments will need to be made to the mortality predictor.

Though crisis standards of care did not have to be used in Colorado and the mortality prediction model has never been used for its intended purpose, Bennett says the exercise of creating the tool was an invaluable experience, demonstrating what could be accomplished with data from an EHR.

“We continue to partner with UCHealth and are in discussion about a broader program to build and deploy models to meet their operational needs,” says Bennett. “The success of this demonstrated what we can do together.”