Forecasting and Situation Assessment for Real-Time Epidemic Science: With the increased interest in epidemic forecasting and situational assessment during the COVID-19 pandemic, our team members have been actively involved with state and federal agencies.
Estimating the true magnitude of infections was one of the significant challenges in combating the COVID-19 outbreak early on. Our inability in doing so allowed unreported infections to drive up disease spread in numerous regions in the US and worldwide. Even today, identifying the true magnitude (the number of total infections) is still challenging, despite the use of surveillance-based methods such as serological studies, due to their costs and biases. In recent work, we propose an information theoretic approach to estimate total infections accurately. Our approach is built on top of ordinary differential equations-based epidemiological models, which have been used extensively in understanding the dynamics of COVID-19, and aims to estimate the true total infections and a parameterization that “best describes” the observed reported infections. Our experiments show that the parameterization learned by our framework leads to a better estimation of total infections and forecasts of the reported infections compared to a “baseline” parameterization, which is learned via usual model calibration. We also demonstrate that our framework can be leveraged to simulate what-if scenarios with non-pharmaceutical interventions. Our results also support earlier findings that most COVID-19 infections were unreported and non-pharmaceutical interventions indeed helped mitigate the COVID-19 outbreak. Our approach gives a general method to use information theoretic techniques to improve epidemic modeling, which can also be applied to other diseases.
The devastating impact of the currently unfolding global COVID-19 pandemic has sharply illustrated our enormous vulnerability to emerging infectious diseases. Forecasting disease trajectories is a non-trivial and important task. Estimating various measures related to the epidemic gives policymakers valuable lead time to plan interventions and optimize supply chain decisions. The Centers for Disease Control and Prevention (CDC) is hosting collaborative forecasting projects under the umbrella of the COVID-19 Forecast Hub. The tasks are predicting the coronavirus disease spread and anticipating the mortality and number of hospitalizations caused by the disease across the country. This critically timed project has attracted submissions from more than 58 teams from industry and academia of leading data scientists, epidemiologists, statisticians, and high-performance computing researchers from national laboratories, public universities, public health institutions, and some private sector agents. Multiple team members have supported the CDC forecasting efforts.
The Georgia Tech team has used deep learning models to forecast specific targets at the national, state, and local levels since the end of April 2020 [1]. In addition to CDC data, we are incorporating many other real-time datasets such as syndromic surveillance data and mobility. We combine these datasets with domain knowledge using end-to-end deep learning models to predict targets on a weekly basis. The CDC synthesizes our weekly predictions with other models to help determine policy and other planning decisions to help communities prepare for and fight the disease. Our model DeepCOVID, has been included in the official ensemble and its probabilistic forecasts were evaluated over a span of 1 year. The CDC-authored evaluation places our model in the top 5 individual models out of 23 evaluated models. Our methodological advances in DeepCOVID and experiences from an AI perspective were published in AAAI-IAAI 2021 [1], and our work on pandemic forecasting was awarded in several COVID-19 data science challenges (1st place in the Facebook/CMU COVID-19 Challenge and the 2nd place in the C3.ai COVID-19 Grand Challenge).
Continued efforts by our Georgia Tech team members in supporting CDC initiatives have been complemented by our participation in the FluSight challenge that requests groups influenza hospitalization forecasts
at state and national levels. We are participating in this challenge under the team GT-FluFNP, which uses our non-parametric neural forecasting model [2]. In addition, we updated our DeepCOVID model with CAMul, a multi-view non-parametric neural model [3], along with statistical corrections based on data revision dynamics [4]. Incorporating these models into our real-time forecasting pipeline has helped us to further improve our forecasts. Georgia Tech’s models appear among the top 5 models even as more teams are joining for over two years and 50 states. In addition, as part of the COVID-19 Forecast Hub, our evaluation paper has been accepted for publication at PNAS (Cramer et al., 2022).
The UVA team has continued to provide weekly forecasts to the CDC-coordinated COVID-19 Fore- cast Hub (The Hub) using the Bayesian model averaging (BMA) ensemble model. The model incorporates the different classes of models; statistical (auto-regressive models and Kalman filters), deep neural networks (LSTM), and compartmental models (multiple variants). In addition to the 1 − 4 week ahead county-level incidence cases forecasting, we have expanded to providing forecasts for daily COVID- 19 hospital admissions (state level), weekly influenza hospital admissions, incidence case forecasting for the European CDC (https://covid19forecasthub.eu/), and COVID-19 incidence cases in Indian states (https://www.isibang.ac.in/incovid19/dash.php). Through this ensemble model, we identified that all model classes are important and contribute to the forecasting performance at different phases of the pandemic.
A new trend in epidemic forecasting is hybrid models which involves combining theory-based mechanistic models and deep learning models. The objective of these methods is to reap the benefits of both models and obtain better forecasting performance. A key aspect of model development involves capturing the spatiotemporal dynamics of disease spread. A network-based compartmental model is the traditional approach. However, it is challenging to employ those models due to their high computational requirements and data-related issues. In order to account for the spatiotemporal dependencies and simultaneously reduce the computational requirements, we developed a Causal-based Neural Network (CausalGNN) [5]. The CausalGNN is designed to jointly learn embeddings related to the spatiotemporal data (i.e., dynamic graphs) and the causal model (referred to as epidemiological context and representing the disease models; Susceptible, Infected, Recovered counts, and other disease-specific parameters). The framework consists of three components: (i) An attention-based dynamic GNN module that embeds spatial and temporal signals from disease dynamics using a connectivity network, (ii) a causal module that encodes the single-patched compartmental model signals to provide epidemiological context. Unlike the traditional metapopulation compartmental models, in our model, the patches are connected via a learned GNN, and the calibration is done through computationally efficient GNN training. (iii) A causal decoder whose purpose is to encode causal features through node embedding and this is incorporated to ensure the flow of spatiotemporal information into the epidemiological model and vice versa.
In the initial phase of the pandemic, universities and colleges had to resort to remote learning to ensure the safety of the students and staff. However, in Fall 2020, several campuses provided an option for in-person learning and there was concern about outbreaks on campuses and in the larger population, especially due to the lack of vaccine availability. Owing to this concern, several campuses resorted to regular testing and close monitoring of cases. The campus-level case data was made available to the public and there was considerable interest in forecasting them and we developed a model for this purpose. In the Fall 2020 semester, we provided continuous up-to-date campus-level weekly COVID- 19 case forecasts and scenario analyses to inform campus officials of the University of Virginia and three other universities (Virginia Tech, Virginia Commonwealth University, and James Madison University). To the best of our knowledge, it is one of the only campus-level real-time forecasting efforts [6]. Although the data was made available, it suffered from inconsistent reporting frequency, formatting, resolution, and corrections. Given the data-related challenges, statistical or purely data-driven models were not feasible at the time. As a result, we employed the SEIR model for each school and the model parameters were calibrated on the respective school’s case time series. The model produced up to 14-day forecasts under each of three scenarios (stable contact rate, contact rate halved, and contact rate doubled). In addition, we provided isocurves that informed intervention efforts by depicting the trade-off between testing frequency and contact rate in resulting new cases. The model performance suffered during the campus reopening phase but as the fall semester progressed the model was able to provide reasonable forecasts.
Effective influenza forecasting still remains a challenge despite increasing research interest. It is even more challenging amidst the COVID pandemic when the influenza-like illness (ILI) counts are affected by various factors such as symptomatic similarities with COVID-19 and shifts in healthcare-seeking patterns of the general population. Under the current pandemic, historical influenza models carry valuable expertise about the disease dynamics but face difficulties adapting to this new scenario. Therefore, we proposed a neural transfer learning architecture that allows us to ’steer’ a historical disease forecasting model to new scenarios where flu and COVID co-exist. Our framework enables this adaptation by automatically learning when it should emphasize learning from COVID-related signals and when from the historical model. In such a way, we exploit representations learned from historical ILI data as well as the relevant but limited COVID-related signals. Specifically, our method is based on heterogeneous domain transfer learning and contains components to avoid negative transfer (by attentive knowledge distillation losses), promote spatial consistency (via Laplacian losses in a recurrent architecture), and also handle data paucity by learning from multiple geographical regions. We show through extensive experiments that our method effectively captures non-trivial atypical trends in COVID-ILI evolution better than competitive baselines. Our approach and results provide guidance for forecasting future emerging infectious pandemics. This work has been published in Rodriguez et al. AAAI-21 [1] and a preliminary version in [7].
Influenza seasons tend to be highly dynamic and have high variability due to numerous factors (e.g., weather, human mobility, virus strains circulating amongst the population) affecting the overall characteristics of the season. However, for such complex dynamics, there is not enough data characterizing its multiple aspects of it. Because of this, purely data-driven models may show undesirable, unexplainable, or otherwise unexpected behavior. To ameliorate these issues, we propose incorporating expert guidance to inform our model of aspects of the epidemic dynamics by enforcing desirable behaviors such that the resulting forecasts will be more actionable. For example, consider influenza incidence during the annual holiday season in the US. During this period, patients typically self-select and refrain from going to health providers, unless the situation is serious. This causes a temporary drop in recorded influenza trajectory. However, as human mobility is high, flu activity rapidly increases in the following weeks. Hence, if we can ensure that the forecasting model’s predictions are reasonably ’smooth’, such behavior can be avoided. We propose to leverage the Seldonian Optimization framework proposed in AI safety to enforce expert guidance. Our framework provides feedback to the user regarding the success or failure in incorporating expert insights. In case the framework fails to incorporate the insight, it communicates the failure to the user, who in turn can take steps to alter/improve the insight or change data or modify model hyper-parameters. We present concrete case studies showing examples of expert guidance motivated by epidemiologist observation, and how our method helps to achieve experts’ requirements. This work has been published in [8].
We tackled the pandemic along two research tracks. The first track centers around the models for the spread of COVID-19, while the second track studies different approaches for speeding up drug discovery.
Social distancing is one of the most widespread and powerful measures deployed against COVID-19. Policymakers require epidemiological models that can estimate the impact of different social distancing behaviors — including school closure, non-essential business closure, religious center closure, and shelter-in-place orders — will have on their local communities. However, current models cannot make effective local-level estimates because each county implemented different social distancing measures, in different orders, at different times. Our work in this area spans: (i) a crowdsourcing effort to collect critical information during the early stages of the epidemic in the US, (ii) scientific visualizations and communication of social distancing measures, (iii) a SARS-CoV-2 spread model that integrates fine-grained, dynamic mobility networks, and (iv) decision-support tools that utilize large-scale data and epidemiological modeling to quantify the impact of changes in mobility on infection rates.
Prompted by a request from epidemiological modelers who are assisting the CDC and other federal and state agencies in the COVID-19 response, we have launched a crowdsourcing effort to collect and update social distancing data such as when each restriction was put into place and relaxed for each of 3000+ counties in the US (https://socialdistancing.stanford.edu/) [9]. This collaborative effort between the Stanford and UVA teams gathered over 2100 submissions across 481 counties, which were used to guide policymakers in making decisions about when and how to relax social distancing, support local residents in exploratory model visualization of the results of relaxing social distancing, and enable scientists to better estimate the effects of each social distancing measure.
Our second goal has been to aid scientific visualization and communication of social distancing measures. We have worked to develop visualizations of counterfactuals for how COVID-19 would spread (or not) based on changes to social distancing behavior. These visualizations were embedded in https://covid-mobility.stanford.edu. One of the visualizations was featured by The New York Times [10].
[1] A. Rodriguez, A. Tabassum, J. Cui, J. Xie, J. Ho, P. Agarwal, B. Adhikari, and B. A. Prakash, “Deep-COVID: An operational deep learning-driven framework for explainable real-time COVID-19 forecasting,” in Proc. AAAI Conference on Artificial Intelligence, 2021.
[2] H. Kamarthi, L. Kong, A. Rodríguez, C. Zhang, and B. A. Prakash, “When in doubt: Explainable and principled uncertainty quantification for epidemic forecasting,” in NeuRIPS, 2021.
[3] H. Kamarthi, L. Kong, A. Rodríguez, C. Zhang, and B. A. Prakash, “Camul: Calibrated and accurate multi-view time-series forecasting,” CoRR, vol. abs/2109.07438, 2021.
[4] H. Kamarthi, A. Rodríguez, and B. A. Prakash, “Back2future: Leveraging backfill dynamics for improving real-time predictions in future,” CoRR, vol. abs/2106.04420, 2021.
[5] L. Wang, A. Adiga, J. Chen, A. Sadilek, S. Venkatramanan, and M. Marathe, “CausalGNN: Causal-based graph neural networks for spatio-temporal epidemic forecasting,” in Proceedings of the 36th AAAI Conference in Artificial Intelligence (to appear), AAAI Press, 2022.
[6] B. Hurt, A. Adiga, M. Marathe, and C. L. Barrett, “Informing university covid-19 decisions using simple compartmental models,” in 2021 Winter Simulation Conference (WSC), pp. 1–12, 2021.
[7] A. Rodríguez, N. Muralidhar, B. Adhikari, A. Tabassum, N. Ramakrishnan, and B. A. Prakash, “Steering a historical disease forecasting model under a pandemic: Case of flu and covid-19,” NeurIPS 2020. Machine Learning in Public Health (MLPH) Workshop, 2020.
[8] A. Rodríguez, B. Adhikari, N. Ramakrishnan, and B. A. Prakash, “Incorporating expert guidance in epidemic forecasting,” arXiv preprint arXiv:2101.10247, 2020.
[9] T. Abate, “Stanford Crowdsourcing site collects county-level policy data to inform decisions about easing social-distancing,” April 2020. https://news.stanford.edu/2020/04/13/stanford-crowdsources-
county-level-covid-19-policy-data/.
[10] Y. Serkez, “The magic number for reducing infections and keeping businesses open,” The New York Times, December 2020.
