Forecasting

Forecasting and Situation Assessment for Real-Time Epidemic Science

With the increased interest in epidemic forecasting and situational assessment during the COVID-19 pandemic, our team members have been actively involved with state and federal agencies. 

Forecasting COVID-19 outcomes

In April 2020, the Centers for Disease Control and Prevention (CDC) in collaboration with academic partners initiated the COVID-19 Forecast Hub (The Hub), a consortium of modeling teams to coordinate forecasting efforts (https://viz.covid19forecasthub.org/). This critically timed project has attracted submissions from more than 58 teams (as of Sept 2020) of leading data scientists, epidemiologists, statisticians, and high-performance computing researchers from national laboratories, public universities, public health institutions, and some private sector agents. Initially, due to the lack of reliable data the efforts were restricted to death forecasting at national and state levels.  Subsequently, with the progression of the pandemic,  there was a greater emphasis from policy makers for high-resolution forecasts. In July 2020, the Hub expanded the efforts to include incident case forecasts at the county level. Our team members have actively participated in these efforts, and have developed novel methods for forecasting.    

The UVA and Georgia Tech team members are using deep learning models to forecast specific targets at the national, regional, state, and local levels since the end of April 2020. In addition to CDC data, we are incorporating many other real-time datasets such as syndromic surveillance data and mobility. We combine these datasets with domain knowledge using end-to-end deep learning models to predict targets on a weekly basis. The CDC synthesizes our weekly predictions with other models to help determine policy and other planning decisions to help communities prepare for and fight the disease. Our model, DeepCOVID, has been included in the official ensemble and its probabilistic forecasts were evaluated over a span of 7 months. The latter evaluation places our model in the top 5 individual models out of 23 evaluated models. Our approach is the first purely data-driven deep learning (DL) model for real time pandemic forecasting in the COVID-19 Forecast Hub, where the majority of models are mechanistic. By our work, we also aimed to address a gap in the literature pertaining to using purely data-driven approaches for emerging pandemics. First, we have to cope with heterogeneous, scarce, and noisy data so that our DL-based model can leverage many heterogeneous signals that are more sensitive to what is happening on the ground. Enabling this allows us to bring a unique perspective closer to the observed data signals with minimal assumptions that is complementary to other methods in the official ensemble, which benefits from diverse perspectives. Apart from excellent predictive performance, we also ensure our framework gives explanations for its forecasts, which are very important for communication and interpretation by both the public and decision makers. Our work in the ensemble has been submitted to journals [14,46].  The methodological advances in DeepCOVID and experiences from an AI perspective were published in [51,52]. Our work on pandemic forecasting was awarded in several COVID-19 data science challenges as well.    

The UVA team has been one of the first few teams to have provided county-level forecasts. We have employed an ensemble of statistical, machine learning, and mechanistic models to forecast the COVID-19 incident cases at the county level [5]. Studies in the literature and our experience suggest that there is no single class of models that shows superior performances across space and time. The ensemble framework enables one to combine forecasts from multiple classes of models in a robust manner. Specifically, we use Bayesian model averaging which incorporates the uncertainty in individual model forecasts and helps to avoid overconfident predictions. As for the individual ML/AI-based methods, we have incorporated autoregressive models, Kalman filters, long short-term memory (LSTM) deep learning models, as well as mechanistic models which have all shown merits in forecasting not only COVID-19 cases but also for other infectious diseases such as influenza and dengue.   

The ensemble is developed as a high resolution real-time forecasting system and the pipeline is run on high performance machines to meet the demands. The one- to four-week ahead forecasts are submitted on a weekly basis to the CDC-coordinated COVID19 Forecast Hub (conglomeration of modeling teams) and our county-level evaluations reveal that our model forecasts rank consistently among the top few models.  Owing to the reliable short-term forecasts of the ensemble, the one-week ahead forecasts also serve as extended ground truth for our scenario-based projections provided by PatchSim, an SEIR model serving state and federal agencies. These forecasts have been provided to the hub for over 6 months on a weekly basis and also shared with DTRA.    

The IU team has also developed new forecasting methods. We are extending our work of integrative deep learning framework for COVID-19 prediction, and comparison with health data currently to 3142 counties for the full period of the pandemic. Early work is published in [19]. We incorporate domain knowledge to model decomposition based on the cause-and-effect relationship by modeling multi-sources time-series data with high uncertainty and extreme events. This improved the deep learning forecasting performance [41].    

In addition, we have attempted to capture the complex interplay between human behavior and disease spread through deep learning [65-67]. Specifically, we considered graph neural networks that provide a framework to capture the spatiotemporal patterns (at county level) of disease spread. The graphs are constructed using mobility data and correlation between case counts of counties. However, it is hard to explain the working of the deep learning methods. We address explainability using a graph neural network framework. We enforce the graph neural network to jointly learn the graph embeddings and the causal space embeddings [65]. The causal model embedding when passed through a trained decoder yielded meaningful causal model parameters which faithfully recreated the case curves when passed through the SEIR model. Finally, in [33], we use high resolution proximity statistics for early warning of epidemic outbreaks during US universities reopening in COVID-19.

Forecasting COVID-19-contaminated influenza

Effective influenza forecasting still remains a challenge despite increasing research interest. It is even more challenging amidst the COVID pandemic, when the influenza-like illness (ILI) counts are affected by various factors such as symptomatic similarities with COVID-19 and shifts in healthcare seeking patterns of the general population. Under the current pandemic, historical influenza models carry valuable expertise about the disease dynamics but face difficulties adapting to this new scenario. Therefore, we proposed a neural transfer learning architecture which allows us to ’steer’ a historical disease forecasting model to new scenarios where flu and COVID co-exist. Our framework enables this adaptation by automatically learning when it should emphasize learning from COVID-related signals and when from the historical model. In such a way, we exploit representations learned from historical ILI data as well as the relevant but limited COVID-related signals. Specifically, our method is based on heterogeneous domain transfer learning and contains components to avoid negative transfer (by attentive knowledge distillation losses), promote spatial consistency (via Laplacian losses in a recurrent architecture), and also handle data paucity by learning from multiple geographical regions. We show through extensive experiments that our method effectively captures non-trivial atypical trends in COVID-ILI evolution better than competitive baselines. Our approach and results provide guidance for forecasting future emerging infectious pandemics. This work has been published in Rodriguez et al.  AAAI-21 [52] and preliminary version in [50].

Incorporating expert guidance in epidemic forecasting

Influenza seasons tend to be highly dynamic and have high variability due to numerous factors (e.g., weather, human mobility, virus strains circulating amongst the population) affecting the overall characteristics of the season. However, for such complex dynamics, there is not enough data characterizing the multiple aspects of it. Because of this, purely-data driven models may show undesirable, unexplainable, or otherwise unexpected behavior. To ameliorate these issues, we propose incorporating expert guidance to inform our model of aspects of the epidemic dynamics by enforcing desirable behaviors such that the resulting forecasts will be more actionable. For example, consider influenza incidence during the annual holiday season in the US. During this period, patients typically self- select and refrain from going to health providers, unless the situation is serious. This causes a temporary drop in recorded influenza trajectory. However, as human mobility is high, flu activity rapidly increases in the following weeks. Hence, if we can ensure that the forecasting model’s predictions are reasonably ’smooth’, such behavior can be avoided. We propose to leverage the Seldonian Optimization framework proposed in AI safety to enforce expert guidance. Our framework provides feedback to the user regarding the success or failure in incorporating the expert insights. In case the framework fails to incorporate the insight, it communicates the failure to the user, who in turn can take steps to alter/improve the insight or change data or modify model hyper-parameters. We present concrete case studies showing examples of expert guidance motivated by epidemiologist observation, and how our method helps to achieve experts’ requirements. This work has been published in [49].

Crowdsourced data collection and visualization

We tackled the pandemic along two research tracks. The first track centers around the models for the spread of COVID-19, while the second track studies approaches for speeding up drug discovery.    Social distancing is one of the most widespread and powerful measures deployed against COVID-19. Policy makers require epidemiological models that can estimate the effect of each social distancing behavior — including school closure, non-essential business closure, religious center closure, and shelter-in-place orders — will have on their local communities. However, current models cannot make effective local-level estimates because each county implemented different social distancing measures, in different orders, at different times. Our work in this area spans: (i) a crowdsourcing effort to collect critical information during the early stages of the epidemic in the US, (ii) scientific visualizations and communication of social distancing measures, (iii) a SARS-CoV-2 spread model that integrates fine-grained, dynamic mobility networks, and (iv) decision- support tools that utilize large-scale data and epidemiological modeling to quantify the impact of changes in mobility on infection rates.    

Prompted by a request from epidemiological modelers who are assisting the CDC and other federal and state agencies in COVID-19 response, we have launched a crowdsourcing effort to collect and update social distancing data such as when each restriction was put into place and relaxed for each the 3000+ counties in the US (https://socialdistancing.stanford.edu/) [1]. This collaborative effort between the Stanford and UVA teams gathered over 2100 submissions across 481 counties, which were used to guide policy makers in making decisions about when and how to relax social distancing, support local residents in exploratory model visualization of the results of relaxing social distancing, and enable scientists to better estimate the effects of each social distancing measure.    

Our second goal has been to aid scientific visualization and communication of social distancing measures. We have worked to develop visualizations of counterfactuals for how COVID-19 would spread (or not) based on changes to social distancing behavior. These visualizations were embedded in https://covid-mobility. stanford.edu. One of the visualizations was featured by the New York Times [60].

References

[1] T. Abate. Crowdsourcing site collects county-level policy data to inform decisions about easing social- distancing, April 2020. https://news.stanford.edu/2020/04/13/stanford-crowdsources-county- level-covid-19-policy-data/.

[2] V. Abeykoon, N. Perera, C. Widanage, S. Kamburugamuve, T. A. Kanewala, H. Maithree, P. Wick- ramasinghe, A. Uyar, and G. Fox. Data engineering for hpc with python. In 2020 IEEE/ACM 9th Workshop on Python for High-Performance and Scientific Computing (PyHPC), pages 13–21. IEEE, 2020.

[3] A. Adiga, J. Chen, M. Marathe, H. Mortveit, S. Venkatramanan, and A. Vullikanti. Data-driven modeling for different stages of pandemic response. Journal of the Indian Institute of Science, pages 1–15, 2020.

[4] A. Adiga, D. Dubhashi, B. Lewis, M. Marathe, S. Venkatramanan, and A. Vullikanti. Mathematical models for covid-19 pandemic: a comparative analysis. Journal of the Indian Institute of Science, pages 1–15, 2020.

[5] A. Adiga, L. Wang, B. Hurt, A. Peddireddy, P. Porebski, S. Venkatramanan, B. Lewis, and M. Marathe. All models are useful: Bayesian ensembling for robust high resolution covid-19 forecasting, 2021. Sub- mitted.

[6] E. Bradley, M. Marathe, M. Moses, W. D. Gropp, and D. Lopresti. Pandemic informatics: Preparation, robustness, and resilience. arXiv preprint arXiv:2012.09300, published as part of CCC Quadrennial pa- pers, 2020. https://cra.org/ccc/resources/ccc-led-whitepapers/#2020-quadrennial-papers.

[7] B. Brubach, D. Chakrabarti, J. P. Dickerson, A. Srinivasan, and L. Tsepenekas.  Fairness,  semi- supervised learning, and more: A general framework for clustering with stochastic pairwise constraints. arXiv preprint arXiv:2103.02013, 2021.

[8] H. L. Carscadden, C. J. Kuhlman, M. V. Marathe, S. S. Ravi, and D. J. Rosenkrantz. Blocking the propagation of two simultaneous contagions over networks. In R. M. Benito, C. Cherifi, H. Cherifi, E. Moro, L. M. Rocha, and M. Sales-Pardo, editors, Proc. 9th International Conference on Complex Networks and Applications (Complex Networks), pages 455–468, Chan, Switzerland, 2020. Springer.

[9] S. Chang, E. Pierson, P. W. Koh, J. Gerardin, B. Redbird, D. Grusky, and J. Leskovec. Mobility network models of covid-19 explain inequities and inform reopening. Nature, 589(7840):82–87, 2020.

[10] J. Chen, S. Levin, S. Eubank, H. Mortveit, S. Venkatramanan, A. Vullikanti, and M. Marathe. Net- worked epidemiology for covid-19. Siam news, 2020.

[11] J. Chen, A. Vullikanti, S. Hoops, H. Mortveit, B. Lewis, S. Venkatramanan, W. You, S. Eubank, M. Marathe, C. Barrett, and M. A.  Medical costs of keeping the us economy open during covid-19. Scientific reports, Oct 2020.

[12] J. Chen, A. Vullikanti, J. Santos, S. Venkatramanan, S. Hoops, H. Mortveit, B. Lewis, W. You, S. Eu- bank, M. Marathe, C. Barrett, and A. Marathe. Epidemiological and economic impact of covid-19 in the us. medRxiv : the preprint server for health sciences, November 2020.

[13] D. Chistikov, G. Lisowski, M. Paterson, and P. Turrini. Convergence of opinion diffusion is pspace- complete. In Proc. AAAI, pages 7103–7110. AAAI Press, 2020.

[14] E. Y. Cramer, V. K. Lopez, J. Niemi, G. E. George, J. C. Cegan, I. D. Dettwiller, W. P. England, M. W. Farthing, R. H. Hunter, B. Lafferty, et al. Evaluation of individual and ensemble probabilistic forecasts of covid-19 mortality in the us. medRxiv, 2021.

[15] S. Eubank, I. Eckstrand, B. Lewis, S. Venkatramanan, M. Marathe, and C. Barrett. Commentary on ferguson, et al.,“impact of non-pharmaceutical interventions (npis) to reduce covid-19 mortality and healthcare demand”. Bulletin of mathematical biology, 82(4):1–7, 2020.

[16] M. C. Fitzpatrick and A. P. Galvani. Optimizing age-specific vaccination. Science, 371(6532):890–891, 2021.

[17] G. Fox. Deep learning based time evolution, 2020. http://dsc.soic.indiana.edu/publications/ Summary-DeepLearningBasedTimeEvolution.pdf.

[18] G. Fox. Deep learning for spatial time series, 2020. http://dsc.soic.indiana.edu/publications/ Deep%20Learning%20for%20Spatial%20Time%20Series.pdf.

[19] G. C. Fox, G. von Laszewski, F. Wang, and S. Pyne. Aicov: An integrative deep learning framework for covid-19 forecasting with population covariates. arXiv preprint arXiv:2010.03757, 2020.

[20] F. Haghpanah, G. Lin, S. Levin, and E. Y. Klein. Analysis of the potential efficacy and timing of covid-19 vaccine on morbidity and mortality. Available at SSRN 3745195, 2021.

[21] A. Haque, M. Thakur, M. Bielskas, and A. V. A Marathe. Persistence of anti-vaccine sentiment in social networks through strategic interactions. In Thirty Fifth AAAI Conference on Artificial Intelligence (AAAI-21), 2021.

[22] D. Harris, T. Pensyl, A. Srinivasan, and K. Trinh. Dependent randomized rounding for clustering and partition systems with knapsack constraints. In International Conference on Artificial Intelligence and Statistics, pages 2273–2283. PMLR, 2020.

[23] P. J. Hotez, R. E. Cooney, R. M. Benjamin, N. T. Brewer, A. M. Buttenheim, T. Callaghan, A. Caplan, R. M. Carpiano, C. Clinton, R. DiResta, et al. Announcing the lancet commission on vaccine refusal, acceptance, and demand in the usa. The Lancet, 2021.

[24] K. Huang, T. Fu, W. Gao, Y. Zhao, Y. Roohani, J. Leskovec, C. W. Coley, C. Xiao, J. Sun, and M. Zitnik. Therapeutics data commons: Machine learning datasets and tasks for therapeutics. Nature Communications. Also as arXiv preprint arXiv:2102.09548, 2021.

[25] H. Kamarthi, L. Kong, A. Rodriguez, C. Zhang, and B. A. Prakash. When in doubt: Explainable and principled uncertainty quantification for epidemic forecasting. Submitted, 2021.

[26] X. Kang,  S. Ranganathan,  L. Kang,  J. Gohlke,  and X. Deng.   Bayesian auxiliary variable model for birth records data with qualitative and quantitative responses. arXiv preprint arXiv:2008.06525, 2020.

[27] R. Laxminarayan, S. Jameel, and S. Sarkar. India’s battle against covid-19: Progress and challenges. The American Journal of Tropical Medicine and Hygiene, 103(4):1343, 2020.

[28] R. Laxminarayan, B. Wahl, S. R. Dudala, K. Gopal, S. Neelima, K. J. Reddy, J. Radhakrishnan, J. A. Lewnard, et al. Epidemiology and transmission dynamics of covid-19 in two indian states. Science, 370(6517):691–697, 2020.

[29] G. Lin, A. T. Strauss, M. Pinz, D. A. Martinez, K. K. Tseng, E. Schueller, O. Gatalo, Y. Yang, S. A. Levin, E. Y. Klein, et al. Explaining the “bomb-like” dynamics of covid-19 with modeling and the implications for policy. medRxiv, 2020.

[30] D. Machi, P. Bhattacharya, S. Hoops, J. Chen, H. Mortveit, S. Venkatramanan, B. Lewis, M. Wilson, A. Fadikar, T. Maiden, and C. L. Barrett. Scalable epidemiological workflows to support covid-19 planning and response. In In Proceedings of the 35th IEEE International Parallel and Distributed Processing Symposium, 2021.

[31] M. Marathe, A. Vullikanti, D. Rosenkratz, S. Ravi, R. Stearns, and S. Levin. Computational challenges and opportunities for forecasting epidemic dynamics using network models, 2020.

[32] N. Meghanathan. Exploring the step function distribution of the threshold fraction of adopted neighbors vs. minimum fraction of nodes as initial adopters to assess the cascade blocking intra-cluster density of complex real-world networks. Springer Applied Network Science (Published as part of Special issue on Epidemics Dynamics & Control on Networks), 5(97):1–33, December 2020.

[33] Z. Mehrab, A. G. Ranga, D. Sarkar, S. Venkatramanan,  Y. C. Baek,  S. Swarup,  and M. V. Marathe. High resolution proximity statistics as early warning for us universities reopening during covid-19, 2020. medRxiv preprint 2020.11.21.20236042.

[34] M. Minutoli, P. Sambaturu, M. Halappanavar, A. Tumeo, A. Kalyanaraman, and A. Vullikanti. Preempt: Scalable epidemic interventions using submodular optimization on multi-gpu systems. In Proc. SC, SC ’20. IEEE Press, 2020.

[35] S. M. Moghadas, M. C. Fitzpatrick, P. Sah, A. Pandey, A. Shoukat, B. H. Singer, and A. P. Galvani. The implications of silent transmission for the control of covid-19 outbreaks. Proceedings of the National Academy of Sciences, 117(30):17513–17515, 2020.

[36] S. M. Moghadas, T. N. Vilches, K. Zhang, C. R. Wells, A. Shoukat, B. H. Singer, L. A. Meyers, K. M. Neuzil, J. M. Langley, M. C. Fitzpatrick, et al. The impact of vaccination on covid-19 outbreaks in the united states. medRxiv, 2020.

[37] A. Moitra and E. Mossel. Hot topics in computing: An invitation to computational epidemiology, March 2021.

[38] D. Morris, F. Rossine, J. Plotkin, and S. Levin. Optimal, near-optimal, and robust epidemic control. communication physics. OSF Preprints, 2020.

[39] Theoretically speaking — computational and statistical tools to control a pandemic: A panel discussion, simons institute, May 2020. https://simons.berkeley.edu/events/covid19.

[40] A. S. Peddireddy, D. Xie, P. Patil, M. L. Wilson, D. Machi, S. Venkatramanan, B. Klahn, P. Porebski, P. Bhattacharya, S. Dumbre, and M. Marathe. From 5vs to 6cs: Operationalizing epidemic data management with covid-19 surveillance. In Proceedings of the IEEE International Conference on Big Data (BigData), 2020.

[41] B. Peng,  J. Li,  S. Akkas,  F. Wang,  T. Araki,  O. Yoshiyuki,  and J. Qiu.  Rank position forecasting in car racing. arXiv preprint arXiv:2010.01707, 2020.

[42] N. Perera, V. Abeykoon, C. Widanage, S. Kamburugamuve, T. A. Kanewala, P. Wickramasinghe, A. Uyar, H. Maithree,  D. Lenadora,  and G. Fox.  A fast,  scalable,  universal approach for distributed data reductions. arXiv preprint arXiv:2010.14596, 2020.

[43] A. Pilehvari, W. You, J. Chen, S. Venkatramanan, J. Krulick, and A. Marathe. Differential impact of social distancing on covid-19 spread in the us: by rurality and social vulnerability, 2021. Submitted.

[44] J. D. Priest, M. V. Marathe, S. S. Ravi, D. J. Rosenkrantz, and R. E. Stearns. Evolution of similar configurations in graph dynamical systems.  In R. M. Benito, C. Cherifi, H. Cherifi, E. Moro, L. M. Rocha, and M. Sales-Pardo, editors, Proc. 9th International Conference on Complex Networks and Applications (Complex Networks), pages 544–555, Chan, Switzerland, 2020. Springer.

[45] J.  Quetzalc´oatl  Toledo-Mar´ın,  G.  Fox,  J.  P.  Sluka,  and  J.  A.  Glazier.   Deep  learning  approaches  to surrogates for solving the diffusion equation for mechanistic real-world simulations. arXiv e-prints, pages arXiv–2102, 2021.

[46] E. L. Ray, N. Wattanachit, J. Niemi, A. H. Kanji, K. House, E. Y. Cramer, J. Bracher, A. Zheng, T. K. Yamana, X. Xiong, et al. Ensemble forecasts of coronavirus disease 2019 (covid-19) in the us. MedRXiv, 2020.

[47] A. Rodr´ıguez, B. Adhikari, A. D. Gonz´alez, C. Nicholson, A. Vullikanti, and B. A. Prakash.  Mapping network states using connectivity queries. Proceedings of IEEE International Conference on Big Data 2020 (IEEE BigData 2020), 2020.

[48] A. Rodr´ıguez, B. Adhikari, A. D. Gonz´alez, C. Nicholson, A. Vullikanti, and B. A. Prakash.  Mapping network states using connectivity queries. NeurIPS 2020 Artificial Intelligence and Humanitarian and Disaster Relief (AI + HADR) Workshop, 2020.

[49] A.  Rodr´ıguez,  B.  Adhikari,  N.  Ramakrishnan,  and  B.  A.  Prakash.   Incorporating  expert  guidance  in epidemic forecasting. arXiv preprint arXiv:2101.10247, 2020.

[50] A. Rodriguez, N. Muralidhar, B. Adhikari, A. Tabassum, N. Ramakrishnan, and B. A. Prakash. Steering a historical disease forecasting model under a pandemic: Case of flu and covid-19. NeurIPS 2020 Machine Learning in Public Health (MLPH) Workshop, 2020.

[51] A. Rodriguez, N. Muralidhar, B. Adhikari, A. Tabassum, N. Ramakrishnan, and B. A. Prakash. Steering a historical disease forecasting model under a pandemic: Case of flu and covid-19. Proceedings of the AAAI Conference on Artificial Intelligence, 2021.

[52] A. Rodriguez, A. Tabassum, J. Cui, J. Xie, J. Ho, P. Agarwal, B. Adhikari, and B. A. Prakash. Deep- COVID: An operational deep learning-driven framework for explainable real-time COVID-19 forecasting. In Proc. AAAI Conference on Artificial Intelligence, 2021.

[53] D. J. Rosenkrantz, M. V. Marathe, S. S. Ravi, and R. E. Stearns. Synchronous dynamical systems on directed acyclic graphs: Complexity and algorithms. In Proc. AAAI-2021 (to appear), 2021. 9 pages.

[54] C. Ruiz, M. Zitnik, and J. Leskovec. Identification of disease treatment mechanisms through the mul- tiscale interactome. bioRxiv, 2020.

[55] C. M. Saad-Roy, N. Arinaminpathy, N. S. Wingreen, S. A. Levin, J. M. Akey, and B. T. Grenfell. Implications of localized charge for human influenza a h1n1 hemagglutinin evolution: Insights from deep mutational scans. PLoS computational biology, 16(6):e1007892, 2020.

[56] C. M. Saad-Roy, B. T. Grenfell, S. A. Levin, L. Pellis, H. B. Stage, P. van den Driessche, and N. S. Wingreen. Superinfection and the evolution of an initial asymptomatic stage. Royal Society open science, 8(1):202212, 2021.

[57] C. M. Saad-Roy, S. A. Levin, C. J. E. Metcalf, and B. T. Grenfell. Trajectory of individual immunity and vaccination required for sars-cov-2 community immunity: a conceptual investigation. Journal of the Royal Society Interface, 18(175):20200683, 2021.

[58] C. M. Saad-Roy, S. E. Morris, C. J. E. Metcalf, M. J. Mina, R. E. Baker, J. Farrar, E. C. Holmes, O. Pybus, A. L. Graham, S. A. Levin, et al. Epidemiological and evolutionary considerations of sars- cov-2 vaccine dosing regimes. medRxiv, 2021. Science.

[59] C. M. Saad-Roy, C. E. Wagner, R. E. Baker, S. E. Morris, J. Farrar, A. L. Graham, S. A. Levin, M. J. Mina, C. J. E. Metcalf, and B. T. Grenfell. Immune life history, vaccination, and the dynamics of sars-cov-2 over the next 5 years. Science, 370(6518):811–818, 2020.

[60] Y. Serkez. The magic number for reducing infections and keeping businesses open, December 2020.

[61] A. Uyar, G. Gunduz, S. Kamburugamuve, P. Wickramasinghe, C. Widanage, K. Govindarajan, N. Per- era,  V. Abeykoon,  S. Akkas,  and G. Fox.   Twister2 cross-platform resource scheduler for big data, 2021.

[62] J. Vekemans, M. Hasso-Agopsowicz, G. Kang, W. P. Hausdorff, A. Fiore, E. Tayler, E. J. Klemm, R. Laxminarayan, P. Srikantiah, M. Friede, et al. Leveraging vaccines to reduce antibiotic use and prevent antimicrobial resistance: a who action framework. Clinical Infectious Diseases, 2021.

[63] C. E. Wagner, J. A. Prentice, C. M. Saad-Roy, L. Yang, B. T. Grenfell, S. A. Levin, and R. Laxmi- narayan. Economic and behavioral influencers of vaccination and antimicrobial use. Frontiers in public health, 8:975, 2020.
[64] F. Wang, C. Widanage, W. Liu, J. Li, X. Wang, H. Tang, and J. Fox. Privacy-preserving genomic computing with sgx-based big-data analytics framework. 20th IEEE International Workshop on High Performance Computational Biology (HiCOMB) [under review], 2021.

[65] L. Wang, A. Adiga, J. Chen, A. Sadilek, S. Venkatramanan, and M. Marathe. CausalGNN: Causal-based graph neural networks for spatio-temporal epidemic forecasting, 2021. Submitted.

[66] L. Wang, A. Adiga, S. Venkatramanan, J. Chen, B. Lewis, and M. Marathe. Examining deep learning models with multiple data sources for covid-19 forecasting, 2020. arXiv preprint arXiv:2010.14491.

[67] L. Wang, X. Ben, A. Adiga, A. Sadilek, A. Tendulkar, S. Venkatramanan, A. Vullikanti, G. Aggarwal, A. Talekar, J. Chen, B. Lewis, S. Swarup, A. Kapoor, M. Tambe, and M. Marathe. Using mobility data to understand and forecast covid-19 dynamics. In Proceedings of the 29th International Joint Conference on Artificial Intelligence Workshop on AI for Social Good., 2021.

[68] L. Wang, D. Ghosh, M. T. G. Diaz, A. K. Farahat, M. Alam, C. Gupta,  J. Chen,  and M. Marathe. Wisdom of the ensemble: Improving consistency of deep learning models. In NeurIPS, 2020.

[69] S. Weekes, P. March, M. Berger, M. Marathe, J. Crawley, C. M. Lois, A. El Bakry, R. Renaut, A. Gelb, F. Santosa, T. Grandine,  P.  Seshaiyer,  and  J.  Hestheven.  Report  on  future  research directions  for  the  national  science  foundation  in  the  era  of  covid-19.  A  Report  by  SIAM  Task Force on COVID-19, 2020. https://www.siam.org/Portals/0/reports/Report%20on%20Future% 20Research%20Directions%20for%20NSF.pdf?ver=2020-12-10-144744-750.

[70] Y. Wei and X. Deng. A simple gaussian process modeling approach for experiments with quantitative- sequence factors. Submitted, 2021.

[71] C. Widanage, N. Perera, V. Abeykoon, S. Kamburugamuve, T. A. Kanewala, H. Maithree, P. Wickra- masinghe, A. Uyar, G. Gunduz, and G. Fox. High performance data engineering everywhere. In 2020 IEEE International Conference on Smart Data Services (SMDS), pages 122–132. IEEE, 2020.

[72] Q. Xiao, Y. Wang, A. Mandal, and X. Deng. Modeling and active learning for experiments with quantitative-sequence factors, revision for journal of the american statistical association. Submitted, 2021.

[73] O. Yagan, A. Sridhar, R. Eletreby, S. A. Levin, J. B. Plotkin, and H. V. Poor. Modeling and analysis of the spread of covid-19 under a multiple-strain model with mutations. Harvard Data Science Review, 2021.

[74] A. Yalaman, G. Basbug, C. Elgin, and A. P. Galvani. Cross-country evidence on the association between contact tracing and covid-19 case fatality rates. Scientific reports, 11(1):1–6, 2021.