7 Reasons Not to Use ODEs for Epidemic Modeling
Epidemic models based on ordinary differential equations (ODEs) are neither right nor useful for epidemic modeling. This does not discourage the scientific community to publish one ODE paper after another: Fitting poor models to low-quality data ultimately generating misleading predictions. Most of these studies are missing basic contextualization and a basic sense of their limitations.
Since the emergence of COVID-19, computational epidemiological modeling has experienced a rapid surge in popularity. ODE-based epidemiological models are the most popular tool, especially among non-epidemiologists. However, ODE-models are not only the simplest epidemic model but also the worst for both analysis and prediction.
What are ODE compartment models?
In the simplest case, the SIR-model, the population is split into three groups (compartments): susceptible (healthy), infected (and infectious), and recovered (immune or deceased) individuals. For a tutorial we refer the reader to this story.
The fraction of individuals in the corresponding compartments are described by a system of three ODEs. Thus, the fractions evolve deterministically and continuously over time. Adding more compartments, for instance, exposed (infected but not yet infectious) individuals is straight-forward.
Why should they not be used in epidemic modeling?
Now, I will summarize the most relevant reasons why this technique, which is based on work from almost 100 years ago [1], is almost useless for modern epidemiology:
- ODE-models are prediction-centered: Epidemiology has come to the conclusion that predicting the spread of an epidemic is “virtually impossible” [2], “had failed” [3], and provides “a case study in model failure” [4]. More importantly, COVID-19 forecasting is also unscientific as “[s]cience is not about making single point predictions but about understanding properties” [5]. In other words, building a black-box that predicts an epidemic is useless if we do not understand which properties of a population are actually relevant and can be controlled. In addition, it is often not communicated for which environment the prediction is supposed to hold. Thus, incorrect predictions can always be attributed to a changing environment, avoiding any falsifiability. And even when a prediction is seemingly correct, this does not mean that any of the underlying assumptions were actually reasonable. This is a problem with many epidemic models, it just seems that people using ODEs have the biggest difficulties understanding this. Interestingly, most researchers who publish ODE papers are (like me) not epidemiologists and seem to misunderstand crucial aspects of ODE-models.
- The saturation in ODE-models is misleading: The only “advantage” an ODE-model has beyond simply fitting a curve is the build-in saturation by recovered individuals. That is, the infection curve naturally flattens. The saturation is the only thing that gives ODE-models any sort of predictive power. This gives the impression that ODE-models provide a mechanistic model of the underlying disease spread. However, on a small scale, this saturation is meaningless because it defies the local nature of an epidemic. On the large scale, it is questionable if we are anywhere in the world at a point where this saturation has a noticeable effect on the dynamics. That is, where herd-immunization has reached a point where it is worth considering compared to any other environmental factor. Sweden, for instance, hoped for a long time that, given the strength of the first wave, the second wave would be significantly mitigated. However, it has shown that this is not the case. If herd immunity, however, will have a significant influence on the pandemic, it will be extremely underestimated by ODE-models as they assume each infected individual has the level of saturation as the whole population while the dynamics is naturally locally correlated [6]. In short, if population-level immunization does not shape the epidemic (beyond any statistical fluctuation), the only aspect that makes an ODE-model somewhat mechanistic becomes completely pointless.
- The infection rate parameter is meaningless: At their core, ODE-models have only one relevant parameter that is the infection rate λ (or λ/β if β is not 1). This parameter encodes both the connectivity of a population and the infectiousness. This renders interpretation attempts of ODE-model hopeless. It also means that you can generate any infection curve you desire by making λ time-dependent. However, modulating λ is no different from directly controlling the effective reproduction number or the daily cases. This shows that having a well-fit ODE model has no inherent meaning and renders the whole approach of using a “mechanistic” model useless. If you want to investigate NPIs like travel restrictions or school closings, there is no principled way of encoding this into the infection rate parameter.
- ODE-models assume a completely homogeneous population: If we learned anything in the COVID-19 pandemic, it is that the heterogeneity in a population is the dominant factor characterizing its propagation and a key to controlling it. The overdispersion (“some COVID-19 patients infect many others, whereas most don’t spread the virus at all” [9]), super-spreaders and super-spreading events, geographical variation, different mobility patterns and susceptibility in sub-populations, etc. shape the pandemic… None of these characteristics are naturally captured by ODE-models. Consequently, they also do not help with questions regarding optimal vaccine distribution. The homogeneity assumption becomes particularly evident when you compare an ODE-generated trajectory with populations in which the interactions are constrained by a graph [6].
- ODE-models are useless for risk-assessment: The deterministic nature of ODE-models makes them highly unsuitable for any kind of risk-assessment or any other type of tasks that relies on handling uncertainties. Of course, you can sample your transition rates from some distribution. However, the resulting uncertainty about the outcome is by no means a mechanistic conceptualization about the uncertainty that arises from the randomness of human nature and limited observations. Relevant aspects such as local die-outs (aka “green zones” [7]) and tail risks [8] cannot be expressed and the model sensitivity remains unprincipled.
- ODE-models are prone to noise: If you fit your ODE-model to real-world data, your results will vary very strongly on small changes in the input data. In any modeling approach, this should be considered a red flag. Estimating the infection rate is notoriously difficult especially in the early phase of an epidemic and when the data is biased [13]. However, small changes have a large effect later on when the infected population is larger. Even under perfect conditions and given high-quality data “unavoidable uncertainties in those parameters, which determine the time at which growth is halted or the overall duration of the pandemic, propagate to the predicted trajectories, preventing reliable prediction of the intermediate and late stages of epidemic spread.” [12]
- ODE-models are unfixable: Researchers tried to fix some of the aforementioned aspects by adding more and more compartments to ODE-models (creating “meta-population” models). However, more intricate compartmentalization does not address the fundamental problems. The twists and turns that are being made to account for things like individual differences are largely unprincipled. It is revealing that often model complexity creepd in due to such ad hoc modifications. Very rarely is examined whether the model actually becomes more expressible. This is somewhat conflicting with the most basic principles of science (Occam’s razor). Specifically, researchers have recognized that “more complex [ODE] models may not be more reliable compared to using a simpler model.” [11]
Some final words about predictions: When you do short-term predictions (you probably shouldn’t), please try to get at least the basics right and use methods that have shown to be useful for past epidemics (for example [14]). Sadly, most models used in recent literature or in online tools provide no conceptualization why (i.e., based on which reasoning) their results should be any more reliable than simply assuming a constant effective reproduction number for the next few weeks. In the end, it is the reasoning and the insights behind a scientific model that makes it valuable, not some black-box prediction results.
Of course, some of these problems persist apart from ODE-models (unsurprisingly, machine learning models are also problematic in this regard). However, having read dozens of ODE papers in the last months, we can confidently state that ODE advocates often fail to communicate the most basic underlying modeling assumptions. They often provide the least amount of self-critical discussion about their limitations, and sometimes show alarming irresponsibility in influencing public discourse with their epidemiological “insights”.
Note that we deliberately decided not to directly reference ODE-papers. A myriad of those can be found on MedRxiv by arbitrary combinations of the words SIR, ODE, compartment, and COVID-19 or in the related work sections of our sources (e.g., [6]).
Sources
[1] Kermack, W. O.; McKendrick, A. G. (1927): A Contribution to the Mathematical Theory of Epidemics
[2] Ellen Kuhl: Data-driven modeling of COVID-19 — Lessons learned: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7427559/pdf/main.pdf
[3] Forecasting for COVID-19 has failed: https://forecasters.org/blog/2020/06/14/forecasting-for-covid-19-has-failed/
[4] Chin et al.: A Case Study in Model Failure? COVID-19 Daily Deaths and ICU Bed Utilisation Predictions in New York State: https://arxiv.org/pdf/2006.15997.pdf
[5] Taleb (aka https://nntaleb.medium.com) et al.: On single point forecasts for fat-tailed variables: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7572356/
[6] (Disclaimer: I participated in this work) Großmann et al.: Importance of Interaction Structure and Stochasticity for Epidemic Spreading: A COVID-19 Case Study: https://link.springer.com/chapter/10.1007/978-3-030-59854-9_16
[7] Siegenfeld et al.: Opinion: What models can and cannot tell us about COVID-19: https://www.pnas.org/content/117/28/16092
[8] Cirillo et al.: Tail risk of contagious diseases: https://www.nature.com/articles/s41567-020-0921-x
[9] Kupferschmidt: Why do some COVID-19 patients infect many others, whereas most don’t spread the virus at all? https://www.sciencemag.org/news/2020/05/why-do-some-covid-19-patients-infect-many-others-whereas-most-don-t-spread-virus-all
[10] Bertozzi et al.: The challenges of modeling and forecasting the spread of COVID-19: https://www.pnas.org/content/117/29/16732
[11] Roda et al.: Why is it difficult to accurately predict the COVID-19 epidemic: https://pubmed.ncbi.nlm.nih.gov/32289100/
[12] Castro et al.: The turning point and end of an expanding epidemic cannot be precisely forecast: https://www.pnas.org/content/117/42/26190
[13] Comunian et al.: Inversion of a SIR-based model: A critical analysis about the application to COVID-19 epidemic: https://www.sciencedirect.com/science/article/pii/S0167278920303912
[14] Stojanović et al.: A Bayesian Monte Carlo approach for predicting the spread of infectious diseases: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0225838
