Criticisms of the IHME COVID-19 model

The IHME model fits to deaths and not cases

The modelers made a concious decision to use the number of daily deaths to estimate the model parameters rather than the number of new daily infections. The rationale for this choice is that there are substantial variations in testing rates in different regions of the country and different countries of the world. This combined with the large number of people who are asymptomatic leads to huge uncertainties in what the number of new infections is really measuring.

The number of deaths in a region, however, is not perfect itself. Many deaths could be uncounted if they occur at home or among infected but undiagnosed patients. The flip side of the coin is that many people who were old and frail may have died with or without the infection.

There is strong empirical evidence from comparing death rates before and after the COVID-19 pandemic that we are greatly underestimating the number of deaths that we attribute to COVID-19.

Whether the underestimate of COVID-19 cases is worse than the underestimate of COVID-19 deaths is an open question. A key point to remember is that a consistent underestimate (everybody misses 25% of the cases or deaths) is not a serious problem for modeling, but a variable underestimate (some regions miss 10% and other miss 50%) is a big problem.

The alternative SEIR models generally use information from the daily infection rate to estimate parameters and build predictions, though they can incorporate information from the daily death rates as well. I have not seen any SEIR models that do this, but there are many SEIR models out there and I’ve only looked at a few.

The IHME model is atheoretical

The most widely criticized feature of the IHME model is that it is atheoretical. That means that the choices that the modelers based were driven solely by what seemed to fit the data best rather than what might be expected based on how infectious diseases are transmitted.

In contrast, most models of infectious disease epidemics use theoretical models. These models typical involve compartments that include groups of patients. The biggest compartment is the susceptible compartment which represents the vast population of people in an area that might get the disease. A second compartment represents infected individuals. This is just a small number at the start of the epidemic, but they interact with the susceptible population and pull some of them into the infected compartment. The people in the infected department don’t stay infected forever. They move over time into a third compartment that represents people who have recovered and are no longer infectious. You can introduce a fourth compartment, such as exposed but not infected, or a fifth, but a theoretical model will. at a minimum, consider how easily someone gets infected and how long that person stay infectious before recovering.

The rates at which people move from susceptible to infected and from infected to recovered are defined as a system of differential equations. The structure of these differential equations is critical in controlling how quickly the epidemic spreads, and how quickly it subsides.

These theoretical models are often referred to by the names of the compartments, so an SEIR model is one that has compartments for susceptible, exposed, infected, and recovered people.

There are simulation models that are very close in spirit to the SEIR models, but which require a much greater degree of computational power. These simulations create random interactions among thousands or millions of individuals where each interaction has a specified probability of spreading the disease during an interaction between an infected individual and a susceptible individual. Other than the far greater amount of computing power needed, these models have the basic features as the SEIR models.

The conflict between atheoretical models and theoretical models has a long, long history. The atheoretical propents believe in “letting the data speak for itself” while the theoretical proponets believe that models need a sound scientific foundation.

You can see some of this conflict play out in the interview that Dr. Murray has on the FiveThirtyEight website (see below). Dr. Murray argues that the theoretical models have too many assumptions and that modelers should let the actual data determine more of the model. The critics of the IHME model contend that the model changes as more data arrives, leading to unstable estimates. It’s important to note here that the model itself is changing and not just the estimates produced by the model.

You will never resolve the controversy over theoretical versus atheoretical models, but in infectious disease modeling, the theoretical models have several important advantages.

First, the SEIR models are battle-hardened. They have been used successfully many times in the past. Every year a new flu season offers a new test of these models and every year they seem to do reasonably well.

Second, the theoretical models are more transparent. You can explain a theoretical model in a paragraph or two and you can easily visualize how one theoretical model differs from another. The IHME model requires a lengthy appendix and is difficult reading even for a specialist like myself.

Third, the theoretical models allow you to more readily make extrapolations beyond the range of the data. You should always be cautious about extrapolations, but we do this all the time. Car manufactures will offer six year warranties on a car model that is less than six months old. The IHME model cannot extrapolate a decline in the daily death rate for a region that has not yet reached its peak as easily as an SEIR model can. We’ll talk more about this below.

Fourth, the theoretical models allow you to simulate scenarios that have not yet occured, such as what happens when we introduce a vaccine or whether a lack of permanent immunity after recovery will lead to a new wave of infections.

The IHME model is based on the standard bell shaped curve

To model the cumulative number of deaths, the IHME model chose something called the error function, known fondly by its nickname, erf. You may not know erf, but you do know its twin, the normal distribution, given the nikcname of bell. Fitting erf to the cumulative number of deaths is mathematically equivalent to fitting the classic bell shaped curve of the normal distribution to the daily number of deaths.

There is no good theoretical reason to expect death rates in an epidemic to follow a bell shaped curve, but the curve does have a fair amount of flexibility and can fit epidemics that start slowly and epidemics that come on quickly. They can fit death rates that peak at a modest number of daily deaths and death rates that reach a very large number of daily deaths.

The one thing that the bell curve cannot handle, however, is an epidemic that rises quickly to a peak, but then fades slowly. The symmetry of the bell shaped curve is a serious limitation.

The modelers fixed this in a revision. The new model identifies a peak and fits a single bell shaped curve to that peak. But then it allows for a mixture of 12 additional bell shaped curves on either side of the peak that can be weighted to produce an asymmetric curve that rises quickly and falls slowly (or vice versa).

This fixes one problem but it causes another. The large number of extra bell curves greatly increases the chances for overfitting. It looks like the modelers have taken steps to control for overfitting, but the whole concept screams for a simpler solution, such as a cubic spline fit.

The IHME model relies heavily on data from China, South Korea, and Italy

If you are in the early phase of an epidemic, everything looks like an exponential curve. Every day you see more deaths than the previous day. This is what that curve looks like.

But then the curve reaches an inflection point. The daily deaths are still increasing, but you can see a deceleration.

Once you are past the inflection point, you can start to forecast further into the future. You can get an initial estimate of when the number of deaths reachs its maximum and when it will die down to zero again. But these estimates are tricky and the confidence limits are very wide.

What you need is to predict when the epidemic might end is some data on the other side of the mountain, data where the number of daily deaths starts its decline.

Fair enough, but the regions that are still early in the pandemic still want predictions on when they will reach their peak and when they will eventually decline to zero. The IHME model does this through a mixed model approach.

The mixed model is a well-established approach in Statistics. It is used a lot in Epidemiology, especially in small area estimation (getting estimated rates in regions that don’t have a lot of data). The mixed model borrows information from other regions that are (hopefully) similar and provides much more precision than you could get by relying on the one region alone.

For the IHME model, the borrowing of information occurs mostly on the far-side (the declining side) of the daily death rate, This is especially true for those regions that still only have data on the near-side (the rising side) of the daily death rate. These regions that have data on the far-side are mostly from China, South Korea, and Italy.

Does it make sense to base the decline in the daily death rate in Chattahoochee county, Georgia on the decline in the daily death rate in Bergamo, Italy? Well, no not really, but what other choice do you have?

The theoretical models in Epidemiology provide an alternate way to extrapolate the far-side of these curves. The shape of the curves are estimated by the rates at which people move from one compartment to another in the SEIR model and variations on that model. You can get partial information on the various rates from the near-side data and then plug in information from earlier epidemics to help with predictions on the far-side.

In either the IHME model or the SEIR models require you to mix apples and oranges. In the IHME model, the oranges are places like Bergamo, Italy. In the SEIR mode, the oranges are epidemics like H1N1 influenza. The SEIR oranges, though, have one advantage of the IMHE oranges. There are many more SEIR oranges to choose from. The IMHE oranges are, at least for now, limited to China, South Korea, and Italy.

The IHME model has changed over time

All models provide different predictions as you accumulate more data. This leads to an inconsistent message over time and is largely unavoidable in any model. The IHME model, however, has undergone changes in the underlying mechanics that have increased its instability over time. Some of these changes were made to address criticisms in the model and some were made to get better fits to the data. One of the changes was noted above in the replacement of a single bell shaped curve with a weighted average of multiple bell shaped curves.

The desire for consistency can be overrated. Stephen Senn talks about a medical biostatistician who doesn’t believe in America because it wasn’t in Christopher Columbus’s original research plan. Still, the changes are a legitimate concern among some of the critics of the IHME model.

Imperial College of London

US Predictions

Iowa State

Wang L, Wang G, Gao L, Li X, Yu S, Kim M, Wang Y, Gu Z. Spatiotemporal Dynamics, Nowcasting and Forecasting of COVID-19 in the United States. Preprint published in Arxiv, 2020-04-30. Available in pdf format.

• This article fits an SEIR model to individual counties in the United States with a large number of county specific covariates. The model produces a nice dashboard.