On the Puerto Rico Hurricane Study

A lot of Trump opponents are claiming that a recent study by the Milken Institute School of Public Health shows that after hurricane María hit Puerto Rico, the Trump administration’s relief effort was so shoddy that 3000 people died.

I’ll admit that letting 3000 people die sounds like something Trump would do, but that’s not actually what the study shows. It’s not an evaluation of the relief effort, and it doesn’t attempt to assign blame for any deaths. The study concludes that 3000 people died as a result of the hurricane, but it doesn’t have much to say about the specific causes of their deaths. In fact, the Milken press release announcing the study specifically says:

Additional research must be done to understand how the hurricane was involved in the excess deaths identified in this study. This would involve interviews of family members and others, as well as in-depth statistical analyses, to learn about the circumstances leading up to individual deaths. 

It’s important to realize that this study does not address the effectiveness of the relief effort. This study is about the inclusive death toll from the hurricane. That would include any deaths caused by an inadequate relief effort, but the study does not and cannot separate out those deaths from any others. So while the Trump administration’s relief effort might have contributed to the excess deaths described in this study, the study itself is silent on the matter.

That doesn’t mean Trump supporters are right about the study either. Let’s start with Trump himself:

“3000 people did not die in the two hurricanes that hit Puerto Rico. When I left the Island, AFTER the storm had hit, they had anywhere from 6 to 18 deaths. As time went by it did not go up by much. Then, a long time later, they started to report really large numbers, like 3000…” –@realDonaldTrump

These deaths were not reported immediately because they did not happen immediately. And nobody reported the numbers until a long time later because it took a long time to gather and analyze the data. Public health studies take time.

…..This was done by the Democrats in order to make me look as bad as possible when I was successfully raising Billions of Dollars to help rebuild Puerto Rico. If a person died for any reason, like old age, just add them onto the list. Bad politics. I love Puerto Rico! — @realDonaldTrump

The Democrats are certainly trying to make him look bad, but that doesn’t make the study wrong, and the study was not exactly done by the Democrats. It’s not hard to understand why Trump thinks this way, however, when you look at what other people are saying. Here’s Lou Dobbs’s nearly logic-free take:

Picking a few choice bits,

The President, by the way, is right. The study the president alluded to is one produced last month by the Milken Institute at George Washington University. Almost a year later, the number went from 65 people killed to 2,975 people, attributed to the storm.

Dobbs is clever with words. Saying “almost a year later, the number went from” implies that there’s some kind of trickery about the delay, rather than the boring truth that it takes time to do public health studies.

The finding wasn’t the result of a death toll count, a body count, nor a study of death certificates, but a public health study that subtracted the number of people who theoretically should have died over the same period from the number of people who were reported dead over that period.

This is why I called Dobbs take almost logic-free. He’s accurately described the study, but then he does nothing with it. He simply pronounces it wrong without ever actually making an argument.

After the report came out on August 28, last month, Puerto Rico’s Democratic governor officially revised the death toll from 65 to the new estimate. And why did he choose to Trust that study? Why not the Harvard study back in June, that found deaths related to the hurricane fell within a, well, a narrow range…are you ready? A narrow range from about 800 to 8000 people. By the way, that’s also an abstract and unconnected-to-any-evidence estimate, that even the liberal Washington Post found ludicrous.

The declaration that these two studies are “unconnected to any evidence” is simply false. The Harvard study was based on a geographically distributed random sampling of over 3000 households which gathered data about 9522 people. Based on the number of deaths found in that sample, they extrapolated to obtain rates for the whole island, and from that estimated excess deaths. This methodology is commonly used to estimate deaths when death records are unavailable, such as poor countries and war zones, and it is understood to give only a very rough answer, thus the extremely wide confidence interval.

The more comprehensive Milken study was based on thousands of official death records from the Puerto Rico Vital Statistics Registry (PRVSR), and its confidence range is a much narrower 2658 to 3290, which is probably one of the reasons the Puerto Rican Government prefers it. It’s just a better study. (Also, the government commissioned it.)

The numbers were inflated, and the President was right to call out the organizations who threw out science, statistics, and evidence to discredit the Trump administration.

Lou Dobbs is an ignorant ass. There’s a lot more legitimate science and statistics in these studies than in anything he said. And as for his assertion that the study is an attempt to discredit the Trump administration, the only actual evidence he offers is the original report of 65 dead immediately after the hurricane. And he gets even that wrong: Until the Milken study was released, the official death toll stood at 64.

John Hinderaker at Powerline has his own take, which is equally silly:

This is what is going on: Some “scientists”–read anti-Trump Democratic Party activists–constructed a theoretical baseline of how many deaths would be expected to occur in Puerto Rico during the months after Hurricane Maria. They then compared this baseline to the actual number of deaths, and voila! The actual number was higher than their hypothetical guess by 3,000.

Hinderaker’s use of scare quotes around “scientists” and his accusations that they are “activists” is complete bullshit that he just made up. There isn’t even any need to attack the study to defend Trump, because the study draws no conclusions regarding the Trump administration or the relief effort.

Hinderaker’s description is, however, a reasonably accurate description of the methodology. The only thing I’d add is that the theoretical baseline is produced by analyzing death rates in Puerto Rico in past years (the Milken study goes back to 2010) to produce a statistical model of deaths, and then using that model to project past trends onto the period of the study to estimate the expected deaths.

So all of those deaths–whether caused by cancer, car accidents, or whatever–are attributed to the hurricane. The study doesn’t even attempt to figure out which 3000 excess deaths are caused by the hurricane. These activists have not made any attempt to count the actual number of hurricane-related deaths.

When discussing “all of those deaths,” it’s important to remember that the death toll of 2,975 is an excess figure. From September 2017 through February 2018 (the period of the study) the PRVSR recorded 16,608 deaths. But the statistical model used in the Milken study estimate the expected number of deaths during that period at 13,633. The difference between the actual and projected death tools is where the figure 2,975 comes from. In other words, according to the study, only about 18% of the deaths during that period were caused by hurricane María. The rest were the result of normal causes, “cancer, car accidents, or whatever,” as Hinderaker says.

But that doesn’t change the fact that there was still a higher than expected number of deaths, which raises the question: If the excess deaths were not caused by the hurricane, if they were due to “cancer, car accidents, or whatever,” then why did those causes of deaths increase? If it wasn’t the hurricane, what was it? Hinderaker offers no alternative.

No one would use such a foolish methodology except for political reasons.

This is actually a standard methodology for estimating deaths in public health science. It’s necessary in situations where it is difficult to identify and observe the causal channels.

Estimating the number of deaths from a disease like Ebola is fairly easy: You find people who tested positive for Ebola and who died from Ebola-like symptoms. Count them, and you’ve got your answer.

Estimating deaths from a disease like HIV is a lot more complicated, because people don’t die from HIV. They die from things like tuberculosis, hepatitis C, Kaposi’s sarcoma, non-Hodgkin’s lymphoma, and a variety of other diseases that kill people whose immune systems have been weakened by HIV. It can be hard to untangle the causes in these cases, and scientists have created the Coding Causes of Death in HIV (CoDe) protocol to help decide, in a consistent way, which deaths to attribute to HIV.

Classifying the causes of death can be complicated. Say a person with HIV is taking drugs which can produce vomiting as a side effect, and one day that person starts vomiting while driving, loses control of their car, and dies in a crash. Does that count as a death from HIV?

I have no idea what CoDe says about that scenario, but in general the answer to questions like that depend on why you want to know. If you are an actuary for an insurance company providing group life insurance to a company that employs people with HIV, it certainly counts, since you will have to pay that claim. And if you’re an economist studying the cost of HIV to society, not only would you count that death, but you would also count the collateral deaths of any passengers or pedestrians killed in the crash as well.

With a hurricane, the causal channels are even harder to analyze. The immediate deaths from storm surge, windblown debris, and collapsed structures are relatively easy to identify, but the hurricane also disrupts the infrastructure of civilization, and that can lead to more deaths. People with diabetes could face hazardous disruptions to their food, their insulin supply, and their healthcare. People on supplemental oxygen could lose power for the oxygen concentrators, and delivery of tanked oxygen could be disrupted. Patients on blood thinners could miss blood tests and fail to adjust their dosage correctly. Kidney dialysis centers could be out of power or supplies, forcing patients to travel further for care.

In fact, with random destruction taking out buildings, people will likely have to travel further for everything, and the extra driving alone will kill some people, even before the added risks from driving on debris-strewn roads with no traffic signals or street lights. People will die because they can’t call an ambulance with the phones down, or because the ambulances are all busy, or because the ambulances have to take longer routes to hospitals, or because hospitals are unable to operate at full capacity, or because hospitals can’t get supplies of needed drugs. People will kill themselves trying to salvage property from collapsed houses. People will fall off their roofs while repairing storm damage.

Hurricanes are a mess of causes that are difficult to sort out, which is why this study used a method that doesn’t depend on knowing the cause of every single death. A conceptually similar methodology was used to establish that cigarettes caused lung cancer. When the global lung cancer epidemic hit humanity at the end of the 19th century, nobody knew the cause. The cellular mechanisms of cancer would not be understood for many decades (and are still the subject of research today), so scientists had no way to determine the causal chain that led to lung cancer in specific patients. Nevertheless, by the 1930s scientists had gathered statistical evidence showing that the lung cancer rate was far higher in people who smoked cigarettes than in people who didn’t smoke. By the 1960s, the link was established well enough (even over the obstructive efforts of tobacco companies) to discourage people from smoking and to affect public policy.

The Milken study is similar in concept, except for the choice of control population. Cigarette studies used non-smokers as controls, because they could be drawn from otherwise similar populations. There’s no similar control population for Puerto Rico — no otherwise identical island population that wasn’t hit by a hurricane — so as a control the researchers used the same population that was hit by the hurricane, but from the years immediately before the hurricane.

By this point, someone will be screaming that “correlation does not prove causation!” That’s true, but it can certainly imply causation in a properly done study, especially when accompanied by a good explanatory theory. In this case, the theory is that hurricanes destroy infrastructure, thus increasing the danger to human life. That’s not a particularly controversial theory, and when you observe a hurricane followed by an increase in deaths, that tends to confirm the theory. Critics are welcome to offer better theories to explain the data.

(I should mention that the Powerline post also argues that Puerto Rico’s death rate declined in 2017. I’m not sure where those figures come from, but the IndexMundi source they site appears to get its data from the CIA World Fact Book, which describes the Puerto Rico Death Rate entry as a “2017 estimate.” Furthermore, since the death rate is calculated as the number of deaths divided by the number of people, it is sensitive to changes in the population size, and I suspect this estimate is based on U.S. census estimates of the Puerto Rican population. However, after María struck, about 8% of the Puerto Rican population, 300,000 people, decided to leave the island. That mass emigration did not make it into census estimates, but the Milken study used travel records to adjust its figures to account for the decline in population.)

My regular nemesis, Jack Marshall, also attacks the study:

I’ve covered the revised hurricane death tolls before. Nobody knows what the real figure is, and it is fair to question the newer estimates, which were produced by public health experts at George Washington University in Washington. Their report was commissioned by the U.S. territory’s governor, Ricardo Rossello, and he was looking for big numbers: the more deaths, the more U.S. aid.

That’s no reason to smear the scientists with accusations of falsifying data.

The news media misrepresented the study as well. Here’s Reuters:

[..] The study found that those deaths could be attributed directly or indirectly to Maria from the time it struck in September 2017 to mid-February of this year.

False! the study didn’t “find” that at all. It assumed it; it theorized it; it argued it. There is a material difference between finding something and assuming it’s there.

The results of a study are commonly referred to as its “findings.” Jack is either (1) an ass for pretending he doesn’t know that or (2) an idiot for not knowing that.

Jack also references a blog post by Peter Grant at Bayou Renaissance Man:

I find this study highly suspect. One can find similar increased death tolls in other areas, but with autopsies, witness statements, etc. that make it possible to analyze them properly. Example: the opioid epidemic that’s ravaging several US states at present. Death rates due to the misuse of opioids are climbing dramatically, but in every case, the cause of death can be measured, medically confirmed, and verified. How do we know that opioids weren’t responsible for at least some of the “excess” deaths in Puerto Rico?

I’m sure opioids were the cause of some of the 16,608 deaths in Puerto Rico in the five months after the hurricane, and it’s possible there was even an increase in opioid-related deaths after the hurricane, and if someone can dig up those figures, the Milken institute should probably revise its data (assuming that the increase in opioid-related deaths is not itself due to the hurricane).

What about deaths caused by vehicles? How do you know whether an accident was due to increased traffic, caused by aid distribution after the storm, or a drunk driver? The first might be blamed on the hurricane; the second, certainly not.

Again, the 2,975 deaths are an excess figure, so offering alternative causes of death is insufficient. Any alternative theory for the deaths needs to explain why the death rate for that cause increased when it did. If some of the excess deaths are due to drunk driving (or opioids for that matter), what happened to make the drunk driving rate increase in the months after the hurricane?

Without medical and other evidence, one can’t assign a definitive cause to each casualty; but the study conducted there did not examine such evidence. It only looked at numbers, and made assumptions.

Obviously, the Milken study is not definitive. We could get a much better idea of the death toll from hurricane María if someone did a careful study of each of the 16,608 deaths during this period to determine the cause. Unfortunately, the data just isn’t available. The CDC has guidelines for a special death certification process to be used after a natural disaster which would have gathered some of the needed data, however the Milken study reports that those guidelines weren’t followed in Puerto Rico, largely due to lack of training of medical personnel and poor communication by authorities. In addition, the PRVSR offices were damaged by the hurricane, and death certificate filings were substantially disrupted.

To gather the necessary data now would require defining a protocol and training a team to review medical records and interview friends and family members of the deceased to determine the complete causal chain leading to their death. That’s the (very expensive) study you’d need to do to lay this question to rest.

But no one has done that study, or any study better than this one. So until someone does, the Milken study remains the best and most accurate attempt ever made to estimate the death toll from hurricane María.