“Seeing Is Not the Same Thing as Looking”: Anupam B. Jena, M.D., Ph.D., on Learning from Natural Experiments

Analysis Group affiliate Anupam B. Jena – the Ruth L. Newhouse Associate Professor of Health Care Policy at Harvard Medical School and a physician at Massachusetts General Hospital – has spoken and published widely on the importance of creative thinking when using economic tools to study difficult questions in health care policy and practice. Most recently, he has been applying this approach to examine assumptions policy leaders and medical practitioners have about effective responses to the COVID-19 pandemic. Professor Jena had a wide-ranging discussion with Managing Principal Stephen Fink on these topics.

As we speak, a lot of media and political attention is being paid to the timeline for developing a safe and effective vaccine for COVID-19. As a medical professional, you understand how important it is to maintain the strict and well-tested guidelines for drug evaluations. But are Phase 3 trials the only solution available to us, particularly as medical practitioners look to develop better courses of treatment for COVID-19?

Of course, randomized controlled trials remain the gold standard for clinical research, but under extreme circumstances doctors, policymakers, and certainly people urgently needing treatment can’t always wait. They are looking for answers now.

And this is where I think we can take a more creative approach and start looking at our immediate challenges through the lens of natural experiments.

Could you explain what natural experiments are, and the appropriate way to apply them?

At the risk of oversimplifying, a natural experiment involves looking at the real world – at observed experience – in a different way. An example I like to use comes from my own life. My wife runs long races (I don’t!). One time, I couldn’t get to the race to see her because the police had blocked all the roads, as they normally do for road races.

Afterwards, my wife pointed out that, if I couldn’t get to the race, what about all the people who were trying to get to a hospital that was on the same route? I thought that was a very interesting question, so I looked into it and found that, indeed, there was a 15% higher mortality rate among the elderly on days of marathons, concentrated among people living near the race route, compared to days with no marathon. I also found that ambulances were delayed on the mornings of marathons, which was consistent with my conjecture about roads being closed and transportation becoming more difficult.

So in this instance, life provided us with an experimental group and a control group – people who had heart attacks on race days, and people who had heart attacks on other days. That’s a natural experiment.

How can we use these natural experiments during the pandemic?

There are a great many applications that might be helpful in formulating our response to the pandemic. For instance, instead of looking at people who need to get to the hospital but can’t, we might ask ourselves, who should be going to the hospital but isn’t? People don’t stop having heart attacks because the emergency rooms are already full of COVID-19 patients. But there may well be people who ignore or downplay their chest pain or other symptoms because they are fearful of visiting a facility that they think will expose them to COVID-19.

This phenomenon may prove especially worrisome for those most at risk of health issues aside from COVID-19, such as the elderly or children. In some of my research, for example, I’ve noted that, even though the CDC [Centers for Disease Control and Prevention] and physician groups have been vocal about the importance of still bringing young children to the doctor for their regular vaccines during the pandemic, many parents have been hesitant.

Is it simply a matter of asking different questions, or should we also be paying attention to how we go about developing answers?

As I’ve said many times before, it’s important to understand that seeing is not the same thing as looking, and being a casual observer is not the same thing as being a thoughtful observer and analyzing what you see. For instance, when the lockdowns first started, I wondered if keeping more people off the road would result in fewer auto-related deaths, perhaps even offsetting some of the deaths from COVID-19. After all, I believe that for the past few years the US has been averaging over 100 fatalities a day from motor vehicle accidents. The question seemed to make sense.

As it turned out, though, a month or so after our governor in Massachusetts ordered an economic shutdown, the number of auto accidents in Massachusetts had been cut in half, but the rate of fatal accidents had doubled. In other words, there were just as many fatalities, even when traffic was half the volume.

To me, this means that we shouldn’t draw a straight line connecting traffic volume, the number of accidents, and the number of fatalities. Rather, we should be looking at whether behavior is any different now. With fewer cars on the road, are the remaining drivers driving faster, perhaps, or with less caution?

 

Can this approach also be applied to medical or clinical practice in treating COVID-19?

Certainly; it can be especially useful for directing attention to the right places. In fact, a couple of years ago I looked into how the social science phenomenon known as the “Hawthorne effect” affected treatment outcomes in hospitals. The Hawthorne effect, which we have known about for nearly a century now, shows that people actively change their behavior when they know they are being observed.

This effect has clear applications for groups such as research subjects, test-takers, and even voters, but my colleagues and I wondered whether it might also affect patient outcomes. So we obtained data for more than 1.7 million Medicare beneficiaries who had been hospitalized either a week before an unannounced hospital inspection, during the week of an inspection itself, or a week after an inspection had ended.

We found a statistically significant decrease in mortality during inspection weeks, which couldn’t be explained by other factors we analyzed, such as differences in patient populations, or reductions in surgeries and elective procedures (something we thought might occur if hospitals clamped down operations to focus their attention on the inspections), for the three study periods. This suggested to us that hospital staff behavior – that is, increased focus, attention, and clinical vigilance – might be having a significant impact on patient care.

Ultimately, we can use similar approaches with different natural experiments to help determine what, if any, external factors are contributing to better COVID-19 treatments and patient outcomes. As health care providers gain more experience treating patients with COVID-19, and as more data become available, there will be ample opportunity to study the various treatment practices and look to determine what factors impact patient outcomes in significant ways.

How are researchers and policymakers already making use of this approach?

Natural experiments are happening on their own all the time, and many have already happened – it’s just a matter of looking for them. For example, a large team of researchers searched through millions of electronic health records from hospitals and clinics in the Los Angeles area for a period in late 2019 and early 2020, which was before the first official recognition of the arrival of the novel coronavirus in the US. They looked at how often the word “cough” appeared as the reason for a visit, and compared that with year-ago levels for the same facilities.

The researchers identified a statistically significant increase in the number of visits related to the presence of a cough. Of course, we now know that a cough can be one of the symptoms of COVID-19, and so it seems possible that the coronavirus may already have been present in the US for at least a few months before it was recognized.

Those findings potentially could be used to improve detection, tracking, and treatment programs. Similarly, as I’ve written about elsewhere, we can look at how practice patterns at different hospitals have changed since the onset of the pandemic. Essentially, the evolution of treatment patterns allows researchers to use each hospital, at different periods in time, as its own control to study the effects of various treatments on patient outcomes.

In many of your talks on this topic, you make a point of saying that, by asking the right questions – by asking creative questions – we can use natural experiments to gain more informed insights into behavior and practices. What kinds of questions should we be asking during the pandemic?

When we try to come up with ideas on how to protect ourselves in this pandemic, and how we can be better prepared for future ones, it can be extremely useful to take off our blinders and start being more creative with the questions we ask. Then we need to look behind the questions at the data the real world can give us.

As a case in point, in July I published a paper with my colleague from Harvard Medical School, Dr. Christopher Worsham, in which we asked whether the month in which a child was born might be related to vaccination rates. You could well ask, why would birth month make a difference?

In fact, we know that accessibility is an important factor in vaccinations, and timing is critical for annual flu vaccinations. We found that children between the ages of two and five who were born in summer months were significantly less likely to receive flu shots, which are administered in the fall.

Given that children born in the earlier months would have their annual physicals scheduled each year before flu vaccinations became available, it seems likely that their parents would have to make separate appointments for these children to receive the shots. In other words, the extra effort that is required may well constitute a barrier to accessibility.

Some have said that vaccines don’t help people, vaccinations do. So these kinds of findings could conceivably help inform planning for the widespread distribution of a COVID-19 vaccine, whenever a safe and effective one has been developed.

Developing a rigorous, well-thought-out, and proactive plan for distributing and administering the vaccine to as many people as possible will be paramount for health officials and government leaders, and that must include thinking about how to overcome any and all potential barriers. If we can identify those who are less likely to seek out or accept a vaccine, or those who can’t get to health care easily, then we can develop more targeted information campaigns and more effective vaccination programs.

Boston Medical Center provides an excellent example of how thinking more creatively can lead to better care. They recognized that the pandemic had made it more difficult for children to go out and get routine vaccinations, as I mentioned earlier. At the same time, the pandemic resulted in fewer 911 calls for medical care. The management and pediatricians at Boston Medical put two and two together, and designed a program for pediatricians to partner with idle ambulance crews and bring the vaccines directly to the children in their homes. By the same token, once a COVID-19 vaccine is available, perhaps nurses could join postal workers on their routes, for example, to bring the vaccine directly to homes. ■