The "Missing Data" Challenge: Using Clinical Trials Results

A critical issue for those performing statistical analysis in support of litigation is how to interpret findings meaningfully when they are based on incomplete data. The "missing data" challenge is particularly pronounced in health care-related disputes. Stakeholders in the US health care system are paying closer attention than ever to the safety and efficacy of treatments and monitoring how pharmaceutical companies report the results of clinical trials. A consequence of this scrutiny is the large number of mass tort and product liability cases in which pharmaceutical companies’ analyses, interpretations, and reporting of results have played a central role.

In these types of disputes, attorneys and analysts often rely on data collected from clinical trials. But as Daniel Scharfstein, a professor of biostatistics at Johns Hopkins University’s Bloomberg School of Public Health, explains, these data are commonly incomplete. For a variety of reasons, patients may drop out and be observed for only part of an entire trial period. Depending on the risk profiles of the people who drop out compared with those who stay on, the standard tools for analyzing clinical trials data may not be valid. Indeed, the issues raised by “informative dropout” can hinder efforts to determine liability, assess damages, or, in the case of mass torts, to determine common impact. Litigators and analysts are left with significant information gaps and do not know whether these gaps matter in their assessments of the case issues at hand.

Dr. Scharfstein recently served on a US Food and Drug Administration panel on the handling of missing data in clinical trials and has collaborated with Analysis Group on cases in which this dropout effect was present. We asked him to discuss ways to draw scientifically valid conclusions despite missing data.

Q: How can patient dropout compromise study results?

Dr. Scharfstein: In a typical clinical trial, eligible study patients are randomized across two drugs according to a strict treatment protocol. This randomization lets researchers balance out, on average, the distributions of known and unknown baseline risk factors among the participants. This way, patients receiving alternative treatments can be compared fairly. During the course of the clinical trial, however, some patients may prematurely terminate their participation. This dropout is "informative" if patients who withdraw are different from those who remain on study with respect to factors related to the outcome under investigation. When informative dropout occurs, the data that researchers observe may be biased relative to what they would have seen had all the patients completed the study. Researchers cannot know the distribution of the missing outcomes, so their analysis of the data then relies in part on unverifiable assumptions about the missing observations.

Q: How does this data dynamic play out in a typical clinical trial?

Dr. Scharfstein: Let me give you a recent example – the COMPANION study compared the effects of three alternative treatments on the survival of patients with chronic heart disease. These treatments were (1) a cardiac pacemaker, (2) a cardiac pacemaker with defibrillator, and (3) drug treatment alone. The researchers were interested in treatment differences with respect to two primary endpoints: death, and hospitalization plus death. About one-quarter of the patients who were enrolled in the drug treatment arm of the study dropped out, compared with only 6% or 7% for the patients who were enrolled in the device arms. Many of the patients who dropped out in the drug treatment arm subsequently asked for implantation of a pacemaker.

These patients were likely at higher risk for hospitalization or death than the patients in the drug treatment arm who remained in the study. An analysis that ignores this fact could provide an inaccurate or biased estimate of the difference in outcomes between the device arms and the drug treatment arm. This could be significant for, say, a device manufacturer facing allegations related to the safety or risk associated with its products.

Q: How can researchers account for missing data?

Dr. Scharfstein: To account for missing data, I advocate the use of sensitivity analysis. First, I recommend an analysis, guided by clinical input, that assumes that dropout can be explained (probabilistically) by time varying the information measured both on patients who dropped out and on patients who completed the study. This assumption is sometimes called “missing at random,” but I like to call it “explainable dropout.” Second, I recommend submitting this analysis to a stress test. This is done by positing clinically plausible deviations from the explainable dropout assumption and seeing whether inferences change. If they don’t change, then you have confidence that results are robust. ■