• Challenges of Analyzing Health Care Data in the COVID-19 Era

    Identifying sources of potential biases introduced by COVID-19 to real-world data (RWD) and clinical trials, and developing statistical methodologies to correct for them, are critical for drawing valid inferential conclusions.

    The massive disruption created by the coronavirus disease 2019 (COVID-19) pandemic extends beyond the health, economic, and societal impacts featured in headlines. COVID-19 is also having a confounding effect on health outcome assessment and comparative effectiveness research. As we discuss below, after identifying sources of biases introduced by COVID-19, researchers will need to develop appropriate statistical methodologies to correct for them when analyzing data generated during the pandemic.

    COVID-19 is having short-term and long-term effects on health care utilization (e.g., hospitalizations, emergency room visits, outpatient clinic visits), treatment patterns, patient behaviors, and health insurance coverage (e.g., through changes in employment status). Pandemic-induced changes in health care data used to analyze these factors may bias the assessment of treatment comparisons in terms of effectiveness, safety, adherence to treatment protocols, natural history, resource utilization, and cost endpoints.

    Biases in real-world data on adverse events

    Consider data from the FDA Adverse Events Reporting System (FAERS) for pharmaceuticals, or from the Manufacturer and User Facility Device Experience (MAUDE) database for medical devices. Adverse event (AE) data can be voluntarily submitted to these systems by health care professionals, such as physicians, nurses, and pharmacists, as well as by others, such as patients, family members, and lawyers.

    The data from these sources – captured through the FDA’s MedWatch System – support post-marketing safety surveillance efforts by regulators and manufacturers. Such data are also sometimes used in a litigation context to support arguments regarding drug safety.

    Even outside of a pandemic, these kinds of data are particularly susceptible to biases introduced by external factors such as media coverage, journal publications, or regulatory activities that can stimulate or otherwise affect the number of AE reports submitted. The current pandemic will likely introduce additional biases that must be taken into account in any analysis of these data, for a number of reasons.


    First, the pandemic has limited patients’ ability or willingness to see physicians for care that is not related to COVID-19, affecting the number of AEs reported by physicians during the pandemic. Because an important channel of AE reporting is through treating physicians during patient visits and examinations, any time-series analysis of AE reports will need to account for the potential impact of such time-period biases.

    Second, patients in areas with high prevalence of COVID-19, or those at risk for severe illness who are likely to take extra precautions, may have their mobility restricted more than others, and so may have fewer visits to the health care providers who report AEs. To the extent that these geographic variations in provider visits are correlated with the use of a particular drug or medical device, or with the likelihood of submitting an AE report, the pandemic may confound the comparison of AE rates or patterns across products.

    Third, the lack of continuity of care during the COVID-19 pandemic may increase the risk of certain AEs. This may escalate the number of AE reports for these conditions beyond those that would have been reported under the normal course of care and treatment.

    Finally, physical distancing measures and social isolations may affect AE reporting through changes in perceived physical function, symptoms, satisfaction with care, quality of life, and psychological well-being.

     

    Biases in real-world data on prescription drug use patterns

    Similarly, COVID-19 may also introduce biases to prescription drug dispensing data. These data are commonly used in pharmaceutical antitrust cases to study the effect of prices on drug prescribing and switching behavior, and to define the relevant market. Due to reduced patient-physician interaction during the pandemic, some industry analysts expected that the number of new prescriptions would decrease for many drugs, both to newly diagnosed patients and to previously diagnosed patients who may have switched to the drug in a world without COVID-19.1

    On the other hand, certain drugs that are typically used to treat other conditions may be diverted to off-label treatment of COVID-19. A prominent example of such drugs is hydroxychloroquine, which is indicated for the treatment of malaria and lupus. Data on such drugs would show large increases in the number of prescriptions that are otherwise unexplained by either newly approved indications or changes in relative drug prices.

    As a result, without proper adjustments the pandemic may bias the analyses of drug substitutability patterns across time, region, or population groups, and may bias the conclusions that can be drawn from these analyses.

    Biases in real-world data in observational studies


    In observational studies using RWD, such as medical records or health insurance claims data, patients are frequently followed over time to assess their clinical and health resource utilization outcomes to determine the relative effectiveness, safety, resource use, and economic profile among drugs of interest. In such studies, changes in health care utilization and reduced patient-physician interactions during the COVID-19 era may also artificially reduce the observation of relevant outcomes.

    In addition, patients who are diagnosed with certain diseases or seek care during the COVID-19 era are likely those with more severe conditions, causing them to breach stay-at-home orders or visit the emergency department. These patients are not representative of the general population of patients with the same condition, and so treatment and outcome data may be skewed towards these more serious cases.

    Also, increased mortality due to COVID-19 may bias the overall survival assessment in oncology patients receiving anti-cancer treatments differentially distributed in time periods before, during, and after the COVID-19 pandemic.

    Finally, similar biases may affect analyses of clinical trial data (in particular, single-arm trials) that may use real-world outcomes of patients outside of the trial for contextualization or indirect treatment comparisons. As COVID-19 may differentially affect patients in and outside of the trial, the comparison of their clinical outcomes may yield biased results.

     

    Far-reaching implications of COVID-19

    In summary, the COVID-19 pandemic has far-reaching implications for the collection and analysis of health care data. Going forward, reliable scientific analysis of such data collected during the COVID-19 era must account for the multitude of biases introduced by the pandemic. As a result, to draw valid conclusions regarding the use and effects of pharmaceutical and medical device products, it will be essential to model the different effects of COVID-19 across different periods, geographic areas, and population cohorts. ■


  • Maral DerSarkissian, Managing Principal
    Mei Sheng Duh, Managing Principal
    Jee-Yeon Lehmann, Managing Principal
    Brian Ellman, Principal
    Rose Chang, Vice President
    Lucia Antras, Consultant
    Andrea Di Miceli, Associate

  • Endnotes

    1. See, for example, Sarah Rickwood, Rising to The Challenge: Five key focus areas for Life Sciences during and after the COVID-19 pandemic, IQVIA (March 2020). See also, “The pandemic will recast America’s health-care industrial complex,” The Economist (May 9, 2020), which reported that “new prescriptions fell by 15% year on year in the week to April 17th.”