Using Assessment Tools to Determine the Validity of Non-Randomized Studies

Using Assessment Tools to Determine the Validity of Non-Randomized Studies

By Joan Largent, Senior Director, Epidemiology and Outcomes Research, IQVIA

As the use of real world evidence (RWE) becomes a more prominent resource for confirming and expanding product benefit-risk profiles and informing regulatory decisions, observational, and non-randomized studies (NRSs) are emerging as valuable generators of real world data (RWD). The FDA’s framework for including RWE in their processes has the goal of bringing therapeutic interventions to patients faster, and its recent series of draft guidance documents for the use of RWD in submissions underscores the increasingly important role for RWE in regulatory decision-making for drugs and biological products. RWE generating studies typically include broader patient populations than traditional randomized clinical trials (RCTs). As a result, an NRS can deliver relevant insights into the safety and effectiveness of treatments when used in actual clinical practice.

An NRS can offer evidence that supplements what we know about medicines from RCTs. For example, insights driven by NRSs are particularly useful for patients with comorbid conditions or in varying age and racial groups, because they may be underrepresented in RCTs due to stringent inclusion/exclusion criteria. The knowledge gained from RWE can contribute to better informed decision-making, and therefore, better patient health outcomes overall. However, NRSs pose higher risk of inadvertent bias than RCTs. This possibility makes it essential to apply a high-level of rigor to determine the validity of the study, or how well the patient population studied represents the target population.

The potential for introducing bias

RWD is useful because it can provide a clearer representation of treatment benefits and risks when used in actual clinical practice. However, the RWD may not be initially collected for study purposes, which can result in inherent limitations. This factor, as well as the more limited control as compared with RCTs, makes NRS methodologies more prone to possible bias for the following reasons:

Selection of patients

In an RCT, the strict inclusion and exclusion criteria allow for greater control over patients enrolled but can limit the generalizability of the study findings to patients with similar characteristics. In real world practice, patients aren’t randomly selected for one treatment or another. Demographic variables such as age, race, ethnicity, as well as access to care and underlying comorbidities can all influence whether a patient receives a particular treatment or not and may also be associated with outcomes. It is important for NRS to include patients for whom the intended results are to be generalized in terms of the range of clinical characteristics, demographics, prior treatments, comorbidities, and co-medications as well as to consider how characteristics influencing treatment selection may also influence outcomes.

Measurement of exposure and outcomes

In RCTs patient exposure to medications or interventions as well as outcome assessments are strictly prescribed per protocol. In contrast, NRS patient exposure and outcome assessments generally follow routine clinical practice with potential for substantial variation. Variables, such as the start and stop dates of medications, side effects, and adverse reactions, require accurate recording and monitoring to mitigate bias. A patient who experiences only mild nausea while using a treatment might continue taking the medication and never mention this side effect to their doctor. Unless the patient is asked directly about reactions, the effect might not be recorded. To ensure an NRS is accurate, we need to have confidence that outcomes under study are recorded in the data. It is also critical to ensure patients in groups to be compared are followed up in a similar manner. For example, if patients receiving one medication require more frequent physician visits than patients receiving another medication, this could also introduce bias as outcomes may be more frequently recorded in patients with more frequent physician visits. Enforcing consistent measurements of exposures and outcomes across groups to be compared is critical to ensure validity of the NRS.

Covariate measurements

When a patient has received another treatment including over-the-counter medications or has another health condition that may not be recorded in the RWD, this also can introduce potential for bias. For example, patients who receive vaccines tend to be healthier and have fewer underlying conditions than those who don’t. Additionally, patients with certain comorbidities may respond differently to the same treatment as healthier patients. If data about other treatments and vaccines or patient comorbidities are not recorded as part of the study, it may confound an association between an intervention and an outcome. Diabetes and body mass index (BMI), for instance, are typically linked. In the case of a NRS in diabetic patients, for example, it’s generally important to have BMI measurements because that metric could be associated with both the treatment and the outcome under study.

Statistical approaches

To address imbalances that may exist between treatment groups to be compared in NRS, statistical analyses should account for such imbalances. Potential confounding variables should be measured and addressed in analyses. Statistical approaches can also assess degree of uncertainty or precision of the estimated effects which can aid interpretation of the study findings.

Optimizing use of NRS data through assessments

For researchers to optimize the acceptance of RWE, it’s essential to minimize the risk of bias in the studies. Assessment tools can help researchers evaluate the likelihood of bias in an NRS. Researchers may use assessment tools, particularly when planning or evaluating real world or non-randomized research that could ultimately increase the probability of a regulator or payer accepting the study results into their decision-making process.

A special interest group comprised of members of the International Society of Pharmacoepidemiology systematically reviewed 44 different assessment tools as part of a broader goal of developing a framework for integrating evidence from NRSs and other RWD with evidence from RCTs to increase understanding of the benefits and risks of medicines.¹ The retrieved assessment tools were evaluated according to quality elements to determine whether they sufficiently addressed the main methodological challenges of NRSs as predefined via a Delphi survey.

Most of the assessment tools reviewed apply to observational study designs, while some focused on prospective or cohort designs, in which enrolled patients receive a treatment or alternative treatment and are then monitored for future outcomes. Other assessment tools focus primarily on retrospective or case-control designs. These designs examine patients with a specific outcome (or absence of), and then review their history to identify what medications they may have been exposed to. Assessment tools reviewed include checklists, summary judgments and rating scales, which offer ways to evaluate the quality of evidence.

The review of these assessment tools identified three tools that are designed to specifically assess studies of comparative safety and effectiveness of pharmacological interventions. Other tools evaluated applied more broadly to observational or NRS. The research highlighted quality domains that are not routinely covered by the assessment tools, and which are important for researchers to assess when considering the quality of NRSs also when integrating RWE with evidence derived from RCTs.

Future expectations

Assessment tools can help researchers and stakeholders critically evaluate the validity of RWE. Verifiable assessment could become an expectation as part of submissions to regulators or Health Technology Assessment (HTA) bodies provided that suitable tools are available to conduct assessments. For example, UK’s National Institute for Health and Care Excellence (NICE) has recommended use of bias assessments relative to study design in their Real World Evidence Framework. In this case, stakeholders could require proof of assessment to evaluate the validity of the RWE.

Researchers planning to leverage NRS methodologies in their work should begin by carefully determining the validity of their study design and mitigating any potential bias that could invalidate results. Assessment tools can help with planning and evaluating the potential for bias and increasing both internal and external validity. By effectively addressing these hurdles including use of relevant assessment tools, researchers can contribute to furthering the use of RWE to improve our understanding of treatments and our ability to positively impact patient outcomes.

Appendix

1. D’Andrea E, Vinals L, Patorno E, et al. How well can we assess the validity of non-randomised studies of medications? A systematic review of assessment tools. BMJ Open 2021;11:e043961. doi: 10.1136/bmjopen-2020-043961