Real-World Data for the Drug Development in the Digital Era

I.INTRODUCTION

Randomized clinical trials (RCTs) have been considered the gold standard to provide evidence about the efficacy of new medications during the drug development process for regulatory approval. Although well-conducted RCTs demonstrate the causality and provide the most valid estimate of the relative efficacy of competing medications or healthcare interventions, RCTs do not necessarily reflect the “real-world” experience. This is because RCTs are usually conducted in a small number of patients that meet a rigorous set of study inclusion and exclusion criteria. In addition, patients in the RCTs are closely monitored and are followed up following strict protocols. It is unknown whether or to what extent the findings from RCTs in homogenous samples of patients under well-controlled experimental conditions could be generalized to various patients in routine clinical practice. Some rare adverse events are impossible to be observed in a small sample of patients during a short time of clinical trial period. RCTs are not feasible or ethical in some situations, such as “is smoking a risk factor for tuberculosis?” and “is dietary fat associated with breast cancer in the population?” Furthermore, the clinical evidence (mainly phase 3 RCTs) may not be sufficient to fully guide clinicians and policy-makers in choosing the optimal treatment for their patients because most medicines (especially newly marketed medications) approved for the same disease or indication have not been directly compared by RCTs. To address these limitations of RCTs and answer some questions that cannot be answered by RCTs, real-world evidence (RWE) obtained from real-world data (RWD) has become increasingly important as a complementary source to RCT data in the digital era [1–7].

II.REAL-WORLD DATA AND REAL-WORLD EVIDENCE

RWD has been defined differently in the literature. In a recent literature review, the authors identified 38 definitions of RWD [1]. The definition by the International Society for Pharmaco economics and Outcomes Research (ISPOR) Task Force on RWD is most commonly used: “data used for decision-making that are not collected in conventional randomized controlled trials” [3] According to the US Food and Drug Administration (FDA) [8], RWD is “data relating to patient health status and/or the delivery of healthcare routinely collected from a variety of sources.” RWE is defined as “clinical evidence regarding the use and potential benefits or risks of a drug derived from analysis of RWD.” Regardless of various definitions, RWD has two principal characteristics. First, RWD is collected in a naturalistic manner or recorded in routine clinical practice setting, and thus RWD is observational in nature. Second, a medicine or treatment regimen is not randomly assigned to patients (pragmatic clinical trials (PCTs) are exceptional) but is decided by healthcare providers and/or patients.

III.RWD SOURCES

RWD in routine clinical practice can be obtained from multiple sources [2,9], including but not limited to data that are already being gathered in real-world settings at the point of care, such as medical charts and electronic medical/health records (EMR/EHR) originating from healthcare providers, imaging and laboratory tests, healthcare administrative claims data, and pharmacy data used to fill prescriptions. Data may also be collected in patient and physician surveys, safety surveillances, disease registries, prospective observational studies, or PCTs (discussed in the following section). Social media are growing data sources [2], which can provide patient perspectives on health topics such as adverse events, reasons for changing treatments and nonadherence, treatment satisfaction, and quality of life. Electronic devices and software applications and internet-based websites have been used more and more widely to collect RWD in the digital era. RWD can be big but are not equal to “big data,” which involves large or complex unstructured datasets, characterized by four high vs: high volume, high velocity, high variety, and high veracity.

EHR datasets and pharmacy and health insurance databases are commonly used for retrospective studies/analyses, while patient registries are a common type of prospective observational real-world studies. EHR datasets are systems into which healthcare providers enter routine clinical and laboratory data during routine clinical practice. Pharmacy and health insurance databases or administrative databases are types of healthcare database systems that are set up by pharmacists or health insurers for billing and reimbursement and other healthcare administration and management, such as monitoring of healthcare service use. Administrative databases can be either commercial claims datasets from private insurers or noncommercial claims data from Medicare and Medicaid in the US, for example. Data elements available in EHRs and healthcare administrative claims databases in the US can be found in the book of Developing a Protocol for Observational Comparative Effectiveness Research: A User’s Guide by the Agency for Healthcare Research and Quality (US) [10].

A patient registry is an organized system that uses observational study methods to collect uniform data (clinical and other) to evaluate specified outcomes for a population defined by a particular disease, condition, or exposure, and that serves a predetermined scientific, clinical, or policy purpose(s) [11]. Registries are cohort studies that prospectively collect, analyse, and disseminate data on a group of patients with specific characteristics in common. According to how their populations are defined, registries can be focused on a disease/condition, medical product, or health service. Data in patient registries include various clinical and biological variables, which can be obtained from medical charts, EHR, imaging and laboratory tests, or linked databases.

IV.REAL-WORLD DATA STUDIES

Real-world data studies (RWDSs) are studies that are conducted in routine clinical practice or using existing RWD that have been collected. RWDSs tend to be large (to assess rare events and subgroup effects) and generalizable. They may also be clinically more relevant because they can address the question of which of the available treatments is best for a specific patient population via comparative effectiveness research [12] rather than the question whether a treatment works or not via RCTs.

RWDSs can be conducted 1) retrospectively by analyzing existing datasets such as electronic health/medical records, healthcare administrative claims databases, and epidemiological data [13,14]; 2) prospectively by collecting new data such as prospective cohort studies and patient registries [15]; or 3) by cross-sectional surveys of patient- and physician-reported treatment outcomes, adherence, and satisfaction [16].

PCTs or large, simple trials are a type of RWDSs or observational studies and a type of RCTs as well. PCTs are defined as “prospective, randomized controlled trials that use large numbers of patients, broad patient inclusion criteria, multiple study sites, minimal data requirements, and electronic registries; their purposes include detecting small and moderate treatment effects, gaining effectiveness data, and improving external validity.” in the National Library of Medicine. PCTs are designed to evaluate a drug or an intervention in a study environment that is closer to real life in terms of study population, intervention, comparator, and outcomes [17]. PCTs are characterized by 1) randomly assigning patients to different treatments/medicines to balance baseline characteristics and unmeasured confounders; 2) routinely or as far as possibly observing patients in the real-world clinical practice setting to maintain usual care throughout the trial; and 3) outcomes and number of end points, which are closely relevant to optimal healthcare policy or clinical treatment decision-makings. Compared with traditional RCTs, PCTs are generally more efficient and less expensive. Results from PCTs are more relevant to real-world populations and thus are more generalizable. In addition, PCTs can detect clinically relevant treatment effects and rare adverse events, minimize the effects of random errors, and are statistically powerful for subgroup analyses because of large sample size. However, PCTs are not suitable for early-phase studies to explore whether a new drug or an intervention has any biological effect [17]. PCTs have been becoming increasingly popular, but they are still less conducted than traditional RCTs because of the limitations of the data routinely collected or recorded in routine clinical practice including EHRs compared with data collected from traditional RCTs.

The process of RWDSs is the same as traditional clinical/health studies, including RCTs, from developing research questions to study report/publication as described in Fig. 1.

Fig. 1. Process of real-world evidence generation using real-world data.

The characteristics of RWDSs compared with traditional RCTs are summarized in Table I.

Table I Characteristics of traditional randomized clinical trials (RCTs) and real-world data studies (RWDSs)

V.APPLICATIONS OF RWE

RWE can be used at different stages of drug development and is useful to healthcare decision and policy-makers in determining the value of various treatment options [4,5,9]. For example, studies based on RWD can provide RWE of utilization and treatment patterns, clinical effectiveness (i.e., clinical efficacy in the real-world clinical practice) and safety, healthcare costs and utilization, and parent-reported outcomes (quality of life and treatment satisfaction) associated with a treatment/medicine. RWE can also be used for different phases of drug development by examining disease prevalence and social/economic burden, identifying unmet medical needs, identifying new indications and rare adverse events, describing the natural history of a disease, and delineating treatment pathways in routine clinical practice. Furthermore, such evidence can be used to develop a product’s benefit–risk profile and clinical practice guidelines for clinical decision-making. RWE is also currently used to generate additional hypotheses for continued clinical development and inform aspects of early drug development such as clinical trial design or the comparative effectiveness of comparator treatments within a given indication.

Both the European Medicines Agency and the US FDA have recently acknowledged the need for studies that examine the effectiveness of a drug in the “real world.” The US FDA has drafted the guidance of use of RWE to support regulatory decision. Under the 21st Century Cures Act (Cures Act), the US FDA is directed to develop a regulatory framework to evaluate how RWE can potentially be used to support approval of new indications for approved drugs or to support or satisfy post-approval study requirements. This framework includes what types of data that could be used and how, methods of study design and conduct, human subject protections, and methods of statistical analysis [8]. The US FDA consistently seeks to advance regulatory science that will optimize the decision-making process for the development of drugs, biological products, and devices. The US FDA currently accepts RWD and RWE to support regulatory decision-making about drug safety, but it is still used less frequently to establish drug effectiveness. Detailed examples of issues that RWD can address at the different stages of the drug development lifecycle (from discovery→early development→full development→registration/market access→lifecycle management) can be found at Bate et al. [18] and Purpura et al. [5].

VI.CHALLENGES/LIMITATIONS

Evidence-based medicine (EBM) is “the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients” as defined by Sackett and colleagues [19]. EBM includes four steps: 1) formulate a clear clinical question from a patient’s problem; 2) search the literature for relevant clinical articles; 3) evaluate (critically appraise) the evidence for its validity and usefulness; and 4) implement useful findings in clinical practice [20]. Evaluating the validity and usefulness are critical to provide robust evidence for clinical decision-making. Figure 2 shows the hierarchy of medical evidence obtained through various methods in the order based on their credibility. The results of meta-analyses of well-conducted large, randomized trials are generally ranked the strongest evidence, followed by large multicentered randomized trials, single-centered randomized trials, non-randomized trials, non-interventional observational studies (i.e., RWDSs), clinical experience, or basic science research.

Fig. 2. Hierarchy of medical evidence (lowest to highest from bottom) for clinical decision-making. RCTs = Randomized clinical trials; EHR = Electronic health record; RWDSs = Real-world data studies.

The evidence from RWD is less strong than RCTs and varies across RWDSs. In general, large-sample prospective observational studies with controls provide the strongest RWE, followed by retrospective database analyses, with the cross-sectional or case reports without controls being the weakest.

It must be cautious when conducting RWDSs and interpreting RWE because RWD has many limitations. The imitations of RWD mainly arise from selection bias and unmeasured confounders due to lack of randomization. Another issue is the accuracy, completeness, and consistency of data. The third issue is information/reporting bias due to lack of blindness. In addition, RWDSs are not well designed and conducted, and RWD is poorly statistically analyzed. The following limitations/questions should be taken into consideration when interpreting and using RWE from a RWDSs.

• 1.RWD may be biased or false. Reporting or information bias is not uncommon. For example, a study using a claims or EMR database is subject to limitations inherent in potential coding errors and missing data.
• 2.RWD from one source or limited patient populations may not be representative of all real-world patients. Findings from a real-world clinical practice patient population may not be generalized to other patients. That is, the experience from the studied clinical practices may not be applied to other clinical practices.
• 3.RWD may not be large enough for powerful statistical analyses. This may be particularly true for subgroup analyses.
• 4.RWD may not be useful to answer some healthcare or clinical questions. For example, administrative databases are useful to describe treatment patterns and estimate healthcare costs associated with a medicine or intervention but may not be good to examine some clinical outcomes, such as disease progression free survival and tumor response.
• 5.RWDSs may not be well conducted, such as small sample sizes, short follow-up, limited clinical practice sites, inappropriate study design, inappropriate statistical analyses, and findings being selectively reported.
• 6.Retrospective/observational RWD analyses study associations between variables but are unable to determine causality.
• 7.The results from RWDSs are limited by potential selection bias because patients were not randomly assigned, and treatment groups often show differences in baseline characteristics.
• 8.Although multivariate analyses or advanced methods such as propensity score matching and inverse probability treatment weighting can be used to adjust for patient demographics and clinical characteristics, confounding factors may still be present due to unmeasured confounders.
• 9.Databases, study populations, study designs, statistical methods, and study end point definitions between studies may vary and should be taken into consideration when evaluating the outcomes. Findings may not be directly compared across real-world data studies.
• 10.Sensitivity analysis, “a method to determine the robustness of an assessment by examining the extent to which results are affected by changes in methods, models, values of unmeasured variables, or assumptions” [21] is important for almost all real-world studies. If the findings are not consistent with primary analysis across different assumptions/methods, such as patient populations, outcomes definitions, follow-up times, and statistical analyses, the findings or conclusions may not be robust.

VII.IN SUMMARY

In the digital era, RWD will be more easily and widely collected. RWD has become more and more important in the process of drug development and provides RWE for clinical decision-making. However, RWD is not a panacea, and it should not be misused and overused [22]. This is because RWD may yield misleading evidence if RWD is not valid and/or RWDSs are not well designed and conducted. Efforts must be made at the stages of database selection, study design, statistical analysis, and interpretation of the results to minimize the biases and confounding effects due to data limitations and non-randomization.