A 2024 Data Quality Framework (DQF) from the European Medicines Agency (EMA) aims to provide some guidance for drug manufacturers. Here is a summary of that document.
Factors to be considered when evaluating RWD
Key factors to be considered in the EMA’s DQF include:
Reliability: Are the data correct?Extensiveness: Are the data enoughCoherence: Are the data homogeneousTimeliness: Are the data collected in a timely mannerRelevance: Is this the right type of data?
Much of the document provides useful but high-level information about assuring the data have sufficiently reliable processes for data collection and the data themselves are useful to the research question at hand. However, how will EMA consider the RWD?
Determination of ‘fit for use’
The key question is whether the data is “fit-for-use” for the research question at hand. To determine this, researchers need to specify relevant design which include:
Study population: Inclusion criteria, cohort size, representativeness of the sampleStudy sampling: case-control, cohort, other methodTreatment/exposure group: Ability to identify treatment, sample size of newly treatedComparator group: Ability to identify treatment, sample size of newly treated Outcomes (primary and secondary): Whether outcome can is captured and accuracy/precision of endpointLength of follow-up: Minimum follow-up time required; typical follow-up time of patients in the dataData lag time: How long the data lag isConfounders: Specify which confounders can be collected in the data and the reliability of the measurement of these confoundersData detail: Is information available at the patient level or only aggregate level? Will data be linked to other data sources and if so how
Ensuring RWD is analyzed in an unbiased manner
Then the key question is whether the data considered can adequately capture all of this information in a timely and unbiased manner and has sufficient samples size to answer the research question with sufficient precision. The DQF states:
Where possible, pre-specify the importance of the quality of data elements in the protocol this assessment should be done in anticipation of the analysis methods (e.g., sample size calculations, use of time-to-event endpoints, sensitivity analyses, statistical adjustment for measurement error), which will impact what is considered acceptable for missing data or errors. While not part of the quality assessment itself, anticipating methods is important to provide context for performing a quality assessment.
Outstanding Questions
While the DQF provides a useful framework for using RWD for regulatory submission, there are still many questions left unanswered. For a specific case, how likely is it for EMA to consider that the data will meet the reliability, extensiveness, coherence, timeliness, and relevance criteria? Is it worth it for drug manufacturers to use RWD for regulatory submissions? While this could reduce their cost for clinical trials and expedite review, there is significant risk that EMA would deem certain RWD not ‘fit for purpose’ and the drug manufacturer may have conduct a standard randomized controlled trial anyway. How should drug manufacturers select RWD? Is reliability, extensiveness, coherence, timeliness, or relevance most important to EMA? This is not clear.
Yet this lack of specificity is not due to a desire to be vague. From the EMA perspective, it is hard to give very definitive rules because the utility of RWD will depend heavily on the specific use case and research question at hand.
In short RWD has tremendous potential to help expedite drug development, but there is still significant uncertainty regarding whether this potential will be realized.
Further Reading
For those interested in more information on the use of real-world data for regulatory submissions, I recommend checking out the GetReal Institute.
