Meta-research on pragmatic trials

Follow us on Twitter @PragMeta_Org

Background

It is often claimed that results from some randomized clinical trials (RCT) may not be adequate to treat patients in real life situations. Pragmatic RCTs provide decision-oriented, real world evidence that is perceived as highly applicable and generalizable to routine care. Conversely, explanatory RCTs conducted in highly controlled research settings are often considered little or not applicable and generalizable. It is widely assumed that treatment effects provided by more pragmatic trials differ from those given by more explanatory trials. However, it is unknown how they differ and which features determine pragmatism, generalizability, and applicability. Our goal is to provide empirical evidence on the strengths, weaknesses, and the impact on treatment effect estimates of pragmatism, applicability, and generalizability beyond the theoretical expectations of their value in the decision-making process of patient care.

Objectives

1.  To set up the PragMeta database, which will include trials investigating the same clinical questions but having different aspects of generalizability, applicability, and pragmatism;
   
2. to descriptively characterize those aspects using, for example, the commonly used PRECIS-2 score (PRagmatic Explanatory Continuum Indicator Summary) and specific trial population- and setting-related determinants of generalizability (for example strict or broad eligibility criteria) and applicability (for example highly controlled environment versus routine care); and
3. to determine their impact on the treatment effect estimates

Design and Methods

This is a meta-epidemiological study of published trials. We will work in a large international research collaboration, with key experts in the field, to incorporate and share in the PragMeta database large already available datasets of pragmatic trials, supplemented by specific systematic searches of pragmatic and explanatory trials on the same clinical questions. Standard methods of systematic reviewing, meta-analysis and meta-epidemiology will be used. We will compare treatment effects from pragmatic versus explanatory trials and from trials with versus without specific determinants of pragmatism, generalizability, and applicability using metaregressions and the ratio of odds ratio method. We plan to include hundreds of trials with pragmatism and explanatory characteristics.

Relevance of the project

The cornerstone of real world evidence is that treatment effects under real world conditions in routine care may differ from that in highly controlled research settings, such as in drug approval trials. The PragMeta project aims to provide a better understanding of the pragmatic and explanatory determinants that may impact the generation and interpretation of real world evidence. It will give practical guidance for clinicians, researchers, regulators, health insurers and health-care policy makers, developers of reporting guidelines, and other stakeholders who develop, fund, conduct, assess, or otherwise use clinical trials. Ultimately, it may help to design the right trial for the right purpose generating evidence that is more useful for patients, clinicians, and other key health care decision-makers.

Protocol and procedures are published  on Open Science Framework (last update: March 21, 2023).