Stages and Phases in Development, Evaluation and Implementation of Clinical Prediction Models: A Systematic Literature Review and Content Analysis

Abstract

Abstract

Background: In an era of “big data”, medical information is increasingly accessible for use in patient care. Recently numerous prediction models have been developed using the association of multiple known clinical factors to predict the health outcomes of patients with a particular condition. Paradoxically, only a minority of the prediction models are currently used in clinical practice. This discrepancy is explained by the current the tendency toward derivation of new model rather than validating the existing one.

Aim: To help structuring the development and validation of the prediction models, the article’s aim is to construct a framework adressing the evidence level throughout the stages of development, evaluation, and implementation of prediction models. This framework strives to promote a more structured and rigorous approach to advancing these models for use in clinical settings.

Method: A first article selection was conducted in PubMed search database (1970 onwards). ASReview was used to prioritize screening and find relevant texts from search output. A second article selection was manually retrieved to include additional articles overlooked by the initial selection process and demonstrating a strong rationale for inclusion. Lastly, a third selection targeted regulatory agencies’ documentations/ guidelines for medical device and biomarkers obtained directly from the respective regulatory agency websites (EMA, FDA, MHRA, IMDRF, MDCG).

Results: Two categories of articles were defined after analyzing the included material. A separation between ”stages” articles and ”phases” articles was determined. The stages outlines the life-cycle of a prediction model from its theoretical background, to its conception and implementation in clinical practice without consideration to the level of evidence. While, phases describe the evidence level of prediction models throughout their development and evaluation process. A total of 45 articles were included in the systematic review (PubMed n= 24, Dossier n=16, Gray n=5). Approximately two third of these papers were categorized as “phases” papers (n=29, 64%), the other third were classified as ”stages” papers (n=14, 31%). The two remaining articles were categorized as both “phases” and “stages” paper (n=2, 4%). This paper presents two frameworks: 1) a 8-stages framework summarizing the key process from analytical to clinical implementation (1= identification unmet medical need, 2=model derivation, 3= external validation, 4=clinical impact assessment, 5= regulatory approval, 6= local adoption, 7= clinical implementation, 8= post implementation monitoring) and 2) a 4 phases framework structuring the evidentiary level of the prediction model based on the study design (1= feasibility, 2= capability, 3= effectiveness, 4= impact).

Conclusion: Although development and evaluation of prediction model might seem straightforward, adherence to methodological standards are essential to develop a clinically relevant and performant model with positive impact on patient care. The proposed frameworks are therefore aimed to guide during the models’ development by addressing both the evidentiary level and lifecycle aspects of the model. These frameworks are a key component of the strategy to structure the models’ development and bridge the gap between models’ development and clinical practice.