AI model evaluation lifecycle at MetReal Cluster Meeting

Hosted by the Danish Medicines Agency and organized by the REDDIE project, the two-day workshop united the MetReal Cluster (REDDIE, More-EUROPA, ONCOVALUE, Real4Reg, INSAFEDARE, and REALM) with regulators, HTA bodies, and industry experts.

Broadly, the cluster aims to pioneer new methods for using real-world and synthetic data in regulatory and HTA decision-making, for pharmaceutical and medical devices sectors. A key takeaway from the discussions was that while Europe possesses world-class RWE initiatives, achieving maximum impact requires stronger alignment and cross-project communication to bridge the gap between innovation and policy.

REALM presented significant advancements in the standardisation and regulatory assessment of medical AI software. Represented by Gokhan Ertaylan (VITO, REALM Scientific Coordinator) and Janaki Raman (VPH, REALM Partner), the team highlighted the broad regulatory landscape  as well as a comprehensive framework for navigating the complex regulatory pathway of AI-enabled medical devices.

A key highlight of the contribution was a visual roadmap, co-drafted by VPH, illustrating the end-to-end lifecycle of AI models within the REALM platform. This framework provides a clear path for manufacturers and regulators to ensure technical performance of AI models, benchmarked by REALM sandbox and living lab environment, through RIANA platform.

Briefly, the REALM platform includes data catalogue first identifies relevant, independent public data sources—such as validation collections found in public repositories and synthetic data, as needed. Subsequently, the technical performance of models is evaluated using the REALM regulatory toolbox, a suite of rigorous performance metrics designed to meet high regulatory conformity. REALM’s federated architectures offers an immutable trace and documentation through model cards, to ensure total transparency on the performance assessment. By utilizing blockchain technology to anchor Model Cards, the REALM platform ensures that every event and performance outcome throughout the AI model’s lifecycle is securely and transparently documented

Impact on Stakeholders

By centralizing these processes in multi-country federated architecture, the REALM platform enables a collaborative, iterative assessment of AI as medical device software. Technology Developers, regulators, and Health Technology Assessment (HTA) actors can critically review and refine the claims and intended purposes of medical device software, backed by verifiable independent data source and rigorous metric, with transparency in the overall process.