Guidelines for Reporting Studies on Large Language Models in Radiology: An International Delphi Expert Survey

Research output: Contribution to journalReview articleContributedpeer-review

Contributors

  • Jonathan Kottlors - , University of Cologne (Author)
  • Andra Iza Iuga - , University of Cologne (Author)
  • Christian Bluethgen - , University of Zurich (Author)
  • Keno Bressem - , Technical University of Munich (Author)
  • Jakob Nikolas Kather - , Department of Internal Medicine I, Else Kröner Fresenius Center for Digital Health, National Center for Tumor Diseases (NCT) Heidelberg (Author)
  • Linda Moy - , New York University (Author)
  • Christoph Wald - , Lahey Clinic Medical Center, Burlington (Author)
  • Wei Wang - , Sun Yat-Sen University (Author)
  • Tianming Liu - , University of Georgia (Author)
  • Erik Ranschaert - , St. Nikolaus Hospital, Ghent University (Author)
  • Thomas Dratsch - , University of Cologne (Author)
  • Jens Kleesiek - , University Hospital Essen (Author)
  • Roman Johannes Gertz - , University of Cologne (Author)
  • Pranav Rajpurkar - , Harvard University (Author)
  • Arash Bedayat - , University of California at Los Angeles (Author)
  • Matthias A. Fink - , Heidelberg University  (Author)
  • Almut Zeeck - , University of Freiburg (Author)
  • Akshay Chaudhari - , Stanford University (Author)
  • Tarik Alkasab - , Massachusetts General Hospital (Author)
  • Honghan Wu - , University of Glasgow, University College London (Author)
  • Felix Nensa - , University Hospital Essen (Author)
  • Benyou Wang - , The Chinese University of Hong Kong, Shenzhen (Author)
  • Nils Große Hokamp - , University of Cologne (Author)
  • Kai Roman Laukamp - , University of Cologne (Author)
  • Thorsten Persigehl - , University of Cologne (Author)
  • David Maintz - , University of Cologne (Author)
  • Daniel Truhn - , RWTH Aachen University (Author)
  • Simon Lennartz - , University of Cologne (Author)

Abstract

Large language models (LLMs) have transformative potential in radiology, including textual summaries, diagnostic decision support, proofreading, and image analysis. However, the rapid increase in studies investigating these models, along with the lack of standardized LLM-specific reporting practices, affects reproducibility, reliability, and clinical applicability. To address this, reporting guidelines for LLM studies in radiology were developed using a two-step process. First, a systematic review of LLM studies in radiology was conducted across PubMed, IEEE Xplore, and the ACM Digital Library, covering publications between May 2023 and March 2024. Of 511 screened studies, 57 were included to identify relevant aspects for the guidelines. Then, in a Delphi process, 20 international experts developed the final list of items for inclusion. Items consented as relevant were summarized into a structured checklist containing 32 items across six key categories: general information and data input; prompting and fine-tuning; performance metrics; ethics and data transparency; implementation, risks, and limitations; and further/optional aspects. The final FLAIR (Framework for LLM Assessment in Radiology) checklist aims to standardize reporting of LLM studies in radiology, fostering transparency, reproducibility, comparability, and clinical applicability to enhance clinical translation and patient care.

Details

Original languageEnglish
Article numbere250913
JournalRadiology
Volume318
Issue number2
Publication statusPublished - Feb 2026
Peer-reviewedYes

External IDs

PubMed 41631991
ORCID /0000-0002-3730-5348/work/211722515

Keywords