Evaluating Large Language Model Literature Reviews in Interdisciplinary Science: A Systems Biology Perspective

Research output: Contribution to book/Conference proceedings/Anthology/ReportConference contributionContributedpeer-review

Contributors

Abstract

We evaluate the effectiveness of current large language model (LLM) literature review systems in interdisciplinary domains. While LLMs can support and accelerate reviewing the scientific literature, it is unclear how they cope with interdisciplinary science, where sources from multiple fields must be integrated according to relevance defined by context. We study this from the perspective of systems biology, a field that combines biology, mathematics, physics, and computer science. Using a set of expert-defined research questions, we assess the ability of LLMs to meaningfully integrate cross-domain knowledge and correctly reflect relevance. Specifically, we evaluate the quality of generated reports and the relevance of retrieved references from five different review models. We find that LLMs are a valuable augmentative tool for literature reviews, but trade off report quality for completeness in interdisciplinary domains. We address these limitations by proposing a novel method, termed AURORA, which is particularly designed for interdisciplinary applications. On the interdisciplinary systems biology benchmark, AURORA offers good coverage with high-quality reports.

Details

Original languageEnglish
Title of host publicationEKAW-PDWT 2024 - Posters and Demos, Workshops, and Tutorials of EKAW 2024
Number of pages6
Publication statusPublished - 2024
Peer-reviewedYes

Publication series

SeriesCEUR Workshop Proceedings
Volume3967
ISSN1613-0073

Conference

Title24th International Conference on Knowledge Engineering and Knowledge Management
SubtitleKnowledge in the Age of Language Models
Abbreviated titleEKAW 2024
Conference number24
Duration26 - 28 November 2024
Website
LocationCentrum Wiskunde & Informatica (CWI)
CityAmsterdam
CountryNetherlands

External IDs

ORCID /0000-0003-4414-4340/work/192042781
ORCID /0000-0002-7227-3441/work/192044681

Keywords

ASJC Scopus subject areas

Keywords

  • Interdisciplinary Science, Large Language Models, Literature Review, Scientific Literature, Systems Biology