Evaluating Large Language Model Literature Reviews in Interdisciplinary Science: A Systems Biology Perspective
Research output: Contribution to book/Conference proceedings/Anthology/Report › Conference contribution › Contributed › peer-review
Contributors
Abstract
We evaluate the effectiveness of current large language model (LLM) literature review systems in interdisciplinary domains. While LLMs can support and accelerate reviewing the scientific literature, it is unclear how they cope with interdisciplinary science, where sources from multiple fields must be integrated according to relevance defined by context. We study this from the perspective of systems biology, a field that combines biology, mathematics, physics, and computer science. Using a set of expert-defined research questions, we assess the ability of LLMs to meaningfully integrate cross-domain knowledge and correctly reflect relevance. Specifically, we evaluate the quality of generated reports and the relevance of retrieved references from five different review models. We find that LLMs are a valuable augmentative tool for literature reviews, but trade off report quality for completeness in interdisciplinary domains. We address these limitations by proposing a novel method, termed AURORA, which is particularly designed for interdisciplinary applications. On the interdisciplinary systems biology benchmark, AURORA offers good coverage with high-quality reports.
Details
| Original language | English |
|---|---|
| Title of host publication | EKAW-PDWT 2024 - Posters and Demos, Workshops, and Tutorials of EKAW 2024 |
| Number of pages | 6 |
| Publication status | Published - 2024 |
| Peer-reviewed | Yes |
Publication series
| Series | CEUR Workshop Proceedings |
|---|---|
| Volume | 3967 |
| ISSN | 1613-0073 |
Conference
| Title | 24th International Conference on Knowledge Engineering and Knowledge Management |
|---|---|
| Subtitle | Knowledge in the Age of Language Models |
| Abbreviated title | EKAW 2024 |
| Conference number | 24 |
| Duration | 26 - 28 November 2024 |
| Website | |
| Location | Centrum Wiskunde & Informatica (CWI) |
| City | Amsterdam |
| Country | Netherlands |
External IDs
| ORCID | /0000-0003-4414-4340/work/192042781 |
|---|---|
| ORCID | /0000-0002-7227-3441/work/192044681 |
Keywords
ASJC Scopus subject areas
Keywords
- Interdisciplinary Science, Large Language Models, Literature Review, Scientific Literature, Systems Biology