Evaluating Large Language Model Literature Reviews in Interdisciplinary Science: A Systems Biology Perspective
Publikation: Beitrag in Buch/Konferenzbericht/Sammelband/Gutachten › Beitrag in Konferenzband › Beigetragen › Begutachtung
Beitragende
Abstract
We evaluate the effectiveness of current large language model (LLM) literature review systems in interdisciplinary domains. While LLMs can support and accelerate reviewing the scientific literature, it is unclear how they cope with interdisciplinary science, where sources from multiple fields must be integrated according to relevance defined by context. We study this from the perspective of systems biology, a field that combines biology, mathematics, physics, and computer science. Using a set of expert-defined research questions, we assess the ability of LLMs to meaningfully integrate cross-domain knowledge and correctly reflect relevance. Specifically, we evaluate the quality of generated reports and the relevance of retrieved references from five different review models. We find that LLMs are a valuable augmentative tool for literature reviews, but trade off report quality for completeness in interdisciplinary domains. We address these limitations by proposing a novel method, termed AURORA, which is particularly designed for interdisciplinary applications. On the interdisciplinary systems biology benchmark, AURORA offers good coverage with high-quality reports.
Details
| Originalsprache | Englisch |
|---|---|
| Titel | EKAW-PDWT 2024 - Posters and Demos, Workshops, and Tutorials of EKAW 2024 |
| Seitenumfang | 6 |
| Publikationsstatus | Veröffentlicht - 2024 |
| Peer-Review-Status | Ja |
Publikationsreihe
| Reihe | CEUR Workshop Proceedings |
|---|---|
| Band | 3967 |
| ISSN | 1613-0073 |
Konferenz
| Titel | 24th International Conference on Knowledge Engineering and Knowledge Management |
|---|---|
| Untertitel | Knowledge in the Age of Language Models |
| Kurztitel | EKAW 2024 |
| Veranstaltungsnummer | 24 |
| Dauer | 26 - 28 November 2024 |
| Webseite | |
| Ort | Centrum Wiskunde & Informatica (CWI) |
| Stadt | Amsterdam |
| Land | Niederlande |
Externe IDs
| ORCID | /0000-0003-4414-4340/work/192042781 |
|---|---|
| ORCID | /0000-0002-7227-3441/work/192044681 |
Schlagworte
ASJC Scopus Sachgebiete
Schlagwörter
- Interdisciplinary Science, Large Language Models, Literature Review, Scientific Literature, Systems Biology