Foundations of LLM Knowledge Materialization: Termination, Reproducibility, Robustness

Publikation: Beitrag in Buch/Konferenzbericht/Sammelband/Gutachten › Beitrag in Konferenzband › Beigetragen › Begutachtung

Abstract

Large Language Models (LLMs) encode substantial factual knowledge, yet measuring and systematizing this knowledge remains challenging. Converting it into structured format—for example through recursive extraction approaches such as the GPTKB methodology (Hu et al., 2025)—is still underexplored. Key open questions include whether such extraction can terminate, whether its outputs are reproducible, and how robust they are to variations. We systematically study LLM knowledge materialization using miniGPTKBs (domain-specific, tractable subcrawls), analyzing termination, reproducibility, and robustness across three categories of metrics: yield, lexical similarity, and semantic similarity. We experiment with four variations (seed, language, randomness, model) and three illustrative domains (from history, entertainment, and finance). Our findings show (i) high termination rates, though model-dependent; (ii) mixed reproducibility; and (iii) robustness that varies by perturbation type—high for seeds and temperature, lower for languages and models. These results suggest that LLM knowledge materialization can reliably surface core knowledge, while also revealing important limitations.

Details

OriginalspracheEnglisch
Titel19th Conference of the European Chapter of the Association for Computational Linguistics, Findings of EACL 2026
Herausgeber (Verlag)Association for Computational Linguistics (ACL)
Seiten2145-2164
Seitenumfang20
ISBN (elektronisch)979-8-89176-386-9
PublikationsstatusVeröffentlicht - März 2026
Peer-Review-StatusJa

Konferenz

Titel19th Conference of the European Chapter of the Association for Computational Linguistics
KurztitelEACL 2026
Veranstaltungsnummer19
Dauer24 - 29 März 2026
Webseite
OrtPalais des Congrès Rabat Bouregreg
StadtRabat
LandMarokko

Externe IDs

ORCID /0000-0002-5410-218X/work/222763835