Foundations of LLM Knowledge Materialization: Termination, Reproducibility, Robustness

Research output: Contribution to book/Conference proceedings/Anthology/Report › Conference contribution › Contributed › peer-review

Abstract

Large Language Models (LLMs) encode substantial factual knowledge, yet measuring and systematizing this knowledge remains challenging. Converting it into structured format—for example through recursive extraction approaches such as the GPTKB methodology (Hu et al., 2025)—is still underexplored. Key open questions include whether such extraction can terminate, whether its outputs are reproducible, and how robust they are to variations. We systematically study LLM knowledge materialization using miniGPTKBs (domain-specific, tractable subcrawls), analyzing termination, reproducibility, and robustness across three categories of metrics: yield, lexical similarity, and semantic similarity. We experiment with four variations (seed, language, randomness, model) and three illustrative domains (from history, entertainment, and finance). Our findings show (i) high termination rates, though model-dependent; (ii) mixed reproducibility; and (iii) robustness that varies by perturbation type—high for seeds and temperature, lower for languages and models. These results suggest that LLM knowledge materialization can reliably surface core knowledge, while also revealing important limitations.

Details

Original languageEnglish
Title of host publication19th Conference of the European Chapter of the Association for Computational Linguistics, Findings of EACL 2026
PublisherAssociation for Computational Linguistics (ACL)
Pages2145-2164
Number of pages20
ISBN (electronic)979-8-89176-386-9
Publication statusPublished - Mar 2026
Peer-reviewedYes

Conference

Title19th Conference of the European Chapter of the Association for Computational Linguistics
Abbreviated titleEACL 2026
Conference number19
Duration24 - 29 March 2026
Website
LocationPalais des Congrès Rabat Bouregreg
CityRabat
CountryMorocco

External IDs

ORCID /0000-0002-5410-218X/work/222763835