Foundations of LLM Knowledge Materialization: Termination, Reproducibility, Robustness
Research output: Contribution to book/Conference proceedings/Anthology/Report › Conference contribution › Contributed › peer-review
Contributors
Abstract
Large Language Models (LLMs) encode substantial factual knowledge, yet measuring and systematizing this knowledge remains challenging. Converting it into structured format—for example through recursive extraction approaches such as the GPTKB methodology (Hu et al., 2025)—is still underexplored. Key open questions include whether such extraction can terminate, whether its outputs are reproducible, and how robust they are to variations. We systematically study LLM knowledge materialization using miniGPTKBs (domain-specific, tractable subcrawls), analyzing termination, reproducibility, and robustness across three categories of metrics: yield, lexical similarity, and semantic similarity. We experiment with four variations (seed, language, randomness, model) and three illustrative domains (from history, entertainment, and finance). Our findings show (i) high termination rates, though model-dependent; (ii) mixed reproducibility; and (iii) robustness that varies by perturbation type—high for seeds and temperature, lower for languages and models. These results suggest that LLM knowledge materialization can reliably surface core knowledge, while also revealing important limitations.
Details
| Original language | English |
|---|---|
| Title of host publication | 19th Conference of the European Chapter of the Association for Computational Linguistics, Findings of EACL 2026 |
| Publisher | Association for Computational Linguistics (ACL) |
| Pages | 2145-2164 |
| Number of pages | 20 |
| ISBN (electronic) | 979-8-89176-386-9 |
| Publication status | Published - Mar 2026 |
| Peer-reviewed | Yes |
Conference
| Title | 19th Conference of the European Chapter of the Association for Computational Linguistics |
|---|---|
| Abbreviated title | EACL 2026 |
| Conference number | 19 |
| Duration | 24 - 29 March 2026 |
| Website | |
| Location | Palais des Congrès Rabat Bouregreg |
| City | Rabat |
| Country | Morocco |
External IDs
| ORCID | /0000-0002-5410-218X/work/222763835 |
|---|