Large language models (LLMs) are increasingly applied in the fields of mechanical engineering and materials science. As models that establish connections through the interface of language, LLMs can be applied for step-wise reasoning through the processing–structure–property–performance (PSPP) chain of materials science and engineering. Current LLMs are built for adequately representing a dataset, which consists of a significant subset of the accessible internet. However, the internet mostly contains nonscientific content. If LLMs should be applied for engineering purposes, it is valuable to investigate models for their intrinsic knowledge: the capacity to generate correct information about materials. In the current work, for the example of the Periodic Table of Elements, the role of vocabulary and tokenization for the uniqueness of material fingerprints and the LLMs’ capabilities of generating factually correct output of different state-of-the-art open models are highlighted. This leads to a material knowledge benchmark for an informed choice, for which steps in the PSPP chain LLMs can be applied and where specialized models are required.