The Asset Administration Shell (AAS) enables semantic interoperability in Industry 4.0, yet the growing number of Submodel Templates (SMTs) raises concerns about their semantic quality. Due to their interdisciplinary creation, SMTs often contain inconsistencies and ambiguities, especially in open domains. However, existing validation methods focus on syntactic checks and fail to capture deeper semantic issues. To address this issue, this paper investigates the use of Large Language Models (LLMs) for improving SMT quality. We examine whether LLMs can detect semantic inconsistencies beyond traditional approaches and how their performance can be evaluated resulting in an LLM-based validation tool. Additionally, we contribute an expert-curated dataset of AAS submodels with injected semantic issues to verify such tools.