Despite their outstanding performance across various NLP tasks, Large Language Models (LLMs) still produce incorrect answers, which can be harmful in safety-critical domains like medicine and autonomous driving. To address this issue, selective prediction systems aim to reject predictions from LLMs that are likely to be incorrect. However, current approaches either rely on querying the LLM multiple times, requiring access to its internals, or fine-Tuning it. Given the significant operational costs of an LLM, we propose a selective prediction system that does not involve the LLM during inference. We conduct an extensive experimental study regarding training data sizes, time consumption, utilized models, and embeddings, improving on the current state-of-The-Art while treating the LLM as a black box, without accessing its internals or requiring fine-Tuning.