This work summarizes our submission to ARQMath-3. We pre-trained Transformer-Encoder-based Language Models for the task of mathematical answer retrieval and employed a Transformer-Decoder Model for the generation of answers given a question from a mathematical domain. In comparison to our submission to ARQmath-2, we could improve the performance of our models regarding all three metrics nDGC’, mAP’ and p’@10 by refined pre-training and enlarged fine-tuning data. In addition, we improved our p’@10 results even further by additionally fine-tuning on annotated test data from ARQMath-2. In summary, our findings confirm that Transformer-based models benefit from domain adaptive pre-training in the mathematical domain.