Based on existing test suites like the ASHRAE 140 BESTEST and EN ISO 13791, a novel test suite for inter-model comparison of building energy simulation tools has been developed. The test cases investigate physical models and implementations individually and in combined tests, hereby identifying sources of deviations between tested tools. The methodology to build test cases incrementally and evaluate changes resulting from new features added to the tests, allows a much better isolation of error sources. The test suite is particularly focused on providing reference solutions for simulation tool developers. Another novel aspect of the test suite is a set of rules that ensures publication of models and simulation input data, such that results can be reproduced and verified easily by other users. Any software certified with the test suite can be considered to fulfill minimum requirements for the anticipated use cases. The article describes the test cases and variants formulated so far, and gives exemplary result for the tested tools NANDRAD, IDA-ICE, TRNSYS, TAS, Model-ica/AixLib, THERAKLES and ETU-Simulation. Key Innovations • Novel inter-model comparison test suite for modern building energy simulation tools • Test and evaluation criteria, including interactive quality tool compliance webpage Practical Implications The availability of independently verified and documented software tests may help increase confidence in numerical simulation tools. The standardization on physical problem description and model complexity increases comparability of results from different simulation tools.