Van der Waals materials exhibit intriguing properties for future electronic and optoelectronic devices. As those unique features strongly depend on the materials' thickness, it has to be accessed precisely for tailoring the performance of a specific device. In this study, a nondestructive and technologically easily implementable approach for accurate thickness determination of birefringent layered materials is introduced by combining optical reflectance measurements with a modular model comprising a 4×4 transfer matrix method and the optical components relevant to light microspectroscopy. This approach is demonstrated being reliable and precise for thickness determination of anisotropic materials like highly oriented pyrolytic graphite and black phosphorus in a range from atomic layers up to more than 100 nm. As a key feature, the method is well-suited even for encapsulated layers outperforming state of-the-art techniques like atomic force microscopy.