Broadband coherent anti-Stokes Raman scattering (BCARS) is a powerful spectroscopy method combining high signal intensity with spectral sensitivity, enabling rapid imaging of heterogeneous samples in biomedical research and, more recently, in crystalline materials. However, BCARS encounters spectral distortion due to a setup-dependent non-resonant background (NRB). This study assesses BCARS reproducibility through a round robin experiment using two distinct BCARS setups and crystalline materials with varying structural complexity, including diamond, 6H-SiC, KDP, and KTP. The analysis compares setup-specific NRB correction procedures, detected and NRB-removed spectra, and mode assignment. We determine the influence of BCARS setup parameters like pump wavelength, pulse width, and detection geometry and provide a practical guide for optimizing BCARS setups for solid-state applications.