Axial vectors, such as current or magnetization, are commonly used order parameters in time-reversal symmetry breaking systems. These vectors also break isotropy in three dimensional systems, lowering the spatial symmetry. We demonstrate that it is possible to construct a three-dimensional medium with average isotropy and inversion symmetry where time-reversal symmetry is systematically broken. We devise a model of an amorphous material with scalar time-reversal symmetry breaking, implemented by hopping through chiral magnetic clusters along the bonds. The presence of only average spatial symmetries—continuous rotation and inversion—is sufficient to protect a topological phase, yielding a statistical topological insulator. We demonstrate the topological nature of our model by constructing a bulk integer topological invariant for the effective continuum model, which guarantees gapless surface spectrum on any surface with an odd number of Dirac nodes, analogous to crystalline mirror Chern insulators. We also show the expected transport properties of a three-dimensional statistical topological in-sulator, which remains critical on the surface for odd values of the invariant.