Bismuth is gaining importance as a key element of functional quantum materials. The effects of spin–orbit coupling (SOC) are at the heart of many exciting proposals for next-generation quantum technologies, including topological materials for efficient information transmission and energy-saving applications. The “heavy” element bismuth and its compounds are predestined for SOC-induced topological properties, but materials design is challenged by a complex link between them and the chemical composition and crystal structure. Nevertheless, a lot can be learned about a certain property by testing its limits with compositional and/or structure modifications. We survey a handful of topological bismuth-based materials that bear structural and chemical semblance to the early topological insulators, antimony-doped elemental bismuth, Bi2Se3 and Bi2Te3. Chemical bonding via p orbitals and modular structure underlie all considered bismuth chalcogenides, subhalides, and chalcogenide halides and allow us to correlate the evolution of chemical bonding and structure with variability of the topological properties, although materials design should not be regarded as a building blocks set. Over the past decade, material discoveries have unearthed a plethora of topological properties, and bismuth is very fertile as a progenitor of a rich palette of exotic quantum materials, ranging from strong and weak 3D and crystalline topological insulators over topological metals and semimetals to magnetic topological insulators, while preserving the general layered structure motif.