Band structures of topological insulators are characterized by nonlocal topological invariants. Consequently, proposals for the experimental detection using local probes are rare. A recent paper [R.-J. Slager, Phys. Rev. B 92, 085126 (2015)PRBMDO1098-012110.1103/PhysRevB.92.085126] has argued, based on theoretical results for a particular class of models, that insulators with topologically trivial and nontrivial band structures in two space dimensions display a qualitatively different response to pointlike impurities. Here, we present a comprehensive investigation of the impurity response of a large set of models of noninteracting electrons on the honeycomb lattice, driven insulating by either broken inversion, broken time-reversal, broken C3, or broken translation symmetry. These cases include Hofstadter bands, strain-induced pseudo-Landau levels, and higher-order topological insulators. Our results confirm that for hopping models respecting the lattice symmetries, the response to a single impurity can indeed distinguish between trivial and nontrivial band topology. However, for modulated or inhomogeneous host systems we find that trivial states of matter can display an impurity response akin to that of topologically nontrivial states, and thus the diagnostic fails.