Numerous digital media repositories have been set up during recent decades, each containing plenty of data about historic cityscapes. In contrast, digital 3D reconstructions of no longer extant cityscapes have so far almost solely been performed for specific cases and via manual modelling techniques. Within the Jena4D research group, we are investigating and developing methods and technologies for transferring historical media and their contextual information into 4D models. The next step will be to automate this process and extend it to the world scale. Our team are working on different facets of that endeavor: retrieving images, text, and object data from public sources; crowdsourcing and collecting qualitative data from contests and pupil labs; processing historical plans and photographs to extract spatial data; location-based mobile visualization; and collection-browsing interfaces. This article is intended to highlight (1) current challenges, (2) the state of the art, (3) our approach to 4D modelling and visualization on the world scale, and (4) results from testing demo cases in Dresden, Jena, and Amsterdam.