This paper presents two different strategies for the exploitation of historical photographs and maps and their usage in a three-dimensional (3D) and four-dimensional (4D) space. In the first approach, over 4000 historical photographs of Jena are collected via a citizen competition. These images are filtered in a semi-automatic way and processed in an adapted Structure-from-Motion workflow leading to multiple historical 3D models. As an innovative approach, two of the resulting reconstructions are refined using two different Neural Rendering approaches. For the first time, this enables the detailed seamless 3D visualization of sparse historical datasets, including defunct buildings. In cases where no photographs of buildings are available, an alternative strategy is demonstrated using historical maps. With an increasing amount of digitized historical maps, it becomes possible to segment and vectorize the building footprints for different points in time. The approach uses a semi-automatic workflow where a part of a historical map of Jena in 1936 is labeled manually in order to derive the remaining footprints automatically. At the moment, this labor-intensive step still has to be transferred to other historical maps to generate varying simple building models for different points in time. A combination of both approaches will allow the generation of detailed urban 4D models at a larger scale.