The process for automatically creating 3D city models from contemporary photographs and visualizing them on mobile devices is well established. 4D city models that can display a temporal dimension are far more complex to generate automatically. In this article, we focus on major challenges in the process of developing an automated pipeline, starting from content-based image retrieval applied to historical images, via automatic historical image orientation, up to visualization of the 4D data in Virtual Reality (VR). The result is an interactive browser-based device-rendered 4D visualization and information system for mobile devices. This pipeline has been in development since 2015. In this article, we present initial results and early-stage findings in the process of overcoming three major challenges on the way to 4D city models: (1) to identify photographs with corresponding views, (2) to reconstruct the position and orientation of photographs and (3) to design a user-centered, browser-based 4D mobile application.