Rainfall simulations are an established method to gain knowledge on small-scale hydrological processes like infiltration, ponding and the formation of surface runoff. Due to limitations in measuring methods, these processes must usually be understood to happen homogeneously within the bounded plot area while it is well known that they actually vary on a subplot scale. Within this study we took high resolution time-lapse images of several plots to observe and quantify the subplot processes of ponding and the formation of connectivity and surface runoff. We investigated the potential of deep learning in the segmentation of water ponding areas in time-lapse images during rainfall simulations and to estimate the ponding time. We trained three different Convolutional Neural Networks (CNNs), considering classification uncertainty and imbalance of the ground-truth data (water pixels) as well as ensemble modeling and spatial correlation between samples. Our findings suggest that addressing ground-truth annotation uncertainty and imbalance was more important in our study than the choice of the CNNs itself, and ensemble models increase the model performance leading to more robust predictions. Overall, our results suggest that CNNs have great potential to segment ponding areas, and thus it is possible to observe their spatio-temporal evolution. When comparing the evolution of water ponding areas to runoff, different behaviors across the plots were observable, which could be related to differences in initial soil moisture and infiltration behaviors. Further, our image-based deep learning approach allows for direct measurement of the ponding time and can be considered a first step to spatially and temporally resolved mapping of infiltration rates.