This study aims to evaluate sequences of raw time series using an autoencoder structure for unsupervised damage detection and localization under varying environmental conditions (ECs). When it comes to structural health monitoring (SHM) for real-world applications, data-driven models need to improve sensitivity and robustness toward damage due to the EC-dependent variance. For systems situated outdoors, changing ECs affects the stiffness properties without causing permanent alterations to the structure. Applying data normalization strategies to consider these natural variations is not easy to conduct and is unfavorable for sensitivity regarding damage. To address these challenges, the model’s input variables are non-standardized to avoid input-related modifications and to feature a higher sensitivity toward structural changes. The autoencoder’s ability to capture structural variations caused by ECs and to handle non-standardized time series data makes it favorable for real-world applications. By quantifying the input-residual correlations, sensitivity, and robustness can be improved; no adjustments to the model have to be made. The autoencoder’s black-box nature is inspected by analyzing a linear dynamic 8DOF system and the Leibniz University Structure for Monitoring (LUMO). The neural network’s structure is identified by tracking the residual correlation. Here, a common test statistic of a whiteness test is used to find an optimal choice of the bottleneck dimension. Significantly increased robustness and sensitivity toward damage when evaluating the input-residual correlations instead of the reconstruction error is observed. To capture the temperature-dependent structural response for experimental validation, 10-min data sets of different structural temperatures are given to the neural network during training. It was derived that for damage detection, an amplitude-related normalization is inevitable due to the different excitation intensities in real life, which was carried out using input-residual correlations quantified by a Pearson coefficient. Considering the results obtained, autoencoders with non-standardized time series and input-residual correlations demonstrate a potent tool for vibration-based damage identification.