During the life-cycle of constructions various influences induce material defects that could affect the behavior of the structural system. Therefore, anomalies that affect heavily stressed constructions, such as bridges, need to be inspected and evaluated regarding their impact on the structural capacity. By using new technologies in the field of damage detection, e.g. laser scanners or unmanned aircraft systems (UAS), this process can be facilitated. However, the classification and assessment of detected anomalies must still be performed in a manual way by human experts due to the lack of machine-processable evaluation methods. In this paper an approach is proposed towards a machine-based damage evaluation, applying semantic web technologies on a new developed method for damage detection on constructions. Thereby, anomalies are detected based on a large amount of high-resolution images from which georeferenced point clouds are calculated by using photogrammetric methods. Using the geometric relations among the image positions and reconstructed points, image features such as anomalies are localized on a 3 dimensional surface. Based on these image features, a web ontology as semantic representation of the recorded damages is generated and linked with an ontology that contains information about the affected construction and its environment. By using predefined rules based on expert knowledge, the detected anomalies are classified and assessed automatically. The inferred information is then used to generate damage representations in a structural analysis model. Furthermore, the geometrical data, which are represented in a model created according to Building Information Modeling (BIM) standards, the semantic data as well as the structural data are linked by utilizing the Multimodel approach.