In the surgical domain, individual clinical experience, which is derived in large part from past clinical cases, plays an important role in the treatment decision process. Simultaneously the surgeon has to keep track of a large amount of clinical data, emerging from a number of heterogeneous systems during all phases of surgical treatment. This is complemented with the constantly growing knowledge derived from clinical studies and literature. To recall this vast amount of information at the right moment poses a growing challenge that should be supported by adequate technology. While many tools and projects aim at sharing or integrating data from various sources or even provide knowledge-based decision support - to our knowledge - no concept has been proposed that addresses the entire surgical pathway by accessing the entire information in order to provide context-aware cognitive assistance. Therefore a semantic representation and central storage of data and knowledge is a fundamental requirement. We present a semantic data infrastructure for integrating heterogeneous surgical data sources based on a common knowledge representation. A combination of the Extensible Neuroimaging Archive Toolkit (XNAT) with semantic web technologies, standardized interfaces and a common application platform enables applications to access and semantically annotate data, perform semantic reasoning and eventually create individual context-aware surgical assistance. The infrastructure meets the requirements of a cognitive surgical assistant system and has been successfully applied in various use cases. The system is based completely on free technologies and is available to the community as an open-source package.