Accurate extraction of roof structures from aerial imagery is a critical step in the creation of detailed 3D models for digital heritage reconstruction. This study explores a hybrid methodology that combines prompt-based segmentation with structured vector reconstruction to enhance the extraction of roof skeletons from Very High Resolution (VHR) orthophotos. Using HEAT (Holistic Edge Attention Transformer) as the primary reconstruction model, we fine-tuned it on a domain-specific dataset containing representative gabled and hipped roofs to adapt to the unique geometries found in the city of Jena, Germany. To test whether prior roof isolation could improve reconstruction performance, we integrated mask outputs from two segmentation models — RobustSAM and LangSAM — into the HEAT pipeline. While segmentation offered visually precise results in several instances, overall evaluation revealed that prior segmentation did not consistently improve HEAT’s reconstruction accuracy. These findings underscore HEAT’s robustness and adaptability, especially when properly fine-tuned. Moreover, while SAM variants did not significantly boost performance here, their ease of use and potential for improvement through domain-specific fine-tuning suggest promising applications in other contexts.