With increasing dexterity and robustness of robotic manipulation systems, their field of applications expands. Due to the still limited capabilities of autonomous agents and the requirement of fallback solutions in case of failure, teleoperation remains an essential functionality for systems that are far remote or in, for humans, inaccessible areas. Recently, a new control approach (TDPA-HD) was developed for teleoperation at extreme delays which ensures safe interactions, but, leads to conservative and late force application of the robot in its environment. In this work, we amalgamate the TDPA-HD with a model-augmentation approach to overcome this limitation and accelerate the force application without sacrificing the safety in the remote robotic interactions. To this end, we make use of a local virtual model of the remote environment which is pre-known or is sensed and created during runtime. Considering the energetic behavior of the haptic interaction of an operator with this local model as a reference for the remote interaction, the robot is allowed to apply interaction forces earlier when compared to pure TDPA-HD. At the same time, safe interactions in case of unexpected contacts with unmodeled objects in the remote environment are ensured. The method is introduced in 6-DoF and validated in 3-DoF experiments in rigid and elastic environments involving complex interaction tasks at up to 1.6 s roundtrip-delay.