Time-delay in bilateral teleoperation systems often leads to passivity violations, necessitating the use of stability controllers. Time domain passivity approach has garnered attention within the haptics and tele-robotics field, due to demonstrating significant robustness while being less conservative compared to other passivity-based approaches. However, it's affected by two types of artifacts, namely position drift and high-frequency force vibration. This paper presents a perceptual-based scaling method to mitigate such vibrations, which would otherwise adversely affect transparency and user perception. The proposed method uses Weber's law to leverage human haptic perception thresholds to scale the delayed feedback force during the energy- out phase on the human operator side. By selectively scaling the feedback force within the human just-noticeable difference (JND) threshold, the activation of impedance-type passivity controller is reduced while maintaining overall system passivity. Experimental validation using a 1-DoF teleoperation system with 200 ms and 500 ms round-trip delay, demonstrates a significant reduction in high-frequency force vibrations.