With the rapid evolution of programmable network devices, the transformation of network infrastructure to a programmable computing platform has become a reality. As a result, such programmable networks make the deployment of in-network AI possible for achieving predictive maintenance, which in turn enables the much desired (sound-based) time-critical anomaly detection in modern smart factories. However, such abnormal sound detection systems face high computational complexity due to their high density and multi-source nature. In this paper, we present our novel You only hear once (Yoho) design for multi-target anomalous sound detection and end-to-end feature extraction. Yoho can detect anomalies directly from a mixed multi-source acoustic signal without recovering each source data. In addition, with a lightweight dual-path backbone design (Yoho-Dual-Path), we achieve low model complexity and high detection accuracy. Our experiments show that the proposed solution can reduce the inference time by 77% and improve the detection accuracy by 51%. This enables the integration of intelligent anomalous sound detection systems into the network.