Railway damage detection is of great significance in ensuring railway safety. The cracks on the rail surface play a key role in studying the formation and development process of rail damage, predicting the occurrence of rail defects, and then improving the service life of the rail. However, due to the small shape of the cracks, the typical detection method is relatively complicated, and the speed is quite slow. Although traditional magnetic particle inspection technology is fairly accurate at detection, it is costly and inconvenient to carry and install, while also limiting the detection speed and affecting the system’s operation. In this paper, a semantic segmentation detection method is developed by using various collected rail surface crack data and deep learning through a neural network. By comparing the inspection of the same rail surface with magnetic particle inspection technology, only inexpensive cameras are used and the inspection speed is increased while maintaining relatively high accuracy. In addition, the method can achieve fast detection speeds if it is extended to be combined with high-frequency cameras. It is an economical, efficient, and environmentally friendly method for future rail surface detection.