The increasing number of passengers and services using railways and the corresponding increase in rail use has caused the acceleration of rail wear and surface defects which makes rail defect identification an important issue for rail maintenance and monitoring to ensure safe and efficient operation. Traditional visual inspection methods for identifying rail defects are time-consuming, less accurate, and associated with human errors. Deep learning has been used to improve railway maintenance and monitoring tasks. This study aims to develop a structured model for detecting railway artifacts and defects by comparing different deep-learning models using ultrasonic image data. This research showed whether it is practical to identify rail indications using image classification and object detection techniques from ultrasonic data and which model performs better among the above-mentioned methods. The methodology includes data processing, labeling, and using different conventional neural networks to develop the model for both image classification and object detection. The results of CNNs for image classification, and YOLOv5 for object detection show 98%, and 99% accuracy respectively. These models can identify rail artifacts efficiently and accurately in real-life scenarios, which can improve automated railway infrastructure monitoring and maintenance.