Railway Track Defect Detection

Augustin Crespin, Ioannis Kostis, Pierre Schaus

Track: Type E (Late-Breaking Abstracts)

Keywords: Railway defect detection, YOLO, Dynamic Time Warping, Retrieval-Augmented Generation (RAG)

Abstract:

The Track Sentry Vision (TSV) project, funded by the Walloon Region, aims to develop a low-cost, autonomous preventative maintenance system tailored for railway sectors in developing countries, with a focus on Africa. Current maintenance practices in these regions are often reactive and costly, relying heavily on corrective interventions. TSV seeks to transform this approach by introducing an autonomous detection system for track defects, designed to integrate seamlessly with commercial trains and operate within the constraints of limited resources. The system’s core objective is to predict defects through continuous track condition analysis, thereby reducing unplanned downtime and prioritizing maintenance interventions based on real-time data.

The project addresses two critical data-science challenges. First, data signal alignment is required to compare multimodal sensor and video streams captured during different runs, where variations in train speed and environmental conditions introduce temporal misalignments. Second, self-supervised anomaly detection must identify defects and obstacles without relying on pre-labeled datasets, which are typically unavailable or impractical to collect in real-world railway environments. To tackle these challenges, we designed a proof-of-concept laboratory setup using a remote-controlled LEGO train equipped with a Raspberry Pi 5, an accelerometer, and a camera. This scalable platform allowed us to test and refine our algorithms in a controlled yet representative setting.

For signal alignment, we applied Dynamic Time Warping (DTW) to accelerometer data collected during runs at varying speeds. DTW successfully aligned the time-series signals by stretching and compressing segments, enabling accurate comparison of track conditions across runs despite speed differences.

For anomaly detection, we explored two complementary approaches: self-supervised feature extraction using DINOv3, and fine-tuning a lightweight YOLOv11n model. The YOLO model, trained on manually annotated video frames, achieved an accuracy of 0.93, precision of 0.95, recall of 0.98, and an F1-score of 0.96, demonstrating robust performance in identifying artificial obstacles with minimal false positives. Meanwhile, DINOv3’s self-supervised features enabled effective clustering of anomalous frames, such as those containing unexpected objects or track irregularities.

The detection pipeline is designed to integrate with a broader maintenance management framework using a Model Context Protocol (MCP) architecture. An Agentic RAG backend will handle the retrieval of technical documentation and maintenance records and allow interaction between these and the end-user, while detection outputs will be directed to a dedicated analytics module for visualization and explainable reporting. This modular design ensures scalability and decouples the detection system from documentation management, allowing for flexible deployment.

Preliminary results from our lab experiments confirm the feasibility of DTW for time-series alignment and the effectiveness of self-supervised and vision-based methods for anomaly detection. However, further validation is required to assess the robustness and scalability of these approaches in real-world operational contexts. The use of a LEGO train as a proof of concept underscores the adaptability of our solution to low-resource settings, where full-scale testing may initially be challenging. Future work will focus on field validation, algorithm optimization for edge deployment, and expanding the Agentic RAG subsystem to support data-driven decision-making. Ultimately, TSV aims to provide a scalable, cost-effective solution that enhances railway safety and operational efficiency in developing countries.

Submission Number: 93