Go to ICLR 2026 Conference homepageSelf-Diagnosing Neural Networks: A Causal Framework for Real-Time Anomaly Detection in Training Dynamics
Keywords: Training dynamics, Real-time anomaly detection, Self-supervised learning, Masked autoencoders, Spatiotemporal modeling, Open-set recognition, Uncertainty quantification
TL;DR: Activations and gradients are recast as a spatiotemporal video, enabling a causal, open-set diagnostician that detects and classifies training failures in real time, decisively outperforming scalar-curve baselines.
Abstract: Monitoring neural network training via scalar curves can obscure early indicators of failure in high-dimensional optimization dynamics. This work studies a framework that treats training as a spatiotemporal signal, converting sequences of internal activations and gradients into internal-state videos. A Dynamics Masked Autoencoder (DynamicsMAE) is pretrained on these videos to learn dynamics-aware representations, and a Temporal Vision Diagnostician (TeViD) with an evidential classification head is then fine-tuned for streaming, open-set diagnosis of training runs under a past-only constraint. Given a short window of internal-state frames, TeViD predicts diagnostic labels (Healthy, Overfitting, Instability, Catastrophic Forgetting, Concept Bias) and can abstain via an Unknown category. Evaluation uses time-to-detect, event-time AUPRC, risk--coverage analysis, and a simple decision-theoretic cost model. On >500 held-out runs from a curated data-factory benchmark with factorially held-out architectures, datasets, optimizers, and anomaly types, the method attains an event-time AUPRC of $0.96 \pm 0.01$ and issues alerts a median of $6.2$ epochs earlier than scalar rule-based baselines at a fixed $5\$% false-alarm rate, suggesting that internal-state video diagnosis can serve as a useful training-time signal alongside existing Machine Learning Operations (MLOps) tools on this benchmark.
Primary Area: learning on time series and dynamical systems
Submission Number: 18440
Loading