Breaking Points: How Transformer Vulnerabilities Reveal Paths to Faster Inference

Agents4Science 2025 Conference Submission277 Authors

Keywords: Transformer Models, Noise Robustness, Layer-wise Analysis, Model Vulnerability, RoBERTa, ELECTRA, BERT, Performance Degradation, Linguistic Processing, Gradient Dynamics, Layer Dropout, Model Efficiency, Noisy Inputs, OCR Errors

TL;DR: We analyze how and where noisy text breaks Transformer models layer-by-layer, finding specific vulnerability points that can be used to make them more robust and efficient.

Abstract: Transformer models exhibit significant performance degradation when exposed to noisy inputs, yet the mechanisms underlying this vulnerability remain poorly understood. We present a comprehensive layer-wise analysis of noise robustness across encoder architectures using 52,500 controlled evaluations (2,100 samples × 5 models × 5 noise types), plus 7,000 real-world validation samples from OCR errors and social media text. Our analysis identifies consistent vulnerability transitions at layers 3 and 8 in 12-layer encoders, marking boundaries between linguistic processing phases: surface features (79% robustness retention), syntactic structure (52% robustness under syntax-specific noise), and semantic encoding (67% robustness retention). RoBERTa maintains 0.787 robustness score where ELECTRA retains only 0.607, with real-world noise proving 15-20% relatively more challenging than synthetic perturbations. Runtime measurements confirm that strategic layer dropout achieves 1.28x actual speedup (1.31x at batch=32) while preserving 92% of the original robustness score (0.92 retention ratio). Cross-model analysis reveals 69.3% average correlation in vulnerability patterns when compared to BERT baseline, with the remaining variance explained by architecture-specific gradient dynamics. We empirically observe that phase transitions align with mutual information inflection points and gradient norm peaks of 1.83x ± 0.12. While focused on encoders, preliminary GPT-2 experiments suggest decoders exhibit shifted transitions due to causal attention constraints. These findings enable practical deployment optimizations and inform the design of robust, efficient transformer architectures.

Submission Number: 277