Latent Space Steering for Controllable Rare Pedestrian Trajectory Generation

Ananya Arvind

Keywords: latent space steering, trajectory prediction, rare behavior generation, autonomous driving simulation

Abstract: Simulation-based testing of autonomous vehicles depends critically on diverse, high-fidelity scenario coverage, yet real-world pedestrian trajectory datasets are dominated by routine, low-activity motion, leaving the tail-event behaviors most relevant for safety assessment severely underrepresented. We investigate latent space steering as a lightweight, retraining-free approach to high-fidelity tail-event simulation: at inference time, we add a learned risk-direction vector to a trajectory encoder's latent representation to controllably shift generated pedestrian behaviors toward rare, high-activity modes without training any additional generative components. Using the Stanford Drone Dataset (SDD), we train and compare two sequential architectures---an LSTM and a Transformer trajectory predictor---and show both substantially outperform a constant-velocity baseline (LSTM: $-48.0\%$ ADE, Transformer: $-58.3\%$ ADE on the held-out test set). Linear probing of the learned latent spaces reveals a striking $2.55 \times$ gap in how linearly each model encodes behavioral risk: Transformer $R^2 = 0.808$ vs. LSTM $R^2 = 0.316$. Structured latent steering outperforms random perturbation by $+0.159$ risk units (Transformer) and $+0.141$ (LSTM) at steering magnitude $\alpha =1.5$, while maintaining physical plausibility above $ 94 \% $ throughout. Our central findings propose latent space steering as a lightweight, retraining-free method for controllable rare-behavior generation in AV simulation, and introduce latent probing $R^2$ as a complementary evaluation metric that captures representation quality invisible to standard ADE/FDE benchmarks. Our behavioral risk score is a kinematic activity proxy derived from speed, acceleration, and turn rate; extending this to scene-aware or learned danger scores is an important direction for future work.

Submission Number: 17