LISA: Latent Inference on the Sphere for A-posteriori sampling

Abstract: Deterministic $\textit{autoencoders}$ (AEs) train stably and reconstruct sharply, but unconditional generation is typically delegated to an ex-post latent density fit (e.g., MVG/GMM) followed by decoding sampled codes. We argue that this pipeline is brittle because reconstruction training does not define a well-conditioned sampling interface in latent space. We support this with two diagnostics: $\textbf{(i)}$ deterministic decoders can be strongly sensitive to latent radius, so decoding can be poorly conditioned along an otherwise unconstrained radial degree of freedom, an effect we quantify via a controlled radial-scaling test in latent space. This implies that any ex-post sampler that perturbs radius can provoke large output changes in simple Euclidean AEs. Secondly, the $\textbf{(ii)}$ learned latent representations can exhibit strong directional concentration, revealed by a spiky directional second-moment spectrum (low effective rank / large leading eigenvalue), making simple density models poorly calibrated for sampling. Motivated by these failure modes, we propose $\textit{Latent Inference on the Sphere for A-posteriori sampling}$ (LISA), a deterministic autoencoder that projects codes onto the hypersphere to remove latent scale and adds a repulsive hyperspherical kernel energy regularizer to promote directional coverage. Combined with projected MVG/GMM samplers, LISA yields a simple plug-in prior for unconditional generation. Across standard image benchmarks and structured generation tasks, this approach achieves lower $\textit{Frechet inception distance}$ (FID) among a broad suite of autoencoder baselines, without stochastic encoders, adversarial training, or learned priors, highlighting latent-geometry conditioning as an effective design principle for deterministic generative modelling. We also introduce $\textit{Grammar-LISA}$ for structured data (arithmetic expressions), demonstrating that the approach extends beyond image data.

Submission Type: Long submission (more than 12 pages of main content)

Assigned Action Editor: ~Yi_Liu12

Submission Number: 7493