On more accurate alignment modeling methods for automatic speech recognition

Albert Zeyer, Tina Raissi, Ralf Schlüter, Hermann Ney

Keywords: speech recognition, CTC, HMM, AED, alignment accuracy, full sum, peaky behavior, separated blank, alignment by input gradient

TL;DR: Renormalizing gradients for class rebalancing; separating blank in CTC; using gradients wrt. inputs to get an alignment.

Abstract:

The connectionist temporal classification (CTC) training criterion
optimizes the conditional log probability of the label sequence given the input,
which involves a sum over all possible alignment label sequences including blank.
It is well known that CTC training leads to peaky behavior
where blank is predicted in most frames and the labels are focused mostly on single frames.
Thus, CTC is suboptimal to obtain accurate word boundaries.
Hidden Markov models (HMMs) can be seen as a generalization of CTC
and trained in the same way with
a generalized training criterion,
and may lead to similar problems.
Label units such as subword units and its vocabulary size or phoneme-based units
also significantly impact the alignment quality.
Here we study different methods of obtaining an alignment
with the goals
to improve alignment quality
while keeping a good performing model,
and to gain better understanding of the training dynamics.
We introduce
(1) a synthetic framework to study alignment behavior,
and compare various models, noise and training conditions,
(2) a new training variant with renormalizing the gradients to counteract the class imbalance of blank,
(3) a novel CTC model variation to use a hierarchical softmax and separating the blank label in CTC,
as another alternative to counteract class imbalance,
(4) a novel way to get alignments via the gradients
of the label log probabilities w.r.t. the input features.
This method can be used for all kinds of models,
and we evaluate it for CTC and attention-based encoder-decoder (AED) subword based models
where it performs competitive and more robustly,
although phoneme-based HMMs still provide the best alignments.

Primary Area: applications to computer vision, audio, language, and other modalities

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 11977