Hitchhikers' Guide to Masked Latent Semantic Modeling

TMLR Paper4699 Authors

Abstract:

Masked Latent Semantic Modeling (MLSM) is a recent pre-training objective which offers a sample efficient alternative to the use of Masked Language Modeling (MLM) for training encoder language models. In this paper, we identify and carefully evaluate previously unexplored important properties of MLSM pre-training. Based on the results of our rigorous experiments, we formulate a series of recommendations and best practices regarding MLSM pre-training. With these experiments, we also aim at advancing the understanding and proper use of MLSM pre-training by filling in important voids of previous empirical investigations.
We release our code for reproducing our experiments at \url{github.com/[MASK]}

Submission Length: Regular submission (no more than 12 pages of main content)

Assigned Action Editor: ~Yu_Meng1

Submission Number: 4699