On the Depth between Beam Search and Exhaustive Search for Text Generation

Anonymous

TL;DR: Analysis of the lookahead strategy of beam search showing there may be room for improvement by searcher deeper.

Abstract:

Beam search and exhaustive search are two extreme ends of text decoding algorithms with respect to the search depth.
Beam search is limited in both search width and depth, whereas exhaustive search is a global search that has no such limitations.
Surprisingly, beam search is not only computationally cheaper but also performs better than exhaustive search despite its higher search error.
Plenty of studies have reported that moderate search widths work the best, but little has been investigated regarding the search depth.
Based on the success of the moderate beam width, we examine a range of search depths to see its effect on performance.
To this end, we introduce Lookahead Beam Search (LBS), a multi-step lookahead search that optimizes the objective considering a fixed number of future steps.
Beam search and exhaustive search are special cases of LBS where the lookahead depth is set to $0$ and $\infty$, respectively.
We empirically evaluate LBS with the lookahead depth of up to 3 and show that it improves upon beam search.
Although LBS is not a practical algorithm on its own because of its computational complexity, the results indicate that beam search with moderate widths still has room for improvement by searching deeper.

Paper Type: long

Research Area: Machine Translation

Contribution Types: Model analysis & interpretability

Languages Studied: English, French, German