Domain-specific Question Answering with Hybrid Search

Dewang Sultania, Franck Dernoncourt, Seunghyun Yoon

Abstract:

Domain-specific question answering is an evolving field that
requires specialized solutions to address unique challenges.
In this paper, we show that a hybrid approach—combining a
fine-tuned dense retriever with keyword-based sparse search
methods—significantly enhances performance. Our system
leverages a linear combination of relevance signals, including cosine similarity from dense retrieval, BM25 scores, and
URL host matching, each with tunable boost parameters. Experimental results indicate that this hybrid method outperforms our single-retriever system, achieving improved accuracy while maintaining robust contextual grounding. These
findings suggest that integrating multiple retrieval methodologies with weighted scoring effectively addresses the complexities of domain-specific question answering in enterprise
settings.