Basilectal-Inspired Health Questions Expose Robustness Gaps in Small Medical QA Models

Saiankit Shankar, BHVSP Subrahmanyam, Neeraj Choudhary

Keywords: Basilectal-Inspired English, Medical Question Answering, Language Robustness, Fluent Incompleteness

TL;DR: Basilectal-inspired health queries expose medical concept loss in small QA models.

Abstract: Small local language models are attractive for medical question answering in settings where connectivity, cost, and privacy limit cloud-based use. Yet users in these settings may express health questions in forms that move away from benchmark-standard English, including compressed, phonetic, and basilectal-inspired phrasing. We present a controlled robustness study using 102 TREC LiveQA Medical questions, paired L0–L3 synthetic basilectal-inspired variants, six small instruction-tuned models, and 7,416 free-form generations. Across the baseline setting, answer quality is stable under mild variation but degrades sharply in the L3 basilectal-inspired condition: BERTScore-style F1 falls from 0.707 at L0 to 0.614 at L3, ROUGE-L from 0.136 to 0.095, and Medical Concept Overlap recall from 0.129 to 0.046. The dominant failure mode is fluent incompleteness: models often continue producing plausible medical text while preserving fewer medically relevant concepts from trusted reference answers. We also examine prompt intervention, quantization, feature ablations, behavioral proxies, and claim-level support. The study is intentionally bounded: it is a controlled robustness benchmark, not a clinical safety certification, not a naturally collected dialect corpus, and not a claim about any specific speech community. Code is available at https://github.com/ankitsblade/Basilectal-Robustness-in-SLMs

Paper Type: Long Paper

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 12