HealthSLM-Bench: Benchmarking Small Language Models for On-device Healthcare Monitoring

ICLR 2026 Conference Submission25511 Authors

Keywords: Small Language Models (SLMs), Mobile Health, On-Device LLMs

TL;DR: This paper benchmarks Small Language Models (SLMs) for mobile health applications, demonstrating efficient, privacy-preserving predictions on wearable devices, with performance comparable to larger language models (LLMs) in health task predictions.

Abstract:

On-device healthcare monitoring play a vital role in facilitating timely interventions, managing chronic health conditions, and ultimately improving individuals’ quality of life. Previous studies on large language models (LLMs) have highlighted their impressive generalization abilities and effectiveness in healthcare prediction tasks. However, most LLM-based healthcare solutions are cloud-based, which raises significant privacy concerns and results in increased memory usage and latency. To address these challenges, there is growing interest in compact models, Small Language Models (SLMs), which are lightweight and designed to run locally and efficiently on mobile and wearable devices. Nevertheless, how well these models perform in healthcare prediction remains largely unexplored.
We systematically evaluated SLMs on health prediction tasks using zero-shot, few-shot, and instruction fine-tuning approaches, and deployed the best performing fine-tuned SLMs on mobile devices to evaluate their real-world efficiency and predictive performance in practical healthcare scenarios. Our results show that SLMs can achieve performance comparable to LLMs while offering substantial gains in efficiency, reaching up to 17$\times$ lower latency and 16$\times$ faster inference speed on mobile platforms. However, challenges remain, particularly in handling class imbalance and few-shot scenarios. These findings highlight SLMs, though imperfect in their current form, as a promising solution for next-generation, privacy-preserving healthcare monitoring.

Primary Area: datasets and benchmarks

Submission Number: 25511