Large Language Models as Medical Codes Selectors: a benchmark using the International Classification of Primary Care

Vinicius Anjos de Almeida, Vinicius de Camargo, Raquel Gomez Bravo, Kees van Boven, Egbert van der Haring, Marcelo Finger, Luis Fernandez Lopez

Keywords: Large Language Models (LLMs), Medical Coding, International Classification of Primary Care (ICPC-2), Semantic Search, Information Retrieval, Benchmark, Natural Language Processing (NLP), Extreme Multi-label Classification (XMC), Zero-shot Learning, Prompt Engineering, Brazilian Portuguese, Healthcare Informatics

TL;DR: This paper benchmarks 33 large language models on their ability to select correct ICPC-2 medical codes from a list of semantic search results, demonstrating that many models have strong potential to automate this task even without fine-tuning.

Abstract:

Background: Medical coding structures healthcare data for research, quality monitoring, and policy. This study assesses the potential of large language models (LLMs) to assign ICPC-2 codes using the output of a domain-specific search engine.

Methods: A dataset of 437 Brazilian Portuguese clinical expressions, each annotated with ICPC-2 codes, was used. A semantic search engine (OpenAI's text-embedding-3-large) retrieved candidates from 73,563 labeled concepts. Thirty-three LLMs were prompted with each query and retrieved results to select the best-matching ICPC-2 code. Performance was evaluated using F1-score, along with token usage, cost, response time, and format adherence.

Results: Twenty-eight models achieved F1-score > 0.8; ten exceeded 0.85. Top performers included gpt-4.5-preview, o3, and gemini-2.5-pro. Retriever optimization can improve performance by up to 4 points. Most models returned valid codes in the expected format, with reduced hallucinations. Smaller models (<3B) struggled with formatting and input length.

Conclusions: LLMs show strong potential for automating ICPC-2 coding, even without fine-tuning. This work offers a benchmark and highlights challenges, but findings are limited by dataset scope and setup. Broader, multilingual, end-to-end evaluations are needed for clinical validation.

Submission Number: 2