Open-Source vs Close-Source: The Context Utilization Challenge

Litu Ou

Blogpost Url: https://d2jud02ci9yv69.cloudfront.net/2025-04-28-llm-context-utilization-73/blog/llm-context-utilization/

Abstract: This blog post aims to evaluate how well the most capable open-source long context large language models (LLMs) utilize context, using the Needle In A Haystack test. We adopt the task of chapter summarization for recently published books to minimize data contamination while ensuring a challenging test. Our results show that open-source models still have room to improve in context utilization compared to close-source models.

Conflict Of Interest: No conflict.

Submission Number: 40