Designing LLM-Based Support for Homelessness Caseworkers

Whitney Nelson, Min Kyung Lee, Eunsol Choi, Victor Wang

Track Selection: Track 1: Developing LLM-powered tools for positive outcomes

Keywords: Large Language Models (LLMs), Homelessness Casework, AI in Public Sector, Text Summarization, User-Centered AI Design, Prompt Engineering, Domain-Specific LLM

TL;DR: This paper explores the challenges of using of LLMs to assist caseworkers in case note summarization due to the diversity in summary standards and identifies further work in prompt engineering, evaluation metrics, and summary verification mechanisms.

Abstract: Our research explores the use of Large Language Models (LLMs) to assist social service delivery for people experiencing homelessness. We follow a human-centered design approach and work with caseworkers who provide homelessness-intensive case management. They identified summarization of case notes as an opportunity to better understand their clients. We then asked caseworkers to generate summaries in order to understand what information summaries need to contain. However, instead of resulting in ``gold standard'' summaries that can be used to train LLMs, we found that there were diversities in summaries that they wanted to see depending on their case management philosophies and roles. (We report our ongoing exploration of LLM summaries to support the diversities.) We also discovered that enabling workers to verify the summary is a key issue. We share implications that summaries should be able to support caseworkers' diverse goals, which requires future work on prompting engineering, evaluative metrics, and summary verification mechanisms.

Submission Number: 15