A Preliminary Study of Identifying Housing Outcomes from Casenotes Using Large Language Models
Keywords: humanities & computational social science, causality, information retrieval, human-in-the-loop machine learning
TL;DR: We explore the use of large language models to annotate unstructured case notes and combine them with expert labels to evaluate the causal impact of street outreach efforts.
Abstract: In collaboration with a nonprofit organization providing homelessness services in New York, we explore the use of machine learning and statistical analysis to evaluate the impact of street outreach efforts. Assessing causal effects presents significant challenges, particularly when outcomes are missing. The first step of this work is to obtain outcome labels. While the ideal gold standard would be to have expert annotations for the entire dataset, that can be very expensive. In this preliminary study, we investigate using large language models (LLMs) to obtain these labels from unstructured case notes from street outreach teams. We compare the accuracy of LLMs when it comes to predicting human labels for four critical outcomes in street outreach. We aim for this study to serve as a proof of concept. In future work, we would like to expand on this evaluation effort and demonstrate how expert labels and LLM annotations can be combined strategically and used for causal effect estimation and evidence-based policy-making with limited data.
Submission Number: 14
Loading