Fostering the Ecosystem of AI for Social Impact Requires Expanding and Strengthening Evaluation Standards

Bryan Wilder, Angela Zhou

Keywords: AI for social impact, evaluation

TL;DR: Researchers and reviewers in AI for Social Impact should adopt broader but more rigorous standards for impact

Abstract: There has been increasing research interest in AI/ML for social impact, and correspondingly more publication venues refining review criteria for practice-driven AI/ML research. However, these review guidelines tend to most concretely recognize projects that simultaneously achieve deployment and novel ML methodological innovation. We argue that this introduces incentives for researchers that undermine the sustainability of a broader research ecosystem of social impact, which benefits from projects that make contributions on one front (applied _or_ methodological) that may better meet project partner needs. Our position is that researchers and reviewers in machine learning for social impact must simultaneously adopt: 1) a more expansive conception of social impacts beyond deployment and 2) more rigorous evaluations of the impact of deployed systems.

Lay Summary: Researchers are increasingly interested in using AI to address social challenges. Right now, such projects tend to be highly valued when they simultaneously create novel AI methods and are deployed in practice. We argue that this way of evaluating research doesn't lead to work that best meets the needs of partner organizations or communities. Our position is that the field as a whole would be better off if evaluations of social impact research recognized a broader range of types of contributions while also adopting higher standards for judging whether deployments of AI were successful.

Submission Number: 550