CausalCite: A Causal Formulation of Paper Citations

Anonymous

Abstract: Evaluating the significance of a paper is pivotal yet challenging for the scientific community. While the citation count is the most commonly used proxy for this purpose, they are widely criticized for failing to accurately reflect a paper’s true impact. In this work, we propose a causal inference method, TextMatch, which adapts the traditional matching framework to high-dimensional text embeddings. Specifically, we encode each paper using the text embeddings by large language models, extract similar samples by cosine similarity, and synthesize a counterfactual sample by the weighted average of similar papers according to their similarity values. We apply the resulting metric, called CausalCite, as a causal formulation of paper citations. We show its effectiveness on various criteria, such as high correlation with paper impact as reported by scientific experts on a previous dataset of 1K papers, and test-of-time awards for past papers, and sub-field stable behavior. We also provide a set of findings that can serve as suggested ways for future researchers to use our metric for a better understanding of a paper’s quality.

Paper Type: long

Research Area: Machine Learning for NLP

Contribution Types: Data analysis, Theory

Languages Studied: English