An Improved Method of Automated Nonparametric Content Analysis for Social Science

Connor Thomas Jerzak, Gary King

Abstract:

Some scholars build models to classify documents into chosen categories. Others,
especially social scientists who tend to focus on population characteristics, instead
usually estimate the proportion of documents in each category — using either para-
metric “classify-and-count” methods or “direct” nonparametric estimation of propor-
tions without individual classification. Unfortunately, classify-and-count methods
can be highly model dependent or generate more bias in the proportions even as the
percent of documents correctly classified increases. Direct estimation avoids these
problems, but can suffer when the meaning of language changes between training
and test sets or is too similar across categories. We develop an improved direct es-
timation approach without these issues by including and optimizing continuous text
features, along with a form of matching adapted from the causal inference literature.
Our approach substantially improves performance in a diverse collection of 73 data
sets. We also offer easy-to-use software that implements all ideas discussed herein