Streaming algorithms for evaluating noisy judges on unlabeled data - binary classification.

Andrés Corrada-Emmanuel

Keywords: unlabeled data, evaluation, ensembles, stream algorithms, algebraic geometry

TL;DR: The failures of an independent stream evaluator for binary classifiers are used to find and monitor nearly independent ensembles.

Abstract:

The evaluation of noisy binary classifiers on unlabeled data is treated as a
streaming task - given a data sketch of the decisions by an ensemble, estimate
the true prevalence of the labels as well as each classifier's accuracy on them.
Two fully algebraic evaluators are constructed to do this. Both are based on the assumption that
the classifiers make independent errors on the test items. The first is based on
majority voting. The second, the main contribution of the paper, is guaranteed
to be correct for independent classifiers. But how do we know the classifiers
are error independent on any given test? This principal/agent monitoring paradox
is ameliorated by exploiting the failures of the independent evaluator to
return sensible estimates. Some of these failures can be traced to producing
algebraic versus real numbers while evaluating a finite test. A
search for nearly error independent trios is empirically carried out on the
\texttt{adult}, \texttt{mushroom}, and \texttt{twonorm} datasets by using
these algebraic failure modes to reject potential evaluation ensembles as
too correlated. At its final steps, the searches are refined by constructing
a surface in evaluation space that must contain the true value point.
The surface comes from considering the algebra of arbitrarily correlated
classifiers and selecting a polynomial subset that is free of any correlation variables.
Candidate evaluation ensembles are then rejected if their data sketches produce
independent evaluation estimates that are too far from the constructed surface.
The results produced by the surviving evaluation ensembles can sometimes be as good as 1\%.
But handling even small amounts of correlation remains a challenge. A Taylor expansion
of the estimates produced when error independence is assumed but the classifiers are, in fact,
slightly correlated helps clarify how the proposed independent evaluator has algebraic `blind spots'
of its own. They are points in evaluation space but the estimate of the independent evaluator
has a sensitivity inversely proportional to the distance of the true point from them.
How algebraic stream evaluation can and cannot help when done for safety or economic
reasons is briefly discussed.

Submission Number: 7767