A One-Step MSE Estimation of Models in Production

Ryuta Matsuno

Keywords: Error estimation, model evaluation, model monitoring, MLOps

Abstract:

In real-world operation of machine learning systems, monitoring the performance of prediction models is crucial.
However, in these scenarios, actual values of target variables are observed with a delay, making real-time evaluation of prediction performance impossible.
In this paper, we propose a novel one-step Mean Squared Error (MSE) estimation method that directly and tightly minimizes the upper bound of the MSE estimation error for regression tasks.
Due to its direct estimation, our method is more efficient at estimating MSE compared to the conventional two-step approach, which approximates the mean and variance of the target variable.
We also provide generalization error bounds for our proposed method based on a theoretical analysis.
Our experiments demonstrate the effectiveness of our method, outperforming existing methods on both synthetic and real data sets.

Primary Area: general machine learning (i.e., none of the above)

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 4684