Fantastic DNN-Classifier Identification without Testing Dataset
Primary Area: general machine learning (i.e., none of the above)
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Keywords: prototype, evaluation, representation, interpretation
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.
TL;DR: We employ prototype based metrics to evaluate quality of DNN without test dataset
Abstract: Deep Neural Networks (DNNs) are trained, validated, and tested with an example dataset. For a given example dataset, several models for different architectures are trained and then using the validation dataset a model is selected. If the models have hyper-parameters, their good values are selected using validation datasets as well. Finally, performance of the selected DNN is tested using a test dataset. This testing method treats the DNN as a black-box and doesn’t attempt to understand its characteristics. On the other hand, many theoretical and empirical studies have used complexity measures for estimating generalization phenomena using the training dataset, with rare exceptions. To the best of our knowledge, no method exists to estimate test accuracy (not generalization) without any testing dataset. We propose a method for estimating test accuracy of a given DNN without any test dataset. Assuming that a DNN is the composition of a feature extractor and a classifier, we propose and evaluate a method for estimating their qualities. The first step of the proposed method is generation of one (input) prototype vector for each class. Then using these seed prototypes, (k − 1) core prototypes are generated for each class. These prototypes are our data for evaluating the qualities of the feature extractor and classifier as well as estimating test accuracy of the given DNN. We have empirically evaluated the proposed method for DNNs trained with CIFAR10, and CIFAR100.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 6708
Loading