Breaking Beyond COCO Object Detection

ali borji

Keywords: object detection, deep learning, performance analysis

TL;DR: An analysis of the state of the art in object detection, the empirical upper bound, and errors in models and datasets

Abstract:

COCO dataset has become the de facto standard for training and evaluating object
detectors. According to the recent benchmarks, however, performance on this
dataset is still far from perfect, which raises the following questions, a) how far can
we improve the accuracy on this dataset using deep learning, b) what is holding
us back in making progress in object detection, and c) what are the limitations
of the COCO dataset and how can they be mitigated. To answer these questions,
first, we propose a systematic approach to determine the empirical upper bound
in AP over COCOval2017, and show that this upper bound is significantly higher
than the state-of-the-art mAP (78.2% vs. 58.8%). Second, we introduce two
complementary datasets to COCO: i) COCO_OI, composed of images from COCO
and OpenImages (from 80 classes in common) with 1,418,978 training bounding
boxes over 380,111 images, and 41,893 validation bounding boxes over 18,299
images, and ii) ObjectNet_D containing objects in daily life situations (originally
created for object recognition known as ObjectNet; 29 categories in common with
COCO). We evaluate models on these datasets and pinpoint the annotation errors
on the COCO validation set. Third, we characterize the sources of errors in modern
object detectors using a recently proposed error analysis tool (TIDE) and find that
models behave differently on these datasets compared to COCO. For instance,
missing objects are more frequent in the new datasets. We also find that models
lack out of distribution generalization. Code and data will be shared.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Submission Guidelines: Yes

Please Choose The Closest Area That Your Submission Falls Into: Infrastructure (eg, datasets, competitions, implementations, libraries)