VisionAD, a software package of performant anomaly detection algorithms, and Proportion Localised, an interpretable metric
Abstract: We release VisionAD, an anomaly detection library in the domain of images. The library forms the largest and most performant collection of such algorithms to date. Each algorithm is written through a standardised API. The library has a focus on fair benchmarking intended to mitigate the issue of cherry-picked results. It is designed enable rapid experimentation, and the integration of new algorithms is straightforward. In addition, we propose a new metric, Proportion Localised (PL). This reports the proportion of anomalies that are sufficiently localised via discretely classifying each anomaly as localised or not. The metric is far more intuitive as it has a real physical relation, which is attractive to industry-based professionals. We also release a thorough benchmarking of the MVTec dataset consisting of the top 15 algorithms available. We call this the VisionAD leaderboard. We are committed to hosting an updated version of this leaderboard online, and encourage researchers to add, tweak and improve algorithms to climb this leaderboard.
Submission Length: Regular submission (no more than 12 pages of main content)
Changes Since Last Submission: We have added another dataset, VisA, and updated our benchmarking to include this other dataset. This makes the benchmarking more robust. VisA is a dataset of 12 classes across industrial and household objects, and is more challenging than MVTec
We have updated PL to allow rotated bounding boxes (such that the metric is not dependent on image rotation). These rotated boxes are the smallest area boxes around a given anomaly. This means the PL score is independent of anomaly rotation. We have updated PL to merge closer bounding boxes. For a group of small and close anomalies, as their scaled bounding box labels would overlap, these anomalies are converted into one anomaly. New Figure 4 shows this clearly. This slightly reduces the number of bounding boxes, and slightly reduces the PL value. This effect is most apparent in the VisA dataset, where there are more small/close anomalies.
We have generalised Figure 2 to compare the metric's behaviour using the real results, as opposed to a synthetic experiment. We believe this Figure clearly shows the superior behaviour of PL, using real results.
Assigned Action Editor: ~bo_han2
Submission Number: 1985
Loading