VisionAD, a software package of performant anomaly detection algorithms, and Proportion Localised, an interpretable metric

Alexander D. J. Taylor, Phillip Tregidgo, Jonathan James Morrison, Neill D. F. Campbell

Abstract: We release VisionAD, an anomaly detection library in the domain of images. The library forms the largest and most performant collection of such algorithms to date. Each algorithm is written through a standardised API, for ease of use. The library has a focus on fair benchmarking intended to mitigate the issue of cherry-picked results. It enables rapid experimentation and straightforward integration of new algorithms. In addition, we propose a new metric, Proportion Localised (PL). This reports the proportion of anomalies that are sufficiently localised via classifying each discrete anomaly as localised or not. The metric is far more intuitive as it has a real physical relation, meaning it is attractive to industry-based professionals. We also release the VisionADIndustrial (VADI) benchmark, a thorough benchmarking of the top anomaly detection algorithms. This benchmark calculates the mean across the pooled classes of the MVTec and VisA datasets. We are committed to hosting an updated version of this leaderboard online, and encourage researchers to add, tweak and improve algorithms to climb this leaderboard. VisionAD code is found at https://github.com/alext1995/VisionAD, and Proportion Localised code is found at https://github.com/alext1995/proportion_localised.

Submission Length: Regular submission (no more than 12 pages of main content)

Changes Since Last Submission:

We have added another dataset, VisA, and updated our benchmarking to include this other dataset. This makes the benchmarking more robust. VisA is a dataset of 12 classes across industrial and household objects, and is more challenging than MVTec

We have updated PL to allow rotated bounding boxes (such that the metric is not dependent on image rotation). These rotated boxes are the smallest area boxes around a given anomaly. This means the PL score is independent of anomaly rotation. We have updated PL to merge closer bounding boxes. For a group of small and close anomalies, as their scaled bounding box labels would overlap, these anomalies are converted into one anomaly. New Figure 4 shows this clearly. This slightly reduces the number of bounding boxes, and slightly reduces the PL value. This effect is most apparent in the VisA dataset, where there are more small/close anomalies.

We have generalised Figure 2 to compare the metric's behaviour using the real results, as opposed to a synthetic experiment. We believe this Figure clearly shows the superior behaviour of PL, using real results.

Video: https://youtu.be/tUqyAjONy8s

Code: https://github.com/alext1995/VisionAD

Assigned Action Editor: ~bo_han2

Submission Number: 1985