ConvBench: A Comprehensive Benchmark for 2D Convolution Primitive Evaluation

Lucas Fernando Alvarenga e Silva, Victor Ferrari, RAFAEL CARDOSO FERNANDES SOUSA, Marcio Pereira, Guido Araujo

Workshop Track: System for Machine Learning

Presentation: In-Person

Keywords: convolution neural network, convolution, benchmark

Presenter Full Name: Lucas Fernando Alvarenga e Silva

TL;DR: This paper proposes ConvBench, a primitive-level benchmark for the evaluation and comparison of convolution algorithms.

Presenter Email: [e-mail]

Abstract: Convolution is a compute-intensive operation placed at the heart of Convolution Neural Networks (CNNs). It has led to the development of many high-performance algorithms, such as Im2col-GEMM, Winograd, and Direct-Convolution. However, the comparison of different convolution algorithms is an error-prone task as it requires specific data layouts and system resources. Failure to address these requirements might lead to unwanted time penalties. Thus, considering all processing steps within convolution algorithms is essential to comprehensively evaluate and fairly compare their performance. Furthermore, most known convolution benchmarking adopts ad-hoc testing suites with limited coverage and handmade operations. This paper proposes ConvBench, a primitive-level benchmark for the evaluation and comparison of convolution algorithms. It assesses 9243 convolution operations derived from 1097 real-world deep learning models, resulting in performance and execution breakdown graphs for a detailed evaluation. ConvBench capability is evaluated across the Sliced Convolution (SConv) algorithm. The experiments showed results faster than Im2col-GEMM in 93.6% of the convolutions. However, the use of ConvBench allowed the delving into the remaining 6.4\% underperforming convolutions, uncovering a critical slowdown of 79.5% on average of SConv's packing step. This analysis underscores a potential source of optimization for SConv, opening up new paths for convolution designers to improve their algorithms.

Presenter Bio: PhD Student on Laboratory of Computer Systems – LSC @ IC-UNICAMP

Paper Checklist Guidelines: I certify that all co-authors have validated the presented results and conclusions, and have read and commit to adhering to the Paper Checklist Guidelines, Call for Papers and Publication Ethics.

Dataset Release: I certify that all co-authors commit to release the dataset and necessary scripts to reproduce the presented results.

Workshop Registration: Yes, at least one of the authors has registered for the workshop (Two-Day Registration at minimum).

Submission Number: 4