SoK: The Faults in our Graph Benchmarks
Keywords: Graph Systems Benchmarking
TL;DR: Systematically characterizing and presenting issues in benchmarking graph processing systems
Abstract: Many modern applications process data naturally described by graphs, e.g., social networks, financial transactions, brain networks, and protein interactions.
There is a need for special-purpose graph processing systems, since conventional data management systems make several assumptions that do not hold for graph-structured data.
Data management systems are usually evaluated along two axes: scalability and performance.
Benchmarking graph processing systems is not a standardized practice, and this can lead to poor baselines and misdrawn conclusions.
In particular, evaluations frequently ignore datasets' statistical idiosyncrasies, which significantly affect system performance.
Moreover, scalability studies in system evaluations often use datasets that easily fit in memory on a modest desktop.
Some studies rely on synthetic graph generators to generate large graphs, but these generators produce graphs with unnatural characteristics that also affect performance, producing misleading results.
Currently, the community has no consistent and principled manner with which to compare systems and provide guidance to developers who wish to select the system most suited to their application.
We provide three different systematizations of benchmarking practices.
First, we present a 12-year literary review of graph processing benchmarking, including a summary of the prevalence of specific datasets and benchmarks used in these papers.
Second, we demonstrate the impact of two statistical properties of datasets that drastically affect benchmark performance. We show how different assignments of IDs to vertices, called \emph{vertex orderings}, dramatically alter benchmark performance due to the different caching behavior they induce. We also show the impact of \emph{zero-degree vertices} on the runtime of benchmarks such as breadth-first search and single-source shortest path.
We show that these issues can cause a performance to change by as much as 38\% on several popular graph processing systems.
Finally, we suggest best practices to account for these issues when evaluating graph systems.
Area: Streaming Systems
Type: Systemization of Knowledge (SoK)
Revision: No
Submission Number: 1
Loading