Extensible Database Simulator for Fast Prototyping In-Database Algorithms

Yifan Wang, Daisy Zhe Wang

Abstract:

With the rapid increasing of data scale, in-database analytics and learning has become one of the most studied topics in data sci-
ence community, because of its significance on reducing the gap between the management and the analytics of data. By extending
the capability of database on analytics and learning, data scientists can save much time on exchanging data between databases and
external analytic tools. For this goal, researchers are attempting to integrate more data science algorithms into database. However,
implementing the algorithms in mainstream databases is super time-consuming, especially when it is necessary to have a deep dive into
the database kernels. Thus there are demands for an easy-to-extend database simulator to help fast prototype and verify the in-database
algorithms before implementing them in real databases. In this demo, we present such an extensible relational database simulator, DBSim, to help data scientists prototype their in-database analytics and learning algorithms and verify the effectiveness of their ideas with minimal cost. DBSim simulates a real relational database by integrating all the major components in mainstream RDBMS, including SQL parser, relational operators, query optimizer, etc. In addition, DBSim provides various interfaces for users to flexibly plug their custom extension modules into any of the major components, without modifying the kernel. By those interfaces, DBSim supports easy extensions on SQL syntax, relational operators, query optimizer rules and cost models, and physical plan execution. To enable accurate evaluation on users’
extensions, DBSim supports connecting with real RDBMS and using their real-world cost estimators to calculate the query plan cost. Furthermore, DBSim provides utilities to facilitate users’ developing
and debugging, like query plan visualizer and interactive analyzer on optimization rules. We develop DBSim using pure Python to support seamless implementation of most data science algorithms into it, since many of them are written in Python.