Toward Fast Query Serving in Key-Value Store Migration with Approximate Telemetry

Alexander Braverman, Zaoxing Liu

Abstract:

Distributed key-value stores scale data analytical process-
ing by spreading data across nodes. Frequent migration of
key-value shards between online nodes is a key technique
to react to dynamic workload changes for load balancing
and service elasticity. During migration, the data is split
between a source and a destination, making it difficult to
query the exact location. Existing solutions aiming to pro-
vide real-time read and write query capabilities during mi-
gration may require querying both source and destination
servers, doubling the compute/network resources. In this
paper, we explore a simple yet effective measurement ap-
proach to track the key-value migration status, in order to
improve the query-serving performance under migration. In
our preliminary prototype, we use a Bloom filter on the des-
tination server to keep track of individual key-value pairs
that have been successfully migrated. For key-value pairs
that have yet migrated, the information stored in the Bloom
filter enables fast forwarding to the source server without the
need to check the database. We prototype this design on a
local cluster with Redis deployments. Our preliminary re-
sults show that this approximate measurement-based design
minimizes query losses during migration.