Does the content defined chunking really solve the local boundary shift problem?

Wenlong Tian, Ruixuan Li, Zhiyong Xu, Weijun Xiao

Abstract: Data chunking is one of the most important issues in a deduplication system, which not only determines the effectiveness of deduplication such as deduplication ratio, but also impacts the modification overhead. It breaks the file into chunks to find out the redundancy by fingerprint comparisons. The content-defined chunking algorithms such as TTTD, BSW CDC, and RC, can resist the boundary shift problem caused by small modifications. However, we observe that there exist a lot of consecutive maximum chunk sequences in various benchmarks. These consecutive maximum chunk sequences will lead to local boundary shift problem when facing small modifications. Based on this observation, we propose a new chunking algorithm, Elastic Chunking. By leveraging dynamic adjustment policy, elastic chunk can quickly find the boundary to remove the consecutive maximum chunk sequences. To evaluate the performance, we implement a prototype and conduct extensive experiments based on synthetic and realistic datasets. Compared with TTTD, BSW CDC and RC algorithms, proposed chunking algorithm can achieve the higher deduplication ratio and throughput.