DIODE: Dynamic Inline-Offline DE Duplication Providing Efficient Space-Saving and Read/Write Performance for Primary Storage Systems

Yan Tang, Jianwei Yin, ShuiGuang Deng, Ying Li

Abstract: Specific requirements and characteristics of primary storage make designing desirable deduplication schemes a sophisticated task. In this paper, we propose DIODE, a Dynamic Inline-Offline DEduplication scheme that provides salient read/write performance and space-saving simultaneously for primary storage systems. DIODE orchestrates inline and offline processes in a dynamic manner. We propose two innovative mechanisms in DIODE: Context-aware Threshold Adjustment (CTA) for inline-phase deduplication and Deferred Priority-based Enforcement (DPE) for offline-phase deduplication, respectively. CTA mechanism enables selective inline deduplication under a dynamically updated threshold. Data skipped during inline-phase is regarded as candidates for offline-phase, and is handled in a prioritized order under the governance of DPE mechanism. Inline deduplication works in concert with offline deduplication so as to complement the weakness of each other.