On Memory and I/O Efficient Duplication Detection for Multiple Self-clean Data Sources

Download PDF
Open Webpage

Ji Zhang, Yanfeng Shu, Hua Wang

Published: 2010, Last Modified: 22 Jan 2026DASFAA Workshops 2010EveryoneRevisionsBibTeXCC BY-SA 4.0
Abstract: In this paper, we propose efficient algorithms for duplicate detection from multiple data sources that are themselves duplicate-free. When developing these algorithms, we take the full consideration of various possible cases given the workload of data sources to be cleaned and the available memory. These algorithms are memory and I/O efficient, being able to reduce the number of pair-wise record comparison and minimize the total page access cost involved in the cleaning process. Experimental evaluation demonstrates that the algorithms we propose are efficient and are able to achieve better performance than SNM and random access methods.