Scalable-DSP: a high scalable distributed storage and processing system for unstructured data in big data environments

Dawei Sun, Shang Gao

Abstract: High scalability is very important for an Internet-scale data storage and processing system in big data era. To achieve scalability, data-relevant issues are identified: unstructured data management, cost of data storage and processing, and cross-domain data management. In this paper, a high scalable distributed storage and processing system for unstructured data is proposed and developed. The paper includes the following contributions. (1) A high scalable distributed architecture is designed. (2) A multilevel, unstructured data storage system is built. (3) A distributed data processing system is implemented to verify the scalable architecture. Experimental results conclusively demonstrate the efficiency and effectiveness of the proposed storage and processing system, which achieves higher data storage efficiency and lower data access time objectives in Internet-scale big data environments.