256 Metaverse Recording Dataset

Patrick Steinert, Stefan Wagenpfeil, Ingo Frommholz, Matthias Hemmje

Abstract: The metaverse is an evolving field and the subject of multimedia research. In this paper, we introduce the 256-MetaverseRecords dataset, a novel and extensive collection of annotated screen recordings in the form of videos from various virtual worlds of the metaverse. We describe the process of creating the dataset, the quality criteria for the annotations, and the exploration of the dataset. We also show four experiments to evaluate the performance of different feature extraction methods for Metaverse Recordings (MVRs): MVR segmentation, audio event detection, and object and interaction detection based on this dataset. Our results demonstrate that existing methods have limitations and leave challenges in dealing with the diversity and complexity of metaverse data, and that more research is needed to develop metaverse-specific techniques. Our dataset can serve as a valuable resource for the research community and foster the development of new applications and solutions for the metaverse.

Relevance To Conference:

This work contributes significantly to multimedia/multimodal processing by addressing the critical need for specialized methods in analyzing complex and diverse data from the metaverse. The introduction of the 256-MetaverseRecords dataset, an extensive collection of annotated screen recordings from various virtual worlds, marks a pioneering step in multimedia research focused on the metaverse. By conducting experiments on MVR segmentation, audio event detection, and object and interaction detection, this study illuminates the challenges existing methods face when applied to the multifaceted and rich media environment of the metaverse.

The findings highlight the limitations of current feature extraction techniques, underscoring the necessity for metaverse-specific research and the development of novel analytical tools tailored to the unique characteristics of metaverse recordings. This initiative advances the understanding of multimedia content within virtual environments and sets the stage for innovative applications and solutions. Consequently, this research extends the frontier of multimedia and multimodal processing, offering a valuable resource that facilitates the exploration of new intellectual directions and tackling cutting-edge challenges in the realm of the metaverse.

Primary Subject Area: [Content] Media Interpretation

Secondary Subject Area: [Experience] Multimedia Applications, [Systems] Data Systems Management and Indexing

Submission Number: 2165