Multimodal Interoperability with the CLAMS Platform

Kelley Lynch, Kyeongmin Rim, Owen King, James Pustejovsky

Abstract: This paper presents the CLAMS (Computational Linguistics Applications for Multimedia Services) platform, which provides a framework for developing and deploying interoperable multimedia analysis tools [7]. CLAMS facilitates the processing of audiovisual content such as broadcast news videos by enabling seamless integration of tools across different media types including text, audio, video, and images. At the core of CLAMS is the Multi-Media Interchange Format (MMIF), a JSON-based annotation format designed to support the exchange of data between different tools in a consistent and structured manner. This ensures that annotations produced by one tool can be readily used by others, creating complex pipelines for automated content analysis. We describe the features provided by the CLAMS software development kit (SDK), present 2 example pipelines of CLAMS applications, and show a visualization tool for exploring data generated using CLAMS.