Semantic Captioning for Bharatanatyam via Mushti Motion Classification

ACL ARR 2026 January Submission6794 Authors

Keywords: Semantics, Bharatanatyam, MediaPipe, Phonology, Hand Gestures

Abstract: This paper presents a semantic analysis aimed at enabling AI closed captioning for Bharatanatyam (BN), with a first computational component that detects the Mushti (closed-fist) gesture and maps its motion to two semiotic meanings: courage and steadiness. Steadiness is defined as a downward motion of a closed fist facing the self, while courage is defined as a rotation of the closed fist that starts fac- ing downwards and turns upwards toward the self. The system is a browser-based demo that runs on webcam input using MediaPipe hand landmarks and exposes its thresholds as live, persistent controls, which keeps the analysis transparent and tunable. The dataset includes 60 short clips (30 for courage and 30 for steadiness), each 2 seconds long. We do not report metrics yet; instead we focus on the interpretive link between gesture and meaning, and show how the Mushti classifier can serve as one measurable anchor for future captioning systems that are sensitive to semantic nuance in classical dance.

Paper Type: Short

Research Area: Discourse, Pragmatics, and Reasoning

Research Area Keywords: Speech acts, pragmatics, morphology, gesture, explainability, multimodal

Contribution Types: Model analysis & interpretability, NLP engineering experiment, Data analysis

Languages Studied: Bharatanatyam mudras (hand gestures)

Submission Number: 6794