Visualizing Events: Simulating Meaning in Language

James Pustejovsky, Nikhil Krishnaswamy

Abstract: In this note, we present results from a new approach to modeling the semantics of natural language, Multimodal Semantic Simulations (MSS), being pursued by the first author and his students. This approach assumes both a richer formal model of events and their participants, as well as a modeling language for constructing 3D visualizations of objects and events denoted by natural language expressions. The Dynamic Event Model (DEM) encodes events as programs in a dynamic logic with an operational semantics, while the language VoxML, Visual Object Concept Modeling Language, is being used as the platform for multimodal semantic simulations in the context of human-computer communication.