A Visual Medium for Programmatic Control of Interactive Applications

Luke S. Zettlemoyer, Robert St. Amant

Abstract: The VisMap system provides for visual manipulation of arbitrary off-the-shelf applications, through an applications graphical user interface. VisMaps API-independent control has advantages for tasks that can benefit from direct access to the functions of the user interface. We describe the design goals and architecture of the system, and we discuss two applications, a user-controlled visual scripting program and an autonomous solitaire-playing program, which together demonstrate some of the capabilities and limitations of the approach.