Encounters between the worlds of visual arts, easy language and AI

University of Eastern Finland DRDHum 2024 Conference Submission23 Authors

Keywords: AI, audio-description, easy language, image, visual art, conversation analysis

TL;DR: Poster presentation

Abstract:

Artificial intelligence produces images, but how does it recognize and interpret them? In my poster, I examine the encounters between the visual arts, easy language and artificial intelligence. How artificial intelligence can assist in producing text material, for different contexts and especially for audio description, a process of linguistic description of visual information in visual works for example for the visually impaired.
The dataset consists of different artistic images that pose different challenges to both audio-desciption and interpretation. AI uses neural networks to identify elements in a work of art. It then uses these elements to describe the image linguistically. How does AI recognize semi-abstract paintings? Compositions, shapes, rhythm? Atmosphere? Light and shadow? Can a line represent hay or water? Thus, I am also interested in how AI is able to identify and interpret not only the visible but also the interpretive elements of the image. The generative language model, which is capable of discussion, creates new content based on the prompts. Especially in this context, as the paintings are not unambiguously representational, the connotations offered in the prompts to AI are important. In the poster, I present examples about conversations with AI representing its ability to interpret the connotative aspects of the painting and if the outcome can assist in creating reliable text for audio description. Moreover, examining the conversational processes along with the prompts I will be asking if the qualitative conversation analytical (CA) approach successful in examining AI's abilities to proceed in the conversation. Additionally, preliminary findings about AI’s ability to produce liguistically simplified (easy-to-read) text is examined in those settings.

Submission Number: 23