IN SEARCH OF THE INVISIBLE: GPT in An Investigation of Hidden Semantic Information
Keywords: artificial intelligence, cognitive semantic theories, LLMs, translation, translation evaluation
TL;DR: The paper evaluates the proficiency of GPT 4.0 in discerning hidden semantic nuances during translation tasks.
Abstract: The surge of interest in artificial intelligence (AI) stems from the remarkable fluency demonstrated by Large Language Models (LLMs) in language comprehension. Our study focuses on uncovering hidden semantic information, particularly how OpenAI’s Generative Pre-trained Transformer (GPT) 4.0 model excels in this area. Informed by documented obstacles in neural machine translation (MT) (Koehn and Knowles, 2017; Wan et al., 2022), challenges in text-generation models (Wang et al., 2023), recent advancements in leveraging LLMs for natural language processing (NLP) tasks (Riemenschneider and Frank, 2023) and human-like translation strategies (He et al., 2024), our aim to evaluate GPT’s proficiency in discerning and articulating semantic nuances, comparing its performance with human judgement.
Employing a comparative analysis framework, our study scrutinises a selection of translated sentences from literary and audiovisual texts that have been extensively studied within the framework of Talmy’s (2000) force dynamics by Wiśniewska (2022 and 2023). This conceptual framework illuminates how language expresses causality and dynamic relationships between entities, often using the metaphor of physical forces acting on objects. For instance, in the sentence She persuaded him to go, the force exerted by she leads to the action of him going.
Methodologically, our approach is multifaceted, integrating quantitative measures with qualitative analysis to assess the fidelity of translations in capturing hidden semantic information. Custom prompts are employed to elicit translation and self-evaluation of GPT output as metadata. We explore relationships between English, Finnish, and Polish, with a focus on verb phrases and sentences embedded with force dynamics meanings. Evaluation encompasses idiomatic correctness and language conventions, providing technical details alongside a nuanced discussion of semantics in JSON format.
Anticipated findings indicate that while GPT demonstrates remarkable proficiency in rendering surface-level meanings, its ability to identify and articulate hidden semantic information varies based on linguistic context, complexity, and the specificity of prompts. Human translation evaluators display a more nuanced understanding, leveraging linguistic intuition to analyse translations rich in hidden meanings.
This study contributes to the growing body of research on AI-assisted translation by shedding light on the capabilities and limitations of LLMs in uncovering implied semantic information. By elucidating the interplay between human translators, AI, and MT systems, this research advances our understanding of translation studies in the digital age of NLP. Drawing on insights from studies such as Alzahrani et al. (2024), it emphasises the importance of rigorous methodology and careful consideration of evaluation metrics in assessing the performance of translation models.
Submission Number: 66
Loading