A Comparative Analysis of LLMs in interpreting Idioms and Proverbs in Konkani

ACL ARR 2025 July Submission672 Authors

Abstract: It is difficult for machines to interpret idioms and proverbs, and more so in lesser known languages such as Konkani. This paper categorizes the ability of the LLMs to comprehend the actual meaning of 50 Konkani idioms and 50 Konkani proverbs. We have compared three language models i.e., GPT-4, Gemini 2.5 Flash, and LLaMA 3 in our work and examined how well each model was able to comprehend the meaning of every idiom and proverb not only literally, but whether it was able to catch the figurative or cultural connotation behind it. Gemini 2.5 Flash provided the best results, whereas GPT-4 performed better in comprehending meanings. Although as observed throughout the research, Gemini 2.5 Flash initially struggled but appeared to get better gradually as if it learned from previous trends. GPT-4 did very well right from the start, although its scores were not always the highest. LLaMA 3 performed comparatively low. Our results indicate the issues which still exist when dealing with regional languages and figurative language.

Paper Type: Short

Research Area: Interpretability and Analysis of Models for NLP

Research Area Keywords: Konkani Idioms, Low resource LLM Performance

Contribution Types: Model analysis & interpretability

Languages Studied: Konkani

Reassignment Request Area Chair: This is not a resubmission

Reassignment Request Reviewers: This is not a resubmission

A1 Limitations Section: This paper has a limitations section.

A2 Potential Risks: N/A

B Use Or Create Scientific Artifacts: Yes

B1 Cite Creators Of Artifacts: Yes

B1 Elaboration: In Section 3 i.e. Proposed Methodology we have cited the information of the LLMs we have tested our data on.

B2 Discuss The License For Artifacts: N/A

B3 Artifact Use Consistent With Intended Use: Yes

B3 Elaboration: yes, we have use the artifacts i.e. LLMs for testing our dataset and we have presented the said results in the paper in the section 4 i.e. Results & Analysis

B4 Data Contains Personally Identifying Info Or Offensive Content: No

B5 Documentation Of Artifacts: N/A

B6 Statistics For Data: N/A

C Computational Experiments: No

C1 Model Size And Budget: N/A

C2 Experimental Setup And Hyperparameters: N/A

C3 Descriptive Statistics: N/A

C4 Parameters For Packages: N/A

D Human Subjects Including Annotators: No

D1 Instructions Given To Participants: N/A

D2 Recruitment And Payment: N/A

D3 Data Consent: N/A

D4 Ethics Review Board Approval: N/A

D5 Characteristics Of Annotators: N/A

E Ai Assistants In Research Or Writing: Yes

E1 Information About Use Of Ai Assistants: Yes

E1 Elaboration: Yes,we have used it for evaluating the model performance, and also we have used it for writing and grammar correction.

Author Submission Checklist: yes

Submission Number: 672