Aya in Action: An Investigation of its Abilities in Aspect-Based Sentiment Analysis, Hate Speech Detection, Irony Detection, and Question-Answering

Julia da Rocha Junqueira, Emerson P. Lopes, Eduarda Abreu Carvalho, Larissa Astrogildo de freitas, Ulisses Brisolara Corrêa

Keywords: Sentiment Analysis, Hate Speech Detection, Irony Detection, Question-Answering, Large Language Models, Few-shot Learning, Portuguese Language.

Abstract:

While resource-rich languages such as English and Mandarin drive considerable advancements, low-resource languages face challenges due to the scarcity of substantial digital and annotated linguistic resources. Within this context,
in 2024, Aya was introduced, a multilingual generative language model supporting 101 languages, over half of which are lower-resourced. This study aims to assess Aya's performance in tasks such as Aspect-Based Sentiment Analysis, Hate Speech Detection, Irony Detection, and Question-Answering, using a few-shot methodology in Brazilian Portuguese. The objective is to evaluate Aya's effectiveness in these tasks without fine-tuning the pre-trained model, thereby exploring its potential to improve the quality and accuracy of outputs in various natural language understanding tasks.
Results indicate that while Aya performs well in certain tasks like Question-Answering, where it surpassed Portuguese-specific models with an Exact Match score of 58.79%, it struggles in others. For the Hate Speech Detection task, Aya's F1-score of 0.64 was significantly lower than the 0.94 achieved by the Sabiá-7B model. Additionally, the model's performance on the Aspect-Based Sentiment Analysis task improved considerably when neutral examples were excluded, but its handling of complex slang and context-dependent features in other tasks remained challenging. These results suggest that multilingual models like Aya can perform competitively in some contexts but may require further tuning to match the effectiveness of models specifically trained for Portuguese.

Primary Area: applications to computer vision, audio, language, and other modalities

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 12430