Language model fine-tuning is essential for modern natural language processing. The effectiveness of fine-tuning is limited by the inclusion of training examples that negatively affect performance. Here we present Information Gain Filtration, a general fine-tuning method, for improving the overall final performance of a fine-tuned model. We define Information Gain of an example as the improvement on a validation metric after training on that example. A secondary learner is then trained to approximate this quantity. During fine-tuning, this learner filters informative examples from uninformative ones. We show that our method is robust and has consistent improvement across datasets, fine-tuning tasks, and language model architectures. 

Improving Language Model Fine-tuning with Information Gain Filtration

The transformer-based architectures have achieved remarkable success in several Natural Language Processing tasks, such as the Question Answering domain. Our research focuses on different transformer-based language models' performance in software development legal domain specialized datasets for the Question Answering task. It compares the performance with the general-purpose Question Answering task. We have experimented with the PolicyQA dataset and conformed to documents regarding users' data handling policies, which fall into the software legal domain. We used as base encoders  BERT, ALBERT, RoBERTa, DistilBERT and LEGAL-BERT and compare their performance on the Question answering benchmark dataset SQuAD V2.0 and PolicyQA. Our results indicate that the performance of these models as contextual embeddings encoders in the PolicyQA dataset is significantly lower than in the SQuAD V2.0. Furthermore, we showed that surprisingly general domain BERT-based models like ALBERT and BERT obtain better performance than a more domain-specific trained model like LEGAL-BERT.

Study of Question Answering on Legal Software Document using BERT based models

This work investigates consumer complaints shared on Consumidor.gov.br, a Brazilian Online Dispute Resolution platform. Leveraging Natural Language Processing (NLP) techniques,  we inquire to what extent complaints by male and female consumers are represented in that platform and whether differences in the language of their complaints can reveal patterns to be explored as linguistic indicators of gender authoring. Our results show differences in the way males and females construe meanings in terms of word POS, psycholinguistic properties and emotions expressed.

NAACL 2022 Workshop LXNLP Regular Deadline Submissions

Improving Language Model Fine-tuning with Information Gain Filtration

Study of Question Answering on Legal Software Document using BERT based models

User Profile Characterization Within a Brazilian Online Dispute Resolution Platform