[Re] Key Point Analysis via Contrastive Learning and Extractive Argument Summarization

Anonymous

Keywords: key point analysis, contrastive learning, argument

Abstract:

Reproducibility Summary

Scope of Reproducibility

The goal of this work is to validate the reproducibility of key point analysis of arguments framework proposed by the main paper.
The authors claimed that they achieved the best performance in the KPA shared task via contrastive learning. For key point generation, they developed a graph-based extractive summarization model that output informative key points of high quality for
a collection of arguments.

Methodology

We used open source code of the authors with slight changes. Simple
parts of code were run on CPU, while the parts that require training and working with deep models were run on a NVIDIA Tesla K80 GPU (with 12GB memory, which is the Google Colab's default GPU) for 2 hours approximately.

Results

We reproduced the results of paper on the provided test set with the following details:

Our ROUGE-1 metric was within 0.99% of the reported value which is acceptable.
Our ROUGE-2 metric was within 7.14% of the reported value which is a little high.
Our ROUGE-L metric was within 1.07% of the reported value which is acceptable.

Check notebook number 6 (available at the git repository > code > src-ipynb ) for this part of results.
There are also some metrics for evaluation of key point matching on validation set with the following details:

Our strict mAP on the validation set was the same as the reported value (with accuracy of one hundredth of a decimal) and relaxed mAP metric was within 1.04\% of the reported value which is acceptable.

Check notebook number 3 for this part of results.

It can be said that the results of reproduction were generally acceptable.

What was easy

It was easy to run and config most parts of the provided code in the repository of the paper, except some parts that we will cover in the next session.

What was difficult

Some parts of code like the notebook number 4 in our repository was unable to run because of timeout errors, which was easy to solve by some exception handling. Furthermore, matching the datasets, because of having two groups of data and having some extra data which was not used in the code, was a little hard.

Communication with original authors

The official implementation is complicated thus not easy to follow. We contacted the first author about the order of running files so the author cleaned the git repository of code but some of files were missing that were available from previous commit.

Paper Url: https://aclanthology.org/2021.argmining-1.19.pdf

Paper Venue: Not in list

Venue Name: Proceedings of The 8th Workshop on Argument Mining

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/key-point-analysis-via-contrastive-learning/code)