NoisyAnnot Causal News Corpus 2022 Causality Detection using Multiple Annotation Decisions Quynh Anh Nguyen1 2Arka Mitra2

2025-05-02 0 0 355.35KB 6 页 10玖币

侵权投诉

NoisyAnnot@ Causal News Corpus 2022: Causality Detection using

Multiple Annotation Decisions

Quynh Anh Nguyen1, 2 Arka Mitra2

1University of Milan 2ETH Zürich

quynguyen@ethz.ch, amitra@ethz.ch

Abstract

The paper describes the work that has been

submitted to the 5th workshop on Challenges

and Applications of Automated Extraction of

socio-political events from text (CASE 2022).

The work is associated with Subtask 1 of

Shared Task 3 that aims to detect causality in

protest news corpus. The authors used dif-

ferent large language models with customized

cross-entropy loss functions that exploit anno-

tation information. The experiments showed

that bert-based-uncased with reﬁned cross-

entropy outperformed the others, achieving a

F1 score of 0.8501 on the Causal News Cor-

pus dataset.

1 Introduction

A causal relationship in a sentence implies an un-

derlying semantic dependency between the two

main clauses. The clauses in these sentences are

generally connected by markers which can have

different parts of tags in the sentence. Moreover,

the markers can be either implicit or explicit and

for these reasons, one cannot rely on regex or

dictionary-based systems. Thus, there is a need

to investigate the context of the sentences. For the

given task, we exploited different large language

models that provide a contextual representation of

sentences to tackle causality detection.

Shared task 3 in CASE-2022 (Tan et al.,2022a)

aims for causality detection in news corpus, which

can be structured as a text classiﬁcation problem

with binary labels. Pre-trained transformer-based

models (Vaswani et al.,2017) have shown success

on tackling a wide range of NLP tasks including

text generation, text classiﬁcation, etc. The authors

look into inter-annotation agreements and number

of experts and how they can be included in the

loss to improve the performance of the pre-trained

models.

The main contributions of the paper are as fol-

lows:

Extensive experimentation with different large

language models.

Incorporation of additional annotation infor-

mation, i.e inter-annotation agreement and the

number of annotators, to the loss.

The remaining paper is formulated as follows:

Section 2reviews the related work, section 3de-

scribes the dataset on which the work has been

done, section 4discusses the methodology used

in the paper, the following section discusses the

results and provides an ablation of the various loss

functions introduced and ﬁnally, section 6con-

cludes the paper and suggests future works.

2 Related Work

Multiple annotations on a single sample reduce

the chances of the labelling to be incorrect or bias

being incorporated into the dataset (Snow et al.,

2008). Including multiple annotators also leads

to disagreement among the labels that have been

provided by them. The ﬁnal or gold annotation is

then usually determined by majority voting (Sabou

et al.,2014) or by using the label of an "expert"

(Waseem and Hovy,2016). There are also different

methodologies which do not use majority voting to

select the "ground truth".

Expectation Maximization algorithm has been

used to account for the annotator error (Dawid and

Skene,1979). Entropy metrics have been devel-

oped to identify the performance of the annota-

tors(Waterhouse,2012;Hovy et al.,2013;Gordon

et al.,2021). Multi-task learning is also used to deal

with disagreement in the labels (Fornaciari et al.,

2021;Liu et al.,2019;Cohn and Specia,2013;

Davani et al.,2022). There are methods which

include the annotation disagreement into the loss

function for part of speech tagging (Plank et al.,

2014;Prabhakaran et al.,2012) on SVMs and per-

ceptron model. The present work considers the

inter-annotator agreement as well as the number

arXiv:2210.14852v2 [cs.CL] 1 Dec 2022

of annotators into the loss function for any model.

The work also compares the performance when the

annotators who disagree with the majority voting

has been ignored.

3 Dataset

The Causal News Corpus dataset (Tan et al.,2022b)

consists of 3,559 event sentences extracted from

protest event news. Each sample in the dataset con-

tains the text, the corresponding label, the number

of experts who annotated the label and the degree

of agreement among the experts. Figure 1shows a

sample from the provided training set. The training

data is fairly balanced, containing 1603 sentences

with a causal structure and 1322 sentences without

a causal structure. Also, the number of causal and

non-causal sentences in the validation set does not

differ signiﬁcantly. Finally, 311 news articles have

been used as test set for evaluation.

Figure 1: A datapoint from the provided training data.

Besides the binary labels, the Causal News Cor-

pus dataset also provides additional information

regarding the number of experts who labeled the

sentence and the percentage of agreement between

them. Figure 1shows that the number of experts

who annotated the text "The farmworkers’strike re-

sumed on Tuesday when their demands were not

met." is 3 (

num_votes = 3

). Also, all of the ex-

perts labeled the sentence to be causal so the agree-

ment is 1.0 (100% agreement) and the label is 1. In

case only one of three experts assigned label 1 to

the previous text, the three predictors

num_votes

agreement

label

would now become

respectively. In this paper, the authors exploit

this information to give the model more prior and

thus potentially improve the model’s performance,

which has been described in more detail in section

4 Methodology

The section discusses the pipeline, the different

types of loss functions that were implemented, and

the experimental details that have been used in the

third shared task for CASE 2022 (Tan et al.,2022a).

4.1 Pipeline

The authors ﬁnetuned large language models with

different loss functions to tackle Subtask 1 in

Shared Task 3 of CASE@EMNLP-2022, causality

detection in a given sentence. The problem can be

reformulated as a binary classiﬁcation where the

model predicts whether the sentence is causal or

not. Since contextual awareness plays an essential

role in handling this speciﬁc task, the authors used

several transformer-based models, namely, BERT

(Devlin et al.,2019), FinBERT (Liu et al.,2020),

XLNET (Yang et al.,2019) and RoBERTa (Zhuang

et al.,2021).

The given sentence is ﬁrst tokenized by a tok-

enizer from the corresponding pretrained model

architecture provided by HuggingFace (Wolf et al.,

2020). The vector output from the tokenization

stage is then fed as input to the model. The most in-

formative token is the classiﬁcation token ([CLS]),

which is a special token that can be used as a

sentence representation. The [CLS] token is then

passed through a feed-forward network to gener-

ate logits. The softmax over the logits gives us

the probability of whether the sentence is causal or

not. For each model, the authors experimented with

cross-entropy loss and proposed two loss functions

described in detail in subsection 4.2.

4.2 Loss Functions

Cross Entropy Loss

The loss of the classiﬁca-

tion task can be represented by a simple cross-

entropy loss, as shown in Equation 1:

L=1

i=1

(−ytrue

ilog(ypred

−(1 −ytrue

i)log(1 −ypred

i))

(1)

where

ytrue

and

ypred

denote the true label and the

predicted label for the

ith

input in a batch of M

sentences.

Noisy Cross Entropy Loss

The dataset not only

provides the standard information about

{text,

label}

, but also contains the information about

the number of experts who annotated the sentence’s

label, and proportion of agreement between them.

The authors have considered the annotation by

each of the experts to be the true label for the sen-

tence. For a sentence with

expert annotations

文档加载中……请稍候！
如果长时间未打开，您也可以点击刷新试试。

下载文档到电脑，查找使用更方便

10 玖币 0人已下载

立即下载

摘要：

NoisyAnnot@CausalNewsCorpus2022:CausalityDetectionusingMultipleAnnotationDecisionsQuynhAnhNguyen1,2ArkaMitra21UniversityofMilan2ETHZürichquynguyen@ethz.ch,amitra@ethz.chAbstractThepaperdescribestheworkthathasbeensubmittedtothe5thworkshoponChallengesandApplicationsofAutomatedExtractionofsocio-politic...

展开>> 收起<<

NoisyAnnot Causal News Corpus 2022 Causality Detection using Multiple Annotation Decisions Quynh Anh Nguyen1 2Arka Mitra2.pdf

共6页,预览2页

还剩页未读，继续阅读

声明：本站为文档C2C交易模式，即用户上传的文档直接被用户下载，本站只是中间服务平台，本站所有文档下载所得的收益归上传人(含作者)所有。玖贝云文库仅提供信息存储空间，仅对用户上传内容的表现方式做保护处理，对上载内容本身不做任何修改或编辑。若文档所含内容侵犯了您的版权或隐私，请立即通知玖贝云文库，我们立即给予删除！

NoisyAnnot Causal News Corpus 2022 Causality Detection using Multiple Annotation Decisions Quynh Anh Nguyen1 2Arka Mitra2

相关推荐

开通VIP享超值会员特权

作者详情

相关内容

热门标签

举报选择: