Causal Intervention-based Prompt Debiasing for Event Argument Extraction Jiaju Lin1Jie Zhou2Qin Chen1 1School of Computer Science and Technology East China Normal University

2025-04-27 0 0 296.51KB 8 页 10玖币

侵权投诉

Causal Intervention-based Prompt Debiasing for Event Argument Extraction

Jiaju Lin1,Jie Zhou2,Qin Chen1

1School of Computer Science and Technology, East China Normal University

2School of Computer Science, Fudan University

jiaju lin@stu.ecnu.edu.cn, jie zhou@fudan.edu.cn, qchen@cs.ecnu.edu.cn,

Abstract

Prompt-based methods have become increasingly

popular among information extraction tasks, es-

pecially in low-data scenarios. By formatting a

ﬁnetune task into a pre-training objective, prompt-

based methods resolve the data scarce problem ef-

fectively. However, seldom do previous research

investigate the discrepancy among different prompt

formulating strategies. In this work, we compare

two kinds of prompts, name-based prompt and

ontology-base prompt, and reveal how ontology-

base prompt methods exceed its counterpart in

zero-shot event argument extraction (EAE) . Fur-

thermore, we analyse the potential risk in ontology-

base prompts via a causal view and propose a de-

bias method by causal intervention. Experiments

on two benchmarks demonstrate that modiﬁed by

our debias method, the baseline model becomes

both more effective and robust, with signiﬁcant im-

provement in the resistance to adversarial attacks.

1 Introduction

Event argument extraction (EAE) plays an independent role

in natural language processing. It has become widely de-

ployed in downstream tasks like natural language understand-

ing and decision making[Zhang et al., 2022a]. During the

past few years, a pretrain-ﬁnetuning paradigm of large lan-

guage models has achieved great success. However, it is un-

avoidable to train a new task-speciﬁc head for every new ap-

pearing type with plenty of labeled instances, that leads to

‘data hungry’ and sets barriers to real-word implementation.

Fortunately, a novel paradigm, namely ‘prompt’, provides

a promising approach to address the data scarce problem

[Liu et al., 2021b]. By formatting a ﬁnetune task into a

pre-training objective, prompt-based methods have become

the best-performed baselines especially in low-data scenar-

ios. Nonetheless, how to design a proper prompt is still an

open problem. Although automatic prompts generating meth-

ods have experiences a great surge in the last years, manual

prompts still dominate information extraction area. Current

state-of-the-art prompt-based methods [Li et al., 2021]are

mainly based on manual designed prompts. Besides, previ-

ous works [Ma et al., 2022a]also verify the ineffectiveness of

auto-prompts in EAE. Based on these observations, we won-

der which is the better prompt design strategy and how this

prompt facilitates extraction.

In this paper, to answer the above questions, we divide

current manual prompts into two categories: 1) name-based

prompts which are formed by concatenating names of the

arguments belonging to an event type. 2) ontology-based

prompts which are derived from the event ontology, the de-

scription of an event type in natural language. We carry

out quantitative analysis on predictions of these two prompts

and ﬁnd that, compared with name-based prompts, ontology-

based prompts can provide additional syntactic structure

information to facilitate extraction. By ﬁltering the im-

proper potential arguments, ontology-based prompts improve

model’s overall performance. Nevertheless, every coin has

its two sides. The hidden risks are introduced along with the

beneﬁcial information. We theoretically identify the spurious

correlation caused by ontology-based prompts from a causal

view. Based on the structural causal model, we ﬁnd that the

model trained with ontology-based prompts may have bias on

entities that share the common syntactic role with the argu-

ment name in the prompt, e.g. both the entity in the sentence

and argument name in the prompt are subject.

We further propose to conduct causal intervention on the

state-of-the-art method. Via backdoor adjustments, inter-

vened model rectiﬁes the confounder bias stem from the sim-

ilar syntax. Experiments are performed on two well-know

benchmarks, namely RAMS and WikiEvents. The enhance-

ments in performance demonstrate the effectiveness of our

proposed approach. Moreover, we evaluate the robustness of

our method via exposing the model to adversarial attacks and

noise in training. The results show that modiﬁed by our ad-

justments, the model becomes more robust than ever before.

Our contributions are threefold:

• We propose a causal intervention-based prompt debi-

asing model for event argument extraction based on

bias found by investigating how ontology-based prompts

work in zero-shot event argument extraction task.

• We rethink the prompt-based event argument extraction

in a causal view to analyze the causal effect among dif-

ferent elements and reduce the biases via backdoor ad-

justments.

• Extensive experiments on the cutting-edge method and

arXiv:2210.01561v1 [cs.CL] 4 Oct 2022

datasets demonstrate the effectiveness and robustness of

our method.

2 Pilot Experiments and Analysis

To investigate the work mechanism of prompts for extraction,

we investigate two varieties of the most advanced prompt

based-method PAIE [Ma et al., 2022b]in zero-shot event ar-

gument extraction. In this setting, events are split into two

parts, top nmost common ‘seen events’ and the rest ‘unseen

events’. Both training and validation sets are composed of

‘seen events’ only, while the test set merely contains the ‘un-

seen events’. In our experiment, we set n=4.

2.1 Model

We apply the most cutting-edge method PAIE [Ma et al.,

2022b], an efﬁcient prompt-based extraction model for both

sentence-level and documental-level EAE. It introduces span

selectors for extraction rather than forms it as a generation

task. Following PAIE’s original setting, we investigate two

different prompt creation strategies:

1name-based prompt creation, which concatenates

names of the arguments belonging to an event type.

2ontology-based prompt creation, where prompts are

derived from the event ontology, which depicts the in-

teractions among prompts in natural language.

2.2 Datasets

We conduct experiments on three common datasets in

event argument extraction: RAMS [Ebner et al., 2020],

WikiEvents[Li et al., 2021]and ACE05[Doddington et al.,

2004]. The former two datasets are annotated in documental-

level. RAMS includes 139 event types and 65 semantic role.

In the WikiEvents dataset, 50 event types and 59 seman-

tic roles are deﬁned. Although it renders the coreference

links of arguments, we only consider conventional arguments.

While ACE05 is a multilingual dataset providing sentence-

level event annotation. Here we use its English set for our

experiments. For data split of ACE05, we follow the prepro-

cessing procedure of DyGIE++ [Wadden et al., 2019].

ACE RAMS WikiEvents

name based 37.3 23.4 26.3

ontology based 43.5 34.3 32.0

Table 1: Overall results for pilot experiments. The metric used here

is span-level F1.

2.3 Results and Analysis

As shown in Table 1, on each dataset, ontology-based method

exceeds name-based remarkably by at least 5.8 point in F1

score. To further investigate how ontology-based prompts

success, we conduct a quantitative error analysis on each pre-

diction set. Imitating previous work [Das et al., 2022], we

deﬁne the mistakes when the ground truth is ‘None’ but the

model extracts some arguments as spurious role error, for it

reﬂects that the model extracts these none-existing arguments

dependent on some spurious relations. Figure 1(a) shows that

(a) Spurious Error Ratio

(b) Syntactic Role Matching Ratio

Figure 1: Detailed comparison between name-based prompt and

ontology-based prompt method.

on each dataset, ontology-base prompts generate less spuri-

ous role errors. Furthermore, by analysing the syntactic role

of predicted arguments, we observe that syntactic similarity

between the instance sentence and ontology-based prompt

promotes the extraction. Speciﬁcally, for every prediction,

we compare the dependency label, like ‘nsubj’ or ‘dobj’, of

each predicted argument in the original sentence and its argu-

ment name in the ontology by spaCy1. The matching degree

is shown in Figure 1(b). On every dataset, the dependency

labels of predictions from ontology-based prompt match the

event ontology better. And there is always a gap around 6

point in the matching ratio.

With above observations, it is reasonable to conclude that

compared with name-based prompt, ontology-based prompts

impose more syntactic level constraint when extracting, rec-

tifying some improper predictions by their syntactic roles.

As case study displayed in Table 2 sample 1, ‘Trump’ is the

subject of the input sentence, matching the role of ‘commu-

nicator’ in the ontology-based prompt. Depending on this

clue, the model extracts the right communicator ‘Trump’.

Although ontology-based prompt partly solves the long-

standing problem about entity type relying, it introduces new

vulnerability. In Table 2 sample 2, owing to the word ‘They’

is in the subject position, the same as where ‘employee’ lo-

cates in the ontology prompt, the model mistakenly identiﬁes

‘They’ as ’employee’. We will discuss such kind of potential

1https://github.com/explosion/spaCy

文档加载中……请稍候！
如果长时间未打开，您也可以点击刷新试试。

下载文档到电脑，查找使用更方便

10 玖币 0人已下载

立即下载

摘要：

CausalIntervention-basedPromptDebiasingforEventArgumentExtractionJiajuLin1,JieZhou2,QinChen11SchoolofComputerScienceandTechnology,EastChinaNormalUniversity2SchoolofComputerScience,FudanUniversityjiajulin@stu.ecnu.edu.cn,jiezhou@fudan.edu.cn,qchen@cs.ecnu.edu.cn,AbstractPrompt-basedmethodshavebecomei...

展开>> 收起<<

Causal Intervention-based Prompt Debiasing for Event Argument Extraction Jiaju Lin1Jie Zhou2Qin Chen1 1School of Computer Science and Technology East China Normal University.pdf

共8页,预览2页

还剩页未读，继续阅读

声明：本站为文档C2C交易模式，即用户上传的文档直接被用户下载，本站只是中间服务平台，本站所有文档下载所得的收益归上传人(含作者)所有。玖贝云文库仅提供信息存储空间，仅对用户上传内容的表现方式做保护处理，对上载内容本身不做任何修改或编辑。若文档所含内容侵犯了您的版权或隐私，请立即通知玖贝云文库，我们立即给予删除！

Causal Intervention-based Prompt Debiasing for Event Argument Extraction Jiaju Lin1Jie Zhou2Qin Chen1 1School of Computer Science and Technology East China Normal University

相关推荐

开通VIP享超值会员特权

作者详情

相关内容

热门标签

举报选择: