U3E Unsupervised and Erasure-based Evidence Extraction for Machine Reading Comprehension 1stSuzhe He

2025-05-06 0 0 299.15KB 7 页 10玖币

侵权投诉

U3E: Unsupervised and Erasure-based Evidence

Extraction for Machine Reading Comprehension

1stSuzhe He

Beijing Institute of Technology

Beijing, China

suzhehe@bit.edu.cn

2ndShumin Shi*

Beijing Institute of Technology

Beijing, China

bjssm@bit.edu.cn

3ndChengHao Wu

Beijing, China

874635423@qq.com

Abstract—More tasks in Machine Reading Comprehension

(MRC) require, in addition to answer prediction, the extraction

of evidence sentences that support the answer. However, the

annotation of supporting evidence sentences is usually time-

consuming and labor-intensive. In this paper, to address this issue

and considering that most of the existing extraction methods are

semi-supervised, we propose an unsupervised evidence extraction

method (U3E). U3E takes the changes after sentence-level feature

erasure in the document as input, simulating the decline in

problem-solving ability caused by human memory decline. In

order to make selections on the basis of fully understanding

the semantics of the original text, we also propose metrics to

quickly select the optimal memory model for this input changes.

To compare U3E with typical evidence extraction methods and

investigate its effectiveness in evidence extraction, we conduct

experiments on different datasets. Experimental results show that

U3E is simple but effective, not only extracting evidence more

accurately, but also signiﬁcantly improving model performance.

Index Terms—Machine Reading Comprehension, evidence ex-

traction, unsupervised learning, feature erasure

I. INTRODUCTION

MRC has attracted more and more attention, and it can

be roughly divided into two categories: extractive and non-

extractive. Extractive MRC requires one or more paragraphs

of text to be selected as the answer, such as SQuAD [1]

and DROP [2]. Non-extractive MRC need reasoning ability

more than the former. It includes multiple-choice MRC [3]–

[5], unanswerable questions [6], [7], veriﬁcation MRC [8],

[9]. As shown in table I, there is no doubt that evidence

sentences can not only help the MRC model to predict the

correct answer, but also help improve the interpretability of

the model. [36] effectively fuse the extracted evidence in

reasoning to enhance the power of relation extraction. [37]

aIteratively performing long document segmentation enhances

text generation performance.

However, it is difﬁcult to label evidence sentences on large-

scale datasets, especially for non-extractive MRC. Because a

large number of these questions are not just extractive (e.g.

87% of questions in RACE( [24], [25])). Answering Such

questions may require more advanced reading comprehension

skills such as single-sentence or multi-sentence reasoning

skills ( [26], [27]).

Considering the high cost of human-labeling evidence sen-

tences, some recent efforts have been devoted to improving

TABLE I

EXAMPLES OF VERIFIED READING COMPREHENSION. BOLD PARTS ARE

EVIDENCE SENTENCES. ANSWER 0MEANS THE DOCUMENT DOES NOT

SUPPORT THE OPTION, 1 MEANS IT DOES.

O: It is legal to own an ar15 in california.

D: ... State of California. California’s 2000 Assault Weapons ban

went further and banned AR-15s made by ...

A: 0

O: Ferguson jenkins is in the hall of fame.

D: ... following his major league career. In 1991, Jenkins became

the ﬁrst Canadian to be inducted into the National Baseball

Hall of Fame.

A: 1

MRC by exploiting noisy evidence labels when training ev-

idence extractors. Some work( [10], [11]) generates remote

tags using handcrafted rules and external sources, [23] apply

remote supervision to generate labels, denoising using a deep

probabilistic logical learning framework. Some studies ( [29])

employ reinforcement learning (RL) to determine the labels of

evidence, but this RL approach suffers from training instabil-

ity. More distant monitoring techniques are also used to reﬁne

noisy labels. However, improving the evidence extractor is still

challenging when golden evidence labels are not available.

So some recent works focus on weakly supervised extraction

of evidence sentences, [12] uses a small number of evidence

annotations combined with a large number of document-level

labels to select evidence, and [13] uses a self-training approach

that uses automatically generated evidence labels to supervise

evidence selection in an iterative process.

In this paper, we propose an unsupervised method U3E for

selecting evidence that is more in line with human intuition.

Inspired by [14], they change the characteristics of paragraph

input (called erasure) by penalizing illogical predictions to

improve the model.

Instead of the loss value used in their work, we choose to use

the change in the predicted value to reﬂect this memory-decay-

like behavior because the predicted value is more sensitive.

We also innovatively apply it to sentence level and the task of

evidence extraction in MRC. Besides, in order to better adapt

to this task, we propose an optimal memory model selection

arXiv:2210.02621v3 [cs.CL] 22 Oct 2022

method BMC (balance model and changes). Experiments show

that our method achieves good results.

II. RELATED WORK

A. Attribution Interpretation Methods

In the ﬁeld of hindsight, there are ”variable importance”

methods and gradient-based methods. The ”variable impor-

tance” method [15], [16] refers to the difference in the

prediction performance of a model when the value of a variable

changes. In gradient-based methods, the magnitude of the

gradient is used as the feature importance score. Gradient-

based methods are suitable for differentiable models [33].

Erasure [14] as a ”variable importance” method, it is model

independent. The advantage of the erasure method is that it

is conceptually simple and can be optimized for well-deﬁned

objectives [34].

B. Evidence Extraction

In the early MRC, some works focus on better represen-

tation of the features of the question and the context [17],

and constantly explore better fusion matching between them

[4], [18]. With the emergence of pre-training models (such as:

BERT [19]), some works want to understand the basis for the

model to predict the answer. Extracting evidences in MRC is

attracting increasing attention, although still quite challenging.

Evidence extraction aims to ﬁnd evidence and relevant infor-

mation for downstream processes in the task, which arguably

improves the interpretability of the task. Evidence extraction

is useful which is intuitive and becomes an important part of

fact veriﬁcation ( [38], [39]), multiple-choice reading com-

prehension ( [40]), open-domain question answering ( [10]),

multi-hop reading comprehension ( [41]) , natural language

inference ( [42]), and a wide range of other tasks ( [43]).

which can be roughly divided into two categories: one

is supervised learning, which requires a lot of resources to

manually label all the evidence sentence labels, such as:

HOTPOTQA [20] Select evidence sentences on the basis of

asking to answer their speciﬁc tasks, and work on it [21]

iteratively sorts the importance of sentences to select evidence

sentences, [22] decomposes the question and becomes a single-

hop MRC extracts the evidence sentence while selecting the

answer.

The second is semi-supervised learning. Because it is dif-

ﬁcult to extract evidence sentences in non-extractive MRC,

some works use semi-supervised methods to extract evidence,

[23] use remote supervision to generate imperfect labels, and

then use deep probabilistic logic learning to remove noise. [12]

label and improve model performance by combining speciﬁc

tasks with weakly supervised evidence extraction. Finally, on

the basis of weakly supervised learning, [28] use reinforcement

learning to obtain better evidence extraction strategies.

Our method U3E tends to use an unsupervised method,

complete the extraction task in stages. Adjust the erasure

method so that the importance of sentences can be obtained

explicitly. Compared with other methods, not only the cost is

small, but also the effect is remarkable.

Fig. 1. The overall structure of U3E, which includes three stages: T&A, S&R,

and A&R.

III. METHOD

The overall architecture of U3E is shown in ﬁgure 1, which

consists of three stages:

Train and Acquire (T&A): train models according to the

speciﬁc task and achieve changes.

Select and Reacquire (S&R): select the optimal memory

model according to our proposed BMC method and use the

model to reacquire changes.

Apply and Retrain (A&R): extract evidence through changes

and retrain according to the evidence.

In the following speciﬁc implementation, we will explain these

stages in order.

A. Task Deﬁnition

Assuming that each sample of the dataset can be formalized

as follows: Given a reference document consisting of multiple

sentences D={S1, S2, ..., Sm}and a statement O(If there

is a question, then Ois represented as the concatenation of

the question and the candidate) . The model should determine

whether to support this statement according to the document,

the support is marked as 1, otherwise it is marked as 0.

It can also use to extract the evidence sentence set E=

{Sj, Sj+1, ..., Sj+k−1}, which contains k(< m)sentences in

B. Train and Acquire

1) Task-speciﬁc Training: We ﬁrst train according to the

speciﬁc task (here is the classiﬁed task), and then save the

model M={M1, M2, ..., Mx}under all epochs, where

x represents the largest epoch trained. The model structure

during training is pretrained model1and linear layer. The input

is in ” [CLS] + Option + [SEP ] + Document + [SEP ]”

format. The hidden representation of the [CLS]token goes

through a linear layer for binary classiﬁcation to predict

whether the document Dsupports the sentence O:

ˆy=softmax(Wphcls)(1)

1different pretrained models used on different datasets

文档加载中……请稍候！
如果长时间未打开，您也可以点击刷新试试。

下载文档到电脑，查找使用更方便

10 玖币 0人已下载

立即下载

摘要：

U3E:UnsupervisedandErasure-basedEvidenceExtractionforMachineReadingComprehension1stSuzheHeBeijingInstituteofTechnologyBeijing,Chinasuzhehe@bit.edu.cn2ndShuminShi*BeijingInstituteofTechnologyBeijing,Chinabjssm@bit.edu.cn3ndChengHaoWuBeijing,China874635423@qq.comAbstractMoretasksinMachineReadingCompr...

展开>> 收起<<

U3E Unsupervised and Erasure-based Evidence Extraction for Machine Reading Comprehension 1stSuzhe He.pdf

共7页,预览2页

还剩页未读，继续阅读

声明：本站为文档C2C交易模式，即用户上传的文档直接被用户下载，本站只是中间服务平台，本站所有文档下载所得的收益归上传人(含作者)所有。玖贝云文库仅提供信息存储空间，仅对用户上传内容的表现方式做保护处理，对上载内容本身不做任何修改或编辑。若文档所含内容侵犯了您的版权或隐私，请立即通知玖贝云文库，我们立即给予删除！

U3E Unsupervised and Erasure-based Evidence Extraction for Machine Reading Comprehension 1stSuzhe He

相关推荐

开通VIP享超值会员特权

作者详情

相关内容

热门标签

举报选择: