Towards Improving Faithfulness in Abstractive Summarization Xiuying Chen1Mingzhe Li2Xin Gao1Xiangliang Zhang31

2025-05-06 0 0 947.87KB 13 页 10玖币

侵权投诉

Towards Improving Faithfulness in Abstractive

Summarization

Xiuying Chen1Mingzhe Li2Xin Gao1∗Xiangliang Zhang3,1∗

1Computational Bioscience Reseach Center, King Abdullah University of Science and Technology

2Ant Group

3University of Notre Dame

{xiuying.chen, xin.gao}@kaust.edu.sa, limingzhe.lmz@antgroup.com,xzhang33@nd.edu

Abstract

Despite the success achieved in neural abstractive summarization based on pre-

trained language models, one unresolved issue is that the generated summaries are

not always faithful to the input document. There are two possible causes of the

unfaithfulness problem: (1) the summarization model fails to understand or capture

the gist of the input text, and (2) the model over-relies on the language model

to generate ﬂuent but inadequate words. In this work, we propose a Faithfulness

Enhanced Summarization model (FES), which is designed for addressing these

two problems and improving faithfulness in abstractive summarization. For the

ﬁrst problem, we propose to use question-answering (QA) to examine whether the

encoder fully grasps the input document and can answer the questions on the key

information in the input. The QA attention on the proper input words can also

be used to stipulate how the decoder should attend to the source. For the second

problem, we introduce a max-margin loss deﬁned on the difference between the

language and the summarization model, aiming to prevent the overconﬁdence of the

language model. Extensive experiments on two benchmark summarization datasets,

CNN/DM and XSum, demonstrate that our model signiﬁcantly outperforms strong

baselines. The evaluation of factual consistency also shows that our model generates

more faithful summaries than baselines2.

1 Introduction

In recent years, text generation has made impressive progress [

]. The abstractive summarization

task, aiming to produce a concise and ﬂuent summary that is salient and faithful to the source

document, has become a research hotspot due to its broad application prospect. The prevalence of

pretrained transformer language models (LM) [

] has largely improved the ﬂuency and salience

of generated summaries. However, studies [

] showed that many summarization models suffer

from unfaithfulness problem, i.e., the generated summary is not entailed by the information presented

in the source document. Durmus et al. [

] highlighted two notions of the unfaithfulness problem in

summarization: one is the manipulation of information presented in the input document (intrinsic

errors), and the other is the inclusion of information not inferable from the input (extrinsic errors).

The Intrinsic error problem is often caused by the failure of document level inference, which is

necessary for abstractive summarization. Speciﬁcally, the summarization model has misinformation

inferred from the input document because of an inadequate encoder that misunderstands the source

semantic information and a poor decoder that cannot fetch relevant and consistent content from the

∗corresponding author

2https://github.com/iriscxy/FES

36th Conference on Neural Information Processing Systems (NeurIPS 2022).

arXiv:2210.01877v1 [cs.CL] 4 Oct 2022

encoder. Several recent summarization models were proposed from this perspective. For example,

Wu et al. [

] proposed a uniﬁed semantic graph encoder to learn better semantic meanings and a

graph-aware decoder to utilize the encoded information. Cao et al. [

] used contrastive learning

to help the model be aware of the factual information. The second type of error, extrinsic error, is

often introduced by excessive attention paid to the LM, which ensures ﬂuency while neglecting to

summarize the source document. For example, a LM is inclined to generate the commonly-used

phrase “score the winner” while the correct phrase is “score the second highest” which is less

frequently used. This type of error has been studied in the neural machine translation task [

], but

has not been addressed in abstractive summarization.

To address these errors, we propose a novel Faithfulness Enhanced Summarization model (FES).

To prevent the intrinsic error problem, we design FES in a multi-task learning paradigm, i.e.,

completing encoding-decoding for the summarization task with an auxiliary QA-based faithfulness

evaluation task. The QA task poses an additional reasoning requirement on the encoder to have a

more comprehensive understanding on the key semantic meanings of the input document and learn

better representations than working only for summarization. The QA attention on the key entities of

the input can also be used to align the decoder state with the encoder outputs for generating a faithful

summary. To address the extrinsic error problem, we propose a max-margin loss to prevent the LM

from being overconﬁdent. Concretely, we deﬁne an indicator of the overconﬁdence degree of the

LM. The risk of outputting extrinsic error tokens with low prediction probabilities is mitigated by

minimizing this overconﬁdence indicator.

We validate the effectiveness of our FES model by conducting extensive experiments on public

benchmark CNN/DM [

] and XSum [

] datasets. Experimental results demonstrate that our

faithfulness enhanced summarization model has superior performance on the ROUGE scores and

improves the faithfulness of news summarization over several strong baselines.

Our main contributions can be summarized as follows. (1) We propose a faithfulness enhanced

summarization model, which alleviates the unfaithfulness problem from the encoder side and decoder

side. (2) Concretely, we propose a multi-task framework to enhance the summarization performance

by automatic QA tasks. We also propose a max-margin loss to control the overconﬁdent problem

of the LM. (3) Experimental results demonstrate that our proposed approach brings substantial

improvements over the most recent baselines on benchmark datasets, and can also improve the

faithfulness of the generated summary.

2 Related Work

Abstractive Summarization.

In recent years, the research on text generation has made impressive

progress [

], which promotes the progress of abstractive summarization. The abstractive

summarization task generates novel words and phrases not featured in the source text to capture the

salient ideas of the source text [

]. Most works apply an encoder-decoder architecture to implicitly

learn the summarization procedure [

]. More recently, applying pretrained language models

as encoder [

] or pre-training the generation process by leveraging a large-scale of unlabeled

corpus [

] brings signiﬁcant improvements. Explicit structure modeling has also been shown to

be effective in summarization tasks. For example, Jin et al. [

] incorporated semantic dependency

graphs to help generate sentences with better semantic relevance, and Wu et al. [

] came up with a

uniﬁed semantic graph to aggregate relevant disjoint context from the input.

Fact Consistency for Abstractive Summarization.

Producing a summary that is entailed by the

information presented in the source document is a key challenge in the summarization task, and less

progress has been made on it. Pioneer works [

] incorporated fact descriptions or entailment

knowledge to enhance faithfulness. More recently, Zhu et al. [

] modeled the facts in the source

article with knowledge graphs based on a graph neural network. Cao et al. [

] proposed to leverage

reference summaries as positive training data and erroneous summaries as negative data, to train

summarization systems that are better at distinguishing between them. Aralikatte et al. [

] introduced

focus attention mechanism to encourage decoders to proactively generate tokens that are similar

or topical to the input document. On the contrary, other works post-edit the generated summaries.

Different from previous works, we enhance the semantic understanding of the document with

faithfulness evaluation as a direct signal and prevent the overconﬁdence of LM which is not addressed

before.

!"# $%#

Multi-task

Encoder

QA-enhanced

Decoder

(a) Existing QA-based faithfulness evaluation model (b) Our faithfulness-enhanced summarization model

Document Generated Summary

Ground-truth Summary

Ground-truth

Answers

Generated

Answers

Questions

Document

Generated

Answers Ground-truth

Answers

Ground-truth Summary

Questions

Generated Summary

ℒ&'

Summarization model training Faithfulness evaluation after training

ℒ(

ℒ%

Figure 1: The comparison of the existing QA-based faithfulness evaluation model and our

faithfulness-enhanced summarization model. The QA task integrated in our model provides an

auxiliary supervision signal to understand the document in the training process and enhance the

faithfulness of the generated summary.

Multi-task Learning.

Multi-task learning is a learning paradigm in machine learning and it aims to

leverage useful information contained in multiple related tasks to help improve the generalization

performance of all the tasks [

]. There is a large quantity of natural language processing tasks

formulated by multi-task learning, such as word segmentation, POS tagging, dependency parsing,

and text classiﬁcation [

]. In this work, we apply multi-task learning to summarization

and question-answering tasks for faithfulness enhancement.

3 Methodology

3.1 Problem Formulation

For an input document

X={x1, ..., xnx}

, we assume there is a ground truth summary

{y1, . . . , yny}

. In our faithfulness enhanced setting,

question answering pairs

Q={Q1, ..., Qnq}

with corresponding answers

A={A1, ..., Anq}

are also attached with

. In the training process,

our model is given QA pairs and document-summary pairs. It tries to extract answers

to the

questions and generate the summary

. In test stage, our model is given document

and questions

, and predicts the answers and summary. The ﬁnal goal is to generate a summary that is not only

informative but also consistent with document X.

Following, we introduce our proposed Faithfulness Enhanced Summarization model, which is

generally built on Transformer [

]. The faithfulness enhancement is implemented from three

aspects: (1) Multi-task Encoder. It improves the semantic understanding of the input document by

examining the quality of the encoded document representations for an auxiliary QA task. The encoded

representation thus captures the key inputs for making faithful summary. (2) QA Attention-enhanced

Decoder. The attention from the multi-task encoder aligns the decoder with the encoder so that the

decoder can fetch more accurate input information to generate the summary. (3) Max-margin Loss.

This is a loss orthogonal to the generation loss. It measures the accuracy of the LM and prevents it

from being overconﬁdent in the generation process.

3.2 Multi-task Encoder

Document

Tra n sforme r Encoder

Questions

Entity nodes:

Sentence nodes:

Question nodes:

Figure 2: Multi-task encoder.

The multi-task encoder is designed for encoding the input doc-

ument for both summarization and question-answering in an

integrated training process, as shown in Figure 1(b). This is

different from the previous work that uses QA in the post-

generation stage for evaluating the faithfulness of the generated

summaries [

], as shown in Figure 1(a). We bring the QA

closer to the encoder instead of leaving it for post-generated

summary, and make the encoder be trained to accomplish the

QA and summarization task in the meantime. This integrated

training of a multi-task encoder includes faithfulness also as an

optimization objective, besides the summary generation quality.

文档加载中……请稍候！
如果长时间未打开，您也可以点击刷新试试。

下载文档到电脑，查找使用更方便

10 玖币 0人已下载

立即下载

摘要：

TowardsImprovingFaithfulnessinAbstractiveSummarizationXiuyingChen1MingzheLi2XinGao1XiangliangZhang3;11ComputationalBioscienceReseachCenter,KingAbdullahUniversityofScienceandTechnology2AntGroup3UniversityofNotreDame{xiuying.chen,xin.gao}@kaust.edu.sa,limingzhe.lmz@antgroup.com,xzhang33@nd.eduAbstra...

展开>> 收起<<

Towards Improving Faithfulness in Abstractive Summarization Xiuying Chen1Mingzhe Li2Xin Gao1Xiangliang Zhang31.pdf

共13页,预览3页

还剩页未读，继续阅读

声明：本站为文档C2C交易模式，即用户上传的文档直接被用户下载，本站只是中间服务平台，本站所有文档下载所得的收益归上传人(含作者)所有。玖贝云文库仅提供信息存储空间，仅对用户上传内容的表现方式做保护处理，对上载内容本身不做任何修改或编辑。若文档所含内容侵犯了您的版权或隐私，请立即通知玖贝云文库，我们立即给予删除！

Towards Improving Faithfulness in Abstractive Summarization Xiuying Chen1Mingzhe Li2Xin Gao1Xiangliang Zhang31

相关推荐

开通VIP享超值会员特权

作者详情

相关内容

热门标签

举报选择: