SentBS Sentence-level Beam Search for Controllable Summarization Chenhui Shen12Liying Cheng13Lidong Bingyy1Yang You2Luo Si1 1DAMO Academy Alibaba Group2National University of Singapore

2025-05-03 0 0 367.5KB 10 页 10玖币

侵权投诉

SentBS: Sentence-level Beam Search for Controllable Summarization

Chenhui Shen ∗1,2 Liying Cheng ∗1,3 Lidong Bing††1Yang You2Luo Si1

1DAMO Academy, Alibaba Group 2National University of Singapore

3Singapore University of Technology and Design

{chenhui.shen, liying.cheng}@alibaba-inc.com

{l.bing, luo.si}@alibaba-inc.com youy@comp.nus.edu.sg

Abstract

A wide range of control perspectives have

been explored in controllable text genera-

tion. Structure-controlled summarization is re-

cently proposed as a useful and interesting re-

search direction. However, current structure-

controlling methods have limited effectiveness

in enforcing the desired structure. To address

this limitation, we propose a sentence-level

beam search generation method (SentBS),

where evaluation is conducted throughout the

generation process to select suitable sentences

for subsequent generations. We experiment

with different combinations of decoding meth-

ods to be used as sub-components by SentBS

and evaluate results on the structure-controlled

dataset MReD. Experiments show that all ex-

plored combinations for SentBS can improve

the agreement between the generated text and

the desired structure, with the best method sig-

niﬁcantly reducing the structural discrepancies

suffered by the existing model, by approxi-

mately 68%. 1

1 Introduction

Controllable text generation is receiving increas-

ing attention due to its wide range of applications.

Depending on the use cases, the controllable gen-

eration tasks may focus on a wide range of control

perspectives, such as entities (Narayan et al.,2022;

Fan et al.,2018a), aspects (Hayashi et al.,2021),

and keywords (Wang et al.,2021;He et al.,2020).

Recently, Shen et al. (2022) propose a sentence-

level labeled meta-review dataset, MReD, for the

controllable summarization task from a new control

perspective that focuses on controlling the struc-

ture of the output summary. The input consists of

several reviews on the same research paper, and a

∗

Chenhui and Liying are under the Joint PhD Program

between Alibaba and their corresponding universities.

†† Corresponding author.

Our code and data are available at

https://github.com

/Shen-Chenhui/SentBS.

control sequence specifying the desired summary

structure. For instance, with a control sequence of

“abstract | strength | decision”, the generated output

should be composed of a sentence that summarizes

the contents of the paper, followed by a sentence

discussing the strengths, then the last sentence giv-

ing the ﬁnal decision.

Previous controllable summarization models are

commonly ﬁne-tuned on pre-trained transformer

architectures (Vaswani et al.,2017) such as BART

(Lewis et al.,2020) and Pegasus (Zhang et al.,

2020a), with the control signals merged into the

text input or prompts (Shen et al.,2022;Narayan

et al.,2022;He et al.,2020;Keskar et al.,2019;

Fan et al.,2018a). Previous works mainly focus

on improving the summary’s similarity with the

gold reference, leaving room for further improve-

ment on the controllability. In particular, the best-

performing model on the recently released MReD

dataset still generates around 29% of the sentences

that do not follow the control structure

, which is

far from satisfactory.

In this paper, we explore how to enhance the

structure-controllability in summarization. Specif-

ically, we notice the following possible pitfalls in

the existing summarization models. First, those

models usually treat generation as a standalone

process, which continuously generates the tokens

solely based on the logits predictions, without stop-

ping to reconsider whether the generated sequences

satisfy the control signals. Moreover, autoregres-

sive models can suffer from error propagation in

generation due to self-attention (Vaswani et al.,

2017). Therefore, if the previous sequences are not

well-controlled, subsequent generations may devi-

ate further from the desired output. Motivated by

this, we propose the

Sent

ence-level

eam

earch

(SentBS) method to address the identiﬁed issues.

For instance, the generated sentence may discuss the

weakness of the paper even though it corresponds to a

“strength” control label.

arXiv:2210.14502v3 [cs.CL] 24 Feb 2023

Sentence Classifier

-0.74

-0.35

-0.92

-1.13

Combined

Score

-0.81

-1.12

-0.49

-0.53

k = 4

n = 1

-1.21

-0.93

-0.65

-0.43

Final Output:

Sent 1: Option 1-1

Sent 2: Option 2-2

Sent L: Option L-0

…

Sent 1

Sequence

Likelihood

Classification

Score

Combined

Score

Sent 2 Combined

Score

Label 1 Label 2 Label L

Sent L

Option 2-1

Option 2-0

Option 2-2

Option 2-3

Option 1-1

Option 1-0

Option 1-2

Option 1-3

Option L-1

Option L-0

Option L-2

Option L-3

Control

Sequence:

Figure 1: Illustration of SentBS. The score values are for illustration purposes only. For simplicity, we only

illustrate for k= 4 and n= 1.

For generating each sentence, SentBS ﬁrst pro-

duces multiple sentence options, evaluates and se-

lects the best sentence according to both the control

structure as well as the model’s log-likelihood, then

continues the generation for the next sentence.

Experiments show that SentBS can signiﬁcantly

improve the model’s structure-controllability. In

particular, our best setting removes up to 68% of

control mistakes produced by the existing model

on MReD without compromising the summariza-

tion quality. The human evaluation further proves

that SentBS signiﬁcantly improves the ﬂuency of

summaries.

To summarize, our main contributions are: (1)

To the best of our knowledge, we are the ﬁrst to

conduct sentence-by-sentence controlled genera-

tion for structure-controllable summarization. (2)

We propose SentBS, which conducts text gener-

ation with continuous evaluation on the sentence

level with respect to the control requirements. This

method can be easily applied to existing autoregres-

sive models. (3) Experiments show that SentBS sig-

niﬁcantly increases the model’s structure controlla-

bility while preserving the summarization quality.

2 Related Work

Conditional Text Generation.

Large pretrained

language models have shown impressive perfor-

mance (Lewis et al.,2020;Zhang et al.,2020a)

on generation tasks. Many controllable generation

tasks (Shen et al.,2022;Narayan et al.,2022;Chia

et al.,2022;He et al.,2020;Cheng et al.,2020b;

Keskar et al.,2019;Fan et al.,2018a) make use of

these models by merging the control signals into

either the sources or targets. Our method differs

from these approaches by breaking the controlled

generation process down to the sentence level, and

we make explicit use of the control signals during

the generation phase.

Decoding Methods.

Common decoding meth-

ods include beam search (Meister et al.,2020;

Stahlberg and Byrne,2019;Graves,2012), nu-

cleus sampling (Holtzman et al.,2019), and beam

sampling (Caccia et al.,2019) (see more details

in Appendix A). In our SentBS method, we use

these methods

as sub-components for generat-

ing multiple sentence options. Hopkins and Kiela

(2017), Ghazvininejad et al. (2016) and Zhang and

Lapata (2014) also works on decoding methods

that enforce linguistic constraints on the genera-

tion structure. However, none of these works use

a sentence-by-sentence generation and evaluation

strategy based on whether the semantic content of

the generation meets the control requirements.

3 Sentence-Level Beam Search

Given the nature of autoregressive models, errors

from previously generated tokens at inference time

can be easily propagated to affect the subsequent

tokens. To better control the structure, we propose

SentBS, to evaluate and select the outputs at the

sentence level during generation. SentBS can be

easily applied to existing generation models during

inference only. In this section, we explain how

SentBS leverages the control sequence where each

control label corresponds to the desired sentence

category. Please see more details on the task and

dataset in Section 4.1.

3.1 Method Details

We illustrate the generation process of SentBS in

Figure 1. Given a generative model, a control se-

3https://huggingface.co/docs/transformers/int

ernal/generation_utils.

文档加载中……请稍候！
如果长时间未打开，您也可以点击刷新试试。

下载文档到电脑，查找使用更方便

10 玖币 0人已下载

立即下载

摘要：

SentBS:Sentence-levelBeamSearchforControllableSummarizationChenhuiShen1,2LiyingCheng1,3LidongBingyy1YangYou2LuoSi11DAMOAcademy,AlibabaGroup2NationalUniversityofSingapore3SingaporeUniversityofTechnologyandDesign{chenhui.shen,liying.cheng}@alibaba-inc.com{l.bing,luo.si}@alibaba-inc.comyouy@comp.nus....

展开>> 收起<<

SentBS Sentence-level Beam Search for Controllable Summarization Chenhui Shen12Liying Cheng13Lidong Bingyy1Yang You2Luo Si1 1DAMO Academy Alibaba Group2National University of Singapore.pdf

共10页,预览2页

还剩页未读，继续阅读

声明：本站为文档C2C交易模式，即用户上传的文档直接被用户下载，本站只是中间服务平台，本站所有文档下载所得的收益归上传人(含作者)所有。玖贝云文库仅提供信息存储空间，仅对用户上传内容的表现方式做保护处理，对上载内容本身不做任何修改或编辑。若文档所含内容侵犯了您的版权或隐私，请立即通知玖贝云文库，我们立即给予删除！

SentBS Sentence-level Beam Search for Controllable Summarization Chenhui Shen12Liying Cheng13Lidong Bingyy1Yang You2Luo Si1 1DAMO Academy Alibaba Group2National University of Singapore

相关推荐

开通VIP享超值会员特权

作者详情

相关内容

热门标签

举报选择: