SubeventWriter Iterative Sub-event Sequence Generation with Coherence Controller Zhaowei Wang1 Hongming Zhang2 Tianqing Fang1 Yangqiu Song1

2025-04-24 0 0 792.67KB 15 页 10玖币

侵权投诉

SubeventWriter: Iterative Sub-event Sequence Generation with Coherence

Controller

Zhaowei Wang1, Hongming Zhang2, Tianqing Fang1, Yangqiu Song1,

Ginny Y. Wong3, & Simon See3

1Department of Computer Science and Engineering, HKUST

2Tencent AI Lab, Bellevue, USA

3NVIDIA AI Technology Center (NVAITC), NVIDIA, Santa Clara, USA

{zwanggy, tfangaa, yqsong}@cse.ust.hk

hongmzhang@global.tencent.com, {gwong, ssee}@nvidia.com

Abstract

In this paper, we propose a new task of sub-

event generation for an unseen process to eval-

uate the understanding of the coherence of sub-

event actions and objects. To solve the prob-

lem, we design SubeventWriter, a sub-event

sequence generation framework with a coher-

ence controller. Given an unseen process, the

framework can iteratively construct the sub-

event sequence by generating one sub-event at

each iteration. We also design a very effective

coherence controller to decode more coherent

sub-events. As our extensive experiments and

analysis indicate, SubeventWriter1can gener-

ate more reliable and meaningful sub-event se-

quences for unseen processes.

1 Introduction

Natural language understanding involves deep un-

derstanding of events. In the NLP community,

there have been many event understanding tasks.

Most of them focus on parsing events into in-

volved entities, time, and locations as semantic

roles (Kingsbury and Palmer,2002;Li et al.,2013;

Lv et al.,2020;Lin et al.,2020;Du and Cardie,

2020;Zhang et al.,2021;Lyu et al.,2021a), or

identifying their binary relations such as tempo-

ral or causal relations (Berant et al.,2014;Smith

et al.,2018;Sap et al.,2019;Wang et al.,2020,

2021). However, our natural language can be used

to describe relations more than binary ones. For ex-

ample, processes (Craig et al.,1998), also known

as scripts (Schank and Abelson,1977) or activi-

ties (Mourelatos,1978), are complex events consti-

tuted by a sequence of sub-events. Understanding

processes can be more challenging than individual

or pair of events.

As shown in Figure 1, to complete the process of

making a chocolate cake, we need to consider a se-

quence of actions, “mix,” “add,” “pour,” and “bake,”

Code is available at

https://github.com/

HKUST-KnowComp/SubeventWriter.

Process: make a chocolate cake 30 cm x 19.05cm

How to make a chocolate cake? Step 1:

mix the dry ingredients. Step 2: [M]

Generate the 2nd sub-event

Beam Search Candidates Problem Coherent?

Add the carrots

Add the coffee and milk

Mix the dry ingredients

Right one

Redundant

Irrelevant

Sub-event Reference

1. Mix the dry ingredients.

3. Pour batter into cake pans.

2. Add the coffee and milk.

4. Bake the cake.

SubeventWriter

Figure 1: A motivating example of SubeventWriter,

which generates one sub-event at a time iteratively. We

show the process “make a chocolate cake” and the sec-

ond iteration of the generation. By considering coher-

ence, we can re-rank candidates and reach the right sub-

event. [M] is a mask token.

which involves different objects, e.g., dry ingredi-

ents, coffee, milk, etc. Those actions should follow

a logically coherent procedure while the objects

should be all related to the target, chocolate cake.

Thus, building such a coherent sequence should

take the whole sub-events into consideration.

There have been two categories of related studies

to processes, namely process induction and narra-

tive cloze tasks. Zhang et al. (2020a) proposed

a task to learn the hierarchical structure called

process induction, where a model needs to gen-

erate a sub-event sequence to ﬁnish a given pro-

cess. Their framework aggregates existing events

so that it can conceptualize and instantiate sim-

ilar processes. However, the aggregation proce-

dure does not consider the coherence of actions

and their objects. In addition, to build the dataset,

they extracted events using a dependency parser

arXiv:2210.06694v3 [cs.CL] 19 Oct 2022

with pre-deﬁned verb-argument templates (Zhang

et al.,2020b,2022). Such structured events might

harm coherence as only head words are retained

after extraction. Consider the ﬁrst sub-event in Fig-

ure 1. After parsing, we lost the indispensable mod-

iﬁer “dry” and the sub-event becomes (mix,ingre-

dients)

, which includes the wet ingredients (e.g.,

“milk”) in the second sub-event. Thus, the logical

relation between the two adjacent sub-events (i.e.,

coherence (Van Dijk,1980)) is defective.

On the other hand, narrative cloze tasks (Cham-

bers and Jurafsky,2008;Granroth-Wilding and

Clark,2016;Chambers,2017;Mostafazadeh et al.,

2016) evaluate whether a model can predict the

missing (usually the last) event in a narrative.

These tasks essentially evaluate the semantic simi-

larity and relatedness between the target event and

the context. However, they did not emphasize how

all events in the contexts are uniﬁed as a whole

process in an ordered and coherent way.

To evaluate complex process understanding, we

propose a new generation-based task to directly

generate sub-event sequences in the free-text form,

as shown in Figure 1. In the task, better genera-

tion of a process means better understanding of

the coherence among action verbs as well as their

operational objects. In fact, we ﬁnd that generat-

ing free-text events is a non-trivial task, even with

existing strong pre-trained models like T5 (Raffel

et al.,2020) and BART (Lewis et al.,2020). First,

generating an overlong piece of text containing

several temporally ordered sub-events at once is

challenging to current pre-trained models (Zhou

et al.,2022;Lin et al.,2021;Brown et al.,2020).

Next, sub-events are generated without considering

the coherence of actions and their objects, which

might give rise to irrelevant or redundant results.

To solve the task, we propose SubeventWriter

to generate sub-events iteratively in the temporal

order. SubeventWriter only generates the next sub-

event in each generation iteration, given the process

and prior generated sub-events. It eases the gen-

eration difﬁculty by decomposing the sub-event

sequence. Moreover, sub-events should be coher-

ently organized to complete a process. To con-

sider coherence in each iteration, we can get a few

sub-event candidates from the beam search and se-

lect the most coherent one, as shown in Figure 1.

In SubeventWriter, we introduce a coherence con-

troller to score whether a candidate is coherent with

2The matched pre-deﬁned template is (verb,object).

the process and prior generated sub-events. As a

result, SubeventWriter can construct more reliable

and meaningful sub-event sequences.

To evaluate our framework, we extract a large-

scale general-domain process dataset from Wiki-

How

, containing over 80k examples. We conduct

extensive experiments with multiple pre-trained

models, and automatic and human evaluations

show that SubeventWriter can produce more mean-

ingful sub-event sequences compared to existing

models by a large margin. Moreover, we con-

duct few-shot experiments to demonstrate that our

framework has a strong ability to handle few-shot

cases. Last but not least, we evaluate the gener-

alization ability of SubeventWriter on two out-of-

domain datasets: SMILE (Regneri et al.,2010) and

DeScript (Wanzare et al.,2016). The results mani-

fest our framework can generalize well.

2 Textual Sub-event Sequence

Generation

We formally deﬁne the sub-event sequence gen-

eration task as follows. Given a process

, we

ask the model to generate sub-event sequences

, which are steps to solve the process. This

task is essentially a conditional language modeling

problem. Speciﬁcally, given a process

consist-

ing of

tokens:

x1, x2, . . . , xn

and a sequence

consists of

sub-events

e1, e2, . . . , em

(each

sub-event refers to a sentence containing

tokens:

yi,1, yi,2, . . . , yi,ti

), models aim to learn the condi-

tional probability distribution by maximizing the

following conditional probabilities in Eq. (1):

Pθ(E|S) =

i=1

Pθ(ei|e<i, S)

Pθ(ei|e<i, S) =

j=1

Pθ(yi,j |yi,<j , e<i, S).

(1)

3 The SubeventWriter Framework

Figure 2illustrates the details of the proposed

SubeventWriter framework. For a given process,

the framework decomposes the generation into

multiple iterations. The sequence-to-sequence

(seq2seq) language model generates a few candi-

dates for the next sub-event in each iteration. We

then leverage a coherence controller to re-rank the

generated candidates by considering whether they

3wikihow.com

…

1 2 k

…

Initial Input:

How to make a chocolate

cake? Step 1:[M]

Candidate 1

Candidate 2

Coherence

Controller

Candidate k

New Input:

How to make a chocolate

cake? Step 1: Mix the dry

ingredients … Step i:

Candidate y.Step i +1: [M]

Step i:

Candidate y

To Step i+1 Take candidate y

with the largest Sum

Step i

Seq2Seq

…

Coherence Input:

How to make a chocolate

cake? Step 1: Mix the dry

ingredients … Step i:

Candidate x

40 cm x19.05cm

➕

…C

Sum

Conditional Probability

Figure 2: The overview of our SubeventWriter. In each iteration, the Seq2Seq language model takes the process

and prior generated sub-events as input and generates a few candidates for the next sub-event. Then the coherence

controller is used to select the most coherent candidate as the next sub-event.

are coherent with the process and prior generated

sub-events. The coherence controller is a discrim-

inative model that can assign a coherence score

to a sub-event sequence. It is ﬁne-tuned indepen-

dently on our synthetic data generated according

to our manually designed coherence rules. Finally,

the framework appends the generated sub-event to

the end of the input to serve as new context and

start the next iteration. The detailed description of

SubeventWriter components is as follows:

3.1 Iterative Event-level Decoding

The iterative event-level decoding scheme is built

on top of seq2seq language models, including

T5 (Raffel et al.,2020) and BART (Lewis et al.,

2020). We describe training and inference details

as follows.

Training:

The seq2seq language models are

ﬁne-tuned to decode one sub-event each time in

chronological order. For each process with its

sub-event sequences in the training data, we create

an augmented set of training examples with each

sub-event in the sequence as the output in turns.

For example, if the valid sequence of a process

consists of temporally ordered sub-events

, and

, we then create four training examples:

S→e1

S∪ {e1} → e2

S∪ {e1, e2} → e3

, and

S∪ {e1, e2, e3} → none

, where “

none

” is a spe-

cial token to end sequences. The order of adding

sub-events

follows the temporal order, which

ensures that the model only needs to predict what

will happen next without a longer-term forecast.

To minimize the gap between pre-training and

ﬁne-tuning, we design a textual prompt template to

construct input in human language. If we want to

generate the

i+ 1

th sub-event given a process

and sub-events

e1, e2, ..., ei

, the template takes the

form of “How to

? Step 1:

... Step i:

. Step

i+1: [M]” as the example shown in Figure 2. [M]

is the mask token of the model. More examples of

input/output are shown in Appendix A.1.

Inference:

During the inference, we apply the

seq2seq language models iteratively to generating

the sub-event sequence of a process. The aforemen-

tioned prompt template is also used. For instance,

the model ﬁrst generates sub-event

for a process

. It then takes

and

as input and generates

the second sub-event

. The model repeats this

process until the special token “

none

” is gener-

ated, which means no more sub-events are required.

Then, generated sub-events are concatenated into a

sequence as the ﬁnal output.

3.2 Coherence Controller

As a sub-event sequence should be coherent to com-

plete a process, we propose a coherence controller

to control the iterative event-level decoding. At

each iteration, the coherence controller considers

whether each sub-event candidate is coherent with

the given process and sub-events generated in previ-

ous iterations. Considering that sub-events (one or

more sentences) are diverse and complicated, here

we employ a coherence model (Jwalapuram et al.,

2021) based on BERT (Devlin et al.,2019) as the

coherence controller to score sub-event candidates.

We train the coherence controller as a binary

classiﬁcation task to discriminate coherent sub-

event sequences from incoherent ones. Following

previous works (Mesgar and Strube,2018;Moon

et al.,2019), we regard a human-written sub-event

sequence as coherent, and we synthetically build

two types of incoherent sub-event sequences by cor-

rupting the local or global coherence of the human-

written one. For

local coherence

, we randomly

文档加载中……请稍候！
如果长时间未打开，您也可以点击刷新试试。

下载文档到电脑，查找使用更方便

10 玖币 0人已下载

立即下载

摘要：

SubeventWriter:IterativeSub-eventSequenceGenerationwithCoherenceControllerZhaoweiWang1,HongmingZhang2,TianqingFang1,YangqiuSong1,GinnyY.Wong3,&SimonSee31DepartmentofComputerScienceandEngineering,HKUST2TencentAILab,Bellevue,USA3NVIDIAAITechnologyCenter(NVAITC),NVIDIA,SantaClara,USA{zwanggy,tfangaa,yq...

展开>> 收起<<

SubeventWriter Iterative Sub-event Sequence Generation with Coherence Controller Zhaowei Wang1 Hongming Zhang2 Tianqing Fang1 Yangqiu Song1.pdf

共15页,预览3页

还剩页未读，继续阅读

声明：本站为文档C2C交易模式，即用户上传的文档直接被用户下载，本站只是中间服务平台，本站所有文档下载所得的收益归上传人(含作者)所有。玖贝云文库仅提供信息存储空间，仅对用户上传内容的表现方式做保护处理，对上载内容本身不做任何修改或编辑。若文档所含内容侵犯了您的版权或隐私，请立即通知玖贝云文库，我们立即给予删除！

SubeventWriter Iterative Sub-event Sequence Generation with Coherence Controller Zhaowei Wang1 Hongming Zhang2 Tianqing Fang1 Yangqiu Song1

相关推荐

开通VIP享超值会员特权

作者详情

相关内容

热门标签

举报选择: