OpenStance Real-world Zero-shot Stance Detection Hanzi Xu Slobodan Vucetic and Wenpeng Yin Temple University

2025-05-02 0 0 271.18KB 11 页 10玖币

侵权投诉

OpenStance: Real-world Zero-shot Stance Detection

Hanzi Xu, Slobodan Vucetic and Wenpeng Yin

Temple University

{hanzi.xu; slobodan.vucetic; wenpeng.yin}@temple.edu

Abstract

Prior studies of zero-shot stance detection

identify the attitude of texts towards unseen

topics occurring in the same document corpus.

Such task formulation has three limitations: (i)

Single domain/dataset. A system is optimized

on a particular dataset from a single domain;

therefore, the resulting system cannot work

well on other datasets; (ii) the model is evalu-

ated on a limited number of unseen topics; (iii)

it is assumed that part of the topics has rich an-

notations, which might be impossible in real-

world applications. These drawbacks will lead

to an impractical stance detection system that

fails to generalize to open domains and open-

form topics.

This work deﬁnes OpenStance: open-

domain zero-shot stance detection, aiming to

handle stance detection in an open world

with neither domain constraints nor topic-

speciﬁc annotations. The key challenge of

OpenStance lies in the open-domain gen-

eralization: learning a system with fully un-

speciﬁc supervision but capable of generaliz-

ing to any dataset. To solve OpenStance,

we propose to combine indirect supervision,

from textual entailment datasets, and weak su-

pervision, from data generated automatically

by pre-trained Language Models. Our single

system, without any topic-speciﬁc supervision,

outperforms the supervised method on three

popular datasets. To our knowledge, this is the

ﬁrst work that studies stance detection under

the open-domain zero-shot setting. All data

and code are publicly released.1

1 Introduction

Stance detection differentiates the attitude (e.g.,

support

oppose

, or

neutral

) of a text to-

wards a topic (Walker et al.,2012a). The topic can

be a phrase or a complete sentence. The same text

can express the author’s positions on many differ-

ent topics. For example, a tweet on climate warm-

1https://github.com/xhz0809/OpenStance

ing may also express attitudes about environmental

policies as well as the debate between electric or

fuel cars. Such compound expression can be seen

on all online platforms, including News outlets,

Twitter, blogs, etc. Therefore, stance detection can

be a complicated task that is essential for develop-

ing the inference capability of NLP models as well

as other disciplines such as politics, journalism,

etc.

Since the textual expressions and the size of top-

ics in the real world are unpredictable, zero-shot

stance detection has become the mainstream re-

search direction in this area: topics in the test set

are unseen during training. For example, Moham-

mad et al. (2016) created a dataset SemT6 based on

tweets with six noun phrases as topics. One of the

topics was reserved for testing and the remaining

were used for training. Allaway and McKeown

(2020) extended the topic size on the domain of

news comments by covering 4,000 topics in train-

ing and 600 unseen topics in testing.

However, despite the change in the domain and

topic size, there are three major limitations in previ-

ous studies which make the task not a real zero-shot

task: (i) the dataset only contains texts from a sin-

gle domain, such as news comments in VAST (All-

away and McKeown,2020) and tweets in SemT6

(Mohammad et al.,2016); (ii) most literature stud-

ied only a limited size of topics with a single textual

form (either noun phrases or sentential claims), e.g.,

(Mohammad et al.,2016;Conforti et al.,2020); (iii)

rich annotation for at least part of the topics is al-

ways required, which is not possible in real-world

applications because data collection can be very

time-consuming and costly (Enayati et al.,2021).

Those limitations lead to an impractical zero-shot

stance detection system that cannot generalize well

to unseen domains and open-form topics.

In this work, we re-deﬁne what a zero-shot

stance detection should be. Speciﬁcally, we deﬁne

OpenStance

: an open-domain zero-shot stance

arXiv:2210.14299v1 [cs.CL] 25 Oct 2022

detection, aiming to build a system that can work in

the real world without any speciﬁc attention to the

text domains or topic forms. More importantly, no

task-speciﬁc supervision is needed. To achieve this,

we propose to combine two types of supervision:

indirect supervision and weak supervision. The in-

direct supervision comes from textual entailment—

we treat the stance detection problem as a textual

entailment task since the attitude toward a topic

should be inferred from the input text. Therefore,

the existing entailment datasets, such as MNLI

(Williams et al.,2018), can contribute supervision

to the zero-shot setting. To collect supervision

that is more speciﬁc to the

OpenStance

task,

we design two MASK choices (

MASK-topic

and

MASK-text

) to prompt GPT-3 (Brown et al.,

2020) to generate weakly supervised data. Given an

input text and a stance label (

support

oppose

neutral

MASK-topic

predicts what topic

is appropriate based on the content; given a topic

and a label,

MASK-text

seeks the text that most

likely holds this stance. The collection of weakly

supervised data only needs the unlabeled texts and

the set of topics that users want to include. The

joint power of indirect supervision and weak super-

vision will be evaluated on VAST, SemT6 and Per-

spectrum (Chen et al.,2019), three popular datasets

that cover distinct domains, different sizes and di-

verse textual forms of topics. Experimental results

show that although no task-speciﬁc supervision is

used, our system can get robust performance on all

three datasets, even outperforming the task-speciﬁc

supervised models (72.6 vs. 69.3 by mean F1 over

the three datasets).

Our contributions are threefold: (i) we deﬁne

OpenStance

, an open-domain zero-shot stance

detection task, that fulﬁlls real-world requirements

while having never been studied before; (ii) we de-

sign a novel masking mechanism to let GPT-3 gen-

erate weakly supervised data for

OpenStance

This mechanism can inspire other NLP tasks that

detect relations between two pieces of texts; (iii)

our approach, integrating indirect supervision and

weak supervision, demonstrates outstanding gen-

eralization among three datasets that cover a wide

range of text domains, topic sizes and topic forms.

2 Related Work

Stance detection.

Stance detection, as a recent

member of the NLP family, was mainly driven by

newly created datasets. In the past studies, datasets

have been constructed from diverse domains like

online debate forums (Walker et al.,2012b;Hasan

and Ng,2014;Abbott et al.,2016), news comments

(Krejzl et al.,2017;Lozhnikov et al.,2018), Twitter

(Mohammad et al.,2016;Küçük,2017;Tsakalidis

et al.,2018)), etc.

Zero-shot stance detection.

Recently, re-

searchers started to work on zero-shot stance

detection in order to build a system that can handle

unseen topics. Most work split the collected

topic-aware annotations into train and test within

the same domain. Allaway and McKeown (2020)

made use of topic similarity to connect unseen top-

ics with seen topics. Allaway et al. (2021) designed

adversarial learning to learn domain-independent

information and topic-invariant representations.

Similarly, Wang and Wang (2021) applied

adversarial learning to extract stance-related but

domain-invariant features existed among different

domains. Liu et al. (2021) utilized common sense

knowledge from ConceptNet (Speer et al.,2017) to

introduce extra knowledge of the relations between

the texts and topics. Most prior systems worked on

a single domain and were tested on a small number

of unseen topics. Li et al. (2021) tried to test on

various unseen datasets by jointly optimizing on

multiple training datasets. However, they still

assumed that part of the topics or domains has rich

annotations. In contrast, our goal is to design a

system that can handle stance detection in an open

world without requiring any domain constraints or

topic-speciﬁc annotations.

Textual entailment as indirect supervision.

Textual entailment studies if a hypothesis can be

entailed by a premise; this was proposed as a uni-

ﬁed inference framework for a wide range of NLP

problems (Dagan et al.,2005). Recently, textual en-

tailment is widely utilized to help solve many tasks,

such as few-shot intent detection (Xia et al.,2021),

ultra-ﬁne entity typing (Li et al.,2022), corefer-

ence resolution (Yin et al.,2020), relation extrac-

tion (Xia et al.,2021;Sainz et al.,2021), event

argument extraction (Sainz et al.,2022), etc. As

far as we know, our work is the ﬁrst one that suc-

cessfully leverages the indirect supervision from

textual entailment for stance detection.

Weak supervision from GPT-3.

As the cur-

rently most popular and (arguably) well-behaved

pre-trained language model, GPT-3 (Brown et al.,

2020) has been a great success on few-shot and

zero-shot NLP. As an implicit knowledge base

fully in the form of parameters, it is not surpris-

ing that researchers attempt to extract knowledge

from it to construct synthetic data, e.g., (Yoo

et al.,2021;Wang et al.,2021). We use GPT-3

to collect distantly supervised data by two novel

masking mechanisms designed speciﬁcally for the

OpenStance.

3 Problem deﬁnition

OpenStance has the following requirements:

•

An instance includes three items: text

, topic

and a stance label

(

l∈

{support,oppose,neutral}

); the task

is to learn the function f(s, t)→l;

•

The text

can come from any domain; the

topic

can be any textual expressions, such

as a noun phrase “gun control” or a sentential

claim “climate change is a real concern”;

•

All labeled instances {(

)} only exist in

test; no train or dev is provided;

•

Previous work used different metrics for the

evaluation. For example, VAST (Allaway and

McKeown,2020) used macro-averaged F1 re-

garding stance labels, while studies on SemT6

(Allaway et al.,2021;Liang et al.,2022) re-

ported the F1 scores per topic. To make sys-

tems be comparable, we unify the evaluation

and use the label-oriented macro F1 as our

main metric.

OpenStance vs. prior zero-shot stance detec-

tion.

Prior studies of zero-shot stance detection

worked on a single dataset

in which all texts

comes from the same domain. Topics

in the

dataset are split into

train

dev

and

test

disjointly.

The main issue is that a model that ﬁts

does not

work well on a new dataset

that may contain

of different domains and unseen

. For example,

a model trained on VAST can only get F1 49.0%

on Perspectrum, which is around the performance

of random guess.

OpenStance

aims at handling

multiple datasets of open domains and open-form

topics without looking at their train and dev.

OpenStance vs. textual entailment.

Stance

detection is essentially a textual entailment prob-

lem if we treat the text

as the premise, and the

stance towards the topic

as the hypothesis. This

motivates us to use indirect supervision from tex-

tual entailment to deal with the stance detection

problem. Nevertheless, there are two distinctions

between them: (i) even though we can match

of stance detection with the labels of textual en-

tailment:

support →entailment

oppose

→contradict

and

neutral →neutral

whether a topic

in stance detection can be treated

as a hypothesis depends on the text form of

. If

noun phrases such as “gun control”,

cannot act as

a hypothesis alone as there is no stance in it; if

a sentential claim such as “climate change is a real

concern”, inferring the truth value of this hypoth-

esis is exactly a textual entailment problem. This

observation motivates us to test

OpenStance

topics of both phrase forms and sentence forms;

(ii) Zero-shot textual entailment means the size of

the annotated instances for labels is zero, while

OpenStance

requires the topics have zero la-

beled examples.

4 Methodology

This section introduces how we collect and com-

bine indirect supervision and weak supervision to

solve OpenStance.

Indirect Supervision.

As we discussed in Sec-

tion 3, stance detection is a case of textual entail-

ment since the stance

towards a topic

should be

inferred from the text

. To handle the zero-shot

challenge in

OpenStance

, textual entailment is

a natural choice for indirect supervision.

Speciﬁcally, we ﬁrst cast stance detection in-

stances into the textual entailment format by com-

bining

and

as a sentential hypothesis

, such

as “it

supports topic

”, and treating the

the premise

; then a pretrained model on MNLI

(Williams et al.,2018), one of the largest entail-

ment dataset, is ready to predict the relationship

between the

and

. An entailed (resp. contra-

dicted or neutral)

means the topic

is supported

(resp. opposed or neutral) by the text s.

Unfortunately, the indirect supervision from tex-

tual entailment may not perform well enough in

real-world

OpenStance

considering the widely

known brittleness of pretrained entailment mod-

els and the open domains and open-form topics in

OpenStance

. Therefore, in addition to the indi-

rect supervision from textual entailment, we will

collect weak supervision that is aligned better with

the texts {x}and the topics {t}.

文档加载中……请稍候！
如果长时间未打开，您也可以点击刷新试试。

下载文档到电脑，查找使用更方便

10 玖币 0人已下载

立即下载

摘要：

OpenStance:Real-worldZero-shotStanceDetectionHanziXu,SlobodanVuceticandWenpengYinTempleUniversity{hanzi.xu;slobodan.vucetic;wenpeng.yin}@temple.eduAbstractPriorstudiesofzero-shotstancedetectionidentifytheattitudeoftextstowardsunseentopicsoccurringinthesamedocumentcorpus.Suchtaskformulationhasthreeli...

展开>> 收起<<

OpenStance Real-world Zero-shot Stance Detection Hanzi Xu Slobodan Vucetic and Wenpeng Yin Temple University.pdf

共11页,预览3页

还剩页未读，继续阅读

声明：本站为文档C2C交易模式，即用户上传的文档直接被用户下载，本站只是中间服务平台，本站所有文档下载所得的收益归上传人(含作者)所有。玖贝云文库仅提供信息存储空间，仅对用户上传内容的表现方式做保护处理，对上载内容本身不做任何修改或编辑。若文档所含内容侵犯了您的版权或隐私，请立即通知玖贝云文库，我们立即给予删除！

OpenStance Real-world Zero-shot Stance Detection Hanzi Xu Slobodan Vucetic and Wenpeng Yin Temple University

相关推荐

开通VIP享超值会员特权

作者详情

相关内容

热门标签

举报选择: