Measuring Fine-Grained Semantic Equivalence with Abstract Meaning Representation Shira Wein

2025-05-02 0 0 733.63KB 11 页 10玖币

侵权投诉

Measuring Fine-Grained Semantic Equivalence

with Abstract Meaning Representation

Shira Wein

Georgetown University

sw1158@georgetown.edu

Zhuxin Wang

Georgetown University

zw85@georgetown.edu

Nathan Schneider

Georgetown University

nathan.schneider@georgetown.edu

Abstract

Identifying semantically equivalent sentences

is important for many cross-lingual and mono-

lingual NLP tasks. Current approaches to

semantic equivalence take a loose, sentence-

level approach to “equivalence,” despite previ-

ous evidence that ﬁne-grained differences and

implicit content have an effect on human un-

derstanding (Roth and Anthonio,2021) and

system performance (Briakou and Carpuat,

2021). In this work, we introduce a novel,

more sensitive method of characterizing se-

mantic equivalence that leverages Abstract

Meaning Representation graph structures. We

develop an approach, which can be used

with either gold or automatic AMR annota-

tions, and demonstrate that our solution is in

fact ﬁner-grained than existing corpus ﬁlter-

ing methods and more accurate at predicting

strictly equivalent sentences than existing se-

mantic similarity metrics. We suggest that our

ﬁner-grained measure of semantic equivalence

could limit the workload in the task of human

post-edited machine translation and in human

evaluation of sentence similarity.

1 Introduction

Translation between two languages is not always

completely meaning-preserving, and information

can be captured by one sentence which is not cap-

tured by the other. For example, consider the par-

allel French and English sentences from the RE-

FreSD dataset (an annotated subset of the French-

English WikiMatrix (Schwenk et al.,2019)) shown

in ﬁgure 1. The French sentence says “tous les

autres édiﬁces” (all other buildings) while the En-

glish speciﬁes “all other religious buildings.” Be-

cause the sentence goes on to list religious build-

ings, it could be inferred from context that the

French is describing other religious buildings. The

French author, for whatever reason, chose to ex-

clude religious; the sentences thus convey the same

overall meaning but are not exactly parallel.

All other religious buildings are mosques or Koranic schools

founded after the abandonment of Old Ksar in 1957.

Tous les autres édiﬁces sont des mosquées ou des écoles

coraniques fondées à l’époque postérieure à l’abondance du

vieux ksar en 1957.

Figure 1: Two parallel sentences from the REFreSD

dataset marked as having no meaning divergence, for

which the AMRs diverge.

Semantic divergence (or conversely, semantic

equivalence) detection aims to pick out parallel

texts which have less than equivalent meaning. Un-

der a strict or close analysis of the translation, these

sentences could be considered divergent, because

the meanings are not identical (in particular, in-

formation made explicit in one is left implicit in

the other). But in the REFreSD corpus (Briakou

and Carpuat,2020), the sentences are annotated

as having no meaning divergence because at the

sentence-level they are essentially equivalent.

Semantic divergence detection plays an impor-

tant role in many cross-lingual NLP tasks, such

as text reuse detection, translation studies, and ﬁl-

tering of parallel corpora or MT output. Though

semantic divergence across sentences in parallel

corpora has been well-studied, current detection

methods fail to capture the full scope of semantic di-

vergence. State-of-the-art semantic divergence sys-

tems rely on perceived sentence-level divergences,

which do not entirely encapsulate all semantic di-

vergences. Two sentences are considered equiv-

alent (non-divergent) at the sentence level if the

same overall information is conveyed, even if there

are minor meaning differences. Finer-grained dif-

ferences are not widely considered in the detection

of semantic divergences, despite the fact that im-

plicit information can be critical to the understand-

ing of the sentence (Roth and Anthonio,2021).

We argue that a ﬁner-grained measure of seman-

tic equivalence is needed: a way to detect strictly se-

mantically equivalent sentence pairs. In this work,

we develop an approach to strict semantic equiva-

arXiv:2210.03018v1 [cs.CL] 6 Oct 2022

lence detection. Our approach to the identiﬁcation

of semantic equivalence moves beyond surface-

level perception of divergence and accounts for

more subtle differences in meaning that cannot be

detected by simply comparing the words of the sen-

tences. We leverage the Abstract Meaning Repre-

sentation (AMR; Banarescu et al.,2013) formalism

to identify these divergences at a ﬁne-grained level.

We present an approach to semantic equivalence

detection via AMR, with analysis of data in two lan-

guage pairs (English-French and English-Spanish).

Drawing on previous work on semantic divergence

(§2), we demonstrate that sentence-level diver-

gence annotations can be coarse-grained, neglect-

ing slight differences in meaning (§3). We ﬁnd that

comparing two AMR graphs is an effective way

to characterize meaning in order to uncover ﬁner-

grained divergences (§4), and this can be achieved

even with automatic AMR parsers (§5). Finally, in

§6we evaluate our AMR-based metric on a cross-

linguistic semantic textual similarity dataset, and

show that for detecting semantic equivalence, it is

more precise than a popular existing model, multi-

lingual BERTScore (Zhang et al.,2020).

Our primary contributions include:

•

Our novel approach to the identiﬁcation of se-

mantic divergence—which moves beyond per-

ceived sentence-level divergences—through

the lens of AMR

•

A simple pipeline algorithm that modiﬁes

Smatch to quantify divergence in cross-lingual

AMR pairs, automating the detection of AMR-

level divergence

•

Studies demonstrating that our AMR-based

approach accurately captures a ﬁner-grained

degree of semantic equivalence than both the

state-of-the-art corpus ﬁltering method and a

semantic textual metric

We will release the code and dataset for this work

upon publication to enable the use of AMR for

semantic divergence detection.

2 Background on Semantic Divergence

Semantic divergences can arise when translating

from one language or another. These divergences

can arise due to different language structure, syn-

tactic differences in the language, or translation

choices (Dorr,1994,1990). Additional divergences

can be introduced when automatically extracting

and aligning parallel resources (Smith et al.,2010;

Zhai et al.,2018;Fung and Cheung,2004).

To address these divergences, a number of sys-

tems have been developed to automatically iden-

tify divergences in parallel texts (Carpuat et al.,

2017;Vyas et al.,2018;Briakou and Carpuat,2020,

2021;Zhai et al.,2020). The approach taken by

Briakou and Carpuat (2020) to detecting sentence-

level semantic divergences involves training mul-

tilingual BERT (Devlin et al.,2018) to rank sen-

tences diverging to various degrees. They intro-

duced and evaluated on a novel dataset called Ra-

tional English-French Semantic Divergences (RE-

FreSD). REFreSD consists of English-French paral-

lel sentences, with crowdsourced annotations clas-

sifying the sentences as having no meaning diver-

gence, having some meaning divergence, or being

unrelated.

Recent work has investigated the differences in

cross-lingual (English-Spanish) AMR pairs within

the framework of translation divergences (Wein

and Schneider,2021). Speciﬁcally, this work de-

veloped an annotation schema to classify the types

and causes of differences between cross-lingual

AMR pairs. We use this dataset to test the per-

formance of our system on English-Spanish gold

AMR pairs. (For English-French, we produce our

own gold judgments of AMR divergence to test our

algorithm.) Additional prior work has explored the

role of structural divergences in cross-lingual AMR

parsing (Blloshmi et al.,2020;Damonte,2019).

3 AMR for Identiﬁcation of Semantic

Equivalence

Semantic representations are designed to capture

and formalize the meaning of a sentence. In partic-

ular, the Abstract Meaning Representation (AMR)

framework aims to formalize sentence meaning as a

graph in a way that is conducive to broad-coverage

manual annotation (Banarescu et al.,2013,2019).

These semantic graphs are rooted and labeled, such

that each node of the graph corresponds to a se-

mantic unit. AMR does not capture nominal or

verbal morphology or many function words, ab-

stracting away from the syntactic features of the

sentence. Attributes are labeled on edges between

nodes (concepts), and these attributes can be either

core roles / arguments, marked with

:ARG0

:ARG1

etc., or non-core roles such as

:time

:domain

, and

:manner

. The root of the AMR is usually the main

predicate of a sentence.

We leverage the semantic information captured

by AMR to recognize semantic equivalence or di-

He later scouted in Europe for the Montreal Canadiens.

(s / scout-02

:ARG0 (h / he)

:ARG1 (c / continent

:wiki "Europe"

:name "Europe")

:ARG2 (c2 / canadiens

:mod "Montreal")

:time (a / after))

Il a plus tard été dépisteur du Canadiens de Montréal en Eu-

rope. (He later scouted for the Montreal Canadiens in Eu-

rope.)

(d / dépister-02

:ARG0 (i / il)

:ARG1 (c / continent

:wiki "Europe"

:name "Europe")

:ARG2 (c2 / canadiens

:mod "Montreal")

:time (p / plus-tard))

Figure 2: A pair of sentences and their human an-

notated AMRs, for which the sentences receive a

“no meaning divergence” judgment in the REFreSD

dataset, and are also equivalent per AMR divergence.

vergence across parallel sentences. Figure 2, for

example, illustrates a strictly meaning-equivalent

sentence pair along with the AMRs. Though the

sentences differ with respect to syntax and lexical-

ization, the AMR graphs are structurally isomor-

phic. If the AMR structures were to differ, that

would signal a difference in meaning.

Two particularly beneﬁcial features of the AMR

framework are the rooted structure of each graph,

which elucidates the semantic focus of the sentence,

as well as the concrete set of speciﬁc non-core

roles, which are useful in classifying the speciﬁc

relation between concepts/semantic units in the

sentence. For example, in ﬁgure 3, the emphasis

on the English sentence is on possession—your

planet—but the emphasis on the Spanish sentence

is on place of origin, asking, which planet are you

from? This difference in meaning is reﬂected in the

diverging roots of the AMRs.

We ﬁnd that non-core roles (such as

:manner

:degree

, and

:time

) are particularly helpful in

identifying parallelism or lack of parallelism be-

tween the sentences during the annotation process.

This is because AMR abstracts away from the syn-

tax (so that word order and part of speech choices

do not affect equivalence), but instead explicitly

codes relationships between concepts via seman-

tic roles. Furthermore, AMRs use special frames

for certain relations, such as

have-rel-role-91

and

include-91

, which can be useful in enforcing

parallelism when the meaning is the same but the

Which is your planet?

(p / planet

:poss (y / you)

:domain (a / amr-unknown))

¿ De qué planeta eres ? (Which planet are you from?)

(s / ser-de-91

:ARG1 (t / tú)

:ARG2 (p / planeta

:domain (a / amr-desconocido)))

Figure 3: Two parallel sentences and AMRs from the

Migueles-Abraira et al. English-Spanish AMR dataset,

which diverge in meaning. The Spanish role labels are

translated into English here for ease of comparison.

Although the sales were slow (admittedly, according to the

band), the second single from the album, "Sweetest Surprise"

reached No. 1 in Thailand within a few weeks of release.

Même si les exemplaires ont du mal à partir (comme l’admet

le groupe), le second single de l’album, Sweetest Surprise,

atteint la première place en Thaïlande la première semaine de

sa sortie.

Figure 4: Two parallel sentences from the REFreSD

dataset (Briakou and Carpuat,2020) marked as having

no meaning divergence, but for which the AMRs di-

verge. Italicized spans indicate the cause of the AMR

divergence.

speciﬁc token is not the same. For example, if the

English and French both have a concession, but the

English marks it with “although” and the French

marks it with “mais” (but), the special frame role

will indicate this concession in the same way, pre-

serving parallelism.

Granularity of the REFreSD dataset.

Another

example, using sentences from the REFreSD

dataset, is shown in ﬁgure 4. These sentences are

marked as having no meaning divergence in the

REFreSD dataset but do have diverging AMR pairs.

The difference highlighted by the AMR pairs is

the

:time

role of reach / atteindre. The English

sentence says that no. 1 is reached “within a few

weeks” of the release, while the French sentence

says that no. 1 is reached the ﬁrst week of the re-

lease (la première semaine).

In examples like this

one it is made evident that sentence-level diver-

gence (as appears in REFreSD) do not capture

all meaning differences.

We explore the ability to discover semantic diver-

gences in sentences either with gold parallel AMR

annotations or with automatically parsed AMRs us-

ing a multilingual AMR parser, in order to enable

the use of this approach on large corpora (consider-

ing that AMR annotation requires training).

We propose that an approach to detecting di-

vergences using AMR will be a stricter, ﬁner-

文档加载中……请稍候！
如果长时间未打开，您也可以点击刷新试试。

下载文档到电脑，查找使用更方便

10 玖币 0人已下载

立即下载

摘要：

MeasuringFine-GrainedSemanticEquivalencewithAbstractMeaningRepresentationShiraWeinGeorgetownUniversitysw1158@georgetown.eduZhuxinWangGeorgetownUniversityzw85@georgetown.eduNathanSchneiderGeorgetownUniversitynathan.schneider@georgetown.eduAbstractIdentifyingsemanticallyequivalentsentencesisimportantf...

展开>> 收起<<

Measuring Fine-Grained Semantic Equivalence with Abstract Meaning Representation Shira Wein.pdf

共11页,预览3页

还剩页未读，继续阅读

声明：本站为文档C2C交易模式，即用户上传的文档直接被用户下载，本站只是中间服务平台，本站所有文档下载所得的收益归上传人(含作者)所有。玖贝云文库仅提供信息存储空间，仅对用户上传内容的表现方式做保护处理，对上载内容本身不做任何修改或编辑。若文档所含内容侵犯了您的版权或隐私，请立即通知玖贝云文库，我们立即给予删除！

Measuring Fine-Grained Semantic Equivalence with Abstract Meaning Representation Shira Wein

相关推荐

开通VIP享超值会员特权

作者详情

相关内容

热门标签

举报选择: