Multilingual BERT has an accent Evaluating English inﬂuences on ﬂuency in multilingual models Isabel Papadimitriou and Kezia Lopez and Dan Jurafsky

2025-04-24 0 0 462.79KB 7 页 10玖币

侵权投诉

Multilingual BERT has an accent:

Evaluating English inﬂuences on ﬂuency in multilingual models

Isabel Papadimitriou* and Kezia Lopez* and Dan Jurafsky

Computer Science Department

Stanford University

{isabelvp,keziakl,jurafsky}@stanford.edu

Abstract

While multilingual language models can im-

prove NLP performance on low-resource lan-

guages by leveraging higher-resource lan-

guages, they also reduce average performance

on all languages (the ‘curse of multilingual-

ity’). Here we show another problem with

multilingual models: grammatical structures

in higher-resource languages bleed into lower-

resource languages, a phenomenon we call

grammatical structure bias. We show this bias

via a novel method for comparing the ﬂuency

of multilingual models to the ﬂuency of mono-

lingual Spanish and Greek models: testing

their preference for two carefully-chosen vari-

able grammatical structures (optional pronoun-

drop in Spanish and optional Subject-Verb or-

dering in Greek). We ﬁnd that multilingual

BERT is biased toward the English-like setting

(explicit pronouns and Subject-Verb-Object or-

dering) as compared to our monolingual con-

trol language model. With our case studies,

we hope to bring to light the ﬁne-grained ways

in which multilingual models can be biased,

and encourage more linguistically-aware ﬂu-

ency evaluation.

1 Introduction

Multilingual language models share a single set of

parameters between many languages, opening new

pathways for multilingual and low-resource NLP.

However, not all training languages have an equal

amount, or a comparable quality of training data

in these models. In this paper, we investigate if

the hegemonic status of English inﬂuences other

languages in multilingual language models. We

propose a novel method for evaluation, whereby

we ask if model predictions for lower-resource lan-

guages exhibit structural features of English. This

is similar to asking if the model has learned some

languages with an “English accent”, or an English

grammatical structure bias.

We demonstrate this bias effect in Spanish and

Greek, comparing the monolingual models BETO

Monolingual

model

Control ratio

Multilingual

model Test ratio

Compare:

Is multilingual model

more English-biased?

English-like corpus:

Spanish with

pronoun

Non-English-like

corpus:

Spanish with

Prodrop

Figure 1: Our method for evaluating English structural

bias in multilingual models. We compare monolingual

and multilingual model predictions on two sets of natu-

ral sentences in the target language: one which is struc-

turally parallel to English, and one which is not.

(Cañete et al.,2020) and GreekBERT (Koutsikakis

et al.,2020) to multilingual BERT (mBERT),

where English is the most frequent language in

the training data. We show that mBERT prefers

English-like sentence structure in Spanish and

Greek compared to the monolingual models. Our

case studies focus on Spanish pronoun drop (pro-

drop) and Greek subject-verb order, two structural

grammatical features. We show that multilingual

BERT is structurally biased towards explicit pro-

nouns rather than pro-drop in Spanish, and subject-

before-verb order in Greek: the structural forms

parallel to English.

Though the effect we showcase here is likely not

captured by the downstream classiﬁcation tasks of-

ten used to evaluate multilingual models (Hu et al.,

2020), it demonstrates the type of ﬂuency that can

be lost with multilingual training — something that

current evaluation methods miss. In fact, though

we choose two clear-cut syntactic features to in-

vestigate, there are many less-measurable features

that make language production ﬂuent: subtleties in

lexical choice, grammatical choice, and discourse

expression, among many others. With this paper,

beyond showing a trend for two speciﬁc grammati-

cal features, we wish to highlight ﬂuency discrepan-

cies in multilingual models, and also call for more

evaluations focused on ﬂuency.

arXiv:2210.05619v2 [cs.CL] 13 Apr 2023

Sparallel: English-like structure Sdifferent: Different structure

Spanish explicit pronoun (pron in red, verb in blue) Spanish prodrop (verb in blue)

Yo volveré para averiguarlo Jamás dan soluciones y siempre [. . . ]

Iwill return to ﬁgure it out [They] Never give solutions and always [. . . ]

El 2004 , ella hizo doblaje a el Inglés [. . . ] Jugó de centrocampista en el Real Zaragoza

In 2004, she did dubbing to English [. . . ] [He/She/You] Played as a midﬁelder in Real Zaragoza

Ella decide pasar sus vacaciones en la granja Habita en Perú .

She decides to spend her vacation in the country [He/She/You] Lives in Peru

Greek Subject-Verb (subject in red, verb in blue) Greek Verb-Subject (subject in red, verb in blue)

Πηγές της Αντιπολίτευσης αναφέρουν ότι [...] Το σκορ του αγώνα άνοιξε οΓουέν Ρούνι

Sources of the Opposition mention that [. . . ] The score of the game opened Wayne Rooney

Σε άλλες πλευρές ο ποταμός κυλά από ψηλούς

βράχους

Εδώ πρέπει να γίνουν μεγαλύτερες προσπά-

θειες.

On other sides, the river ﬂows from tall boulders Here must happen bigger efforts

Ηεκπαίδευση και η μόρφωση απέκτησαν

επιτέλους προτεραιότητα

Απασχόληση στο εξωτερικό ψάχνουν οι μισοί

΄Ελληνες σε παραγωγική ηλικία

Training and education have ﬁnally acquired priority Employment in foreign countries search half of Greeks

Table 1: Examples from our dataset for Sparallel and Sdifferent in Spanish and Greek, along with roughly word-

by-word gloss translations in English. In all cases, we’ve underlined w(x), the word we use to represent the

construction in our calculations. These examples are not randomly selected and have been chosen to be signiﬁcantly

shorter than the average sentence in our datasets in order to be presentable in a table.

Our proposed method can be expanded, without

the need for manual data collection, to any lan-

guage with a syntactic treebank and a monolingual

model. Since our method focuses on ﬁne-grained

linguistic features, some expert knowledge of the

target language is necessary for evaluation. Multi-

lingual evaluation so far has been largely translated

or automatically curated, and the methods for cre-

ating such datasets have allowed for the creation

of resources in many languages for which there

there were none. Fluency evaluation requires some

linguistic expertise to set up, and as such is more

restricted in the languages the research community

can reach. Nevertheless, such evaluation has been

missing from the multilingual NLP literature, and

our work bridges this gap by proposing ﬂuency

testing for multilingual models.

Our work builds off of a long literature on mul-

tilingual evaluation which has until now mostly

focused on downstream classiﬁcation tasks (Con-

neau et al.,2018;Ebrahimi et al.,2022;Clark et al.,

2020;Liang et al.,2020;Hu et al.,2020;Raganato

et al.,2020;Li et al.,2021). With the help of

these evaluation methods, research has pointed out

the problems for both high- and low-resource lan-

guages that come with adding many languages to a

single model (Wang et al.,2020;Turc et al.,2021;

Lauscher et al.,2020, inter alia). Methods for cre-

ating more equitable models have been proposed,

through identifying or reserving language-speciﬁc

parameters for each language (Ansell et al.,2022;

Pfeiffer et al.,2022), through training models with-

out tyoplogically distant languages that dominate

the training data (Ogueji et al.,2021;Virtanen

et al.,2019;Ògúnr

mí and Manning,2023), as

well as through adding model capacity (Conneau

et al.,2020;Xue et al.,2021;Lepikhin et al.,2021;

Liang et al.,2023). We hope that our work can add

to these analyses and methodologies by pointing

out issues beyond downstream classiﬁcation perfor-

mance that can arise with multilingual training, and

aid towards building and evaluating more equitable

multilingual models.

2 Method

Our method relies on ﬁnding a variable construc-

tion in the target language which can take two struc-

tural surface forms: one which is parallel to English

(

Sparallel

) and one which is not (

Sdifferent

). Surface

forms parallel to English are those which mirror

English structure. For example, English has strict

Subject-Verb-Object word order, and so a parallel

structure in another language is one where the verb

文档加载中……请稍候！
如果长时间未打开，您也可以点击刷新试试。

下载文档到电脑，查找使用更方便

10 玖币 0人已下载

立即下载

摘要：

MultilingualBERThasanaccent:EvaluatingEnglishinuencesonuencyinmultilingualmodelsIsabelPapadimitriou*andKeziaLopez*andDanJurafskyComputerScienceDepartmentStanfordUniversity{isabelvp,keziakl,jurafsky}@stanford.eduAbstractWhilemultilinguallanguagemodelscanim-proveNLPperformanceonlow-resourcelan-guage...

展开>> 收起<<

Multilingual BERT has an accent Evaluating English inﬂuences on ﬂuency in multilingual models Isabel Papadimitriou and Kezia Lopez and Dan Jurafsky.pdf

共7页,预览2页

还剩页未读，继续阅读

声明：本站为文档C2C交易模式，即用户上传的文档直接被用户下载，本站只是中间服务平台，本站所有文档下载所得的收益归上传人(含作者)所有。玖贝云文库仅提供信息存储空间，仅对用户上传内容的表现方式做保护处理，对上载内容本身不做任何修改或编辑。若文档所含内容侵犯了您的版权或隐私，请立即通知玖贝云文库，我们立即给予删除！

Multilingual BERT has an accent Evaluating English inﬂuences on ﬂuency in multilingual models Isabel Papadimitriou and Kezia Lopez and Dan Jurafsky

相关推荐

开通VIP享超值会员特权

作者详情

相关内容

热门标签

举报选择: