rich lexical variations across different languages.
On the other hand, despite that AMR is advo-
cated to act as an interlingua (Xue et al.,2014;
Hajiˇ
c et al.,2014;Damonte and Cohen,2018), lit-
tle work has been done to reflect on the ability of
AMR to have impact on subsequent tasks. In order
to advance research in AMR and its applications,
multilingual sentence embedding can be seen as
an important benchmark for highlighting its abil-
ity to abstract away from surface realizations and
represent the core concepts expressed in the sen-
tence. To our knowledge, this is the first attempt
to leverage the AMR semantic representation for
multilingual NLP.
We learn AMR embeddings with contrastive
siamese network (Gao et al.,2021) and AMR
graphs derived from different languages (Cai et al.,
2021). Experiment results on 10 STS tasks and
5 transfer tasks with four state-of-the-art embed-
ding methods show that retrofitting multilingual
sentence embeddings with AMR improves the per-
formance substantially and consistently.
Our contribution is three-fold.
•
We propose a new method to obtain high-quality
semantic vectors for multilingual sentence rep-
resentation, which takes advantage of language-
invariant Abstract Meaning Representation that
captures the core semantics of sentences.
•
We present a thorough evaluation of multilingual
sentence embeddings, which goes beyond seman-
tic textual similarity and includes various transfer
tasks in downstream applications.
•
We demonstrate that retrofitting multilingual sen-
tence embeddings with Abstract Meaning Repre-
sentation leads to better performance on both se-
mantic textual similarity and transfer tasks.
2 Related Work
Universal Sentence Embeddings
Our work
aims to learn universal sentence representations,
which should be useful for a broad set of ap-
plications. There are two lines of research for
universal sentence embeddings: unsupervised ap-
proaches and supervised approaches. Early unsu-
pervised approaches (Kiros et al.,2015;Hill et al.,
2016;Gan et al.,2017;Logeswaran and Lee,2018)
design various surrounding sentence reconstruc-
tion/prediction objectives for sentence representa-
tion learning. Jernite et al. (2017) exploit sentence-
level discourse relations as supervision signals for
training sentence embedding model. Instead of us-
ing the interactions of sentences within a document,
Le and Mikolov (2014) propose to learn the embed-
dings for texts of arbitrary length on top of word
vectors. Likewise, Chen (2017); Pagliardini et al.
(2018); Yang et al. (2019b) calculate sentence em-
beddings from compositional
n
-gram features. Re-
cent approaches often adopt contrastive objectives
(Zhang et al.,2020;Giorgi et al.,2021;Wu et al.,
2020;Meng et al.,2021;Carlsson et al.,2021;Kim
et al.,2021;Yan et al.,2021;Gao et al.,2021) by
taking different views—from data augmentation or
different copies of models—of the same sentence
as training examples.
On the other hand, supervised methods (Con-
neau et al.,2017;Cer et al.,2018;Reimers and
Gurevych,2019;Gao et al.,2021) take advan-
tage of labeled natural language inference (NLI)
datasets (Bowman et al.,2015;Williams et al.,
2018), where a sentence embedding model is fine-
tuned on entailment or contradiction sentence pairs.
Furthermore, Wieting and Gimpel (2018); Wieting
et al. (2020) demonstrate that bilingual and back-
translation corpora provide useful supervision for
learning semantic similarity. Another line of work
focuses on regularizing embeddings (Li et al.,2020;
Su et al.,2021;Huang et al.,2021) to alleviate the
representation degeneration problem.
Multilingual Sentence Embeddings
Recently,
multilingual sentence representations have at-
tracted increasing attention. Schwenk and Douze
(2017); Yu et al. (2018); Artetxe and Schwenk
(2019) propose to use encoders from multilingual
neural machine translation to produce universal
representations across different languages. Chi-
dambaram et al. (2019); Wieting et al. (2019); Yang
et al. (2020); Feng et al. (2020) fine-tune siamese
networks (Bromley et al.,1993) with contrastive
objectives using parallel corpora. Reimers and
Gurevych (2020) train a multilingual model to map
sentences to the same embedding space of an exist-
ing English model. Different from existing work,
our work resorts to multilingual AMR, a language-
agnostic disambiguated semantic representation,
for performance enhancement.
Evaluation of Sentence Embeddings
Tradition-
ally, the mainstream evaluation for assessing the
quality of English-only sentence embeddings is
based on the Semantic Textual Similarity (STS)
tasks and a suite of downstream classification
tasks. The STS tasks (Agirre et al.,2012,2013,