GRAPE Knowledge Graph Enhanced Passage Reader for Open-domain Question Answering

2025-05-06 0 0 1.42MB 13 页 10玖币

侵权投诉

GRAPE: Knowledge Graph Enhanced Passage Reader

for Open-domain Question Answering

Mingxuan Ju1∗, Wenhao Yu1∗, Tong Zhao2, Chuxu Zhang3, Yanfang Ye1

1University of Notre Dame, 2Snap Inc., 3Brandeis University

1{mju2,wyu1,yye7}@nd.edu; 2tzhao@snap.com; 3chuxuzhang@brandeis.edu

Abstract

A common thread of open-domain question

answering (QA) models employs a retriever-

reader pipeline that ﬁrst retrieves a handful of

relevant passages from Wikipedia and then pe-

ruses the passages to produce an answer. How-

ever, even state-of-the-art readers fail to cap-

ture the complex relationships between entities

appearing in questions and retrieved passages,

leading to answers that contradict the facts.

In light of this, we propose a novel knowl-

edge Graph enhanced passagereader, namely

GRAPE, to improve the reader performance

for open-domain QA. Speciﬁcally, for each

pair of question and retrieved passage, we ﬁrst

construct a localized bipartite graph, attributed

to entity embeddings extracted from the inter-

mediate layer of the reader model. Then, a

graph neural network learns relational knowl-

edge while fusing graph and contextual repre-

sentations into the hidden states of the reader

model. Experiments on three open-domain

QA benchmarks show GRAPEcan improve

the state-of-the-art performance by up to 2.2

exact match score with a negligible overhead

increase, with the same retriever and retrieved

passages. Our code is publicly available at

https://github.com/jumxglhf/GRAPE.

1 Introduction

Open-domain question answering (QA) tasks

aim to answer questions in natural language

based on large-scale unstructured passages such

as Wikipedia (Chen and Yih,2020;Zhu et al.,

2021). A common thread of modern open-domain

QA models employs a retriever-reader pipeline, in

which a retriever aims to retrieve a handful of rele-

vant passages w.r.t. a given question, and a reader

aims to infer a ﬁnal answer from the received pas-

sages (Guu et al.,2020;Karpukhin et al.,2020;

Lewis et al.,2020;Izacard and Grave,2021). Al-

though these methods have achieved remarkable

* Equal contribution.

Question: What is the primary language of China?

Retrieved Wikipedia passage (ID: 253645):

Language of the People’s Republic of China and of the

Republic of China (Taiwan), and one of the official lang-

uages of Singapore. Standard Mandarin Chinese now do-

minates public life in mainland China. Outside China and

Taiwan, the only varieties of Chinese commonly taught in

university courses are Mandarin and Cantonese.

Answer (SoTA reader): Cantonese

Answer (ours): Standard Mandarin

Factual Triplet from Wikidata:

Question: What does the Missouri river bisect?

Retrieved Wikipedia passage (ID: 19591):

The Missouri River is the longest river in North America.

Rising in the Rocky Mountains of western Montana the

Missouri flows east and south for before entering the

Mississippi River north of St. Louis, Missouri. The river

drains a sparsely populated, …500,000 square miles,

which includes parts of ten U.S. States

and two Canadian provinces.

Answer (SoTA reader): Missouri

Answer (ours): Mississippi River

Factual Triplet from Wikidata:

language used

ID 19359:

Standard Mandarin

ID 5405:China

ID 19591: Missouri River

ID 19579:

Mississippi River

tributary

Figure 1: The answers produced by the SoTA reader

FiD contradict the facts in the knowledge graph.

advances on various open-domain QA benchmarks,

the state-of-the-art readers, such as FiD (Izacard

and Grave,2021), still often produce answers that

contradict the facts. As shown in Figure 1, the

FiD reader fails to produce correct answers due to

inaccurate understanding of the factual evidence.

Therefore, instead of improving the retrievers to

saturate the readers with higher answer coverage

in the retrieved passages (Yu et al.,2021;Oguz

et al.,2022;Yu et al.,2022a), in this work, we aim

at improving the readers by leveraging structured

factual triples from the knowledge graph (KG).

A knowledge graph, such as Wikidata (Vran-

deˇ

ci´

c and Krötzsch,2014), contains rich relational

information between entities, many of which can

be further mapped to corresponding mentions in

questions and retrieved passages. To verify the

possible improvements brought by the KG, we con-

duct a simple analysis to examine the percentage

of related fact triples present on the KG in the data,

arXiv:2210.02933v2 [cs.CL] 10 Oct 2022

Dataset Fact-related examples Error rate

NQ 736 (20.4%) 31.9%

TriviaQA 3,738 (33.0%) 17.4%

WebQ 1,181 (58.1%) 42.5%

Table 1: The error rate of state-of-the-art reader (i.e.,

FiD base) on the subset of data examples in the test set

that have related fact triplets on the knowledge graph.

i.e., entities in questions are neighbors of answer

entities in retrieved passages through any relation.

We also wonder how many of the above examples

are correctly answered by state-of-the-art readers.

Table 1shows that a great portion of examples

(e.g., 58.1% in WebQ) can be matched to related

fact triplets on the KG. However, without using the

KG, FiD frequently produces incorrect answers to

questions on these subsets, leaving us signiﬁcant

room for improvement. Therefore, a framework

that leverages not only textual information in re-

trieved passages but also fact triplets from the KG

is urgently desired to improve reader performance.

In this paper, we propose a novel knowledge

Gra

ph enhanced

assag

reader, namely GRAPE,

to improve the reader performance for open-

domain QA. Considering the enormous size of KGs

and complex interweaving between entities (e.g.,

over 5 million entities and over 30 neighbors per

entity on Wikidata), direct reasoning on the entire

graph is intractable. Thus, we ﬁrst construct a lo-

calized bipartite graph for each pair of question and

passage, where nodes represent entities contained

within them, and edges represent relationships be-

tween entities. Then, node representations are ini-

tialized with the hidden states of the corresponding

entities, extracted from the intermediate layer of

the reader model. Next, a graph neural network

learns node representations with relational knowl-

edge, and passes them back into the hidden states

of the reader model. Through this carefully curated

design, GRAPEtakes into account both aspects of

knowledge as a holistic framework.

To the best of our knowledge, we are the ﬁrst

work to leverage knowledge graphs to enhance the

passage reader for open-domain QA. Our experi-

ments demonstrate that, given the same retriever

and the same set of retrieved passages, GRAPE

can achieve superior performance on three open-

domain QA benchmarks (i.e., NQ, TriviaQA, and

WebQ) with up to 2.2 improvement on the exact

match score over the state-of-the-art readers. In

particular, our proposed GRAPEnearly doubles

the improvement gain on the subset that can be

enhanced by fact triplets on the KG.

2 Related Work

Text-based open-domain QA

Mainstream

open-domain QA models employ a retriever-

reader architecture, and recent follow-up work

has mainly focused on improving the retriever

or the reader (Chen and Yih,2020;Zhu et al.,

2021). For the retriever, most of them split text

paragraphs on Wikipedia pages into over 20

million disjoint chunks of 100 words, each of

which is called a passage. Traditional methods

such as TF-IDF and BM25 explore sparse retrieval

strategies by matching the overlapping contents

between questions and passages (Chen et al.,

2017;Yang et al.,2019). DPR (Karpukhin et al.,

2020) revolutionized the ﬁeld by utilizing dense

contextualized vectors for passage indexing.

Furthermore, other research improved the per-

formance by better training strategies (Qu et al.,

2021), passage re-ranking (Mao et al.,2021) or

directly generating passages (Yu et al.,2022a).

Whereas for the reader, extractive readers aimed to

locate a span of words in the retrieved passages as

answer (Karpukhin et al.,2020;Iyer et al.,2021;

Guu et al.,2020). On the other hand, FiD and

RAG, current state-of-the-art readers, leveraged

encoder-decoder models such as T5 to generate

answers (Lewis et al.,2020;Izacard and Grave,

2021). Nevertheless, these readers only used text

corpus, failing to capture the complex relationships

between entities, and hence resulting in produced

answers contradicting the facts.

KG-enhanced methods for open-domain QA

Recent work has explored incorporating knowledge

graphs (KGs) into the retriever-reader pipeline for

open-domain QA (Min et al.,2019;Zhou et al.,

2020;Oguz et al.,2022;Yu et al.,2021;Hu et al.,

2022;Yu et al.,2022b). For example, Unik-QA

converted structured KG triples and merged un-

structured text together into a uniﬁed index, so

the retrieved evidence has more knowledge cov-

ered. Graph-Retriever (Min et al.,2019) and GNN-

encoder (Liu et al.,2022) explored passage-level

KG relations for better passage retrieval. KAQA

(Zhou et al.,2020) improved passage retrieval by

re-ranking according to KG relations between can-

didate passages. KG-FiD (Yu et al.,2021) utilized

KG relations to re-rank retrieved passages by a KG

ﬁne-grained ﬁlter. However, all of these retriever-

enhanced methods focused on improving the qual-

ity of retrieved passages before passing them to the

reader model. So, they still suffered from factual

errors. Instead, our GRAPEis the ﬁrst work to

leverage knowledge graphs to enhance the reader,

which is orthogonal to these existing KG-enhanced

frameworks and our experiments demonstrate that

with the same retriever and the same set of retrieved

passages, GRAPEcan outperform the state-of-the-

art reader FiD by a large margin.

3 Proposed Method: GRAPE

In this section, we elaborate on the details of the

proposed GRAPE. Figure 3shows its overall archi-

tecture. GRAPEadopts a retriever-reader pipeline.

Speciﬁcally, given a question, it ﬁrst utilizes DPR

to retrieve top-

relevant passages from Wikipedia

(

3.1). Then, to peruse the retrieved passages,

it constructs a localized bipartite graph for each

pair of question and passage (

3.2.1). The con-

structed graphs possess tractable yet rich knowl-

edge about the facts among connected entities. Fi-

nally, with the curated graphs, structured facts are

learned through a relation-aware graph neural net-

work (GNN) and fused into token-level representa-

tions of entities in the passages (§3.2.2).

3.1 Passage Retrieval

Given a collection of

passages, the goal of

the retriever is to map all the passages in a low-

dimensional vector, such that it can efﬁciently re-

trieve the top-

passages relevant to the input ques-

tion. Note that

can be very large (e.g., over 20

million in our experiments) and

is usually small

(e.g., 100 in our experiments).

Following DPR (Karpukhin et al.,2020), we em-

ploy two independent BERT (Devlin et al.,2019)

models to encode the question and the passage sep-

arately, and estimate their relevance by computing

a single similarity score between their

[CLS]

token

representations. Speciﬁcally, given a question

and a passage

pi∈ {p1, p2, ..., pK}

, we encode

by a question encoder

EQ(·) : q→Rd

and en-

codes

by a passage encoder

EP(·) : p→Rd

where

is the hidden dimension of the used BERT.

The ranking score

w.r.t

is calculated as:

q=EQ(q)|·EP(pi).(1)

We select

passages whose ranking scores

are

top-

highest among all

passages. Before pass-

ing the retrieved passages into the reader model,

Question: <QENT> Swan lake, <QENT> the sleeping beauty

and <QENT> the nutcracker are three famous ballet by?

Q Special Token Q Entity

P Special Token P Entity

Passage: 'The Nutcracker' is an 1892 two-act ballet,

originally choreographed by <PENT> Marius Petipa and

<PENT> Lev Ivanov with a score by <PENT> Pyotr Ilyich

Tchaikovsky (Op. 71). The libretto is adapted from E. T. A.

Hoffmann's story "The Nutcracker and the Mouse King”…

Sleeping beauty

Swan Lake

Nutcracker

Tchaikovsky

Le Ivanov

Localized Bipartite Graph for This Pair

Marius Petipa

Entity in Question

Entity in Passage

Composer/

Notable work

Possessed by spirit

Figure 2: Given a pair of question and passage, the pro-

posed GRAPEconstructs a localized bipartite graph.

we process each question and passage by insert-

ing special tokens before each entity. For entities

in each passage, we use the special token

<PENT>

;

for those in the question, we use another special

token

<QENT>

, as shown in Figure 2. The special to-

kens play an important role in our proposed reader

model, which is illustrated in more detail in

3.2.2.

3.2 KG-enhanced Passage Reader

3.2.1 Graph Construction

Given the retrieved and processed passages, our

proposed GRAPEutilizes the factual triplets from

KGs to construct localized bipartite graphs for each

question-passage pair. A KG is deﬁned as a set of

triplets

KG ={(eh, r, et)}

, where

, and

refer to a head entity, a tail entity, and a correspond-

ing relation between them, respectively. Knowl-

edge graphs represent facts in the simple format

of triplets, which can easily be leveraged to enrich

our knowledge. Taking the question-passage pair

in Figure 2as an example, without any prior knowl-

edge about the authorship of the ballets, the se-

lection of answers between “Marius Petipa”, “Lev

Ivanov” and “Tchaikovsky” is difﬁcult. Nonethe-

less, factual triplets from the KG show that these

three ballets are only “possessed by spirit” by “Mar-

ius Petipa” and “Lev Ivanov”. And their “com-

poser” relations with “Tchaikovsky” make the an-

swer obvious. By fusing such relational facts from

KG triplets, the reader can better comprehend the

concrete facts between involved entities and hence

improve the performance for open-domain QA.

One naive solution could be fetching a sub-graph

文档加载中……请稍候！
如果长时间未打开，您也可以点击刷新试试。

下载文档到电脑，查找使用更方便

10 玖币 0人已下载

立即下载

摘要：

GRAPE:KnowledgeGraphEnhancedPassageReaderforOpen-domainQuestionAnsweringMingxuanJu1,WenhaoYu1,TongZhao2,ChuxuZhang3,YanfangYe11UniversityofNotreDame,2SnapInc.,3BrandeisUniversity1fmju2,wyu1,yye7g@nd.edu;2tzhao@snap.com;3chuxuzhang@brandeis.eduAbstractAcommonthreadofopen-domainquestionanswering(QA)...

展开>> 收起<<

GRAPE Knowledge Graph Enhanced Passage Reader for Open-domain Question Answering.pdf

共13页,预览3页

还剩页未读，继续阅读

声明：本站为文档C2C交易模式，即用户上传的文档直接被用户下载，本站只是中间服务平台，本站所有文档下载所得的收益归上传人(含作者)所有。玖贝云文库仅提供信息存储空间，仅对用户上传内容的表现方式做保护处理，对上载内容本身不做任何修改或编辑。若文档所含内容侵犯了您的版权或隐私，请立即通知玖贝云文库，我们立即给予删除！

GRAPE Knowledge Graph Enhanced Passage Reader for Open-domain Question Answering

相关推荐

开通VIP享超值会员特权

作者详情

相关内容

热门标签

举报选择: