Retrieval Augmentation for T5 Re-ranker using External Sources Kai HuiTao Chen Zhen Qin Honglei Zhuang Fernando Diaz Michael Bendersky Donald Metzler

2025-04-29 0 0 219.76KB 7 页 10玖币

侵权投诉

Retrieval Augmentation for T5 Re-ranker using External Sources

Kai Hui∗Tao Chen Zhen Qin Honglei Zhuang

Fernando Diaz Michael Bendersky Donald Metzler

kaihuibj@google.com

Google Research

Abstract

Retrieval augmentation has shown promising

improvements in different tasks. However,

whether such augmentation can assist a large

language model based re-ranker remains un-

clear. We investigate how to augment T5-

based re-rankers using high-quality informa-

tion retrieved from two external corpora — a

commercial web search engine and Wikipedia.

We empirically demonstrate how retrieval aug-

mentation can substantially improve the ef-

fectiveness of T5-based re-rankers for both

in-domain and zero-shot out-of-domain re-

ranking tasks.

1 Introduction

Large language models (LLMs) have demonstrated

strong performance for both retrieval and re-

ranking tasks (Qiao et al.,2019;Nogueira et al.,

2020;Lin et al.,2021;Tay et al.,2022). Mean-

while, recent research has enhanced LLMs by aug-

menting them with retrieval capabilities. These re-

trieval augmentation strategies have demonstrated

promising results on tasks such as question answer-

ing (Guu et al.,2020;Lewis et al.,2020;Borgeaud

et al.,2021) and knowledge-grounded conversa-

tion (Shuster et al.,2021;Cohen et al.,2022). How-

ever, it remains an open question as to whether re-

trieval augmentation can improve the effectiveness

of LLM-based re-rankers.

Inspired by the uses of the external expan-

sions (Diaz and Metzler,2006), we explore how

query representations can be augmented using in-

formation retrieved from high-quality corpora to

improve re-ranking quality. Similar to RAG (Lewis

et al.,2020) and RETRO (Borgeaud et al.,2021),

we augment a given query with information re-

trieved from external resources before perform-

ing inference (i.e., re-ranking in this case). We

consider external resources from two corpora —

∗Corresponding Author

a commercial web search engine and Wikipedia.

We consider two approaches to augment the orig-

inal query with retrieved information. The ﬁrst

approach augments the query with a sequence of

terms while the second augments the query with a

bag of words. Finally, we investigate the efﬁcacy

of query augmentation strategies for in-domain

and zero-shot out-of-domain settings. We evaluate

these conditions on three datasets: open-domain

Natural Questions (Kwiatkowski et al.,2019), MS

MARCO passage re-ranking (Nguyen et al.,2016),

and TREC DL Track 2019 (Craswell et al.,2020)

and 2020 (Craswell et al.,2021).

This paper makes three contributions: (1) To the

best of our knowledge, this is the ﬁrst investiga-

tion of retrieval augmentation for LLM-based re-

rankers; (2) Retrieval augmentation is empirically

analyzed using four T5-based reranker variants for

in-domain and zero-shot out-of-domain tasks; (3)

A comprehensive experimental study is presented

using two retrieval corpora and two different query

augmentation strategies.

2 Related Work

Retrieval augmentation for NLP tasks.

Due

to the opaque nature of knowledge stored in the

parameters of LLMs, retrieval augmentation has

been introduced for a variety of different NLP

tasks. For example, on question answering tasks,

REALM (Guu et al.,2020), RAG (Lewis et al.,

2020) augment inputs with a document corpus,

enriching the representation using top-ranked re-

trieved items via Maximum Inner Product Search

(MIPS). Meanwhile, RETRO (Borgeaud et al.,

2021) uses retrieval to augment at the granularity

of small chunks of tokens. It has also been shown

that retrieval augmentation can help provide better-

grounded text in dialogue systems (Shuster et al.,

2021;Cohen et al.,2022) and in the evaluation

of hallucination (Honovich et al.,2021). Inspired

arXiv:2210.05145v1 [cs.IR] 11 Oct 2022

by these successes, our work investigates retrieval

augmentation for re-ranking using a ﬁxed retrieval

component.

Query expansion and pseudo-relevance feed-

back (PRF).

In early work, Diaz and Metzler

(2006) showed it is effective to incorporate infor-

mation from an external corpus into a non-neural

language modeling framework. We exploit such in-

formation when using a pre-trained language model

for re-ranking by directly augmenting the original

query with the top-ranked results from an external

corpus. An orthogonal research direction is to im-

prove re-ranking models by incorporating pseudo-

relevance feedback (PRF) signals as in (Li et al.,

2018;Padaki et al.,2020;Zheng et al.,2020;Yu

et al.,2021;Naseri et al.,2021). One essential

component therein identiﬁes the relevant informa-

tion from the pseudo relevance, avoiding the topic

shift. Besides, these methods are involved with ex-

pensive multiple iterations to collect the PRF and

use that for re-ranking. In contrast, our model con-

sumes high-quality external augmentation text and

requires one single iteration.

3 Method

We adopt Nogueira et al.’s method for re-ranking

with LLMs (Nogueira et al.,2019). Let

be the

query string,

be the document string, and

a string that represents the binary relevance of a

document, e.g., “True” or “False”. We construct a

(string) instance xas,

x="Query:qDocument:dRelevant:y"(1)

The model is trained to generate the ﬁnal token (i.e.

) based on the ground-truth relevance of the query-

document pair. To score a new query-document

pair, the normalised score of the ﬁnal token is used

for re-ranking.

We are interested in augmenting

with infor-

mation from an external corpus. We assume that

access to the external corpus is mediated through a

retrieval service

such that

f(q) = [σ1, . . . , σm]

where

σi

is a retrieved passage (e.g. web search

snippet, indexed passage). It is important to note

that the retrieval service can only retrieve items

from a given external corpus and cannot re-rank or

re-score documents in the target corpus.

We represent the information

f(q)

as an aug-

menting string

˜q

. We can directly concatenate the

passages to construct

˜q

; we refer to this as natu-

ral language expansion. Although we expect the

natural language expansion to be more compatible

with LLMs, the ﬁxed capacity of LLM modeling

can result in situations where informative text is

obscured by ‘linguistic glue’ often discarded as

stop words (Tay et al.,2020). Alternatively, we can

extract the most salient topical terms from

f(q)

in (Dang and Croft,2013). Speciﬁcally, we se-

lect terms using the KL2 method (Carpineto et al.,

2001;Amati,2003). In this method, we select

terms from all of the terms in

f(q)

using each in-

dividual words’ contribution in the KL-divergence

between the language model in

f(q)

(denoted as

A) and the corpus (denoted as C).

w(t, A) = P(t|A)log2

P(t|A)

P(t|C)(2)

We estimate the corpus language model using the

target retrieval dataset. We refer to this as topical

term expansion. In both expansion methods, we

truncate the concatenated snippets, paragraphs, or

ordered set of topical words (according to Eq. 2) to

a maximum sequence length.

To incorporate retrieved information, repre-

sented as

˜q

(the expansion terms), we add the text

as a new subsequence (“Description”) in x,

x="Query:qDescription:˜q

Document:dRelevant:y"

Because we are representing instances as strings

with a terminal relevance label, we can easily adopt

the same re-ranking method as Nogueira et al.

(2019).

4 Experiments

Training data.

We use two training datasets,

namely, Natural Questions (NQ) originally pro-

posed in (Kwiatkowski et al.,2019), and, the MS

MARCO (Nguyen et al.,2016) passage re-ranking

dataset. The NQ dataset includes 79k user queries

from the Google search engine. The subset of NQ

derived in (Karpukhin et al.,2020) are used. The

data has the form (question, passage, label), where

only the queries with short answers are included.

The task is to retrieve and re-rank the chunked para-

graphs from Wikipedia with up to 100 words for

the queries. Meanwhile, we use the MS MARCO

triplet training dataset (Nguyen et al.,2016), which

includes 550k positive query-passage pairs. For

validation purposes, we measure Success@20 (also

called Hits@20) on the 8757 questions in the NQ

文档加载中……请稍候！
如果长时间未打开，您也可以点击刷新试试。

下载文档到电脑，查找使用更方便

10 玖币 0人已下载

立即下载

摘要：

RetrievalAugmentationforT5Re-rankerusingExternalSourcesKaiHuiTaoChenZhenQinHongleiZhuangFernandoDiazMichaelBenderskyDonaldMetzlerkaihuibj@google.comGoogleResearchAbstractRetrievalaugmentationhasshownpromisingimprovementsindifferenttasks.However,whethersuchaugmentationcanassistalargelanguagemodelbas...

展开>> 收起<<

Retrieval Augmentation for T5 Re-ranker using External Sources Kai HuiTao Chen Zhen Qin Honglei Zhuang Fernando Diaz Michael Bendersky Donald Metzler.pdf

共7页,预览2页

还剩页未读，继续阅读

声明：本站为文档C2C交易模式，即用户上传的文档直接被用户下载，本站只是中间服务平台，本站所有文档下载所得的收益归上传人(含作者)所有。玖贝云文库仅提供信息存储空间，仅对用户上传内容的表现方式做保护处理，对上载内容本身不做任何修改或编辑。若文档所含内容侵犯了您的版权或隐私，请立即通知玖贝云文库，我们立即给予删除！

Retrieval Augmentation for T5 Re-ranker using External Sources Kai HuiTao Chen Zhen Qin Honglei Zhuang Fernando Diaz Michael Bendersky Donald Metzler

相关推荐

开通VIP享超值会员特权

作者详情

相关内容

热门标签

举报选择: