REASTAP Injecting Table Reasoning Skills During Pre-training via Synthetic Reasoning Examples Yilun Zhao1Linyong Nan1Zhenting Qi2Rui Zhang3Dragomir Radev1

2025-04-29 0 0 607.89KB 14 页 10玖币

侵权投诉

REASTAP: Injecting Table Reasoning Skills During Pre-training

via Synthetic Reasoning Examples

Yilun Zhao1Linyong Nan1Zhenting Qi2Rui Zhang3Dragomir Radev1

1Yale University 2Zhejiang University 3Penn State University

{yilun.zhao, linyong.nan}@yale.edu

Abstract

Reasoning over tabular data requires both ta-

ble structure understanding and a broad set of

table reasoning skills. Current models with

table-speciﬁc architectures and pre-training

methods perform well on understanding table

structures, but they still struggle with tasks

that require various table reasoning skills. In

this work, we develop REASTAP to show

that high-level table reasoning skills can be

injected into models during pre-training with-

out a complex table-speciﬁc architecture de-

sign. We deﬁne 7 table reasoning skills, such

as numerical operation, temporal comparison,

and conjunction. Each reasoning skill is asso-

ciated with one example generator, which syn-

thesizes questions over semi-structured tables

according to the sampled templates. We model

the table pre-training task as a sequence gen-

eration task and pre-train REASTAP to gen-

erate precise answers to the synthetic exam-

ples. REASTAP is evaluated on four bench-

marks covering three downstream tasks includ-

ing: 1) WIKISQL-WEAK and WIKITQ for

Table Question Answering; 2) TABFACT for

Table Fact Veriﬁcation; and 3) LOGICNLG

for Faithful Table-to-Text Generation. Exper-

imental results demonstrate that REASTAP

achieves new state-of-the-art performance on

all benchmarks and delivers a signiﬁcant im-

provement on low-resource setting. Our code

is publicly available at https://github.

com/Yale-LILY/ReasTAP.

1 Introduction

Inspired by the massive success of pre-trained lan-

guage models (LM) on free-form natural language

(NL) tasks (Devlin et al.,2019;Dong et al.,2019;

Raffel et al.,2020;Lewis et al.,2020), researchers

have attempted to extend the pre-training to table

data. Tables are a valuable form of data that or-

ganize information in a structured way. They of-

ten contain data that is organized in a more ac-

cessible manner than in unstructured texts. To

Conjunction:

Q: What was the Company Name when the Industry is Oil and gas and

Headquarters is China? A: National Petroleum, Sinopec Group

Counting:

Q: How many Headquarters have Employees was 662,575? A: 1

Numerical Comparison:

(… abbreviate other reasoning skills …)

REASTAP Wa lm ar t

Supervise

Which Company Name, with Headquarter was United States, has the 4th Profit?

Retail | $ 386,064 |[ROW] 5 Sinopec Group | Oil and gas | … | 553,833 | China

3Amazon Retail 386,064$ 21,331$ 1,298,000 United States

7CVS Health Healthcare 268,706$ 7,179$ 256,500 United States

1Walmart Retail 559,151$ 13,510$ 2,300,000 United States

4National Petroleum Oil and gas 283,958$ 4,575$ 1,242,245 Chin a

10 Volkswagen Automotive 253,965$ 10,104$ 662,575 Germany

2State Grid Electricity 386,618$ 5,580$ 896,360 China

8UnitedHealth Healthcare 257,141$ 15,403$ 330,000 United States

6Apple Electronics 274,515$ 57,511$ 147,000 United States

9Toyota Automotive 256,722$ 21,180$ 366,283 Japan

5Sinopec Group Oil and gas 283,728$ 6,205$ 553,833 China

Employ ees

Headquarters

Rank

Company Name

Industry

Revenue

($ Million)

Profit

($ Million)

Synthesize examples with various pre-defined table reasoning skills

Pre-training

an example of data serialization

Q: Which Company Name, with the Headquarter was United States, has the

4th Profit? A: Walm art

concatenation

Figure 1: The illustration of REASTAP pre-training.

The tables are crawled from Wikipedia. During pre-

processing, we perturb the table row order to alleviate

unwanted bias brought by table encoding. The colored

cells are relevant facts necessary to answer the given

question. Each color corresponds to a different table

reasoning skill. And each reasoning skill corresponds

to an example generator, which synthesizes QA pairs

over tables according to the sampled templates. We

model the pre-training task as a sequence generation

task and pre-train REASTAP to generate correct an-

swers given the ﬂatten table and synthetic question.

adapt the pre-training paradigm on structured tab-

ular data, previous works mainly focus on design-

ing models with table-speciﬁc architectures and

pre-training methods. This includes introducing

a structure-aware attention mechanism (Yin et al.,

2020;Deng et al.,2020;Zayats et al.,2021), adding

arXiv:2210.12374v1 [cs.CL] 22 Oct 2022

auxiliary structure indicative embeddings (Herzig

et al.,2020;Eisenschlos et al.,2020;Wang et al.,

2021b), and designing table-speciﬁc pre-training

objectives (Yin et al.,2020;Yu et al.,2021a;Wang

et al.,2021b;Liu et al.,2022b,a). While these

methods are effective in understanding table struc-

tures, they increase the modeling complexity and

lack interpretability on why models learns table

reasoning skills during pre-training.

This paper presents a new table pre-training ap-

proach, named REASTAP, which enables a model

to efﬁciently learn table structure understanding

and table reasoning skills during pre-training. We

ﬁrst deﬁned 7 table reasoning skills, such as numer-

ical operation and temporal comparison. As shown

in Figure 1, for each reasoning skill, a correspond-

ing example generator was applied to synthesize

Question Answering (QA) examples over tables.

We modeled the pre-training task as a sequence gen-

eration task and pre-trained a sequence-to-sequence

(seq2seq) LM to generate the answer to the syn-

thetic questions. REASTAP is theoretically appli-

cable to any seq2seq LM without a table-speciﬁc

architecture design. Our key insight is that if a

language model can be pre-trained to generate the

answers to synthetic questions, which require var-

ious table reasoning skills, it should have a great

table structure understanding and table reasoning

capacity, thereby conferring beneﬁts to downstream

tasks. The main contributions of our work can be

summarized as follows:

•

We develop a new table reasoning example

generation pipeline, which produces a large-

scale table QA corpus that requires various

reasoning skills over semi-structured tables.

•

We propose a new table pre-training method,

REASTAP, which helps the model to learn ta-

ble structure understanding and various table

reasoning skills during pre-training without

any table-speciﬁc architecture design.

•

REASTAP is evaluated on four downstream

benchmarks. Experimental results demon-

strate that REASTAP achieves new state-of-

the-art results on all of them, and delivers a

great improvement on low-resource setting.

2 Pre-training Corpus

2.1 Table Source and Pre-processing

We chose publicly available semi-structured tables

as the table source. Speciﬁcally, we extracted ta-

bles from English Wikipedia

, which covered a

wide range of domains including popular culture,

geography, politics, and science. We kept tables

with 8-30 rows and at least three columns, resulting

in around 600K tables. For each extracted table, a

pre-processing script was applied to automatically

annotate table columns with their data types (i.e,

string, number, and date), which allows us to gen-

erate questions that involve manipulating numbers

and dates. Furthermore, recent work (Yang et al.,

2022;Wang et al.,2022) demonstrates that existing

table pre-training approaches might encode table

row order as an unwanted bias. For example, the

pre-trained model being aware of row order infor-

mation is inclined to select the ﬁrst or last row of

tables when answering superlative-type questions

without truly understanding the table content. To

alleviate this problem, we randomly shufﬂed table

rows during pre-processing.

2.2 Example Generation

We deﬁned 7 types of table reasoning skills, with

examples and explanations shown in Table 1. The

example generation pipeline was adapted from

Yoran et al. (2021). Each reasoning skill is as-

sociated with one example generator and several

question templates. The example generator was

implemented as a function that takes a table

and generates several reasoning examples (

) according to the template, where

denotes the

question, and adenotes the answer.

Each template contains typed variables that are

instantiated with content from the extracted ta-

ble. Speciﬁcally, column

col

and cell value

val

are indexed to specify that

val:i

must be in-

stantiated by a cell value from the

-th column.

Some templates also regulate that the selected

column and cell value must be date or number

type.

OPERATOR

and

ORDINAL

correspond to

operators and ordinal numerals that are instanti-

ated according to the speciﬁc reasoning skill. And

CONDITION:i

can be 1) a cell value from the

-th column; or 2) a number/temporal comparison

statement if the

-th column is date or number type.

For example, the question from Figure 1"Which

Company Name, with Headquarter was United

States, has the 4th Proﬁt?" are generated from one

of the "Numerical Comparison" templates: "Which

col:1

, with

col:2

was

CONDITION:2

, has

We parsed the 02-20-2022 Wikipedia dump using WikiEx-

tractor Tools from

https://github.com/attardi/

wikiextractor

Reasoning Example Templates Example Questions & Answers %Data

Conjunction

What was the

col:1

when the

col:2

was

CONDITION:2

and the

col:3

was CONDITION:3?

What was the

Television Service

when the

Country was Italy and the Content was Sport?

A: Sky OMC Sports, ESPN, Gazzetta TV, ...

21.6%

Quantiﬁers

Only/Every

Does

OPERATOR col:1

, with

col:2

was

CONDTION:2

, have

col:3

CONDITION:3?

Does every

Company

, with

Headquarter

was

Paris, have Industry Financials?

A: Yes

Does only

Company Name

, with

Founded

Year

was later than 1964, have

Employee Num-

ber greater than 30,000?

A: No

10.3%

Temporal

Comparison

Which

col:1

, with

col:2

was

CONDITION:2

, happened the

ORDINAL according to col:3?

Which

Romaji

, with

Sales

was greater than

203,471, happened the 4th according to Date?

A: Hepburn

14.5%

Date

Difference

how much time had passed between

when the

col:1

was

val:1

and when

the col:2 was val:2?

how much time had passed between when the

Candidate

was John Kufuor and when the

Candi-

date was Paul McCartney?

A: 16 years

5.7%

Counting

How many

col:1

have

col:2

CONDITION:2?

How many

Event Location

have

Attendance

greater than 10,235?

A: 7

18.0%

Numerical

Operation

What was the OPERATOR of

col:1

when the

col:2

was

CONDITION:2

What was the sum of

GDP Estimate ($ US

Million)

when the

GDP Estimate ($ US Million)

was greater than 841,969?

A: 1,574,013

15.9%

Numerical

Comparison

Which

col:1

, with

col:2

was

CONDITION:2

, has the

ORDINAL

col:3?

Which

Franchise

, with

Owner(s)

was Nin-

tendo, has the 5th Total revenue($ US Billion)?

A: Pokemon

14.0%

Table 1: 7 reasoning skills with example for pre-training REASTAP. Variable names indicate permissible instanti-

ations. col denotes a column name, val denotes a cell value, and indices denote that a cell value must originate

from the speciﬁed column. OPERATOR and ORDINAL correspond to operators and ordinal numeral that are in-

stantiated according to the speciﬁc reasoning skill, e.g., for ‘Temporal Comparison’, ORDINAL is replaced with a

reasonable ordinal numeral such as "4th". And CONDITION:i can be 1) a cell value from the i-th column, or 2)

number/temporal comparison statement (e.g. "later than 1967") if the i-th column is of number or date type.

the ORDINAL col:3?"

Once all variables in the sampled template were

instantiated, we obtained question

. Then the

example generator would programmatically return

the corresponding answer a.

2.3 Example Sampling

After generating a vast number of QA examples for

each reasoning skill, we had to sample pre-training

data from these synthetic examples. In our setting,

the portion of pre-training examples (Table 1) cor-

responding to each reasoning skill roughly matches

the portion of logical operations deﬁned in Tab-

Fact (Chen et al.,2020b). We raised the portion of

numerical operation skill as numerical reasoning is

more challenging for models to learn. To increase

the diversity of pre-training corpus, for each reason-

ing skill, we also sampled {SQL query, execution

result} pairs from TAPEX (Liu et al.,2022b) pre-

training corpus as complementary QA examples.

The sampled pairs were categorised according to

their function (e.g.,

COUNT

SUM

). As a result, we

obtained a total of 4M pairs of reasoning examples

as the pre-training corpus for REASTAP.

3 Pre-training REASTAP

Task Formulation

Each example in the syn-

thetic pre-training corpus contains a question

and a semi-structured table

as the model input.

The task objective is to generate an accurate answer

string

a=(a1, a2, . . . , an)

given the question

and input table T:

a=argmax

∏

i=1

P(ai∣a<i,q, T ;θ),(1)

where θdenotes the parameters of a seq2seq LM.

Model Architecture

Our method is theoretically

applicable to any seq2seq LM, such as T5 (Raf-

文档加载中……请稍候！
如果长时间未打开，您也可以点击刷新试试。

下载文档到电脑，查找使用更方便

10 玖币 0人已下载

立即下载

摘要：

REASTAP:InjectingTableReasoningSkillsDuringPre-trainingviaSyntheticReasoningExamplesYilunZhao1LinyongNan1ZhentingQi2RuiZhang3DragomirRadev11YaleUniversity2ZhejiangUniversity3PennStateUniversity{yilun.zhao,linyong.nan}@yale.eduAbstractReasoningovertabulardatarequiresbothta-blestructureunderstandingan...

展开>> 收起<<

REASTAP Injecting Table Reasoning Skills During Pre-training via Synthetic Reasoning Examples Yilun Zhao1Linyong Nan1Zhenting Qi2Rui Zhang3Dragomir Radev1.pdf

共14页,预览3页

还剩页未读，继续阅读

声明：本站为文档C2C交易模式，即用户上传的文档直接被用户下载，本站只是中间服务平台，本站所有文档下载所得的收益归上传人(含作者)所有。玖贝云文库仅提供信息存储空间，仅对用户上传内容的表现方式做保护处理，对上载内容本身不做任何修改或编辑。若文档所含内容侵犯了您的版权或隐私，请立即通知玖贝云文库，我们立即给予删除！

REASTAP Injecting Table Reasoning Skills During Pre-training via Synthetic Reasoning Examples Yilun Zhao1Linyong Nan1Zhenting Qi2Rui Zhang3Dragomir Radev1

相关推荐

开通VIP享超值会员特权

作者详情

相关内容

热门标签

举报选择: