MetaLogic Logical Reasoning Explanations with Fine-Grained Structure Yinya Huang12Hongming Zhang2yRuixin Hong3Xiaodan Liang14y Changshui Zhang3Dong Yu2

2025-05-02 0 0 3.84MB 27 页 10玖币

侵权投诉

MetaLogic: Logical Reasoning Explanations with Fine-Grained Structure

Yinya Huang1,2∗Hongming Zhang2†Ruixin Hong3Xiaodan Liang1,4†

Changshui Zhang3Dong Yu2

1Shenzhen Campus of Sun Yat-sen University 2Tencent AI Lab, Seattle

3Tsinghua University 4Pengcheng Laboratory

yinya.huang@hotmail.com, {hongmzhang, dyu}@global.tencent.com,

hrx20@mails.tsinghua.edu.cn, zcs@mail.tsinghua.edu.cn,

xdliang328@gmail.com

Abstract

In this paper, we propose a comprehensive

benchmark to investigate models’ logical rea-

soning capabilities in complex real-life scenar-

ios. Current explanation datasets often employ

synthetic data with simple reasoning struc-

tures. Therefore, it cannot express more com-

plex reasoning processes, such as the rebuttal

to a reasoning step and the degree of certainty

of the evidence. To this end, we propose a com-

prehensive logical reasoning explanation form.

Based on the multi-hop chain of reasoning, the

explanation form includes three main compo-

nents: (1) The condition of rebuttal that the

reasoning node can be challenged; (2) Logical

formulae that uncover the internal texture of

reasoning nodes; (3) Reasoning strength indi-

cated by degrees of certainty. The ﬁne-grained

structure conforms to the real logical reason-

ing scenario, better ﬁtting the human cogni-

tive process but, simultaneously, is more chal-

lenging for the current models. We evaluate

the current best models’ performance on this

new explanation form. The experimental re-

sults show that generating reasoning graphs re-

mains a challenging task for current models,

even with the help of giant pre-trained lan-

guage models.

1 Introduction

Being able to generate reasonable explanations is

a crucial capability for a reliable reasoning sys-

tem. Most current works try to ask models to gen-

erate reasoning chains as profound explanations.

From simple rationales (DeYoung et al.,2020) to

more complex multi-step explanations (Inoue et al.,

2020;Jhamtani and Clark,2020;Saha et al.,2021)

and deductive chains of reasoning (Clark et al.,

2020;Tafjord et al.,2021;Dalvi et al.,2021), pre-

vious works attempt to encompass comprehensive

∗

This work was done when Y. Huang was an intern at

Tencent AI Lab.

†

X. Liang and H. Zhang are the co-corresponding authors.

Sent1

Certainty: contingent

Node Formula: N/A

Sent4

Certainty: necessary

Node Formula:

□(v𝟐→ □v𝟑)

Condition of rebuttal

Sent3

Certainty: necessary

Node Formula: □(v𝟑⋀v𝟐)

Sent2

Certainty: contingent

Node Formula: N/A

Sent5

Certainty: unnecessary

Node Formula:

¬□(¬□v𝟑→ v𝟏)

Logic Metagraph

sent1: v1: doctor : v2: recent pharmaceutical advances will lead

the way in weight loss .

sent2: v1: prior to these advancements , v2: obesity -related

deaths outnumbered all other causes of death by a wide margin .

sent3: v1: the new drugs will v2: curb appetite and v3: increase

metabolism .

sent4: v1: thanks to v2: these advancements , v3: obesity will

dramatically decline in the near future .

sent5: v1: most people will not be able to afford these

prescriptions v2: since v3: the majority of health care plans will not

cover the new drugs .

Passage

Figure 1: A logical passage and the corresponding

logic metagraph in the proposed MetaLogic. Given a

logical passage, the goal is to generate the full meta-

graph including the chain of reasoning with conditions

of rebuttal, the node formulae, and the degrees of cer-

tainty.

information. However, the current explanation de-

sign still has limitations for logical reasoning texts

in real scenarios. As current explanations lack a

ﬁne-grained structure, three remarkable features

are not included in current explanations for the sake

of real-world logical reasoning: multiple relation

types, hierarchical structure, and certainty. As a re-

sult, we cannot comprehensively evaluate models’

reasoning capabilities in real-life scenarios.

Figure 1shows examples of the crucial reasoning

components that are well studied by previous cog-

nitive science literature (Toulmin,2003;Garson,

2021) but overlooked by previous work in the ma-

chine learning community. First, the inference re-

buttal. Previous work (Tafjord et al.,2021) mostly

only focuses on the inferences of conjunction and

arXiv:2210.12487v1 [cs.AI] 22 Oct 2022

entailment among different statements while ignor-

ing the rebuttal ones, which could be crucial in

real applications. For example,

sent5

counters

sent4

as a condition of exception and we can-

not construct the correct reasoning graph without

the rebuttal relation. Second, there could exist in-

ternal logical relations inside each statement. For

example,

sent5

contains two atomic sentences

connected by a logical implication relation. Third,

real-life statements could have different degrees of

certainty. For example, “He is hungry” and “He is

likely to be hungry” are not identical but relevant

because of the certainty. However, most previous

work simply treats them completely separately in-

stead of considering their relevance and trying to

model the difference (i.e., certainty).

Motivated by previous cognitive science work

(i.e., Toulmin Model

(Toulmin,2003) and modal

logic theory

(Garson,2021)), we propose a new

explanation form, logic metagraphs, to address the

aforementioned limitations of previous work. As

demonstrated in Figure 1, the logical metagraphs

are directed acyclic graphs with meta nodes con-

nected by two types of edges, support and rebut,

representing the inferences between the statements

over a logical passage. The meta structure uncovers

the chain of reasoning from evidence to the con-

clusion, along with the challenges from the rebut-

tal sentences. Each meta node stores information

about a logically sound statement formulated as a

propositional formula in a standard modal logic S5

system (Hughes et al.,1996), a direct extension of

ﬁrst-order propositional logic with two certainty

operators. The formulae have atomic sentences

as logical variables that denote events or beliefs,

which are modiﬁed by three unary operators on

their certainty (negation

, necessity

, and possi-

bility

) and are joined by three binary operators

on their logical relations (implication

→

, conjunc-

tion

∧

, disjunction

∨

). As a result, the logic meta-

graphs are comprehensive with multi-hop reason-

ing paths, inference rebuttal, the internal structure

The Toulmin Model is a canonical theory that helps for-

mat and understand arguments. It provides a general pattern

to assign logical roles to the sentences in the argument, which

clarify the overall logical relations. Especially, the rebuttal

components challenge the derivation from existing evidence

to the conclusion by providing additional information such as

giving a counterexample or proposing an additional condition.

The modal logic theory extends classic ﬁrst-order propo-

sitional logic with two modal operators about certainty and

several corresponding rules. This facilitates us to keep the

logical variables and relations found in the text and, at the

same time, introduce degrees of certainty to the graph.

of the statements, and reasoning strength denoted

by the degrees of certainty. We collect 1,000 log-

ical passages from the ReClor dataset (Yu et al.,

2020) and build the MetaLogic dataset.

Based on our new explanation form, we exam-

ine the current best models’ ability to understand

logical reasoning profoundly. The models need

to generate the logic metagraphs given a logical

passage. Performances are evaluated by matching

scores for the overall structure as well as the three

ﬁne-grained components: (1) The inference steps

between meta nodes; (2) The per-statement formu-

lae with multiple logical triples; (3) The degrees of

certainty. Our evaluation results indicate that gener-

ating a comprehensive logical reasoning structure

is still challenging for existing giant models.

Our contributions are three-fold:

We propose a new explanation form, the logic

metagraphs, with a comprehensive logical struc-

ture and rich logical information, and the corre-

sponding metagraph generation task.

We build a high-quality dataset, MetaLogic, on

real-world logical passages.

We conduct experiments on three generative

models in different frameworks and locate the

challenges for current models.

2 Related Works

Explanations

Explanation in the context of natu-

ral language understanding tasks (e.g., QA) pro-

vides interpretability about how models solve the

problem. The strategies include asking the models

to generate rationales while answering the ques-

tions (DeYoung et al.,2020;Inoue et al.,2020),

and deriving multi-hop chains of reasoning (Jham-

tani and Clark,2020;Dalvi et al.,2021). The

single-sentence rationale provides justiﬁcation for

the question answering but does not uncover the

reasoning procedure. While the form of multi-hop

chains of reasoning uncovers the reasoning proce-

dure and remedies the simple justiﬁcation of ra-

tionale, it still lacks critical clues about the mech-

anism within the reasoning steps. Our proposed

ﬁne-grained explanation form extends the chain of

reasoning by unwrapping the ﬁne-grained texture

within each reasoning step. As a result, it allows

the reasoning chains to include multiple inference

types (e.g., rebuttal) and broader reasoning types

sent1: v1: doctor : v2: recent pharmaceutical advances will lead the way in weight loss .

sent2: v1: prior to these advancements , v2: obesity - related deaths outnumbered all

other causes of death by a wide margin .

sent3: v1: the new drugs will v2: curb appetite and v3: increase metabolism .

sent4: v1: thanks to v2: these advancements , v3: obesity will dramatically decline in the

near future .

sent5: v1: most people will not be able to afford these prescriptions v2: since v3: the

majority of health care plans will not cover the new drugs .

Binary logical operators

→, ⋀, ⋁

Unary logical operators

¬, □(necessity), ◇(possibility)

Sent4

Certainty: necessary

Node Formula:

□(v𝟐→ □v𝟑)

Sent3

Certainty: necessary

Node Formula: □(v𝟑⋀v𝟐)

Sent2

Certainty: contingent

Node Formula: N/A

Sent1

Certainty: contingent

Node Formula: N/A

Sent5

Certainty: unnecessary

Node Formula:

¬□(¬□v𝟑→ v𝟏)

→(labeled)

♢(parsed)

¬(parsed)

___ root of dependency

parsing tree

bold w.r.t. global operator

Support

Rebut

◇¬(◇¬v𝟑→ v𝟏)

Reduction

Node Formula: ¬□(¬□v𝟑→ v𝟏)

Annotation

Global Operators

Certainty: unnecessary

Sent5:v1: most people will not be

able to afford these prescriptions v2:

since v3: the majority of health care

plans will not cover the new drugs .

Figure 2: The overall logical reasoning explanation task is deﬁned as follows. Given a passage, a model recon-

structs the ﬁne-grained logical structure with the meta support or rebut relations, the inner node formulae, and

degrees of certainty for each node. Given a logical statement, the formula is constructed from the labeled logical

triples with the parsed unary operators, which can then be reduced to canonical forms. The certainty label should

follow the global operators.

such as abductive reasoning with the hidden world-

knowledge assumption.

Logical Reasoning

Machine logical reasoning re-

quires models to conduct hidden symbolic reason-

ing processes through question answering (Yu et al.,

2020;Liu et al.,2020;Cui et al.,2020), or explicitly

perform symbolic reasoning via natural language

(Clark et al.,2020;Tafjord et al.,2021;Dalvi et al.,

2021). The QA-based reasoning data is mostly col-

lected from real-life scenarios without correspond-

ing structural information. To perform reasoning,

symbolic modules (Huang et al.,2021;Ouyang

et al.,2021) or learning strategies (Wang et al.,

2022) are designed to approximate the reasoning

structure. On the other hand, explicitly generat-

ing chains of reasoning can better uncover models’

reasoning processes. However, recent work mostly

focuses on deductive reasoning, where models with

iterative strategy (Tafjord et al.,2021) or reasoning

modules (Hong et al.,2022) show superior perfor-

mances. To encourage more advanced reasoning

capabilities, we propose a comprehensive reason-

ing structure with ﬁne-grained factors.

Argumentation / Discourse Structures

works (Lawrence and Reed,2019;Li et al.,2022)

such as argumentation mining (Stab and Gurevych,

2014b,a,2017) or discourse parsing (Carlson et al.,

2001;Webber et al.,2019) study document struc-

ture prediction. Given a passage, a model is re-

quired to predict the argument components or the

discourse relations between them. Instead of iden-

tifying the rhetorical structure of a passage, the

proposed logic metagraphs aim at simulating the

logical reasoning process, where the model needs

to select the relevant knowledge out of a pool to

ﬁnish the reasoning. Besides, unlike directly con-

sidering a sentence or a text span as a reasoning

node, MetaLogic explores a schema with ﬁner gran-

ularity. Each reasoning node is further decomposed

into logical variables with relations and modal op-

erators so that the inner structure as well as the

certainty are considered.

3 Task Deﬁnition

Overall Generation Task

The desideratum is that

a model reconstructs the ﬁne-grained logic expla-

nation for a given passage, which uncovers the

model’s understanding of the logic between the

lines. The logic explanation is formatted as logic

metagraphs with support or rebut inference steps,

per-node logical formulae, and degrees of certainty,

as demonstrated in Figure 2.

The input for the models is a passage with mul-

tiple statements

(S(0), S(1), ..., S(N))

and atomic

sentences

p(n)

∗⊆S(n)

, according to which they

generate the logic metagraph. The logic meta-

graph has three main components: (1) The meta

structure

G= (V,E)

, where

E=ESSER

, and

and

are the two meta edge types, support

and rebut, respectively, between the meta nodes

u(n)∈ V, n ≤N

. (2) The set of node formulae

where

u(n):= fn∈ F

. Each formula is joined by

Senses

Classic Morality Tense Belief

The proposition

is necessary.

pis morally obligatory.

It will always be the

case that p.

Things a person knows to

be true.

3pThe proposition pis possible.pis morally permissible.

It will sometimes be the

case that p.

Things that may be true

as far as a person knows.

Deﬁnitions

2p:= ¬3¬pIt is necessary that p.:= It is not possible that not-p.

3p:= ¬2¬pIt is possible that p.:= It is not necessary that not-p.

Reduction Rules

2¬p=¬3p,3¬p=¬2p,22p=2p,33p=3p,23p=3p,32p=2p.

Degrees of Certainty

2:= 4 (necessary), 3:= 3 (possible), N/A := 2 (contingent), ¬2:= 1 (unnecessary), ¬3:= 0 (impossible)

Table 1: Senses, mutual deﬁnitions, reduction rules, and degrees of certainty of modal logic operators.

logical triples.

fn=Tr(m(p(n)

i),m(p(n)

j))

, where

i6=j

r∈ {→,∧,∨}

, and

is a combination in

{¬,2,3}

. (3) The set of degrees of certainty

deﬁned by the combination format of {¬,2,3}.

4 The Logic Metagraph

In this section, we introduce the proposed logic

metagraph in details.3

4.1 Meta Node and Edge

Each meta node corresponds to a logically sound

statement (e.g., premise, or conclusion). The meta

edges are either support or rebut, relating to a single

step of inference. The support edges join the meta

nodes to form a chain of reasoning to the conclu-

sion, whereas the rebut edges indicate challenges

from the condition of rebuttal to one of the meta

nodes in the chain, which are evidence or claims

about exceptional conditions. Each inference step

allows multiple premises.

4.2 Internal Structure of Meta Node

The internal structure of a statement is formulated

as a propositional logic formula. The logical vari-

ables denote the atomic sentences in the statement

that corresponds to separate events or beliefs. The

logical relations between such events or beliefs are

denoted by binary propositional operators. There

are three logical relations: logical implication, con-

junction, and disjunction (

→

∧

∨

). Multiple such

logical triples are joined by conjunctions (

∧

). Fur-

thermore, each logical variable and the overall for-

mula are modiﬁed by negation (

) and modal (

3An example is shown on the left side of Figure 2.

and

) operators, representing the degrees of cer-

tainty of each atomic sentence as well as the whole

statement, respectively. A more detailed introduc-

tion can be found in Section 4.3.

4.3 Certainty with Modal Operators

Modal logic (Garson,2021) is an extension of ﬁrst-

order propositional logic with two modal operators,

necessity (

) and possibility (

). They are unary

operators, and Table 1presents examples of their

senses in natural language (Hughes et al.,1996).

For example,

denotes that the proposition

necessary, while

means

is possible, in the

classic deﬁnition. In another sense of tense,

represents that the evidence

is true at all times,

whereas

represents that

is only true some-

times. In general, the modal operators indicate

certainty information of the propositions.

The two modal operators can deﬁne each other

with the negation operator (

). Multiple reduction

rules are deﬁned. As a result, any complex formu-

lae composed of modal operators could be reduced

to one of the ﬁve degree-of-certainty forms listed

in Table 1, which is also known as the classic S5

system (Hughes et al.,1996) and makes the logic

metagraph deﬁned in a complete set.

5 MetaLogic

In this section, we introduce the construction de-

tails of the MetaLogic dataset. Since the logic meta-

graphs have ﬁne-grained structures with multiple

evaluation dimensions, which are all dispensable

and supplement each other, we design a rigorous

annotation process for the construction.

5.1 Preparation

Source Data

We use ReClor (Yu et al.,2020) as

the source data, where the multiple-choice ques-

tions are collected from GMAT and LSAT. As a

pilot study on logical reasoning explanation, we

start with the standard text questions so that the

explanation form can beneﬁt from precise and com-

prehensive logical information. Each question con-

tains a logical passage, a question, and multiple

answer options. The original dataset contains 17

reasoning types, which can be mainly categorized

into two folds: complete reasoning composed of

the logical passage and the option (e.g., the types

Necessary Assumptions, Sufﬁcient Assumptions,

Strengthen, Weaken); ﬂawed in-context reasoning

structure (e.g., the types Technique, Identify a Flaw,

or Dispute). As we aim to study models’ under-

standing of the complete reasoning process over

the whole passage, we consider data from the ﬁrst

category, from which we randomly choose 1,000

samples. Examples of the selected questions can

be found in Appendix A.

Data Preprocessing

We ﬁrst ﬁlter out incoherent

options from the questions for logical structure

coherence. For ordinary questions, the incoher-

ent options are the distracting ones. Conversely,

for the inverse questions with “EXCEPT”, we ran-

domly select one of the distracting options and

remove the others. We further split the passage into

sentences as the initial meta nodes and per meta

node sentence into clauses as the initial logical

variables. This follows the convention of applying

linguistic-based segments as reasoning components

in related studies (Dalvi et al.,2021;Huang et al.,

2021;Wang et al.,2022;Xu et al.,2022). Besides,

considering the label hierarchy that the logical vari-

ables are conditioned on the meta nodes, the initial

segments help build the desired metagraph sketch.

Moreover, the initial delimitation is trivial with

punctuation marks and provides the least machine

guidance to the annotators, who are free to modify

the segments on their understanding of reasoning

units, which will be demonstrated in Section 5.2.

From the experts’ view, 27 of 30 randomly sam-

pled annotated graphs are of high quality, which

indicates the high reliability of starting with the

initial segments.

As a result, the text presented to the annotators

contains the original text with the passage, the ques-

tion, and the coherent option, along with a list of

delimited sentences.

Meta Structure Meta Node

M-Node M-Edge L-Variable L-Relation

κ57.80†42.82†65.46‡56.81†

Table 2: IAA with Cohen’s Kappa coefﬁcients. M-

Node: meta node, M-Edge: meta edge, L-Variable: log-

ical variable, L-Relation: logical relation. ‡indicates

very high agreement with κover 60%.†indicates high

agreement with κbetween 40% and 60%.

5.2 Annotation

As all annotation tasks require a global understand-

ing of the overall passage, we recruit the same

annotator to ﬁnish all tasks in the same passage.

The annotation procedure has four steps. (1) Read

through the text and have a rough idea about the

logical role of each initial meta node (e.g., being a

conclusion or rebuttal). If an initial meta node does

not provide complete evidence, then the annotator

needs to merge it with another node to form com-

plete evidence. (2) Annotate the inference types

between the meta nodes. After this stage, we obtain

the chain of reasoning and the rebuttal steps. (3)

For each meta node, annotate the logical variables

by reﬁning the span boundaries of the given initial

logical variables. (4) Annotate the logical binary

operator between the logical variables. The annota-

tion platform is demonstrated in Appendix D.

We recruit annotators from crowd-sourcing plat-

forms. We ﬁrst train annotators with a carefully

designed annotation guideline

and require them

to pass an exam before the annotation to guarantee

the annotation quality. For each passage, we invite

two annotators

. On average, we pay $2.2 for each

logical passage.

For unary logical operators (

), as dis-

cussed by (Toulmin,2003), there exist conventional

clue words for the negation and modality. Follow-

ing that, we leverage such in-context clue words

for the annotation. Given a set of conventional in-

dicators (demonstrated in Table 13 in Appendix C),

we parse each meta node sentence into a depen-

dency parsing tree, then detect those words within

3-hops to the root node, and assign the correspond-

ing operators to the formula. The consecutive unary

operators are ordered by the distance from the indi-

cators to the parsing root node. This results in the

global unary operators. For local unary operators of

4Details are shown in Appendix B.

For the inconsistent annotation, we invite a third annotator

to make the judgement.

文档加载中……请稍候！
如果长时间未打开，您也可以点击刷新试试。

下载文档到电脑，查找使用更方便

10 玖币 0人已下载

立即下载

摘要：

MetaLogic:LogicalReasoningExplanationswithFine-GrainedStructureYinyaHuang1,2HongmingZhang2yRuixinHong3XiaodanLiang1,4yChangshuiZhang3DongYu21ShenzhenCampusofSunYat-senUniversity2TencentAILab,Seattle3TsinghuaUniversity4PengchengLaboratoryyinya.huang@hotmail.com,{hongmzhang,dyu}@global.tencent.com,hr...

展开>> 收起<<

MetaLogic Logical Reasoning Explanations with Fine-Grained Structure Yinya Huang12Hongming Zhang2yRuixin Hong3Xiaodan Liang14y Changshui Zhang3Dong Yu2.pdf

共27页,预览5页

还剩页未读，继续阅读

声明：本站为文档C2C交易模式，即用户上传的文档直接被用户下载，本站只是中间服务平台，本站所有文档下载所得的收益归上传人(含作者)所有。玖贝云文库仅提供信息存储空间，仅对用户上传内容的表现方式做保护处理，对上载内容本身不做任何修改或编辑。若文档所含内容侵犯了您的版权或隐私，请立即通知玖贝云文库，我们立即给予删除！

MetaLogic Logical Reasoning Explanations with Fine-Grained Structure Yinya Huang12Hongming Zhang2yRuixin Hong3Xiaodan Liang14y Changshui Zhang3Dong Yu2

相关推荐

开通VIP享超值会员特权

作者详情

相关内容

热门标签

举报选择: