KSAT Knowledge-infused Self Attention Transformer - Integrating Multiple Domain-Specific Contexts

2025-05-03 0 0 949.4KB 5 页 10玖币

侵权投诉

KSAT: Knowledge-infused Self Aention Transformer -

Integrating Multiple Domain-Specific Contexts

Kaushik Roy

kaushikr@email.sc.edu

Articial Intelligence Institute

University of South Carolina

USA

Yuxin Zi

yzi@email.sc.edu

Articial Intelligence Institute

University of South Carolina

USA

Vignesh Narayanan

vignar@sc.edu

Articial Intelligence Institute

University of South Carolina

USA

Manas Gaur

manas@umbc.edu

KAI2, University of Maryland

Baltimore County

USA

Amit Sheth

amit@sc.edu

Articial Intelligence Institute

University of South Carolina

USA

ABSTRACT

Domain-specic language understanding requires integrating mul-

tiple pieces of relevant contextual information. For example, we

see both suicide and depression-related behavior (multiple con-

texts) in the text “I have a gun and feel pretty bad about my life,

and it wouldn’t be the worst thing if I didn’t wake up tomorrow”.

Domain specicity in self-attention architectures is handled by

ne-tuning on excerpts from relevant domain specic resources

(datasets and external knowledge - medical textbook chapters on

mental health diagnosis related to suicide and depression). We

propose a modied self-attention architecture Knowledge-infused

Self Attention Transformer (KSAT) that achieves the integration

of multiple domain-specic contexts through the use of external

knowledge sources. KSAT introduces knowledge-guided biases in

dedicated self-attention layers for each knowledge source to accom-

plish this. In addition, KSAT provides mechanics for controlling

the trade-o between learning from data and learning from knowl-

edge. Our quantitative and qualitative evaluations show that (1) the

KSAT architecture provides novel human-understandable ways to

precisely measure and visualize the contributions of the infused

domain contexts, and (2) KSAT performs competitively with other

knowledge-infused baselines and signicantly outperforms base-

lines that use ne-tuning for domain-specic tasks.

KEYWORDS

knowledge graphs, language models, knowledge-infusion

1 MOTIVATION

Solving domain-specic tasks such as mental health diagnosis

(MHD), and triaging, requires integrating relevant contextual infor-

mation from data and knowledge sources. Self-Attention based Lan-

guage Models (SAMs) capture an aggregated broader context from

domain-agnostic, voluminous training corpora [

]. Fine-tuning

SAMs on domain-specic corpora achieves domain-specic con-

text capture [

]. However, SAM architectures are black-box in

nature [

]. Consequently, ne-tuned SAM architectures do not lend

themselves to the robust evaluation of the open research aims: (R1)

Relevant domain-specic context coverage, and (R2) The inu-

ence of knowledge context traded-o against the data context in

downstream tasks [

]. We propose a modied self-attention ar-

chitecture Knowledge-infused Self Attention Transformer (KSAT) to

address these aims. KSAT performs well on select domain-specic

tasks (see 2.2) while lending itself to a robust human-understandable

evaluation of R1 and R2. Thus KSAT provides a substantial step

towards fostering AI-user trust, and satisfaction [7, 8].

2 BACKGROUND

2.1 Related Work

Prior approaches that are relevant to R1 and R2 and incorporate

multiple knowledge contexts can be broadly categorized based on

the knowledge-infusion technique as (1) knowledge modulated

SAMs and (2) knowledge infused input embedding-based SAMs

[

]. The former uses knowledge to guide the self-attention

mechanism in SAMs, and the latter embeds the knowledge into a

vector space before passing the inputs into SAMs. Here, we briey

summarize their contributions towards R1 and R2. Both Cate-

gory (1), and Category (2) methods’ domain coverage is evaluated

through performance on domain-specic task descriptions (R1).

These methods’ ablations highlight contributions of knowledge

context (R2). However, inspecting the numerical outputs from the

model components (projection matrices and vectors) does not easily

lend themselves to human-understandable scrutiny. Explainable AI

techniques (post-processing of the numerical outputs that trans-

form them into human-understandable information) are required

to conrm the author(s) perspectives [

]. Post-processing-based

explanations are local approximations of the SAM reasoning for

particular inputs and therefore do not present the global picture,

casting doubts on the SAM evaluation validity. KSAT presents a

SAM architecture whose numerical outputs lend themselves to

robust human-understandable evaluations of R1 and R2.

2.2 Task Description, Data, and External

Knowledge Sources

Although the KSAT architecture broadly applies to any domain-

specic task, we choose the specic task of Mental Health Diag-

nostic Assistance for Suicidal Tendencies by Gaur et al. [

]. We

denote this dataset as MHDA. The data contains high-quality ex-

pert annotations on Reddit posts from suicide-related subreddits.

arXiv:2210.04307v2 [cs.CL] 24 Jun 2023

文档加载中……请稍候！
如果长时间未打开，您也可以点击刷新试试。

下载文档到电脑，查找使用更方便

10 玖币 0人已下载

立即下载

摘要：

KSAT:Knowledge-infusedSelfAttentionTransformer-IntegratingMultipleDomain-SpecificContextsKaushikRoykaushikr@email.sc.eduArtificialIntelligenceInstituteUniversityofSouthCarolinaUSAYuxinZiyzi@email.sc.eduArtificialIntelligenceInstituteUniversityofSouthCarolinaUSAVigneshNarayananvignar@sc.eduArtificial...

展开>> 收起<<

KSAT Knowledge-infused Self Attention Transformer - Integrating Multiple Domain-Specific Contexts.pdf

共5页,预览1页

还剩页未读，继续阅读

声明：本站为文档C2C交易模式，即用户上传的文档直接被用户下载，本站只是中间服务平台，本站所有文档下载所得的收益归上传人(含作者)所有。玖贝云文库仅提供信息存储空间，仅对用户上传内容的表现方式做保护处理，对上载内容本身不做任何修改或编辑。若文档所含内容侵犯了您的版权或隐私，请立即通知玖贝云文库，我们立即给予删除！

KSAT Knowledge-infused Self Attention Transformer - Integrating Multiple Domain-Specific Contexts

相关推荐

开通VIP享超值会员特权

作者详情

相关内容

热门标签

举报选择: