Causal Inference for Chatting Handoff Shanshan Zhong Jinghui Qin Zhongzhan Huang Daifeng Liy School of Computer Science and Engineering

2025-04-27 0 0 903.58KB 11 页 10玖币

侵权投诉

Causal Inference for Chatting Handoff ∗

Shanshan Zhong, Jinghui Qin, Zhongzhan Huang, Daifeng Li†

School of Computer Science and Engineering

Sun Yat-sen University

Abstract

Aiming to ensure chatbot quality by predicting

chatbot failure and enabling human-agent col-

laboration, Machine-Human Chatting Handoff

(MHCH) has attracted lots of attention from

both industry and academia in recent years.

However, most existing methods mainly fo-

cus on the dialogue context or assist with

global satisfaction prediction based on multi-

task learning, which ignore the grounded re-

lationships among the causal variables, like

the user state and labor cost. These variables

are signiﬁcantly associated with handoff de-

cisions, resulting in prediction bias and cost

increasement. Therefore, we propose Causal-

Enhance Module (CEM) by establishing the

causal graph of MHCH based on these two

variables, which is a simple yet effective mod-

ule and can be easy to plug into the existing

MHCH methods. For the impact of users, we

use the user state to correct the prediction bias

according to the causal relationship of multi-

task. For the labor cost, we train an auxil-

iary cost simulator to calculate unbiased labor

cost through counterfactual learning so that a

model becomes cost-aware. Extensive exper-

iments conducted on four real-world bench-

marks demonstrate the effectiveness of CEM

in generally improving the performance of ex-

isting MHCH methods without any elaborated

model crafting.

1 Introduction

In recent years, with the rapid development of deep

learning (He et al.,2016;Ren et al.,2015), more

and more service-oriented organizations have de-

ployed chatbots to alleviate the problem of limited

service resources. Although these chatbots can

respond in real-time and save labor cost, they suf-

fer from inappropriate responses and invalid con-

versations due to the limited quantity of available

high-quality training data and the inherent biases

∗

Work in progress

†

Corresponding authour

I want to buy a ticket to Chicago.

Hi,what can I do for you?

Hi, what can I do for you?

Chatbot

User When are you going to leave?

Today at 5pm.

User

I have to go today!

Chatbot

Sorry, the tickets have been sold out.

User

Help me please!!

Customer

Service

Global satisfaction:

......

User

I want to buy a ticket to Chicago Today.

Local sentiment Handoff label

Normal Transferable

Chatbot

Figure 1: An example of MHCH. Handoff label in-

cludes two types "normal" & "transferable", which de-

notes whether the chatbot should be transferred to hu-

man service.

(Xu et al.,2019;Liang et al.,2022) of neural net-

works. Moreover, the human utterances sometimes

are elusive since they are rich in acronyms, slang

words, and even content without logic or grammar,

which are too obscure for a chatbot to compre-

hend. To alleviate these drawbacks, researchers

introduced a human-agent collaboration mecha-

nism named Machine-Human Chatting Handoff

(MHCH) to allow a human to take over the dia-

logue while a robot agent feel confused so that

a dialogue can be continued to avoid a bad user

arXiv:2210.02862v1 [cs.AI] 6 Oct 2022

experience and reduce the risk of customer churn

(Liu et al.,2021a,b). As shown in Fig.1, when a

chatbot try to address the user’s needs by giving

an inappropriate response, the user will feel disap-

pointed, and give a low global satisfaction score

for current dialogue, which means a service failure

and may lead to customer loss. If deploying with

MHCH mechanism, a human can take over the dia-

logue and give a satisfactory response to meet the

user’s needs, thus ensuring the user experience and

service quality (Radziwill and Benton,2017).

In fact, a high-quality MHCH service should

consider multiple factors, such as dialogue context,

local sentiments, global satisfaction, user state, and

labor cost, etc. However, most existing MHCH

methods mainly concerned on the dialogue context

(Liu et al.,2021a) or assisting with global satisfac-

tion prediction under the multi-task learning setting

(Liu et al.,2021b), ignoring the grounded relation-

ships among the other causal variables of MHCH,

like the user state and human cost.

To address above issues and improve the perfor-

mance of MHCH, we propose a general Causal-

Enhance Module (CEM), which can be plugged

into existing MHCH networks (Liu et al.,2021a,b),

to incorporate the considerations of other potential

causal variables of MHCH. Speciﬁcally, we ﬁrst

analyzes MHCH task based on causal graph by

mining all potential causal variables and deduce

that user states and labor cost are the other two

causal variables that should be considered for high-

quality customer service. Then, to incorporate the

consideration of user state, we train a user state

network mainly driven by local sentiments to main-

tain the changes of user state during the dialogue

and adjust the handoff predictions by correcting the

prediction bias according to the causal relationship

between user states and handoff decisions. To con-

sider the labor cost of customer service and reduce

it as much as possible while maintaining the same

service quality, we construct a counterfactual-based

cost simulator to regress the cost of a dialogue as

an auxiliary task which can make the MHCH back-

bone become cost-aware and minimize the labor

cost as much as possible.

The contributions of our CEM can be summa-

rized as follows:

•

We conduct causal analysis based on causal

graph for MHCH and identify the other two

causal variables: user state and human cost,

which should be considered to build high-

quality MHCH service.

•

To consider the impact of user state, the user

state is applied to correct the handoff predic-

tion bias according to the causal relationship

between user states and handoff decisions.

•

To minimize the labor cost of customer ser-

vice while maintaining the same service qual-

ity, we construct a counterfactual-based cost

simulator to regress the cost of a dialogue as

an auxiliary task, which can make the MHCH

backbone become cost-aware.

2 Related Work

Machine-Human Chatting Handoff.

The re-

search on MHCH is originated in 2018. Using

the idea of reinforcement learning, Huang et al.

(2018) proposed a dialogue robot to choose an as-

sistant. Rajendran et al. (2019) utilize a reinforce-

ment learning framework to maximize success rate

and minimize human workload. Liu et al. (2021a,b)

regraded the MHCH as a classiﬁcagtion problem

and focused on identifying which sentence should

be transferred to the human service.

Causal inference and counterfactual learning.

For structural causal models (Halpern et al.,2005),

related studies (Heskes,2013;Claassen et al.,2014;

Xia et al.,2021) utilize graph neural networks for

directed acyclic graph structure learning. For Ru-

bin causal models, Rubin (2006) and Bengio et al.

(2019) use neural networks to approximate the

propensity scores, matching weights, etc., which

can satisfy the covariate balancing (Kallus,2020;

Kuang et al.,2017); The representation learning

(Huang et al.,2020b;Liang et al.,2020) can also

be used to matched the covariate balance between

the test group and the reference group (Shalit et al.,

2017;Louizos et al.,2017;Lu et al.,2020). Several

studies (Yoon et al.,2018;Yuan et al.,2019;Liu

et al.,2020) uses counterfactual methods based on

the generative models over the observed distribu-

tions to causal inference.

Multi-task learning in dialogue systems.

et al. (2020) uses multi-task learning for auxiliary

pre-training tasks of dialogue data. Qin et al. (2020)

combines dialogue behavior recognition and sen-

timent classiﬁcation. Ide and Kawahara (2021)

proposes a model which includes generation and

classiﬁcation tasks.

User net MHCH net

Tradictional method

LSY

Neural network

CEM Neural network(Pre-trained)

Original CEM (C)CEM (U)

Encoder

LSY

User net

MHCH net

Encoder

User net

MHCH net

Encoder

User net

MHCH net

Encoder

CEM (full)

Figure 2: Causal graphs and model structures. D: Dialog, Y: prediction of MHCH, LS: local sentiment, GS: global

satisfaction, US: user state, C: labor cost. The solid lines represent causality, the dashed line is adjustment, and

the dotted line outlines the part that MHCH classiﬁcation models doesn’t have. a, b, c are the causal graphs

of MHCH, traditional multi-task methods and CEM based on multi-task learning, respectively. dis the original

model structure based on b(Song et al.,2019;Liu et al.,2021a,b). e,fand gare the model structures enhanced by

CEM(U), CEM(C) and CEM(full).

3 Preliminary

A given dialogue

D= [u1, u2, . . . , uL]

contains

utterances and have a label sequence

Yh=

[yh

1, . . . , yh

, where

is the handoff label of

1≤t≤L

. The handoff labels

have two

kinds of labels, i.e.,

"normal"

and

"transferable"

where "normal" means that the utterance is no need

to transfer, and "transferable" means that the utter-

ance needs to be transferred to the manual service.

The dialogue

also have a global satisfaction label

{"satisfactory","neutral","dissatisﬁed"}

. Then,

the local sentiment of each utterance

is measured

by an open-source tool SnowNLP, which includes

three labels {"positive","neutral","negative"}.

4 Methodology

In this section, we analyse the impact of variables

on MHCH from a fundamental view of causality.

Then we present our CEM framework that elimi-

nates the bad effect of ignored causal variables.

4.1 Causal analysis of MHCH

Causal graph is a directed acyclic graph where

a node denotes a variable and an edge denotes a

causal relation between two nodes (Pearl,2009).

It is widely used to describe the process of data,

which can guide the design of predictive models

(Zhang et al.,2021). Fig.2(a) shows the causal

graph of MHCH. The rationality of this causal

graph is explained as follows:

• D denote the dialogue D= [u1, . . . , uL].

•

Y =

[p1, p2, ..., pL]

is the prediction of MHCH,

where

pt,1≤t≤L

is the probability of that

the handoff label of utis "transferable".

•

LS is the local sentiments of each utterance in

a dialogue.

•

GS represents the user’s subjective evaluation

of the current dialogue.

•

US is a state for a given dialogue. Unlike

GS, it is a variable that describes the objective

文档加载中……请稍候！
如果长时间未打开，您也可以点击刷新试试。

下载文档到电脑，查找使用更方便

10 玖币 0人已下载

立即下载

摘要：

CausalInferenceforChattingHandoffShanshanZhong,JinghuiQin,ZhongzhanHuang,DaifengLiySchoolofComputerScienceandEngineeringSunYat-senUniversityAbstractAimingtoensurechatbotqualitybypredictingchatbotfailureandenablinghuman-agentcol-laboration,Machine-HumanChattingHandoff(MHCH)hasattractedlotsofattentio...

展开>> 收起<<

Causal Inference for Chatting Handoff Shanshan Zhong Jinghui Qin Zhongzhan Huang Daifeng Liy School of Computer Science and Engineering.pdf

共11页,预览3页

还剩页未读，继续阅读

声明：本站为文档C2C交易模式，即用户上传的文档直接被用户下载，本站只是中间服务平台，本站所有文档下载所得的收益归上传人(含作者)所有。玖贝云文库仅提供信息存储空间，仅对用户上传内容的表现方式做保护处理，对上载内容本身不做任何修改或编辑。若文档所含内容侵犯了您的版权或隐私，请立即通知玖贝云文库，我们立即给予删除！

Causal Inference for Chatting Handoff Shanshan Zhong Jinghui Qin Zhongzhan Huang Daifeng Liy School of Computer Science and Engineering

相关推荐

开通VIP享超值会员特权

作者详情

相关内容

热门标签

举报选择: