Why did the Model Fail Attributing Model Performance Changes to Distribution Shifts

2025-05-06 0 0 1.49MB 29 页 10玖币

侵权投诉

“Why did the Model Fail?”: Attributing Model Performance Changes to

Distribution Shifts

Haoran Zhang * 1 Harvineet Singh * 2 Marzyeh Ghassemi 1Shalmali Joshi 3

Abstract

Machine learning models frequently experience

performance drops under distribution shifts. The

underlying cause of such shifts may be multiple

simultaneous factors such as changes in data qual-

ity, differences in speciﬁc covariate distributions,

or changes in the relationship between label and

features. When a model does fail during deploy-

ment, attributing performance change to these

factors is critical for the model developer to iden-

tify the root cause and take mitigating actions. In

this work, we introduce the problem of attributing

performance differences between environments

to distribution shifts in the underlying data gener-

ating mechanisms. We formulate the problem as

a cooperative game where the players are distribu-

tions. We deﬁne the value of a set of distributions

to be the change in model performance when only

this set of distributions has changed between en-

vironments, and derive an importance weighting

method for computing the value of an arbitrary set

of distributions. The contribution of each distribu-

tion to the total performance change is then quan-

tiﬁed as its Shapley value. We demonstrate the

correctness and utility of our method on synthetic,

semi-synthetic, and real-world case studies, show-

ing its effectiveness in attributing performance

changes to a wide range of distribution shifts.

1. Introduction

Machine learning models are widely deployed in dynamic

environments ranging from recommendation systems to

personalized clinical care. Such environments are prone to

distribution shifts, which may lead to serious degradations

in model performance (Guo et al.,2022;Chirra et al.,2018;

Equal contribution

MIT

New York University

Columbia

University. Correspondence to: Haoran Zhang

hao-

ranz@mit.edu>.

Proceedings of the

40 th

International Conference on Machine

Learning, Honolulu, Hawaii, USA. PMLR 202, 2023. Copyright

2023 by the author(s).

Koh et al.,2021;Geirhos et al.,2020;Nestor et al.,2019;

Yang et al.,2023). Importantly, such shifts are hard to

anticipate and reduce the ability of model developers to

design reliable systems.

When the performance of a model does degrade during de-

ployment, it is crucial for the model developer to know not

only which distributions have shifted, but also how much a

speciﬁc distribution shift contributed to model performance

degradation. Using this information, the model developer

can then take mitigating actions such as additional data col-

lection, data augmentation, and model retraining (Ashmore

et al.,2021;Zenke et al.,2017;Subbaswamy et al.,2019).

In this work, we present a method to attribute changes in

model performance to shifts in a given set of distributions.

Distribution shifts can occur in various marginal or condi-

tional distributions that comprise variables involved in the

model. Further, multiple distributions can change simulta-

neously. We handle this in our framework by deﬁning the

effect of changing any set of distributions on model perfor-

mance, and use the concept of Shapley values (Shapley et al.,

1953) to attribute the change to individual distributions. The

Shapley value is a co-operative game theoretic framework

with the goal of distributing surplus generated by the players

in the co-operative game according to their contribution. In

our framework, the players correspond to individual distri-

butions, or more precisely, mechanisms involved in the data

generating process.

Most relevant to our contributions is the work of Budhathoki

et al. (2021), which attributes a shift between two joint

distributions to a speciﬁc set of individual distributions.

The distributions here correspond to the components of

the factorization of the joint distribution when the data-

generating process is assumed to follow causal structural

assumptions. This line of work deﬁnes distribution shifts as

interventions on causal mechanisms (Pearl & Bareinboim,

2011;Subbaswamy et al.,2019;2021;Budhathoki et al.,

2021;Thams et al.,2022). We build on their framework

to justify the choice of players in our cooperative game.

We signiﬁcantly differ from the end goal by attributing a

change in model performance between two environments to

individual distributions. Note that each shifted distribution

may inﬂuence model performance differently and may result

arXiv:2210.10769v3 [cs.LG] 6 Jun 2023

“Why did the Model Fail?”: Attributing Model Performance Changes to Distribution Shifts

in signiﬁcantly different attributions than their contributions

to the shift in the joint distribution between environments.

In this work, we focus on explaining the discrepancy in

model performance between two environments as measured

by some metric such as prediction accuracy. We emphasize

the non-trivial nature of this problem, as many distribution

shifts will have no impact on a particular model or metric,

and some distribution shifts may even increase model perfor-

mance. Moreover, the root cause of the performance change

may be due to distribution shifts in variables external to

the model input. Thus, explaining performance discrepancy

requires us to develop specialized methods. Speciﬁcally, we

want to quantify the contribution to the performance change

of a ﬁxed set of distributions that may change across the

environments. Given such a set, we develop a model-free

importance sampling approach to quantify this contribution.

We then use the Shapley value framework to estimate the at-

tribution for each distribution shift. This framework allows

us to expand the settings where our method is applicable.

We make the following contributions1:

•

We formalize the problem of attributing model perfor-

mance changes due to distribution shifts.

•

We propose a principled approach based on Shapley val-

ues for attribution, and show that it satisﬁes several desir-

able properties.

•

We validate the correctness and utility of our method on

synthetic and real-world datasets.

2. Problem Setup

Notation. Consider a learning setup where we have some

system variables denoted by

consisting of two types of

variables

V= (X, Y )

, which comprises of features

and

labels

such that

V∼ D

. Realizations of the variables are

denoted in lower case. We assume access to samples from

two environments. We use

Dsource

to denote the source dis-

tribution and

Dtarget

for the target distribution. Subscripts on

refer to the distribution of speciﬁc variables. For example,

DX1

is the distribution of feature

X1⊂X

, and

DY|X

is the

conditional distribution of labels given all features X.

Let

XM⊆X

be the subset of features utilized by a given

model

. We are given a loss function

ℓ((x, y), f)7→ R

which assigns a real value to the model evaluated at a spe-

ciﬁc setting

of the variables. For example, in the case

of supervised learning, the model

maps

into the la-

bel space, and a loss function such as the squared error

ℓ((x, y), f) := (y−f(xM))2

can be used to evaluate model

performance. We assume that the loss function can be com-

puted separately for each data point. Then, performance

Code:

https://github.com/MLforHealth/expl_

perf_drop

of the model in some environment with distribution

summarized by the average of the losses:

Perf(D) := E(x,y)∼D[ℓ((x, y), f)]

This implies that a shift in any variables

in the system may

result in performance change across environments, includ-

ing those that are not directly used by the model, but drive

changes to the features XMused by the model for learning.

Setup. Suppose we are given a candidate set of (marginal

and/or conditional) distributions

over

that may ac-

count for the model performance change from

Dsource

Dtarget

Perf(Dtarget)−Perf(Dsource)

.Our goal is to at-

tribute this change to each distribution in the candidate

set

.For our method, we assume access to the model

and samples from Dsource as well as Dtarget (see Figure 1).

We assume that dependence between variables

is de-

scribed by a causal system (Pearl,2009). For every variable

Xi∈V

, this dependence is captured by a functional rela-

tionship between

and the so-called “causal parents” of

(denoted as

parent(Xi)

) driving the variation in

. The

causal dependence induces a Markov distribution over the

variables in this system. That is, the joint distribution

can be factorized as,

DV=QXi∈VDXi|parent(Xi)

. This de-

pendence can be summarized graphically using a Directed

Acyclic Graph (DAG) with nodes corresponding to the sys-

tem variables and directed edges (

parent(Xi)→Xi

) in

the direction of the causal mechanisms in the system (see

Figure 1for an example).

Example. We provide an example that illustrates that the

performance attribution problem is ill-speciﬁed without

knowing how the mechanisms can change to result in the

observed performance difference. Suppose we are predict-

ing

from

with a linear model

f(x) := ϕx

under the

squared loss function. Consider two possible scenarios for

data generation – (1)

X←Y

where

changes from

source to target while

DX|Y

remains the same, (2)

X→Y

where

changes from source to target while

DY|X

re-

mains the same. The performance difference of

f(x)

is the

same in both the cases. Naturally, we want an attribution

method to assign all of the difference to the mechanism for

in the ﬁrst case and to the mechanism of

in the second

case. Thus, for the same performance difference between

source and target data, we would like a method to output dif-

ferent attributions depending on whether the data generating

process is case (1) or (2). Note that, in general, it is im-

possible to ﬁnd the appropriate attributions by ﬁrst ﬁnding

the direction of the causal mechanisms. This follows from

the fact that learning the structure is in general, impossible

purely from observational data (Peters et al.,2017). Hence

knowledge of the data-generating mechanisms is necessary

for appropriate attribution.

More concretely, suppose the processes are (1)

Y∼

“Why did the Model Fail?”: Attributing Model Performance Changes to Distribution Shifts

If actual shifts:

Attributions:

Attribute

performance change

for a given model

Input Data OutputKnown Causal Graph

Candidate Distr. Shifts:

Loss:

Figure 1: Inputs and outputs for attribution. Input: Causal graph, where all variables are observed providing the candidate

distribution shifts we consider. The goal is to attribute the model’s performance change

∆

between source and target

distributions to these candidate distributions. Here, out of the three candidate distributions, the marginal distribution of

and the conditional distribution of

given

change. Our method attributes changes to each one such that the attributions

sum to the total performance change

∆

. Note that nodes in the causal graph may be vector-valued, which allows our method

to be used on high-dimensional data such as images.

N(µ1,1), X ∼Y+N(0,1)

. The mean of

shifts to

µ2

in target, and (2)

X∼N(µ1,1), Y ∼X+N(0,1)

where the mean of

shifts to

µ2

in target. For the model

f(x) := ϕx

, the performance difference

∆

in both cases

(1 −ϕ)2(µ2

2−µ2

. This example illustrates the need

for specifying how the mechanisms can shift from source

to target to solve the attribution problem. In this work, we

use partial causal knowledge, in terms of the causal graph

only, to specify the data-generating mechanisms.

In general, this partial knowledge further allows us to iden-

tify potential shifts to consider. Speciﬁcally, the number

of marginal and conditional shifts that can be deﬁned over

(X, Y )

is exponential in the dimension of

. The factoriza-

tion induced by the causal graph or equivalently knowledge

of the data-generating mechanism reduces the space of pos-

sible shifts to consider for attribution. See Section 3for

additional advantages of using a causal framework.

3. Method

We now formalize our problem setup and motivate a game

theoretic method for attributing performance changes to

distributions over variable subsets (See Figure 1for a sum-

mary). We proceed with the following Assumptions.

Assumption 3.1. The causal graph corresponding to the

data-generating mechanism is known and all variables in

the system are observed. Thus, the factorization of the joint

distribution DVis known.

Assumption 3.2. Distribution shifts of interest are due to

(independent) shifts in one or more factors of DV.

Given these assumptions, we now describe our game theo-

retic formulation for attribution.

3.1. Game Theoretic Distribution Shift Attribution

We consider the set of candidate distributions

as the

players in our attribution game. A coalition of any subset

of players determines the distributions that are allowed to

shift (from their source domain distribution to the target

domain distribution), keeping the rest ﬁxed. The value

for the coalition is the model performance change between

the resulting distribution for the coalition and the training

distribution. See Figure 2for an overview of the method.

Value of a Coalition. Consider a coalition of distributions

C⊆CD

. This coalition implies a joint distribution over

system variables

, where members in the coalition con-

tribute their target domain distribution, and non-members

contribute their source domain distribution:

D=

Y

i:DXi|parent(Xi)∈

Dtarget

Xi|parent(Xi)



| {z }

Coalition



Y

i:DXi|parent(Xi)̸∈

Dsource

Xi|parent(Xi)



| {z }

Not in Coalition

(1)

The above factorization follows from Assumptions 3.1

and 3.2. Note that the coalition only consists of distribu-

“Why did the Model Fail?”: Attributing Model Performance Changes to Distribution Shifts

Figure 2: Sketch of the game theoretic attribution

method. Each causal mechanism is a player that, if present

in the coalition, changes to the target distribution and, if

absent, remains ﬁxed at the source distribution. This deﬁnes

the distribution of the resulting coalition

. Performance

is estimated using importance sampling from training

data samples. After computing values for each possible

coalition, Shapley value (Eq. 3) gives the attribution to each

player. Thus, we estimate the performance change under all

possible ways to shift the mechanisms from source to target

and use these to distribute the total performance change

among the individual distributions.

tions that are allowed to change across environments. All

other relevant mechanisms are indeed ﬁxed to the source

distribution. We present an example of a coalition of two

players in Figure 2. The value of the coalition

with the

coalition distribution e

Dis now given by

Val(e

C) := Perf(e

D)−Perf(Dsource)(2)

Thus, our assumptions allow us to represent a factorization

where only members of the coalition change, while all other

mechanisms correspond to the source distribution. If we

consider the change in performance for all combinatorial

coalitions, we can estimate the total contribution of a spe-

ciﬁc distribution by aggregating the value for all possible

coalitions a candidate distribution is a part of. This is exactly

the Shapley value applied to a set of distributions. The Shap-

ley value framework thus allows us to obtain the attribution

of each player d∈CDusing Equation 3.

Abstractly, the Shapley values framework (Shapley et al.,

1953) is a game theoretic framework which assumes that

there are

C:= {1,2, . . . , n}

players in a co-operative game,

achieving some total value (in our case, model performance

change). We denote by

Val : 2C7→ R

, the value for any

subset of players, which is called a coalition. Shapley val-

ues correspond to the fair assignment of the value

Val(C)

to each player

d∈C

. The intuition behind Shapley values

is to quantify the change in value when a player (here, a

distribution) enters a coalition. Since the change in model

performance depends on the order in which players (distribu-

tions) may join the coalition, Shapley values aggregate the

value changes over all permutations of

. Thus the Shapley

attribution Attr(d)for a player dis given by:

Attr(d) = 1

|C|X

C⊆C\{d}|C| − 1

C|−1Val(e

C∪ {d})−Val(e

C)

(3)

where we measure the change in model performance (de-

noted by Val) after adding

to the coalition averaged over

all potential coalitions involving

. The computational com-

plexity of estimating Shapley values is exponential in the

number of players. Hence we rely on this exact expression

only when the number of candidate distributions is small.

That is, the causal graph induces a factorization that results

in smaller candidate sets. For larger candidate sets, we

use previously proposed approximation methods (Castro

et al.,2009;Lundberg & Lee,2017;Janzing et al.,2020)

for reduced computational effort.

Choice of Candidate Distribution Shifts. We motivate

further the choice of candidate distributions that will in-

form the coalition. As mentioned before, without the

knowledge of the causal graph, many heuristics for choos-

ing the candidate sets are possible. For example, a can-

didate set could be the set of all marginal distributions

on each system variable,

CD={DX1,DX2,· · · }

, or dis-

tribution of each variable after conditioning on the rest,

CD={DX1|V\X1,DX2|V\X2,· · · }

. Since we have com-

binatorially many shifts that can be deﬁned on subsets of

V= (X, Y )

, choosing candidate sets that would then in-

form the coalition is challenging. The causal graph, on the

other hand, speciﬁes the factorization of the joint distribu-

tion into a set of distributions. We form the candidate set

constituting each distribution in this factorization. That is,

CD={DX1|parent(X1),· · · ,DXi|parent(Xi),· · · }i=1,··· ,|V|

For a node without parents in the causal graph, the parent set

can be empty, which reduces

DXi|parent(Xi)

to the marginal

distribution of

. This choice of candidate set has three

main advantages. First, it is interpretable since the candi-

date shifts are speciﬁed by domain experts who constructed

the causal graph. Second, it is actionable since identifying

the causal mechanisms most responsible for performance

change can inform mitigating methods for handling distri-

bution shifts (Subbaswamy et al.,2019). Third, it will lead

to succinct attributions due to the independence property.

Consider the case where only one conditional distribution

D(Xi|parent(Xi))

changes across domains. This will result

in a change in distributions of all descendants of

(due to

the above factorization). In this case, a candidate set deﬁned

by all marginals is not succinct, as one would attribute

“Why did the Model Fail?”: Attributing Model Performance Changes to Distribution Shifts

performance changes to all marginals of descendants of

Instead, the candidate set determined by the causal graph

will isolate the correct conditional distribution.

Crucially, to compute our attributions, we need estimates

of model performance under

. Note that we only have

model performance estimates under

Dsource

and

Dtarget

, but

not for any arbitrary coalition where only a subset of the

distributions have shifted. To estimate the performance of

any coalition, we propose to use importance sampling.

3.2. Importance Sampling to Estimate Performance

under a Candidate Distribution Shift

Assumption 3.3.

support(Dtarget

Xi|parent(Xi))⊆

support(Dsource

Xi|parent(Xi))for all Dtarget

Xi|parent(Xi)∈CD.

Importance sampling allows us to re-weight samples drawn

from a given distribution, which can be

Dsource

Dtarget

, to

simulate expectations for a desired distribution, which is the

candidate e

Din our case. Thus, we re-write the value as

Val(e

C) = Perf(e

D)−Perf(Dsource)(4)

=E(x,y)∼

D[ℓ((x, y), f)] −E(x,y)∼Dsource [ℓ((x, y), f)]

=E(x,y)∼Dsource "e

D((x, y))

Dsource((x, y)) ℓ((x, y), f)#−

E(x,y)∼Dsource [ℓ((x, y), f)]

The importance weights are themselves a product of ratios

of source and target distributions corresponding to the causal

mechanisms in CDas follows:

C((x, y)) := e

D((x, y))

Dsource((x, y)) =Y

d∈

Dtarget

d((x, y))

Dsource

d((x, y))

=: Y

d∈

wd((x, y))

(5)

By Assumption 3.3, we ensure that all importance weights

are ﬁnite.

Computing Importance Weights. There are multiple

ways to estimate importance weights

wd((x, y))

, which are

a ratio of densities (Sugiyama et al.,2012). Here, we use

a simple approach for density ratio estimation via train-

ing probabilistic classiﬁers as described in Sugiyama et al.

(2012, Section 2.2).

Let

be a binary random variable, such that when

1, Z ∼ Dtarget

d(Z)

, and when

D= 0, Z ∼ Dsource

d(Z)

. Sup-

pose d=DXi|parent(Xi), then

wd=P(D= 0|parent(Xi))

P(D= 1|parent(Xi)) ·P(D= 1|Xi,parent(Xi))

P(D= 0|Xi,parent(Xi)),

where each term is computed using a probabilistic classiﬁer

trained to discriminate data points from

Dsource

and

Dtarget

from the concatenated dataset. We show the derivation of

this equation in Appendix A. In total, we need to learn

O(|CD|)models for computing all importance weights.

3.3. Properties of Our Method

Under perfect computation of importance weights, the Shap-

ley attributions resulting from the performance-change game

have the following desirable properties, which follow from

the properties of Shapley values. We provide proofs of these

properties in Appendix B.

Property 1. (Efﬁciency)

d∈CD

Attr(d) = Val(CD) =

Perf(Dtarget)−Perf(Dsource)

Property 2.1. (Null Player)

Dsource

d=Dtarget

d=⇒

Attr(d)=0.

Property 2.2. (Relevance) Consider a mechanism

Perf(e

C∪ {d}) = Perf(e

for all

C⊆CD\d

, then

Attr(d)=0.

Property 3. (Attribution Symmetry) Let

AttrD1,D2(d)

denote the attribution to some mechanism

when

D1=

Dsource

and

D2=Dtarget

. Then,

AttrD1,D2(d) =

−AttrD2,D1(d)∀d∈CD.

Thus, the method attributes the overall performance change

only to distributions that actually change in a way that af-

fects the speciﬁed performance metric. The contribution

of each distribution is computed by considering how much

they impact the performance if they are made to change in

different combinations alongside the other distributions.

3.4. Analysis using a Synthetic Setting

We derive analytical expressions for attributions in a simple

synthetic case with the following data generating process.

Source :X∼ N (µ1, σ2

Y∼θ1X+N(0, σ2

Target :X∼ N (µ2, σ2

Y∼θ2X+N(0, σ2

The model that we are investigating is

f(X) = ϕX

, and

l((x, y), f) = (y−f(x))2.

We show the attribution of our method, along with the attri-

bution using the joint method from Budhathoki et al. (2021),

in Table 1. The complete derivation, along with experi-

mental veriﬁcation of the derived expressions, can be found

in Appendix C. We highlight several advantages that our

method has over the baseline.

First, our attribution takes the model parameter

into

account in order to explain model performance changes,

文档加载中……请稍候！
如果长时间未打开，您也可以点击刷新试试。

下载文档到电脑，查找使用更方便

10 玖币 0人已下载

立即下载

摘要：

“WhydidtheModelFail?”:AttributingModelPerformanceChangestoDistributionShiftsHaoranZhang*1HarvineetSingh*2MarzyehGhassemi1ShalmaliJoshi3AbstractMachinelearningmodelsfrequentlyexperienceperformancedropsunderdistributionshifts.Theunderlyingcauseofsuchshiftsmaybemultiplesimultaneousfactorssuchaschangesi...

展开>> 收起<<

Why did the Model Fail Attributing Model Performance Changes to Distribution Shifts.pdf

共29页,预览5页

还剩页未读，继续阅读

声明：本站为文档C2C交易模式，即用户上传的文档直接被用户下载，本站只是中间服务平台，本站所有文档下载所得的收益归上传人(含作者)所有。玖贝云文库仅提供信息存储空间，仅对用户上传内容的表现方式做保护处理，对上载内容本身不做任何修改或编辑。若文档所含内容侵犯了您的版权或隐私，请立即通知玖贝云文库，我们立即给予删除！

Why did the Model Fail Attributing Model Performance Changes to Distribution Shifts

相关推荐

开通VIP享超值会员特权

作者详情

相关内容

热门标签

举报选择: