An Active Learning Reliability Method for Systems with Partially Deﬁned Performance Functions Jonathan Sadeghi_2

2025-04-30 0 0 2.29MB 12 页 10玖币

侵权投诉

An Active Learning Reliability Method for Systems

with Partially Deﬁned Performance Functions

Jonathan Sadeghi

Five AI Ltd.

jonathan.sadeghi@five.ai

Romain Mueller

Five AI Ltd. John Redford

Five AI Ltd.

Abstract

In engineering design, one often wishes to calculate the probability that the per-

formance of a system is satisfactory under uncertainty. State of the art algorithms

exist to solve this problem using active learning with Gaussian process models.

However, these algorithms cannot be applied to problems which often occur in the

autonomous vehicle domain where the performance of a system may be undeﬁned

under certain circumstances. To solve this problem, we introduce a hierarchical

model for the system performance, where undeﬁned performance is classiﬁed

before the performance is regressed. This enables active learning Gaussian process

methods to be applied to problems where the performance of the system is some-

times undeﬁned, and we demonstrate the effectiveness of our approach by testing

our methodology on synthetic numerical examples for the autonomous driving

domain. The code to generate these experiments is available as an open source

repository: https://github.com/fiveai/hGP_experiments/

1 Introduction

Estimating the probability that the performance of a system is satisfactory under uncertain or variable

operating circumstances is an important step towards deploying such systems safely in the real

world. This is especially important in safety critical application such as autonomous driving, where

ﬁnding rare but catastrophic failures has been identiﬁed as a important challenge [

]. Powerful

active learning approaches based on Gaussian processes (GP) have been proposed as a solution to

this problem [

–

] and achieve state of the art performance (we describe related work in detail in

Section 2). However, such approaches cannot be applied to problems where the performance of

the system may be undeﬁned under certain speciﬁc circumstances, a situation which often occurs

in the autonomous vehicle domain [

–

]. For example, consider the case of an autonomous

vehicle waiting to join a road at a T-junction, where other cars are travelling on the road at constant

velocity. Assuming that the autonomous vehicle behaves deterministically with respect to variables

like the initial starting positions and velocities of the vehicles, one could specify a measure of safe

system performance, for example the distance of closest approach between the autonomous vehicle

and the other vehicles as a function of these variables. However, in some cases the closest distance

function could be undeﬁned because the autonomous vehicle never joined the road at all, for example

if the autonomous vehicle’s planner deems the situation to be unsafe. Naïvely masking away regions

where the performance of the system is undeﬁned would introduce discontinuities and leads to poor

performance since Gaussian processes with stationary kernels are not well suited to the regression of

discontinuous targets [

]. In this work, we extend these methods from ﬁrst principles to the case

where the performance function can be undeﬁned by using a hierarchical model (termed hGP) for the

system performance, where undeﬁned performance is classiﬁed before the performance is regressed.

We consider a system whose performance is described by a function

g:X 7→ R∪ {NaN}

, where

x∈ X ⊆ Rk

are random environmental variables affecting the system,

g(x)<0

denotes an

Preprint. Under review.

arXiv:2210.02168v2 [cs.LG] 2 Nov 2022

undesirable event (a failure), and

g(x) = NaN

is an event of unspeciﬁed performance. An undeﬁned

value does not indicate that an undesirable event has occurred for a particular

, and therefore we

wish to classify these

differently to the failure events. The rate of failures is quantiﬁed using the

probabilistic threshold robustness (PTR) of the system, which we deﬁne as

pf=ZX

1[g(x)<0∩g(x)6=NaN]p(x)dx,(1)

where

is the indicator function and

p(x)

is the probability density (mass) function of

[

]. Eq.

(1)

represents the probability that the system is in the failure state, while disregarding the ‘uninteresting’

cases where the performance is undeﬁned. Note that we are not attempting to model the distribution

of environment variables and treat

p(x)

as given. Estimating

in Eq.

(1)

using a vanilla Monte

Carlo simulation can be computationally expensive since identifying a failure rate lower than



will

typically require at least

1/

tests [

]. In order to reduce the required number of samples, we use the

Adaptive Kriging Monte Carlo Simulation (AK-MCS) algorithm, a simple active learning technique

based on Gaussian processes which was shown to provide an extremely efﬁcient evaluation of the

PTR measure for previously studied problems [

], and extend it to partially undeﬁned performance

functions. We perform experiments comparing our proposed methodology to several naïve baselines

on problems for which the results are known analytically. We ﬁnd that our approach produces a more

accurate estimation of

and also that the surrogate model is a more accurate representation of the

true performance function.

2 Related Work

The PTR measure has recently become of interest in the robust optimisation literature [

], for

example Inatsu et al.

[4]

show how system designs can be adjusted to optimise the measure. The

measure has a much longer history in reliability engineering [

]. Moustapha et al.

[6]

and Teixeira

et al.

[15]

provide reviews of active learning methods for estimating this measure. The Adaptive

Kriging Monte Carlo methodology is perhaps the most well known of these methods, and achieves

close to state of the art results [

]. Efﬁcient methods of estimating the PTR measure also exist in

reinforcement learning [

]. Beglerovic et al.

[16]

use a Bayesian optimisation approach to identify

failure cases for an autonomous vehicle but do not exhaustively search for all

such that

f(x)<0

and also do not calculate the PTR measure,

. Similarly, Wang et al.

[17]

use a realistic LiDAR

simulator to modify real-world LiDAR data which can then be used to test end-to-end autonomous

driving systems while searching for adversarial trafﬁc scenarios with Bayesian Optimisation. A

related problem in the autonomous vehicle space is ﬁnding the most likely

, i.e. with largest

p(x)

leading to

f(x)<0

[

]. This is closely related to ﬁrst order methods for estimating the PTR

measure [14].

Although there exists literature related to Gaussian Process modelling for discontinuous targets [

there is little literature on active learning speciﬁcally for the PTR measure for discontinuous targets.

3 Approach

We modify the AK-MCS active learning algorithm by using a different Gaussian Process and

acquisition function to Echard et al.

[2]

. We use a hierarchical Gaussian process model for the rule

function, consisting of separate regression and classiﬁcation Gaussian processes, and a modiﬁed

acquisition function which minimises the catastrophic event classiﬁcation error to yield an optimal

surrogate model of the rule function. Otherwise our proposed algorithm proceeds in the same way

as the AK-MCS algorithm, i.e. an initial training set is chosen to train a Gaussian process, and

then subsequent evaluations of the performance function,

, are chosen iteratively by maximising a

function of the Gaussian process known as the acquisition function, which are then used to retrain

the Gaussian process. The algorithm terminates when the coefﬁcient of variation (CoV) of the

failure probability computed using the Gaussian process is below some threshold, and the predicted

misclassiﬁcation probability is also below some threshold. This algorithm is shown in Algorithm 1.

Let

y∗

be the predicted performance for the test input

x∗∈ X

where

y∈R∪ {NaN}

, and let

the dataset of training examples

D={(xi, yi)|i= 1, ..., n}

. We model the predictive distribution

p(y∗|x∗,D)hierarchically as

p(y∗|x∗,D) = p(y∗|x∗,D, y∗6=NaN)p(y∗6=NaN|x∗,D)if y∗6=NaN,

p(y∗=NaN|x∗,D)if y∗=NaN,(2)

where

p(y∗|x∗,D, y∗6=NaN)

is the predicted regression distribution for

y∗

at the test input

x∗

given that

y∗

is deﬁned, and

p(y∗=NaN|x∗,D)

is the predicted classiﬁcation probability that

y∗

is undeﬁned for

x∗

. We model these distributions with separate GPs; for

p(y∗|x∗,D, y∗6=NaN)

GP regression is used, and for

p(y∗=NaN|x∗,D)

GP classiﬁcation is used. The conditional

prediction of the failure event can easily be calculated as

p(y∗<0, y∗6=NaN|x∗,D)

, which can be

used to deﬁne an acquisition function,

pmisclassiﬁcation(x)

, based on probability of misclassiﬁcation of

y∗<0∩y∗6=NaN

, as in Echard et al.

[2]

. We give more details about our modelling approach in

Appendix B. Finally, our hierarchical model Eq.

(2)

can be used to compute the failure probability as

pf≈ZX

1[p(y∗<0, y∗6=NaN|x∗,D)>0.5]p(x∗)dx∗.(3)

The termination criteria for the AK-MCS algorithm will determine the error in the failure probability

computed using the Gaussian process in Eq.

(3)

, in addition to bounding the error of the hierarchical

Gaussian process model. This ensures that the model is sufﬁciently accurate to be used by engineers

to make predictions about the behaviour of the system.

Note that in this paper we only consider systems with a single performance function, however Yun

et al.

[3]

demonstrate how acquisition functions for multiple performance functions can be combined

when one is interested in a combined PTR measure for the performance functions. This can be applied

to our hierarchical model directly. Finally, we note that our modiﬁcations to the AK-MCS algorithm

are fairly general and only involve changing the performance function model and acquisition function,

and therefore these changes could possibly also be used with different active learning algorithms. We

do not explore these possible applications in this paper and instead leave this as a topic for future

research.

Algorithm 1: Hierarchical Gaussian Process PTR Active Learning Method

Input: GP prior GP(0, k), termination threshold η, model g(x)

Deﬁne proposal set S: sample nmc points from p(x∗).

Deﬁne initial design of experiments: sample

points uniformly from

and evaluate with model

g(x)to deﬁne ˆ

S={(xi, yi)|i= 1, ..., nE}

while CoV > 0.1do

while maxxpmisclassiﬁcation(x)> η do

Train GP on ˆ

Compute µ(x), σ(x), pnan(x)from hierarchical GP for all x∈ S.

Choose most likely misclassiﬁed x:x∗= arg maxx∈S pmisclassiﬁcation(x)

Observe y∗=g(x∗)and Add (x∗, y∗)to ˆ

end while

Estimate pfusing Monte Carlo simulation with Gaussian Process on Susing Eq. (3)

Calculate CoV =q(1−pf)

pf|S|

Sample nmc points from p(x∗)and evaluate with model g(x)to add to S

end while

Output: Fitted hierarchical GP and pfcomputed using Eq. (3).

4 Experiments

Benchmark tasks:

We evaluate our methodology on two benchmark problems where the system

performance is partially undeﬁned and for which pfcan be calculated analytically:

•Toy function The system performance (plotted in Fig. 2a) is given by

g(x) = NaN if 0.215 <x<0.6,

cos(8x)otherwise,(4)

where the uncertain variable xis distributed with p(x) = U[0,1].

文档加载中……请稍候！
如果长时间未打开，您也可以点击刷新试试。

下载文档到电脑，查找使用更方便

10 玖币 0人已下载

立即下载

摘要：

AnActiveLearningReliabilityMethodforSystemswithPartiallyDenedPerformanceFunctionsJonathanSadeghiFiveAILtd.jonathan.sadeghi@five.aiRomainMuellerFiveAILtd.JohnRedfordFiveAILtd.AbstractInengineeringdesign,oneoftenwishestocalculatetheprobabilitythattheper-formanceofasystemissatisfactoryunderuncertainty...

展开>> 收起<<

An Active Learning Reliability Method for Systems with Partially Deﬁned Performance Functions Jonathan Sadeghi_2.pdf

共12页,预览3页

还剩页未读，继续阅读

声明：本站为文档C2C交易模式，即用户上传的文档直接被用户下载，本站只是中间服务平台，本站所有文档下载所得的收益归上传人(含作者)所有。玖贝云文库仅提供信息存储空间，仅对用户上传内容的表现方式做保护处理，对上载内容本身不做任何修改或编辑。若文档所含内容侵犯了您的版权或隐私，请立即通知玖贝云文库，我们立即给予删除！

An Active Learning Reliability Method for Systems with Partially Deﬁned Performance Functions Jonathan Sadeghi_2

相关推荐

开通VIP享超值会员特权

作者详情

相关内容

热门标签

举报选择: