Finding Islands of Predictability in Action Forecasting Dan Scarafoni

2025-05-06 0 0 1.08MB 13 页 10玖币

侵权投诉

Finding Islands of Predictability in Action

Forecasting

Dan Scarafoni

danscarafoni@gatech.edu

Irfan Essa

irfan@gatech.edu

Thomas Pl¨otz

thomas.ploetz@gatech.edu

October 17, 2022

Abstract

We address dense action forecasting: the problem of predicting future

action sequence over long durations based on partial observation. Our

key insight is that future action sequences are more accurately modeled

with variable, rather than one, levels of abstraction, and that the optimal

level of abstraction can be dynamically selected during the prediction pro-

cess. Our experiments show that most parts of future action sequences

can be predicted conﬁdently in ﬁne detail only in small segments of future

frames, which are eﬀectively “islands” of high model prediction conﬁdence

in a “sea” of uncertainty. We propose a combination Bayesian neural

network and hierarchical convolutional segmentation model to both accu-

rately predict future actions and optimally select abstraction levels. We

evaluate this approach on standard datasets against existing state-of-the-

art systems and demonstrate that our “islands of predictability” approach

maintains ﬁne-grained action predictions while also making accurate ab-

stract predictions where systems were previously unable to do so, and

thus results in substantial, monotonic increases in accuracy.

1 Introduction

Dense action forecasting aims to understand and anticipate future action se-

quences before they occur given an initial observation period. This is vital

for a broad spectrum of problems. Robots collaborating with humans, for ex-

ample, must be able to anticipate actions in order to safely work with their

partners [1, 2]. Long term action sequences, in particular, are often very un-

predictable, and predictors must adapt to the fundamental uncertainties of the

video [3, 4, 5, 6]. One solution is to use coarser-grained labels to “hedge a bet”

and predict an action abstraction when a concrete, ﬁne-grained prediction is

impossible. [7].

We propose a system to ﬁnd and replace inaccurate, ﬁne-grained future

action sequence predictions with accurate coarse-grained ones. This system ex-

arXiv:2210.07354v1 [cs.CV] 13 Oct 2022

(a) The activity diagram from which ac-

tion sequences were sampled for 50 Salads

demonstrates that many action sequences

are, to a degree, random by design [8]

(b) We can use uncertainty estimation (1)

to ﬁnd the predictable areas (2) and pre-

dict correct coarse-grained actions else-

where to improve forecasting accuracy (3).

tends uncertainty estimation to dense action forecasting and leverages temporal

convolutional networks to select the correct level of abstraction at each time

step to maintain high prediction accuracy. Our experiments show that most

parts of future action sequences can be predicted conﬁdently in ﬁne detail only

in small segments of future frames, which are eﬀective “islands” of high model

prediction conﬁdence in a “sea” of uncertainty. This we contrast with existing

methods which try to make ﬁne-grained predictions at every point in the fu-

ture [9, 3]. In the standard 50 Salads dataset, for example, ingredients can be

chopped in many diﬀerent orders; however, dressing is almost always poured on

the salad at the end. Thus, predicting what ingredients will be chopped next is

often impossible (see Fig. 1a), but it is almost certain that the dressing will be

poured onto the salad near the end of the video.

Our system gives “partial credit” for guesses where correct coarse actions

are substituted for ﬁne. Since the relative value of coarse or ﬁne predictions

varies depending on the speciﬁc use case, and so, there is a need to bias the

system towards coarse or ﬁne-grained guesses depending on which preference

will ultimately maximize performance. To this end, we introduce Granularity

Loss, which further tunes predictions to the data and evaluation scheme. We

perform extensive testing across all possible evaluation schemes with standard

metrics on benchmark datasets and demonstrate that our method substantially

improves performance on state-of-the-art systems. Fig. 1b demonstrates that

substantial gains in accuracy can be made using our system.

Our primary contributions are (1) a novel system that leverages uncertainty es-

timation and hierarchical temporal segmentation to ﬁnd the predictable sections

(“islands”) of future action sequences and substitute accurate, coarse-grained

predictions where none such exist. (2) a novel loss function, Granularity Loss,

for tuning the system to speciﬁc evaluation schemes. (3) The introduction of

uncertainty estimation to the dense action forecasting problem domain. (4)

Extensive experimentation on standard datasets demonstrates that our method

substantially improves dense action forecasting for state-of-the-art systems.

2 Related work

Future action prediction has seen explosive growth in recent years. Our work

deals with dense action forecasting: the prediction of long-term (on the order

of several minutes), action sequences and their durations [10, 3, 11, 12, 13].

Despite signiﬁcant gains, dense action forecasting remains very diﬃcult,

rarely exceeding 30% accuracy for the longest term predictions on standard

data sets [4, 14, 11]. Some datasets are known to be highly unpredictable and

were designed as such (see Fig. 1a) [8, 6]. Recent work has shown that often

coarser-grained actions can be reliably predicted even when ﬁne-grained actions

cannot [7]. This motivates accurate prediction of coarser-grained activity la-

bels outside of the “islands.” In this work, we posit a method to estimate such

predictable “islands” and make multiple-granularity predictions to accurately

estimate the future.

2.1 Uncertainty estimation in video action prediction

Predicting the correctness of forecast actions requires uncertainty estimation.

More generally, uncertainty estimation is an important area of research in com-

puter vision. Equipping a model with conﬁdence about its prediction allows

a system to understand and cope with sensor and model error. Classiﬁcation

logits do not necessarily reﬂect model conﬁdence, and thus more sophisticated

methods are necessary [15]. Generalized methods for uncertainty estimation in

computer vision have made great strides in recent years, and there have been

many attempts to model dense action forecasting systems with probability dis-

tributions [3, 11, 16, 17]; However, no such techniques exist for dense action

forecasting [15, 18]. There is a need to develop uncertainty estimation for this

problem, as prediction conﬁdence is essential to “hedging bets” as to how precise

a prediction should be [7].

3 Finding islands of predictability

3.1 Problem statement

Given per-frame action predictions for multiple action granularities, our system

selects the ones that both maximize accuracy and granularity. Formally, given

a video of Tframes with t0observed frames, g∈Glevels of granularity, and a

prediction for each granularity level in data point nat frame tˆat,n

g, the goal is

to design a Granularity Selector system Hwith parameters θwhich maximizes

θ∗=arg maxθ

n=1

t=tn

0+1

A(H(ˆat,n

1. . . ˆat,n

G;θ), at,n

g∗, β) (1)

Ais a weighted accuracy metric [7]. It is parameterized by β, which determines

the accuracy weight or “partial credit” given to correct coarser levels of granu-

larity if the ﬁne-grained actions are not predicted. His the Granularity Selector

which selects a ﬁnal granularity for each frame from options at diﬀerent levels

H: ˆa1. . . ˆaG→ˆag∗(2)

文档加载中……请稍候！
如果长时间未打开，您也可以点击刷新试试。

下载文档到电脑，查找使用更方便

10 玖币 0人已下载

立即下载

摘要：

FindingIslandsofPredictabilityinActionForecastingDanScarafonidanscarafoni@gatech.eduIrfanEssairfan@gatech.eduThomasPlotzthomas.ploetz@gatech.eduOctober17,2022AbstractWeaddressdenseactionforecasting:theproblemofpredictingfutureactionsequenceoverlongdurationsbasedonpartialobservation.Ourkeyinsightist...

展开>> 收起<<

Finding Islands of Predictability in Action Forecasting Dan Scarafoni.pdf

共13页,预览3页

还剩页未读，继续阅读

声明：本站为文档C2C交易模式，即用户上传的文档直接被用户下载，本站只是中间服务平台，本站所有文档下载所得的收益归上传人(含作者)所有。玖贝云文库仅提供信息存储空间，仅对用户上传内容的表现方式做保护处理，对上载内容本身不做任何修改或编辑。若文档所含内容侵犯了您的版权或隐私，请立即通知玖贝云文库，我们立即给予删除！

Finding Islands of Predictability in Action Forecasting Dan Scarafoni

相关推荐

开通VIP享超值会员特权

作者详情

相关内容

热门标签

举报选择: