Oflib Facilitating Operations with and on Optical Flow Fields in Python Claudio S. Ravasio120000000264535376 Lyndon Da

2025-04-24 0 0 1019.81KB 15 页 10玖币

侵权投诉

Oﬂib: Facilitating Operations with and on

Optical Flow Fields in Python

Claudio S. Ravasio1,2[0000−0002−6453−5376], Lyndon Da

Cruz3[0000−0002−7695−6354], and Christos Bergeles2[0000−0002−9152−3194]

1University College London (UCL), United Kingdom

2King’s College London (KCL), United Kingdom

3Moorﬁelds Eye Hospital, London, United Kingdom

Abstract. We present a robust theoretical framework for the charac-

terisation and manipulation of optical ﬂow, i.e 2D vector ﬁelds, in the

context of their use in motion estimation algorithms and beyond. The

deﬁnition of two frames of reference guides the mathematical deriva-

tion of ﬂow ﬁeld application, inversion, evaluation, and composition op-

erations. This structured approach is then used as the foundation for

an implementation in Python 3, with the fully diﬀerentiable PyTorch

version oflibpytorch supporting back-propagation as required for deep

learning. We verify the ﬂow composition method empirically and pro-

vide a working example for its application to optical ﬂow ground truth

in synthetic training data creation. All code is publicly available.

Keywords: Optical ﬂow; Flow ﬁeld; Flow vector; Flow composition;

Python; PyTorch; NumPy

1 Introduction

Optical ﬂow as an expression of motion encoding and feature correspondence is

one of the oldest tasks in computer vision, with seminal works such as by Lucas,

Kanade et al [8] dating back to the early 80s. After decades of advances us-

ing variational methods, the extremely successful convolutional neural network

based FlowNet method in 2015 [3] heralded the arrival of well-performing and

eﬃcient end-to-end deep learning methods, usually implemented in Python. This

has quickly become the dominant approach, with performance continuously im-

proving and ever more complex benchmarks being proposed, such as MPI-Sintel,

KITTI, or FlyingThings3D [13,10,9].

In this context, handling optical ﬂow easily and eﬃciently is of increasing

importance. Many algorithms or their training protocols involve operations with

or on optical ﬂow ﬁelds, such as the creation of complex synthetic data [12] or

working with “cues” calculated from bidirectional ﬂow as proposed by Hoﬁn-

ger et al [6]. Implementing this from scratch can be laborious and error-prone.

While there are a great number of publicly available algorithm implementa-

tions as well as methods in Python libraries for the estimation of optical ﬂow

ﬁelds [2,4,7], no such wealth of resources exists for their further manipulation.

arXiv:2210.05635v2 [cs.CV] 14 Oct 2022

2 C. S. Ravasio et al.

An extensive search brought only little Python code to light, all being either

algorithm-dependent or severely limited in their scope. Flow visualisation is an

important topic, and the toolboxes flowvid [14] as well as flow-vis [15] have

some interesting capabilities in this regard. In addition to that, there are pack-

ages such as flowpy [17] which also allow for basic ﬂow warping, and add some

utilities with a narrow focus on speciﬁc tasks such as reading and writing ﬂows.

The aim of oflib on the other hand is to oﬀer a structured approach to

the concept of ﬂow ﬁelds, guided by a framework derived from ﬁrst principles,

and to provide all methods necessary to perform operations within a reasonable

scope. This involves taking into account the two possible frames of reference for

the ﬂow vectors, as well as tracking undeﬁned areas in outputs. The rigorous

method ensures the mathematically correct implementation of a wide range of

ﬂow operations, including more complex functions – such as ﬂow composition –

not found in any of the previously listed python packages. Full interoperability

with any Python code using NumPy [5] or PyTorch [11] lends oflib a high

potential for reuse by the larger research community. The option to perform

operations batched and on a GPU in particular yields signiﬁcant speedups, while

diﬀerentiability in the context of the Pytorch autograd module allows for the

use in any deep learning algorithm relying on back-propagation for optimisation.

2 Theory

The theoretical framework underpinning oflib is derived from ﬁrst principles to

ensure a coherent and rigorous approach. This section will ﬁrst address the two

possible reference frames for optical ﬂow ﬁelds and then present the theoretical

basis for the main functionality provided by oflib, focusing on the concrete

operations needed to eventually translate the mathematical deﬁnition into code.

2.1 Optical Flow Deﬁnition

An optical ﬂow ﬁeld is deﬁned as a spatial mapping of coordinates at time t1to

coordinates at time t2:



2:= X17→ X2;F1



2=X2−X1(1)

where Xtcorresponds to the set of continuous feature coordinates xbeing

mapped at time t, and F1



2is the resulting array of ﬂow vectors between the

feature sets at times t1and t2. In the context of image sequences, this is equiv-

alent to creating a mapping between the image feature coordinates iXtin the

frame at time t1to coordinates in the frame at time t2. We distinguish between

two possible frames of reference, “source” and “target”4, illustrated in Figure 1:

4The terms “forward” and “backward” ﬂow often used in literature are avoided here,

as it can lead to confusion e.g. in the context of reverse “backward” ﬂows.

Oﬂib: Facilitating Operations with and on Optical Flow Fields in Python 3

– Source, “s”: In this case, coordinates on a discretised regular grid at time

t1, termed the “source domain”, are mapped to coordinates in continuous

space at time t2. Applied to images, this indicates each pixel in the ﬁrst

image is matched with some position in the second image - but not every

pixel at time t2has a known source correspondence at time t1.

– Target, “t”: The second option means that for each coordinate on a discre-

tised regular grid at time t2, or the “target domain”, there is a mapping to a

coordinate in continuous space at time t1. Each pixel in the second image is

matched with some position in the ﬁrst image - but not every pixel at time

t1has a known target correspondence at time t2.

Equation (1) can therefore be extended as follows:

“Source” reference: F1



2:= G17→ X2;F1



2=X2−G1

“Target” reference: F1



2:= X17→ G2;F1



2=G2−X1

(2)

where the underlined number in F1



2indicates whether the source or the target

of the mapping is on a discretised regular grid Gin 2D space, spanning the pixel

range from 0 to H−1 vertically and 0 to W−1 horizontally, where Hand W

are the ﬂow ﬁeld height and width, respectively.

Note that while F1



2̸=−F2



1, as the inverse mapping of G17→ X2is not

G27→ X1, the following relationships do hold true:



2=F2



1



inv ⇒G17→ X2= (X27→ G1)inv ⇒F1



2=−F2



2=F2



1



inv ⇒X17→ G2= (G27→ X1)inv ⇒F1



2=−F2



(3)



Fig. 1: Two frames of reference, points at time t1in red, at t2in blue. Left:

“source” means all pixels at time t1, i.e. coordinates on a discrete grid G, are

mapped to a new location at time t2.Right: “target” means all pixels on this

grid Gat time t2are matched with a diﬀerent previous location at time t1.

2.2 Flow Application

Given data on the spatial grid Gat time t1such as an image iG1, an optical

ﬂow ﬁeld F1



2on the same grid Gcan be applied to it to calculate the warped

文档加载中……请稍候！
如果长时间未打开，您也可以点击刷新试试。

下载文档到电脑，查找使用更方便

10 玖币 0人已下载

立即下载

摘要：

Oflib:FacilitatingOperationswithandonOpticalFlowFieldsinPythonClaudioS.Ravasio1,2[0000−0002−6453−5376],LyndonDaCruz3[0000−0002−7695−6354],andChristosBergeles2[0000−0002−9152−3194]1UniversityCollegeLondon(UCL),UnitedKingdom2King’sCollegeLondon(KCL),UnitedKingdom3MoorfieldsEyeHospital,London,UnitedKin...

展开>> 收起<<

Oflib Facilitating Operations with and on Optical Flow Fields in Python Claudio S. Ravasio120000000264535376 Lyndon Da.pdf

共15页,预览3页

还剩页未读，继续阅读

声明：本站为文档C2C交易模式，即用户上传的文档直接被用户下载，本站只是中间服务平台，本站所有文档下载所得的收益归上传人(含作者)所有。玖贝云文库仅提供信息存储空间，仅对用户上传内容的表现方式做保护处理，对上载内容本身不做任何修改或编辑。若文档所含内容侵犯了您的版权或隐私，请立即通知玖贝云文库，我们立即给予删除！

Oflib Facilitating Operations with and on Optical Flow Fields in Python Claudio S. Ravasio120000000264535376 Lyndon Da

相关推荐

开通VIP享超值会员特权

作者详情

相关内容

热门标签

举报选择: