1 Planning Coordinated Human-Robot Motions with Neural Network Full-Body Prediction Models

2025-04-30 0 0 3.33MB 12 页 10玖币

侵权投诉

Planning Coordinated Human-Robot Motions

with Neural Network Full-Body Prediction Models

Philipp Kratzer, Marc Toussaint, and Jim Mainprice

Abstract—Numerical optimization has become a popular ap-

proach to plan smooth motion trajectories for robots. However,

when sharing space with humans, balancing properly safety,

comfort and efﬁciency still remains challenging. This is notably

the case because humans adapt their behavior to that of the robot,

raising the need for intricate planning and prediction. In this

paper, we propose a novel optimization-based motion planning

algorithm, which generates robot motions, while simultaneously

maximizing the human trajectory likelihood under a data-driven

predictive model. Considering planning and prediction together

allows us to formulate objective and constraint functions in

the joint human-robot state space. Key to the approach are

added latent space modiﬁers to a differentiable human predictive

model based on a dedicated recurrent neural network. These

modiﬁers allow to change the human prediction within motion

optimization.

We empirically evaluate our method using the publicly avail-

able MoGaze [1] dataset. Our results indicate that the proposed

framework outperforms current baselines for planning handover

trajectories and avoiding collisions between a robot and a human.

Our experiments demonstrate collaborative motion trajectories,

where both, the human prediction and the robot plan, adapt to

each other.

I. INTRODUCTION

WHILE robots have been working alongside humans in

factories since the 1960’s, to this day, robots are still

fenced in cages and relegated to repetitive tasks. One of the

main challenges for robots to operate freely in the environment

is the difﬁculty to deﬁne safety and comfort objectives. An

example of a good human-robot space-sharing strategy is one

that does not intervene in the human plan while maintaining

the ability for the robot to achieve its goal. In this context the

capacity to reason on some kind of predictive human motion

model is essential.

Human motion is the result of complex biomechanical

processes that are challenging to model. As a consequence,

state-of-the-art work on full-body motion prediction resort

to data-driven models, such as recurrent neural networks

(RNN) [2]–[4]. A drawback of these architectures is that they

purely forecast human motion. Adapting the human prediction

in order to avoid collision with the environment or plan

coordinated motions, such as a handovers, is not possible.

In our prior work [5], [6], we proposed to optimize online

data-driven models by minimizing the deviation from the

Philipp Kratzer and Jim Mainprice are with the Machine Learning

and Robotics Lab, University of Stuttgart, Germany and the

Humans to Robots Motions Research Group, University of Stuttgart,

Germany philipp.kratzer@ipvs.uni-stuttgart.de;

jim.mainprice@ipvs.uni-stuttgart.de

Marc Toussaint is with the Learning and Intelligent Systems lab, TU Berlin,

Germany toussaint@tu-berlin.de

Fig. 1. Co-optimized Human-Robot Motion trajectories. Top Left: human and

robot avoiding each other. Top Right: human and robot performing a handover.

Bottom left: robot backing up to avoid human. Bottom Right: Human picking

up a plate and handing it over to the robot.

model prediction, while accounting for environmental con-

straints using penalty terms. In [6] we have also shown a

preliminary experiment to jointly plan a robot trajectory and

predict human motion.

In this work, we build on these results to propose a

framework for a human-aware motion planner that uses pre-

dictive human short-term dynamics, learned by a RNN. In

order to formulate Human-Robot Collaboration (HRC) motion

planning problems, such as handovers or collision avoidance

(see Figure 1), as trajectory optimization, we introduce two

modiﬁcations to the predictive model.

First, we adapt the architecture of the RNN by adding

latent space modiﬁers to the decoder inputs, leading to a

controllable dynamics function for the human. Second, we

add differentiable Human-Robot interaction constraints to the

output of the RNN, which take both: the robot state and the

human state as input.

These modiﬁcations to the motion prediction system allow

to formulate the problem as trajectory optimization, with

decision variables for both, the robot and the human over the

entire planning horizon. Our joint objective and constraints are

entirely deﬁned as computational graphs including the robot

model, the learned human dynamics and kinematic model,

from which we can compute gradients efﬁciently through

automatic differentiation.

We make use of a quasi-Newton (i.e., Hessian empirical

estimate) primal-dual interior-point method [7] to solve the

corresponding nonlinear program (NLP). The result of the

arXiv:2210.13317v1 [cs.RO] 24 Oct 2022

optimizer is a likely trajectory of human motion and a planned

trajectory for the robot motion.

The main contributions of our work are:

•Formulation of space sharing human-robot motion plan-

ning problems formulated as trajectory optimization with

shooting in the joint human and robot state-space

•Introduction of latent space modiﬁers that can be used to

change the human prediction.

•Efﬁcient gradient objective and constraints computational

models by using monolithic computational graphs.

The rest of the paper is organized as follows: In Section II,

we discuss relevant prior work. In Section III, we introduce our

framework theoretically and explain implementation details.

In Section IV we evaluate our prediction framework on real

motion data. We further discuss the framework in Section V.

Conclusions are drawn in Section VI.

II. RELATED WORK

A. Human-Aware Motion Planning

The rapidly growing research ﬁeld of HRC is focusing on

robotic systems that are able to perform joint actions with

humans, in order to fulﬁll a common task. A main challenge

in close proximity interaction is blanching safety and comfort,

with time and energy efﬁcient execution [8], [9]. Pro-activity

has also been investigated in many scenarios [10], [11].

In order to achieve this, the human partner needs to be

taken into account explicitly when planning the robot’s motion,

leading to human-aware motion planning systems. Human-

aware motion planning has been shown to improve human-

robot team ﬂuency and human worker satisfaction [12]. One

way to introduce human-awareness is to incorporate a cost

function to evaluate the safety of a robot path [13], or

predict which part of the workspace will be occupied by

the human and avoid this area [14], [15]. In order to ensure

human comfort, reasoning explicitly on human’s kinematics,

ﬁeld of view, posture, and preferences is possible [16]. For

robot navigation, Proxemics, considering public, personal and

private spaces, are important to ensure human comfort [17].

In contrast to prior work in human-aware motion planning,

we co-optimize robot and human motion, using a predictive

human behavior model. We incorporate interaction paradigms,

such as Proxemics, as constraints in trajectory optimization.

B. Human Behavior Prediction in Robotics

In close proximity interactions with humans, the ability to

anticipate the actions of the human partner is key. Hence,

intent prediction, which often consists of predicting a discrete

action or a goal position, has been investigated in [18], [19].

Object affordances can be used to improve the prediction of

human intent [20], [21]. It is often required to know the full

trajectory of the human. For example, it might be important

to know which part of the workspace the human will occupy.

This is often done in a second step in the aforementioned

work, for example, by using social forces [19]. Our method

can be combined with intent prediction similarly, as we have

demonstrated in our prior work [22], [23].

RNN

RNN RNN

x0H

4x0H

5x0H

h1h2h3h4h5

Encoder input

Decoder output

Fig. 2. Human Motion Prediction with a RNN. The observed trajectory (blue)

is fed into RNN cells (red) and future states can be predicted (green).

Many works on predictive behavior models focus on directly

forecasting human motion. While 2d human motion prediction

is especially important for robot navigation [24]–[26], we are

not only interested in modeling navigation, but also in pick and

place tasks or handovers and thus require a full-body motion

prediction model.

C. Human Full-Body Prediction Models

Early methods for full-body or arm prediction, for example,

use inverse optimal control [27], [28]. However, the availabil-

ity of larger human motion data-sets and recent advances in

neural networks make deep learning techniques state-of-the-

art. Due to the sequential structure of motion data, RNNs

are suitable for full-body motion prediction [29]. A typical

encoder-decoder structure can be seen in Figure 2. The archi-

tecture can be further improved by adding residual connection

in the loop function [2]. It has also been shown that the rotation

representation and loss is important, for instance, using a

quaternion representation improved over prior work [3], [30].

Including a velocity connection can make predictions more

stable for longer time horizons [4]. Recently, motion prediction

using graph neural networks [31] or transformers [32] has been

shown to slightly improve the prediction performances.

Motion prediction based on neural networks promises good

results for predicting short-term motion. However, the models

have the issues that 1) they purely forecast human motion and

do not incorporate workspace geometry 2) they are not con-

trollable and thus can not be changed during motion planning.

In our work we tackle these issues by adding modiﬁers to

the network architecture, which allows for optimization-based

motion prediction.

D. Motion Optimization

Gradient-based optimization algorithms are widely used

in the ﬁeld of robotics and optimal control for optimizing

trajectories [33]–[39]. These techniques have been shown to

successfully generate motions with a variety of kinematic

and dynamic objectives and constraints, such as obstacle

constraints [34].

Motion optimization has also been used to synthesize human

behavior for animating characters and is able to generate

realistic motions [40], [41]. In contrast, our work focuses on

Oﬄine Online Final Result

Fig. 3. Overview of the framework. Ofﬂine a human model is trained on motion data and a robot model is constructed. Online the states and controls of

human and robot are jointly optimized, accounting for task, environment and, coordination constraints. The ﬁnal result is a coordinated motion plan.

forecasting human motion using a data-driven model and uses

motion optimization for planning robot motion while adapting

the human prediction to the robot plan.

Since we ﬁrst proposed this idea in [6], similar ideas have

been proposed in the meantime.

For example, Fishman et al. propose a method to simulta-

neously plan for the robot arm movements while predicting

human actions a ﬂoating sphere model [42]. Schaefer et al.

use gradient information of a neural network for planning 2D

trajectories within a crowd of pedestrians [43].

In contrast to those works, our approach makes use of a full-

body motion prediction model for the human, which makes it

possible to plan motions using a single framework for tasks

such as handovers, or tasks where navigation and pick and

place actions are combined.

III. JOINT HUMAN-ROBOT TRAJECTORY OPTIMIZATION

AND MOTION PREDICTION

A. Problem Statement

1) Agents: We consider scenarios where multiple agents a

are involved, who have to solve speciﬁc tasks in the environ-

ment. To simplify, we consider and evaluate our framework

with one human agent Hand one robot agent R.

We discretize time with a small constant time change of ∆t

between consecutive timesteps. At a timestep t, each agent a∈

{H, R}has a state xa

t∈Rdawith dimensionality da. When

applying controls ua

t∈Rnaa discrete dynamics function fa

can be used to compute the state at the next timestep:

t+1 =fa(xa

t,ua

t)(1)

2) Formulation as NLP: We formulate our HRC problem,

which aims to plan a robot trajectory xR

k:Twhile predicting a

likely human trajectory xH

k:T, as the following NLP:

min

k:T,uH

k:T

k:T,uR

k:T

αHcH+αRcR−log p(xH

k:T|xH

0:k,D)(2)

subject to:

t+1 =fa(xa

t,ua

t)∀t∈(k, T )(3)

ha(xa)=0, ga(xa)≤0(4)

h(xH,xR)=0, g(xH,xR)≤0(5)

with cH(xH

k:T,uH

k:T)and cR

t(xR

k:T,uR

k:T)being cost functions

associated with the trajectories of human and robot, αH

and αRare hyperparameters used to weight the inﬂuence

of the respective agents, and the likelihood of the human

motion given dataset Dand observed motion xH

0:kis given

by p(xH

k:T|xH

0:k,D).

Equation (3) shows constraints ensuring the dynamics func-

tions for human and robot. Additional equality and inequality

constraints for human or robot (Equation 4) can be used

to ensure environment-dependent and task constraints. Joint

constraints between human and robot (Equation 5) are useful

to ensure collaborative interaction paradigms.

3) Trajectory optimization with shooting: One possibility

to solve the trajectory optimization problem is a collocation

approach, which means having both, the controls and the

states as part of the decision variables and specifying the

dynamics as explicit constraints. However, since the recurrent

neural network introduces additional hidden states h0:Tper

timestep, those would also need to be speciﬁed as equality

constraints, leading to an enormous number of constraints and

high memory requirements.

Nevertheless, when future controls ua

t:Tare available, the

future states xa

t+1:Tcan be computed by forward simulation

using the unrolled dynamics ˜

fa, which applies farecursively:

t+1:T=˜

fa(xa

t,ua

t:T)

=fa(xa

t,ua

t), fa(fa(xa

t,ua

t),ua

t+1), . . .(6)

This is the concept underlying the shooting approach to tra-

jectory optimization. As a consequence, dynamic constraints

are directly fulﬁlled and only controls, ua

k:Tare part of the

decision variables. A disadvantage of this approach is that the

controls at a speciﬁc timestep ua

thave an inﬂuence on all states

at later timesteps xa

t+1:T. As a consequence, it is required

to propagate the gradient through the unrolled dynamics for

optimization.

B. Solving the NLP

Using the shooting method and the assumption that the hu-

man likelihood is integrated into the human dynamics function

(see Section III-D2), the optimization problem simpliﬁes to:

min

k:T,uR

k:T

αHcH+αRcR(7)

subject to:

ha(xa)=0, ga(xa)≤0(8)

h(xH,xR)=0, g(xH,xR)≤0(9)

文档加载中……请稍候！
如果长时间未打开，您也可以点击刷新试试。

下载文档到电脑，查找使用更方便

10 玖币 0人已下载

立即下载

摘要：

1PlanningCoordinatedHuman-RobotMotionswithNeuralNetworkFull-BodyPredictionModelsPhilippKratzer,MarcToussaint,andJimMainpriceAbstractNumericaloptimizationhasbecomeapopularap-proachtoplansmoothmotiontrajectoriesforrobots.However,whensharingspacewithhumans,balancingproperlysafety,comfortandefciencyst...

展开>> 收起<<

1 Planning Coordinated Human-Robot Motions with Neural Network Full-Body Prediction Models.pdf

共12页,预览3页

还剩页未读，继续阅读

声明：本站为文档C2C交易模式，即用户上传的文档直接被用户下载，本站只是中间服务平台，本站所有文档下载所得的收益归上传人(含作者)所有。玖贝云文库仅提供信息存储空间，仅对用户上传内容的表现方式做保护处理，对上载内容本身不做任何修改或编辑。若文档所含内容侵犯了您的版权或隐私，请立即通知玖贝云文库，我们立即给予删除！

1 Planning Coordinated Human-Robot Motions with Neural Network Full-Body Prediction Models

相关推荐

开通VIP享超值会员特权

作者详情

相关内容

热门标签

举报选择: