1 Creating a Dynamic Quadrupedal Robotic Goalkeeper with Reinforcement Learning

2025-04-28 0 0 4.96MB 8 页 10玖币

侵权投诉

Creating a Dynamic Quadrupedal Robotic

Goalkeeper with Reinforcement Learning

Xiaoyu Huang2∗, Zhongyu Li1∗, Yanzhen Xiang1, Yiming Ni1, Yufeng Chi1, Yunhao Li1, Lizhi Yang1,

Xue Bin Peng3, and Koushil Sreenath1

Abstract—We present a reinforcement learning (RL) frame-

work that enables quadrupedal robots to perform soccer goal-

keeping tasks in the real world. Soccer goalkeeping using

quadrupeds is a challenging problem, that combines highly

dynamic locomotion with precise and fast non-prehensile object

(ball) manipulation. The robot needs to react to and intercept a

potentially ﬂying ball using dynamic locomotion maneuvers in a

very short amount of time, usually less than one second. In this

paper, we propose to address this problem using a hierarchical

model-free RL framework. The ﬁrst component of the framework

contains multiple control policies for distinct locomotion skills,

which can be used to cover different regions of the goal. Each

control policy enables the robot to track random parametric end-

effector trajectories while performing one speciﬁc locomotion

skill, such as jump, dive, and sidestep. These skills are then

utilized by the second part of the framework which is a high-level

planner to determine a desired skill and end-effector trajectory in

order to intercept a ball ﬂying to different regions of the goal. We

deploy the proposed framework on a Mini Cheetah quadrupedal

robot and demonstrate the effectiveness of our framework for

various agile interceptions of a fast-moving ball in the real world.

I. INTRODUCTION

Developing a robotic goalkeeper is an appealing but chal-

lenging problem. This task requires the robot to perform

highly agile maneuvers such as jumps and dives in order

to accurately intercept a fast moving ball in a short amount

of time. Solving this problem is attractive because it can

offer us solutions to combine dynamic legged locomotion

with fast and precise non-prehensile arm manipulation. Recent

developments in quadrupedal robots, which allow for more

agile and versatile maneuvers, provides a suitable hardware

platform for tackling this task. Furthermore, recent advances in

model-free reinforcement learning (RL) has shown promising

results on developing controllers for dynamic motor skills

on quadrupedal robots [1]–[3]. However, previous efforts on

applying RL on quadrupedal robots mainly focus on low-

level locomotion control, such as tracking a desired walking

velocity [3] or mimicking a reference motion [1], without

extending the learned locomotion skills to a higher level task,

such as precisely intercepting a fast-moving soccer ball using

agile maneuvers. This is challenging because it is a combina-

tion of highly dynamic locomotion control and accurate non-

prehensile manipulation of a fast moving object, each of which

is already a difﬁcult task on its own. Therefore, there have been

This work was in part supported by NSF Grant CMMI-1944722.

∗Authors contributed equally

1University of California, Berkeley, 2Georgia Institute of Technology, 3

Simon Fraser University. zhongyu_li@berkeley.edu.

Figure 1: A quadrupedal robot goalkeeper, Mini Cheetah, saves a

ﬂying soccer ball towards the goal using the proposed hierarchical RL

framework with multiple locomotion control policies and a motion

planning policy. The ball ﬂying time is only around 0.5second. Video

is at https://youtu.be/iX6OgG67-ZQ.

few prior attempts on developing goalkeeping controllers with

agile maneuvers using quadrupeds in the real world.

In this work, we propose to address the goalkeeping task us-

ing a hierarchical model-free RL framework. This framework

decomposes the goalkeeping task into two sub-problems: 1)

low-level locomotion control to enable the robot to perform

various agile and highly-dynamic locomotion skills, and 2)

high-level planning to decide an optimal skill and motion to

perform in order to intercept the ball.

A. Related Work

The soccer goalkeeping problem using quadrupedal robots

can be viewed as a combination of three domains of robotics

research: robotic manipulation to intercept a fast moving

object, locomotion control to enable a quadruped to perform

highly dynamic maneuvers, and the robot soccer.

1) Robotic Catching/Hitting of Fast Moving Objects: En-

abling robots to catch or hit fast moving objects, such as a

ball, has been studied extensively in the robotic manipulation

ﬁeld. Typically, robotic arms, with a ﬁxed base [4] or a mobile

base [5], and quadrocopter [6] are used for these tasks. A

common approach to tackling catching tasks is to separate it

into two sub-tasks: prediction of the ball’s trajectory based on

the estimated ball position and velocity using models of the

ball’s dynamics [4], [7], [8], and generation of a trajectory for

the robot’s end-effector based on robot’s dynamics model [6],

[7], [9] or model-free RL [10], [11] to catch the ball at the

predicted interception point. An alternative approach [12] is to

learn an end-to-end policy in simulation that directly takes the

camera’s RGB image as input, followed by ﬁne-tuning in the

arXiv:2210.04435v1 [cs.RO] 10 Oct 2022

real world [13], [14]. However, for quadrupeds, the previous

model-based methods which require accurate modeling of

the ball and the robot will be hard to utilize due to the

complexity of the dynamics models, while previous model-free

RL methods have not been applied to control such dynamic

legged robot for manipulation tasks.

2) Dynamic Locomotion Control for Quadrupeds: In re-

cent years, there have been considerable advances in legged

robot hardware and control algorithms that enable quadrupedal

robots to preform highly dynamic locomotion maneuvers, such

as jumping [1], [15]–[19] or running [2], [3], [20], in the real

world. One approach is to utilize an optimal control framework

with the robot’s dynamics models, which can be the robot’s

full-order models and optimized ofﬂine [15], [16], [18], or

simpliﬁed models and deployed online [17], [20]. Another

approach is to leverage model-free deep RL to train the

quadrupedal robots through trail-and-error in simulation ﬁrst

and then transfer to the real robot [1]–[3], [19]. However, most

previous work only focuses on a speciﬁc dynamic locomotion

skill without attaining a more diverse repertoire of maneuvers

based on learned skills to achieve a longer horizon task, such

as jumping while tracking different swing leg trajectories to

intercept a ball.

3) Legged Robot Soccer: Developing robots that can one

day compete with humans in soccer games has been an

enduring goal in the robotics community, and a notable soccer

robot game is RoboCup [21]. Related to the goalkeeping

problem of this work, there are some efforts to develop an

intelligent goalkeeper using holonomic wheeled robots [22]–

[24]. However, most previous work only consider the robot

moving in 2D plane to intercept a ball rolling on the ground

at low speeds [22], [23]. Intercepting balls in a 3D and at

high speeds, like a ﬂying ball with a speed up to 8 m/s, as

in this work, has not been studied in robot soccer. Legged

robots, such as humanoid robots and quadrupedal robots, are

also used in RoboCup, but most presented soccer skills by

legged robots, such as shooting [25], kicking [26], and goal-

keeping [27], are based on rule-based motion primitives due to

their challenging dynamics. Most recently, by leveraging deep

RL, a quadrupedal robot demonstrates the capacity to dribble

a soccer ball to a target at a low walking speed [28], and a

quadruped is also trained to precisely shoot a soccer ball to a

random given target while the robot is standing with a single

shooting skill [29]. However, enabling legged robots to play

soccer while performing multiple highly dynamic locomotion

skills, such as using jump and dive skills, and precise ball

manipulation has not yet been demonstrated.

B. Contributions

The core contribution of this work is the creation of an

agile and dynamic quadrupedal goalkeeper for robot soccer.

This work presents one of the ﬁrst solutions that combines

both highly dynamic locomotion and precise object intercep-

tion (manipulation) on real quadrupedal robots by using a

hierarchical reinforcement learning framework. The proposed

method allows quadrupeds to track parametric trajectories

with its end-effector(s) while engaging in dynamic locomotion

maneuvers. The hierarchical framework is used to learn and

compose a diverse set of low-level locomotion skills, and to

select the most appropriate skill and motion for the robot to

intercept a ﬂying ball. We show that our system can be used

to directly transfer dynamic maneuvers and goalkeeping skills

learned in simulation to a real quadrupedal robot, with an

87.5% successful interception rate of random shots in the real

world. We note that human soccer goalkeepers average around

a 69% save rate, [30]. Although, this is against professional

players shooting towards regulation sized goals, we hope this

paper takes us one step closer to enabling robotic soccer

players to compete with humans in the near future.

II. HIERARCHICAL RL FRAMEWORK FOR GOALKEEPING

TASK WITH MULTI-SKILLS

In this section, we introduce the Mini Cheetah robot which

is the experimental platform for this work. We also provide a

brief overview of the framework for developing goalkeeping

skills as illustrated in Fig. 2.

A. The Mini Cheetah Quadrupedal Robot

As shown in Fig. 1, Mini Cheetah [20] is a quadrupedal

robot having a weight of 9kg and height of 0.4m when it

is fully standing. It has 12 actuated motors qm∈R12 and

a6degree-of-freedoms (DoFs) ﬂoating base, representing its

translational qx,y,z (sagittal, lateral, and vertical) positions and

orientation qψ,θ,φ (roll, pitch, yaw), respectively.

B. Locomotion Skills for Goalkeeping

Inspired by human goalkeepers, we propose a collection

of skills for intercepting a ball ﬂying to different regions of

the goal, as illustrated in Fig. 3. The main concern underlying

the design of goalkeeping locomotion skills is that the robot

needs to react very quickly, since the total timespan of a ball’s

ballistic trajectory is typically under 1sec. Therefore, from

an initial standing pose in the middle of the goal, the robot

needs to perform very dynamic maneuvers to intercept the ball.

To accomplish this, our system uses three locomotion skills:

sidestep,dive, and jump to cover different goal regions.

1) Sidestep: During a sidestep, the robot takes a quick step

in the lateral direction to intercept the ball when it is rolling

on the ground or ﬂying toward the goal at a low attitude.

Depending on the size of the step, the robot may only need

to swing up one of its front leg while the rest can remain in

the stance phase. But for larger steps, the stance legs may also

need to leave the ground, resulting in a small sideways hop.

However, the sidestep skill may not be able to cover regions

that are farther away from the robot, such as the lower corners

of the goal or the upper regions.

2) Dive: The dive skill is based on quadrupedal jumping

behaviors [16], which allows the robot to cover a larger area

of the goal. Using the dive skill, the robot should ﬁrst pitch its

body up onto the rear legs, then turn to the lateral side towards

the direction that the ball is traveling, extend its two swing legs

to reach the ball, and ﬁnally land back on its feet. This skills

enables the robot to quickly block the lower corners of the

goal. During the dive, the rear legs may or may not leave the

ground, depending on how far the robot needs to travel.

文档加载中……请稍候！
如果长时间未打开，您也可以点击刷新试试。

下载文档到电脑，查找使用更方便

10 玖币 0人已下载

立即下载

摘要：

1CreatingaDynamicQuadrupedalRoboticGoalkeeperwithReinforcementLearningXiaoyuHuang2,ZhongyuLi1,YanzhenXiang1,YimingNi1,YufengChi1,YunhaoLi1,LizhiYang1,XueBinPeng3,andKoushilSreenath1AbstractWepresentareinforcementlearning(RL)frame-workthatenablesquadrupedalrobotstoperformsoccergoal-keepingtasksint...

展开>> 收起<<

1 Creating a Dynamic Quadrupedal Robotic Goalkeeper with Reinforcement Learning.pdf

共8页,预览2页

还剩页未读，继续阅读

声明：本站为文档C2C交易模式，即用户上传的文档直接被用户下载，本站只是中间服务平台，本站所有文档下载所得的收益归上传人(含作者)所有。玖贝云文库仅提供信息存储空间，仅对用户上传内容的表现方式做保护处理，对上载内容本身不做任何修改或编辑。若文档所含内容侵犯了您的版权或隐私，请立即通知玖贝云文库，我们立即给予删除！

1 Creating a Dynamic Quadrupedal Robotic Goalkeeper with Reinforcement Learning

相关推荐

开通VIP享超值会员特权

作者详情

相关内容

热门标签

举报选择: