Real-Time Constrained 6D Object-Pose Tracking of An In-Hand Suture Needle for Minimally Invasive Robotic Surgery Zih-Yun Chiu1 Florian Richter1Student Member IEEE and Michael C. Yip1Senior Member IEEE

2025-04-29 0 0 786.17KB 7 页 10玖币

侵权投诉

Real-Time Constrained 6D Object-Pose Tracking of An In-Hand Suture

Needle for Minimally Invasive Robotic Surgery

Zih-Yun Chiu1, Florian Richter1Student Member, IEEE, and Michael C. Yip1Senior Member, IEEE

Abstract— Autonomous suturing has been a long-sought-after

goal for surgical robotics. Outside of staged environments,

accurate localization of suture needles is a critical foundation

for automating various suture needle manipulation tasks in the

real world. When localizing a needle held by a gripper, previous

work usually tracks them separately without considering their

relationship. Because of the signiﬁcant errors that can arise

in the stereo-triangulation of objects and instruments, their

reconstructions may often not be consistent. This can lead to

unrealistic tool-needle grasp reconstructions that are infeasible.

Instead, an obvious strategy to improve localization would

be to leverage constraints that arise from contact, thereby

constraining reconstructions of objects and instruments into

a jointly feasible space. In this work, we consider feasible

grasping constraints when tracking the 6D pose of an in-hand

suture needle. We propose a reparameterization trick to deﬁne

a new state space for describing a needle pose, where grasp con-

straints can be easily deﬁned and satisﬁed. Our proposed state

space and feasible grasping constraints are then incorporated

into Bayesian ﬁlters for real-time needle localization. In the

experiments, we show that our constrained methods outperform

previous unconstrained/constrained tracking approaches and

demonstrate the importance of incorporating feasible grasping

constraints into automating suture needle manipulation tasks.

I. INTRODUCTION

Automating surgical procedures such as suturing has

drawn increased interest within the robotics community

during the past two decades [1]. The advantage of automation

is that it relieves surgeons from time-consuming, tedious, and

challenging tasks that often emerge in Minimally Invasive

Surgeries [2]–[4]. One of the key components of achieving

autonomous procedures is the accurate localization of surgi-

cal instruments in the surgical scene [5]–[7]. This localization

ability serves as the foundation for automating various surgi-

cal tasks in previous work, including needle regrasping [8],

[9], knot tying [10], [11], and blood suction [12].

A surgical scene often contains multiple surgical instru-

ments, and previous studies localize them separately without

considering their physical interactions [8], [9], [13]–[15].

This can lead to unrealistic environmental reconstruction

when combining tracking results of different tools. For

example, if a needle is held by a surgical manipulator,

tracking them separately can result in the needle being in a

non-feasible grasp conﬁguration (e.g., in-collision or ﬂoating,

as shown in Fig. 1). This can be a dangerous situation be-

cause dropping needles can result in damage to surrounding

This project was funded by the US Army Telemedicine and Advanced

Technologies Research Center and NSF CAREER award #2045803. F.

Richter was supported on an NSF Graduate Research Fellowship.

1Zih-Yun Chiu, Florian Richter, and Michael Yip are with the Electrical

and Computer Engineering Dept., University of California San Diego, La

Jolla, CA 92093 USA. {zchiu, frichter, yip}@ucsd.edu

Unconstrained

Constrained

Side ViewTop View

Fig. 1: Live image of a daVinci robot instrument grasping a

suture needle, top and side views of tool reconstruction from

unconstrained and our constrained needle tracking results.

The scene reconstruction of our constrained method is always

feasible. This feasibility is not ensured by unconstrained

approaches, even when the top view of tool reconstruction

aligns well with the live image.

tissue and additional trauma with repetitive needle pick-

up. Therefore, in this work, we focus on considering the

physical interactions between a suture needle and a surgical

manipulator to ensure feasible results in real-time tracking of

an in-hand suture needle. Real-time localization is necessary

since, in practice, grasping a needle causes it to re-orient in

the gripper, and further re-orientation or slippage can happen

once the needle interacts with the environment.

A. Related Work

Current literature on suture needle localization mostly

focuses on the features of a needle extracted from camera

data and assumes the needle can be anywhere in the space.

Several methods reconstruct the pose of a static needle by

observing detected markers or learned segmentation [9], [13],

[16]. Others have considered uncertainty in the features

and motions of a needle and use Bayesian ﬁlters with

different observation models to track its pose [14], [15],

[17]. However, these methods do not consider the physical

interactions between the needle and the environment and thus

do not guarantee that the needle pose is feasible.

Some studies in suture needle localization consider the

physical interactions between a surgical manipulator and a

needle when the manipulator holds the needle. One way

arXiv:2210.11973v1 [cs.RO] 21 Oct 2022

is to perform tracking and assume that the conﬁguration

between the needle and the manipulator tip is known and

remains unchanged over time [18]. Then the robot Jacobian

and joint-sensor readings are used to estimate the motions

of the needle. However, getting to this known state is

nontrivial, as grasping a needle itself is a non-deterministic

action, and grasp pose is situation-dependent, such as during

regrasping [8]. Thus, the work in [19] does not assume a

known conﬁguration of the needle held by an end-effector,

and its motions are estimated by a tool-tracking method [7]

that tracks the pose of the end-effector. These approaches

take into account that the needle should move concurrently

with the gripper when held by it. Nonetheless, they do not

ensure that the suture needle pose lies inside the feasible

grasping manifold of the gripper.

Tracking the poses of an in-hand needle is a constrained

pose tracking problem, where the needle should always

lie inside the feasible grasping manifold of the gripper.

However, there is no uniﬁed approach to deﬁne a feasible

grasping manifold since grippers and grasped objects can

be in arbitrary shapes, making this task highly nonlinear. To

incorporate constraints into nonlinear tracking problems, pre-

vious work follow two approaches: acceptance/rejection sam-

pling [20] and optimization [21], [22]. Acceptance/rejection

methods are known to reduce the diversity of the tracked

pose [22] and require an excessive number of feasibility

checks, making them not desirable for real-time track-

ing [23]. On the other hand, optimization methods project

the estimated pose onto a feasible manifold. However, they

require the manifold to be deﬁned as equality or inequality

constraints [22], [23], and describing the feasible grasping

manifold in such a way would be highly nontrivial.

B. Contributions

In this work, we achieve state-of-the-art performance

for real-time suture needle tracking in robotic surgery by

incorporating grasping constraints. To this end, we present

the following novel contributions:

1) the ﬁrst approach to probabilistically track a suture

needle in real-time with grasping constraints,

2) a state-space to describe a grasped suture needle for

efﬁcient sampling on the feasible grasping manifold,

3) and a comparison of Bayesian ﬁlter approaches that

incorporate the grasping constraints.

The proposed methods are evaluated in both simulation

and real-world environments. In simulation environments, we

demonstrate that our proposed methods outperform other un-

constrained/constrained tracking approaches. Moreover, we

evaluate different tracking methods on the suture needle re-

grasping task [8], [9]. The results indicate that incorporating

grasping constraints makes the regrasping policy more robust

to noise in detections. In real-world environments, we use

marker-less feature detections from a Deep Neural Network

(DNN) as needle observations and reconstruct the tracked

tool poses from ex-vivo images. An example is shown in

Fig. 1. The results demonstrate that our constrained approach

ensures a feasible estimated pose, and an unconstrained

method can lead to unrealistic reconstructions.

II. METHODS

A. Problem Formulation

We aim to solve the in-hand suture needle pose, st,

tracking problem probabilistically from a sequence of ob-

servations, o0:t, which can be formulated as:

Track pt|t(st):=p(st|a0:t−1,o0:t)

s.t. st∈ Ft

where st=f(st−1,at−1,wt−1)∼pf(·|st−1,at−1)

ot=h(st,vt)∼ph(·|st)

(1)

where Ftis the feasible grasping space, f(·)and h(·)are

the motion and observation models with noise wt−1and vt

respectively, and at−1is the action applied to the suture

needle.

In our task, Ftin (1) is the feasible grasping manifold

of the surgical manipulator that is holding a suture needle at

time step t. Usually, a grasping manifold should consider two

feasibility constraints: geometric and dynamic constraints.

Geometric constraints include [24]:

1) The object’s surface should be in contact with the

gripper’s surface, i.e., Surface(st)∩Surface(et)6=∅,

where etis the state of the gripper at time step t.

2) The object should not penetrate with the gripper, i.e.,

Interior(st)∩Interior(et) = ∅.

Dynamic constraints include that if there is no external force

except gravity acting on both the object and the gripper,

the linear and angular velocities of the object relative to the

gripper should be 0. Hence, the feasible grasping manifold

Ftcan be represented as

Ft={st|st∈ Gt∩ Dt},(2)

where Gt={st|Surface(st)∩Surface(et)6=∅and

Interior(st)∩Interior(et) = ∅},(3)

Dt={st|If ExternalF orce \Gravity =∅,

LinearV elocity(st,et) = 0and

AngularV elocity(st,et) = 0}.(4)

Due to the special design and property of surgical manipu-

lators and suture needles, we can simplify the requirements

of deﬁning the feasible grasping manifold for an in-hand

needle. More speciﬁcally, the dynamic constraints in (4) are

ignored because (1) a suture needle is very light compared

to the gripper, and (2) grippers for surgical manipulators

are designed to increase the friction between themselves and

the objects they are holding (e.g., Needle Drivers). Hence,

Ft=Gt,∀t∈[1, . . . , T ].

Since the robot end-effector or the grasped object can have

a complex shape, the feasible grasping manifold Ftin (2) is

difﬁcult to deﬁne for the object pose, [b>

tq>

t]>, where bt∈

R3is the position, and qt∈R3is the axis-angle orientation.

The object pose, which is described in a global frame such as

the camera frame or in the ego-centric end-effector frame,

文档加载中……请稍候！
如果长时间未打开，您也可以点击刷新试试。

下载文档到电脑，查找使用更方便

10 玖币 0人已下载

立即下载

摘要：

Real-TimeConstrained6DObject-PoseTrackingofAnIn-HandSutureNeedleforMinimallyInvasiveRoboticSurgeryZih-YunChiu1,FlorianRichter1StudentMember,IEEE,andMichaelC.Yip1SeniorMember,IEEEAbstractAutonomoussuturinghasbeenalong-sought-aftergoalforsurgicalrobotics.Outsideofstagedenvironments,accuratelocalizati...

展开>> 收起<<

Real-Time Constrained 6D Object-Pose Tracking of An In-Hand Suture Needle for Minimally Invasive Robotic Surgery Zih-Yun Chiu1 Florian Richter1Student Member IEEE and Michael C. Yip1Senior Member IEEE.pdf

共7页,预览2页

还剩页未读，继续阅读

声明：本站为文档C2C交易模式，即用户上传的文档直接被用户下载，本站只是中间服务平台，本站所有文档下载所得的收益归上传人(含作者)所有。玖贝云文库仅提供信息存储空间，仅对用户上传内容的表现方式做保护处理，对上载内容本身不做任何修改或编辑。若文档所含内容侵犯了您的版权或隐私，请立即通知玖贝云文库，我们立即给予删除！

Real-Time Constrained 6D Object-Pose Tracking of An In-Hand Suture Needle for Minimally Invasive Robotic Surgery Zih-Yun Chiu1 Florian Richter1Student Member IEEE and Michael C. Yip1Senior Member IEEE

相关推荐

开通VIP享超值会员特权

作者详情

相关内容

热门标签

举报选择: