A DirectX-Based DICOM Viewer for Multi-User Surgical Planning in Augmented Reality Menghe Zhang Weichen Liu Nadir Weibel and Jürgen P. Schulze

2025-04-30 0 0 6.61MB 12 页 10玖币

侵权投诉

A DirectX-Based DICOM Viewer for Multi-User

Surgical Planning in Augmented Reality?

Menghe Zhang, Weichen Liu, Nadir Weibel, and Jürgen P. Schulze

University of California, San Diego

{mez071, wel008, weibel, jschulze}@ucsd.edu

Abstract. Preoperative medical imaging is an essential part of surgical

planning. The data from medical imaging devices, such as CT and MRI

scanners, consist of stacks of 2D images in DICOM format. Conversely,

advances in 3D data visualization provide further information by assem-

bling cross-sections into 3D volumetric datasets. As Microsoft unveiled

the HoloLens 2 (HL2), which is considered one of the best Mixed Reality

(XR) headsets in the market, it promised to enhance visualization in 3D

by providing an immersive experience to users. This paper introduces a

prototype holographic XR DICOM Viewer for the 3D visualization of

DICOM image sets on HL2 for surgical planning. We ﬁrst developed a

standalone graphical C++ engine using the native DirectX11 API and

HLSL shaders. Based on that, the prototype further applies the OpenXR

API for potential deployment on a wide range of devices from vendors

across the XR spectrum. With native access to the device, our prototype

unravels the limitation of hardware capabilities on HL2 for 3D volume

rendering and interaction. Moreover, smartphones can act as input de-

vices to provide another user interaction method by connecting to our

server. In this paper, we present a holographic DICOM viewer for the

HoloLens 2 and contribute (i) a prototype that renders the DICOM im-

age stacks in real-time on HL2, (ii) three types of user interactions in

XR, and (iii) a preliminary qualitative evaluation of our prototype.

Keywords: Volume Rendering ·HoloLens 2 ·Mixed Reality ·3D User

Interaction ·DICOM.

1 Introduction

Surgical planning with medical imaging in a DICOM format is a demanding

process. Traditionally, viewing and planning on 2D screens make it challenging

for clinical practitioners to extract 3D information such as anatomical structures,

the regions of interest, and spatial relationships among the image set.

3D volumetric visualization in Mobile Augmented Reality (MAR) solves this

problem by integrating the volumetric object with the physical environment for

?This research was funded in part by a grant from The Leona M. and Harry B.

Helmsley Charitable Trust. The authors would also like to thank the volunteers who

provided their medical data for us to work with under UCSD IRB protocol #182146.

arXiv:2210.14349v1 [cs.MM] 25 Oct 2022

mobile devices. Our previous work on Android[16] developed an application to

load, view, and interact with 3D medical volumes in traditional 2D screen and

AR scenarios.

To further allow hands-free operations, we adapt it to HoloLens 2 with our

graphical C++ MR rendering engine using the native DirectX11 API and HLSL

shaders to boost the device’s performance. We also show three user interaction

methodologies for diﬀerent tasks: 1) we design our own interaction set (3D in-

teractions) and interfaces (virtual 2D plane touch) for unique user interactions

using OpenXR APIs; 2) with matched functionalities to our Android DICOM

Viewer, we make Android phones as input devices to MR applications. Fig. 1

shows the categorized feature list. We will detail the implementation of the ex-

tended features in this section.

Fig. 1: List of the functionalities on GLES-based and D3D-based applications.

2 Related Work

2.1 Holographic DICOM Viewer Prototypes

Mojica et al. [9] proposed an AR DICOM Viewer prototype for the ﬁrst gen-

eration of the HoloLens. To accelerate volume rendering, they only render the

plane under a speciﬁc setting to avoid rendering the whole data volume. Remote

rendering is another solution. Frohlich et al. [2] proposed it as part of a dedicated

workstation with a wireless connection that enables real-time computation on

MR headsets. Their latest work [10] is deployed in a two-CPU fashion: parts run

on the HoloLens and parts run on an external PC connected through Wi-Fi to

enable real-time experience.

HoloDICOM [3] is a standard DICOM image stack visualization tool targets

at the HoloLens 2. It contains tools for volume manipulation, a 2D slicer, and

an intuitive ﬂoating 2D user interface. However, only one instructor can interact

with the application, while others could only observe the interfaces.

2.2 Interaction with 3D Objects

3D selection in mid-air is commonly used in XR applications. However, inter-

acting with far away objects and dense object selection still create problems in

terms of user fatigue [4]. We have been exploring to apply 2D touch to achieve

3D selection in AR [14][15]

The idea of tuning smartphones info input devices for HMDs attracted the

designers’ interest in various interactions. Karan et al. [5] enhanced the AR

game experience by creating new interaction methods through a smartphone

used as a game controller. Their remote input SDK makes connections between

Unity applications developed for XR devices. Buschel et al. [1] contributes an

exploration of smartphone-based interaction techniques for 3D pan and zoom,

which are two essential tasks in HMD applications. They designed ﬁve 2D touch

screen gestures mapped to pan &zoom operations in 3D. MARVIS [7] combines

smartphones and HMD AR for data analysis.

Until now, few medical imaging-based surgical planning solutions on the

HoloLens take full advantage of the ﬂexibilities of the native engine.

Meta Data

Image Stack

DICOM Files

Standalone Graphical C++ Engine

(D3D)

Volumetric

Visualization

Core Module

OpenXR Application Interface

3D User

Interaction

gRPC Server

Smartphone

as Controller

Fig. 2: System structure: Oﬄine DICOM data preprocessing and anonymization

(left), Direct3D based DICOM viewer prototype structure (middle), and smart-

phone as input device (right)

文档加载中……请稍候！
如果长时间未打开，您也可以点击刷新试试。

下载文档到电脑，查找使用更方便

10 玖币 0人已下载

立即下载

摘要：

ADirectX-BasedDICOMViewerforMulti-UserSurgicalPlanninginAugmentedReality?MengheZhang,WeichenLiu,NadirWeibel,andJürgenP.SchulzeUniversityofCalifornia,SanDiego{mez071,wel008,weibel,jschulze}@ucsd.eduAbstract.Preoperativemedicalimagingisanessentialpartofsurgicalplanning.Thedatafrommedicalimagingdevices...

展开>> 收起<<

A DirectX-Based DICOM Viewer for Multi-User Surgical Planning in Augmented Reality Menghe Zhang Weichen Liu Nadir Weibel and Jürgen P. Schulze.pdf

共12页,预览3页

还剩页未读，继续阅读

声明：本站为文档C2C交易模式，即用户上传的文档直接被用户下载，本站只是中间服务平台，本站所有文档下载所得的收益归上传人(含作者)所有。玖贝云文库仅提供信息存储空间，仅对用户上传内容的表现方式做保护处理，对上载内容本身不做任何修改或编辑。若文档所含内容侵犯了您的版权或隐私，请立即通知玖贝云文库，我们立即给予删除！

A DirectX-Based DICOM Viewer for Multi-User Surgical Planning in Augmented Reality Menghe Zhang Weichen Liu Nadir Weibel and Jürgen P. Schulze

相关推荐

开通VIP享超值会员特权

作者详情

相关内容

热门标签

举报选择: