Photo-realistic 360Head Avatars in the Wild Stanislaw Szymanowicz12 Virginia Estellers1 Tadas Baltruˇ saitis1 and Matthew Johnson1

2025-05-02 0 0 3.59MB 8 页 10玖币

侵权投诉

Photo-realistic 360◦Head Avatars in the Wild

Stanislaw Szymanowicz1,2⋆, Virginia Estellers1, Tadas Baltruˇsaitis1, and

Matthew Johnson1

1Microsoft

2University of Oxford

stan@robots.ox.ac.uk,

{virginia.estellers,tadas.baltrusaitis,matjoh}@microsoft.com

Abstract. Delivering immersive, 3D experiences for human communi-

cation requires a method to obtain 360◦photo-realistic avatars of hu-

mans. To make these experiences accessible to all, only commodity hard-

ware, like mobile phone cameras, should be necessary to capture the data

needed for avatar creation. For avatars to be rendered realistically from

any viewpoint, we require training images and camera poses from all

angles. However, we cannot rely on there being trackable features in the

foreground or background of all images for use in estimating poses, es-

pecially from the side or back of the head. To overcome this, we propose

a novel landmark detector trained on synthetic data to estimate cam-

era poses from 360◦mobile phone videos of a human head for use in a

multi-stage optimization process which creates a photo-realistic avatar.

We perform validation experiments with synthetic data and showcase

our method on 360◦avatars trained from mobile phone videos.

1 Introduction

Immersive interaction scenarios on Mixed Reality devices require rendering hu-

man avatars from all angles. To avoid the uncanny valley eﬀect, these avatars

must have faces that are photo-realistic. It is likely that in the future virtual

spaces will become a ubiquitous part of every-day life, impacting everything

from a friendly gathering to obtaining a bank loan. For this reason we believe

high-quality, 360◦avatars should be aﬀordable and accessible to all: created from

images captured by commodity hardware, e.g., from a handheld mobile phone,

without restrictions on the surrounding environment.

Obtaining data to train a 360◦photo-realistic avatar ‘in the wild’ is challeng-

ing due to the potential diﬃculty of camera registration: traditional Structure-

from-Motion pipelines rely on reliable feature matches of static objects across

diﬀerent images. Prior work limits the captures to a 120◦frontal angle which

allows the use of textured planar objects that are amenable to traditional feature

detectors and descriptors (e.g., a book, markers, detailed wall decoration). How-

ever, in many 360◦captures from a mobile phone in unconstrained environments

neither the background nor the foreground can be depended upon to provide a

source of such matches.

⋆Work done while at Microsoft.

arXiv:2210.11594v1 [cs.CV] 20 Oct 2022

2 S. Szymanowicz et al.

360ophone capture Dense landmarks 360oNeRF avatar, optimized camera poses

Fig. 1. Our system creates photo-realistic 360◦avatars from captures from a mobile

phone capture and without constraints on the environment. Cameras are registered

from full 360◦pose variation, and our multi-stage optimization pipeline allows for high

quality avatars.

There are several properties of 360◦captures in the wild which pose serious

challenges to camera registration and avatar model learning. First, the space be-

ing captured is likely to either have plain backgrounds (e.g., white walls) and/or

portions of the capture in which the background is an open space, leading to de-

focus blur and the inclusion of extraneous, potentially mobile objects (e.g., pets,

cars, other people). Second, in order to obtain the needed details on the face

and hair the foreground subject will likely occupy much of the frame. While

the face can provide some useful features for camera registration, its non-planar

nature combined with changes in appearance due to lighting eﬀects make it less

than ideal. Further, while the back of the head can produce many features for

tracking the matching can become highly ambiguous due to issues with hair,

i.e., specular eﬀects and repeated texture.

To address the challenges of 360◦captures we propose a multi-stage pipeline

to create 3D photo-realistic avatars from a mobile phone camera video. We

propose using head landmarks to estimate the camera pose. However, as most

facial landmark detectors are not reliable at oblique or backward-facing angles,

we propose using synthetic data to train landmark detectors capable of working

in full 360◦range. We use the predicted landmarks to provide initialization of

the 6DoF camera poses, for a system which jointly optimizes a simpliﬁed Neural

Radiance Field with the camera poses. Finally, we use the optimized camera

poses to train a high quality, photo-realistic NeRF of the subject.

The contributions of our work are three-fold: (1) a reliable system for camera

registration which only requires the presence of a human head in each photo,

(2) a demonstration of how to leverage synthetic data in a novel manner to

obtain a DNN capable of predicting landmark locations from all angles, and

(3) a multi-stage optimization pipeline which builds 360◦photo-realistic avatars

with high-frequency visual details using images obtained from a handheld mobile

phone ‘in the wild’.

文档加载中……请稍候！
如果长时间未打开，您也可以点击刷新试试。

下载文档到电脑，查找使用更方便

10 玖币 0人已下载

立即下载

摘要：

Photo-realistic360◦HeadAvatarsintheWildStanislawSzymanowicz1,2⋆,VirginiaEstellers1,TadasBaltruˇsaitis1,andMatthewJohnson11Microsoft2UniversityofOxfordstan@robots.ox.ac.uk,{virginia.estellers,tadas.baltrusaitis,matjoh}@microsoft.comAbstract.Deliveringimmersive,3Dexperiencesforhumancommuni-cationrequi...

展开>> 收起<<

Photo-realistic 360Head Avatars in the Wild Stanislaw Szymanowicz12 Virginia Estellers1 Tadas Baltruˇ saitis1 and Matthew Johnson1.pdf

共8页,预览2页

还剩页未读，继续阅读

声明：本站为文档C2C交易模式，即用户上传的文档直接被用户下载，本站只是中间服务平台，本站所有文档下载所得的收益归上传人(含作者)所有。玖贝云文库仅提供信息存储空间，仅对用户上传内容的表现方式做保护处理，对上载内容本身不做任何修改或编辑。若文档所含内容侵犯了您的版权或隐私，请立即通知玖贝云文库，我们立即给予删除！

Photo-realistic 360Head Avatars in the Wild Stanislaw Szymanowicz12 Virginia Estellers1 Tadas Baltruˇ saitis1 and Matthew Johnson1

相关推荐

开通VIP享超值会员特权

作者详情

相关内容

热门标签

举报选择: