A Hybrid System for Real-Time Rendering of Depth of Field Effect in Games Yu Wei Tan1

2025-04-27 0 0 8.07MB 12 页 10玖币

侵权投诉

A Hybrid System for Real-Time Rendering of Depth of Field Effect in

Games

Yu Wei Tan1 a, Nicholas Chua1, Nathan Biette1 b and Anand Bhojan1 c

1School of Computing, National University of Singapore

{yuwei, nicholaschuayunzhi, nathan.biette}@u.nus.edu, banand@comp.nus.edu.sg

Keywords: Real-Time, Depth of Field, Ray Tracing, Post-Processing, Hybrid Rendering, Games.

Abstract: Real-time depth of ﬁeld in game cinematics tends to approximate the semi-transparent silhouettes of out-of-

focus objects through post-processing techniques. We leverage ray tracing hardware acceleration and spatio-

temporal reconstruction to improve the realism of such semi-transparent regions through hybrid rendering,

while maintaining interactive frame rates for immersive gaming. This paper extends our previous work with a

complete presentation of our technique and details on its design, implementation, and future work.

1 INTRODUCTION

We present the design and evaluation of a novel real-

time hybrid rendering approach for the Depth of Field

(DoF) effect which incorporates post-process based

DoF with temporally and spatially reconstructed ray

trace based DoF. By adaptively combining the output

of different passes, we achieve more accurate semi-

transparencies of foreground geometry to reveal back-

ground objects. We believe that our hybrid DoF tech-

nique is the ﬁrst to integrate a ray-traced output with

a traditional post-processing pipeline.

Building on our previous work (Tan et al., 2020a),

the key contributions of this paper are as follows.

• Design and implementation of a real-time hybrid

rendering pipeline for DoF.

• Visual quality evaluation of the hybrid method,

speciﬁcally, the accuracy of semi-transparencies.

• Performance evaluation and trade-offs in the use

of ray tracing for DoF.

1.1 Background Information

Current DoF implementations in game engines

typically use the thin lens model (Potmesil and

Chakravarty, 1982) to approximate the behaviour of

cameras. The zone of focus is the part of the scene

ahttps://orcid.org/0000-0002-7972-2828

bhttps://orcid.org/0000-0001-7827-1538

chttps://orcid.org/0000-0001-8105-1739

where the objects look sharp. The Circle of Confu-

sion (CoC) (Demers, 2004) of points in the zone of

focus are smaller than a cell on the sensor, yielding a

single pixel in the image, whereas points outside the

zone of focus appear as a spot on the image based on

their CoC. For such points which lie on the same ob-

ject, an overall blur of the object is produced.

Bokeh shapes, which are bright spots created by

a beam of unfocused light hitting the camera sensor,

appear in areas out of the zone of focus. They usu-

ally take the shape of the camera’s aperture and can

have circular or polygonal frames depending on the

number of blades in the camera shutter.

Blurred foreground objects also have a slightly

transparent silhouette through which background

colour can be observed. These semi-transparent edges

cannot be properly rendered in games with post-

processing as the image does not store any informa-

tion behind a foreground object (Kraus and Strengert,

2007). However, such approaches are widely used in

real-time rendering as images produced by rasteriza-

tion are in sharp focus (McGraw, 2015). According to

Jimenez (2014), many techniques can only perform

an approximation of the background colour locally

using neighbouring pixels like in Abadie (2018) or

grow blur out of the silhouette of foreground objects

onto background colour, reusing foreground informa-

tion to avoid reconstructing the missing background.

However, shifting the blur outwards from foreground

objects produces inaccuracies with regards to their ac-

tual geometries, especially when the amount of ex-

tended area is comparable to the size of the objects

themselves. Objects with more elaborate shapes also

arXiv:2210.06158v1 [cs.GR] 11 Oct 2022

become fat and deformed at areas with large CoC.

Nonetheless, such inaccuracies do not exist in ray-

traced DoF (Cook et al., 1984) as we can simulate

a thin lens and query the scene for intersections, not

being limited to what is rendered in the rasterized im-

age. Nonetheless, achieving interactive frame rates

with ray tracing is difﬁcult due to the high computa-

tional costs of calculating ray-geometry intersections

and multiple shading for each pixel, even with the lat-

est GPUs developed for ray tracing. Hence, hybrid

rendering, which aims to combine existing rasteriza-

tion techniques with ray tracing, is being researched.

2 RELATED WORK

2.1 Hybrid Rendering

Examples of hybrid rendering on related effects in-

clude Macedo et al. (2018) and Marrs et al. (2018)

which invoke ray tracing for reﬂections and anti-

aliasing respectively only on pixels where rasteri-

zation techniques are unable to achieve realistic or

desirable results. Beck et al. (1981), Hertel et al.

(2009) and Lauterbach and Manocha (2009) employ

the same strategy to produce accurate shadows.

The concept of hybrid rendering can also be ex-

tended to general rendering pipelines. For example,

Cabeleira (2010) uses rasterization for diffuse illu-

mination and ray tracing for reﬂections and refrac-

tions. Barr´

e-Brisebois et al. (2019) is also one such

pipeline that has replaced effects like screen-space re-

ﬂections with their ray trace counterparts to achieve

better image quality. Another commonly-used ap-

proach is Chen and Liu (2007), the substitution of

primary ray generation with rasterization in recursive

ray tracing by Whitted (1979). Andrade et al. (2014)

improves upon this technique by observing a render

time limit through the prioritization of only the most

important scene objects for ray tracing.

2.2 DoF

Many DoF rendering techniques have been devised

over the years. Potmesil and Chakravarty (1982)

ﬁrst introduced the concept of CoC for a point based

on a thin lens model which simulates the effects of

the lens and aperture of a physical camera. It em-

ploys a post-processing technique that converts sam-

pled points into their CoCs. The intensity distribu-

tions of CoCs overlapping with each pixel are then

accumulated to produce the ﬁnal colour for the pixel.

Haeberli and Akeley (1990) integrates images ren-

dered from different sample points across the aper-

ture of the lens with an accumulation buffer. On the

other hand, Cook et al. (1984) traces multiple rays

from these different sample points on the lens into the

scene using a technique now commonly known as dis-

tributed ray tracing, for which improvements in ray

budget have been made in Hou et al. (2010) and Lei

and Hughes (2013).

For rendering with real-time performance con-

straints, spatial reconstruction and temporal accumu-

lation approaches have also been developed. For in-

stance, Dayal et al. (2005) introduces adaptive spatio-

temporal sampling, choosing to sample more based

on colour variance in the rendered image with selec-

tive rendering by Chalmers et al. (2006) and favouring

newer samples for temporal accumulation in dynamic

scenes. Schied et al. (2017) also uses temporal ac-

cumulation to raise the effective sample count on top

of image reconstruction guided by variance estima-

tion. Such techniques have been applied for DoF such

as in Hach et al. (2015), Leimk¨

uhler et al. (2018),

Weier et al. (2018), Yan et al. (2016) and Zhang et al.

(2019). More advanced reconstruction techniques for

DoF have also been introduced, such as Belcour et al.

(2013), Lehtinen et al. (2011), Mehta et al. (2014) and

Vaidyanathan et al. (2015) which sample light ﬁelds

as well as Shirley et al. (2011) which selectively blurs

pixels of low frequency content in stochastic sam-

pling. A more adaptive temporal accumulation ap-

proach from Schied et al. (2018) which is responsive

to changes in sample attributes such as position and

normal has also been proposed to mitigate ghosting

and lag in classic temporal accumulation approaches.

Micropolygon-based techniques have also proven

to be capable of DoF like in Fatahalian et al. (2009)

and Sattlecker and Steinberger (2015). Catmull

(1984) solves for per-pixel visibility by performing

depth sorting on overlapping polygons for each pixel.

Following this, approaches based on multi-layer im-

ages like Franke et al. (2018), Kraus and Strengert

(2007), Lee et al. (2008), Lee et al. (2009) and Sel-

grad et al. (2015) have also been introduced where

the contributions from each layer are accumulated to

produce the ﬁnal image. Such layered approaches

are computationally expensive although they can gen-

erate relatively accurate results in terms of semi-

transparencies. Bukowski et al. (2013), Jimenez

(2014), Valient (2013) and state-of-the-art Unreal En-

gine approach Abadie (2018) divide the scene into the

background and foreground, and runs a gathering ﬁl-

ter separately for each. We adopt such a technique,

which performs better in terms of rendering time even

in comparison to Yan et al. (2016), which avoids the

problem of separating the scene by depth by factoring

high-dimensional ﬁlters into 1D integrals.

Hach et al. (2015) acquires a rich lens archive de-

rived from a real target cinematic lens and uses it to

synthesize a point spread function (PSF) for convo-

lution in blurring. For each pixel, Leimk¨

uhler et al.

(2018) splats its PSF using a sparse representation

of its Laplacian. Time-dependent edge functions for

Akenine-M¨

oller et al. (2007) and complex plane pha-

sors for Garcia (2017) have also been used to pro-

duce DoF. Such approaches involve complex compu-

tations and seem to be more suitable for ofﬂine ren-

dering. More recently, convolutional neural network

approaches like Zhang et al. (2019) perform post-

processing for DoF by predicting the amount of blur

to generate through the analysis of past frames in real-

time but require copious amounts of training data.

McGraw (2015) and McIntosh et al. (2012)

are post-process techniques that produce polygonal

bokeh shapes based on the silhouette of the camera

aperture. McGraw also supports bokeh shapes of non-

uniform intensities, including bokeh shapes which are

lighter or darker at the rim due to spherical aberration

of the lens. Our approach currently generates circu-

lar bokeh shapes of uniform intensities but can be ex-

tended to produce alternative bokeh shapes such as

polygons by changing the shape of our sampling ker-

nel, and bokeh shapes of varying intensities by adjust-

ing the relative weight of samples within the kernel.

Our hybrid DoF technique is novel as we augment

conventional post-process approaches with ray trac-

ing, generating more accurate semi-transparencies of

foreground geometry in real-time.

3 DESIGN

Our approach in Figure 1 combines post-process

based DoF with temporally-accumulated and

spatially-reconstructed ray trace based DoF, to

produce a hybrid DoF effect that recreates accurate

semi-transparencies. Using deferred shading, a Ge-

ometry Buffer (G-Buffer) is ﬁrst produced, together

with textures containing other derived information

needed for the post-process and ray trace stages. A

sharp all-in-focus rasterized image of the scene is

also generated. This image subsequently undergoes

post-process ﬁltering while parts of the scene deemed

inaccurate with post-processing undergo distributed

ray tracing augmented with spatio-temporal recon-

struction. The images are ﬁnally composited together

with a temporal anti-aliasing (TAA) pass.

We split our scene into the near ﬁeld and the far

ﬁeld. Points in the scene in front of the focus plane

are in the near ﬁeld, and points behind the focus plane

are in the far ﬁeld further away, as shown in Figure 2.

We perform this split for both the post-process and ray

trace images, in order to merge post-processed colour

with ray trace colour on a per-ﬁeld basis later on.

3.1 Post-Process

For our post-process technique, we adapted the DoF

implementation by Jimenez (2014) which uses a gath-

ering approach directly inspired by Sousa (2013) for

ﬁltering to produce blur. Following Jimenez, the ini-

tial rasterized image is downscaled to half its resolu-

tion to speed up the ﬁltering process.

3.1.1 Preﬁlter Pass

For circular bokeh shapes, Jimenez uses a 49-tap 3-

ring main ﬁlter kernel scaled to the size of the max-

imum CoC in the tile neighbourhood of the target

pixel. However, to ﬁght undersampling, a downsam-

pling 9-tap bilateral preﬁlter is ﬁrst applied to ﬁll the

gaps of the main ﬁlter. We decided to use 81 taps with

an additional ring of samples as shown in Figure 3 for

better visual quality. Hence, our preﬁlter kernel has

a diameter of 1/8 instead of 1/6 the maximum CoC

size as in the original design. In cases where the max-

imum CoC is too small as most pixels in the neigh-

bourhood are in focus, the size of the preﬁlter kernel

is capped at √2 (diagonal length of 1 pixel) to avoid

sampling the same pixel multiple times.

3.1.2 Main Filter Pass

Jimenez performs alpha blending on the foreground

and background layers with the normalized alpha of

the foreground. However, the implementation result

was unsatisfactory as the normalized alpha calculated

was too small, producing an overly transparent fore-

ground. Hence, we used a normalized weighted sum

of foreground and background contributions for the

post-process colour vpinstead as shown.

vp=vf+vb

∑81

i=1D(0, i)·sampleAlpha(ri)

(1)

In the above equation, rirefers to the CoC radius

of sample iwhile vfand vbrepresent the total ac-

cumulated colour for the foreground and background

respectively. D(0,i)refers to the comparison of the

CoC of sample ito its distance to the centre tap of the

kernel. If the radius of the sample’s CoC is greater

than its distance to the kernel centre, the sample con-

tributes to the target pixel’s colour.

To combat aliasing, we jitter the camera’s position

with pseudorandom number values. We also gather

the proportion of samples with high specular values

for each pixel to be used to composite ray trace and

post-process colour on bright bokeh shapes later on.

文档加载中……请稍候！
如果长时间未打开，您也可以点击刷新试试。

下载文档到电脑，查找使用更方便

10 玖币 0人已下载

立即下载

摘要：

AHybridSystemforReal-TimeRenderingofDepthofFieldEffectinGamesYuWeiTan1a,NicholasChua1,NathanBiette1bandAnandBhojan1c1SchoolofComputing,NationalUniversityofSingaporefyuwei,nicholaschuayunzhi,nathan.bietteg@u.nus.edu,banand@comp.nus.edu.sgKeywords:Real-Time,DepthofField,RayTracing,Post-Processing,Hybr...

展开>> 收起<<

A Hybrid System for Real-Time Rendering of Depth of Field Effect in Games Yu Wei Tan1.pdf

共12页,预览3页

还剩页未读，继续阅读

声明：本站为文档C2C交易模式，即用户上传的文档直接被用户下载，本站只是中间服务平台，本站所有文档下载所得的收益归上传人(含作者)所有。玖贝云文库仅提供信息存储空间，仅对用户上传内容的表现方式做保护处理，对上载内容本身不做任何修改或编辑。若文档所含内容侵犯了您的版权或隐私，请立即通知玖贝云文库，我们立即给予删除！

A Hybrid System for Real-Time Rendering of Depth of Field Effect in Games Yu Wei Tan1

相关推荐

开通VIP享超值会员特权

作者详情

相关内容

热门标签

举报选择: