Computational imaging with the human brain G. Wang1 D. Faccio1 School of Physics Astronomy University of Glasgow G12 8QQ Glasgow UK

2025-04-27 0 0 7.77MB 12 页 10玖币

侵权投诉

Computational imaging with the human brain

G. Wang1, D. Faccio1˚

School of Physics & Astronomy, University of Glasgow, G12 8QQ Glasgow, UK

(Dated: October 10, 2022)

Brain-computer interfaces (BCIs) are enabling a range of new possibilities and routes for aug-

menting human capability. Here, we propose BCIs as a route towards forms of computation, i.e.

computational imaging, that blend the brain with external silicon processing. We demonstrate ghost

imaging of a hidden scene using the human visual system that is combined with an adaptive com-

putational imaging scheme. This is achieved through a projection pattern ‘carving’ technique that

relies on real-time feedback from the brain to modify patterns at the light projector, thus enabling

more eﬃcient and higher resolution imaging. This brain-computer connectivity demonstrates a form

of augmented human computation that could in the future extend the sensing range of human vision

and provide new approaches to the study of the neurophysics of human perception. As an example,

we illustrate a simple experiment whereby image reconstruction quality is aﬀected by simultaneous

conscious processing and readout of the perceived light intensities.

Keywords: neurofeedback, SSVEP, ghost imaging, human brain

Introduction. Neurotechnologies and speciﬁcally

brain-computer interfaces (BCIs) provide a route to

augmenting human cognitive abilities, with applications

ranging from decision making to memory enhancement

[1–10]. Visual control of BCIs is a speciﬁc example of

interface that typically relies on the so-called steady

state visual evoked potential (SSVEP) and that can

be read-out either with implanted electrodes or more

readily, using an electroencephalogram (EEG) [11–14].

In this case, it is the visual system that acts as sensor

of the surrounding environment and/or controller of

the computer. SSVEP requires a periodically repeating

illumination pattern or light modulation, typically in the

3-4 Hz up to 30-40 Hz region, to stimulate a steady-state

(periodic) response in the brain. A well-known feature

of SSVEP is also the strong nonlinearity in the form

of multiple harmonics in the output power spectrum

[15–17].

A question that we address here, builds upon BCIs

based on visually evoked responses in the brain and

relates to whether the brain can be integrated into forms

of computational imaging.

Computational imaging is the use of computer-based

approaches to complement or enhance machine vision or

imaging. Notable examples relevant to this work include

ghost imaging, i.e. image formation with just one single

detector pixel and non-line-of-sight imaging, i.e. the

ability to see behind corners.

In its simplest version, ghost imaging (GI) relies on

illuminating an object with a series of light patterns

and then detecting only the corresponding reﬂected or

transmitted gray-scale intensity values that will vary

due to the diﬀerent spatial overlap of each pattern

with the object. By weighting each pattern with the

corresponding measured intensity value and summing

over all patterns, one can reconstruct an image of the

˚Correspondence email address: daniele.faccio@glasgow.ac.uk

object. Signiﬁcant research has been devoted also to the

problem of optimising the shape or required number of

illumination patterns, including also compressive sensing

techniques [18–32].

Non-line-of-sight imaging instead, refers to the ability

to reconstruct images of scenes that are hidden from

sight by e.g. a wall or obstacle. The more common

approaches to this involve using a pulsed light source

to illuminate the hidden scene. The reﬂected light is

collected in the form of transient waveforms that are

reﬂected from a secondary surface, e.g. another wall and

are recorded with very high (i.e. picosecond) temporal

resolution. Various inverse retrieval or processing

techniques including artiﬁcial neural networks (ANNs)

can then retrieve the ﬁnal image. GI protocols have also

been implemented for forms of NLOS imaging that rely

on the fact the GI only requires collecting an intensity

value that is retro-reﬂected from the hidden scene image,

as long as one knows the exact shape of the illumination

patterns [24, 33–38].

A key point of these and other computational tech-

niques is that they rely on some form of machine-based

detection, i.e. cameras or single pixel sensors and these

are then combined with computational algorithmic

approaches to retrieve scene images.

In this work we propose a route towards brain-computer

forms of computational imaging. We demonstrate a

ghost imaging protocol that relies on relaying light

intensity information reﬂected from a surface and that is

read-out as an SSVEP from the brain. This information

is then processed by a computer-based algorithm and an

artiﬁcial neural network that reconstructs an image from

the SSVEP power spectrum. This imaging process is

made more eﬃcient by an adaptive computational loop

whereby the SSVEP signal also indicates how to select

the appropriate illumination patterns that are sent on

to the scene being imaged. We then show preliminary

results whereby the reconstructed imaging quality is

used to quantify the diﬀerence between nonconscious

processing of the light intensity (through the EEG

arXiv:2210.03400v1 [cs.CV] 7 Oct 2022

Projector

Object

Reected light

Human

Laptop

EEG headset

Gray wall

White wall

FIG. 1. The setup used for adaptive ghost imaging. A light

projector illuminates an object cut out from a cardboard sup-

port. Transmitted light is diﬀused by a ground glass that is in

contact with the cardboard support and illuminates a white,

observation wall. This part of the setup is obscured from

the observer by a wall. The distance of both the object and

the observer from this secondary wall is „0.5´1 m. The

EEG signal from the observer is recorded and processed on a

computer.

signal) and explicit conscious processing (by asking the

participant to either verbally communicate or type on a

keyboard the perceived light intensity).

Imaging protocol. Computational ghost imaging

relies on a light source that can project a series of

typically binary (black and white) patterns, Pn. These

light patterns are then reﬂected (or transmitted) from

the object or scene we wish to image and collected with

a bucket detector (i.e. sensitive only to total energy),

an. Then, summation of all the bucket value-weighted

patterns will produce an image, O“řanPn. A very

common choice of patterns are the Hadamard set, H,

that can be recursively deﬁned.

Over the years, researchers have optimised ghost

imaging by using diﬀerent light sources, detectors or

computational algorithms. Recent attempts have also

used the human visual system as a detector where the

visual persistence time of the retina is used to directly

perform the summation operation described above, i.e.

a series of pre-weighted patterns are pre-calculated and

are then visualised at suﬃciently high rates that they

are eﬀectively perceived by the eye as an accumulated

sum [39–41]. Conversely, here we implement a form of

computational ghost imaging in which the human visual

cortex processes visual data and also provides feedback

that allows to adapt the projected patterns in real-time

so as to minimise measurement time.

Ghost imaging with the brain. A schematic

overview of the experiments is shown in Fig. 1. We

project a series of binary Hadamard patterns using a

standard digital light projector (DLP) onto an object.

The light transmitted past the object is then observed in

reﬂection from a secondary white surface (white wall).

Each binary pattern is periodically switched on/oﬀ for

several periods with a frame rate that chosen in the

3-30 Hz region. We detect the SSVEP generated by

the visual cortex activity from a single electrode placed

at Oz, the medial visual cortex region (see SI). This

SSVEP is then analysed in the spectral domain and the

corresponding fundamental (i.e. at the same frequency

of the light modulation) and higher harmonic (due to

neuron nonlinearity) amplitudes are extracted. These

are then used to reconstruct an image of the object,

which as shown in the schematic overview, is hidden

behind a wall.

Linearity. The ﬁrst step for any form of imaging

requires calibration of the detection system and iden-

tiﬁcation of linear regions or at least regions in which

the system response is monotonic with increasing input

intensity. In this case, the ‘system’ is the visual system

and SSVEP read-out, which is known to exhibit signif-

icant nonlinearity. We characterised the (non)linearity

of the SSVEP readout with a standard LCD screen that

displayed a ﬂickering uniform intensity with frequency

between 3 and 30 Hz and that was varied across the

full 8 bit range of the screen, i.e. in values from 0

to 255, corresponding to completely black (no light)

and very bright (corresponding to 125 Lumens). The

EEG signal is then Fourier transformed [42, 43]. Clear

harmonic peaks are observed as expected [13] and we

then consider the maximum values of the individual

harmonics (up to the fourth) as well as the total sum

of these values (the total SSVEP energy). The SSVEP

energy heatmap for each individual harmonic shows a

complicated and typically non-monotonic dependence

for varying screen intensity and ﬂicker frequency (see

Supplementary Information).

Figure 2a shows the total SSVEP energy. Here we

can identify two ideal ﬂicker frequency regions at 6

and 15 Hz, shown in Fig. 2b. The region around 15

Hz shows a clear monotonic increase of SSVEP energy

with increasing illumination and a similar behaviour

occurs also at 6 Hz, albeit only for a more limited screen

intensity range (between 0 and „125 bits, i.e. between 0

and „75 Lumens). The same calibration measurements

performed across three diﬀerent people resulted in a

similar behaviour (see Supplementary Information). We

therefore perform most of our experiments at either 15

Hz (using the full 0-125 Lumens intensity range) or 6 Hz

(using a limited intensity range).

Ghost Imaging results. Using the setup shown in

Fig. 1, objects are illuminated with Hadamard patterns

that are each periodically ﬂickered (see SI for full

details).

Figures 3(a) and (b) show results for the standard

ghost imaging approach for a 4 ˆ4 pixel object with a

6 Hz ﬂicker frequency and for 4 s and 2 s illumination

time for each of the ﬁrst 16 Hadamard patterns. The

columns show the ghost image reconstruction obtained

using each individual harmonic SSVEP energy and then

for the total energy (sum over all harmonics). Only the

total SSVEP energy allows to reconstruct a clear image,

(a) (b)

5 10 15 20 25

Frequency (Hz)

100

150

200

250

Screen Intensity (bits)

0.2

0.4

0.6

0.8

Screen Intensity (bits)

Normalised total SSVEP energy

0 50 100 150 200 250 300

0.2

0.4

0.6

0.8

6Hz

15Hz

FIG. 2. (a) Heatmap of the measured total SSVEP energy

(sum of all harmonic peaks). bTotal SSVEP energy at 6 Hz

and 15 Hz.

(b)

(c)

(d)

(a)

7.50x

3.55x

6.12x

7.64x

1.47x

1.38x

1.17x

1.26x

27.99x

6.62x

14.60x

9.22x

5.13x

4.87x

3.90x

3.73x

Sum of H

Obj

0.5

FIG. 3. Standard ghost imaging results. (a) Inverted “L”

shape (4 sec/pattern illumination time; total acquisition

(illumination) time of 84 seconds). (b) Inverted “L” shape

(2 sec/pattern illumination time; total acquisition time

of 42 seconds). (c) Letter “T” (8 sec illumination time;

total acquisition time of 512 seconds). (d) Letter “T”

(4 sec illumination time; total acquisition time of 256

seconds). The columns, from left to right, show the ghost

images that are reconstructed from the SSVEP fundamental

(1stH), second harmonic (2ndH), third harmonic (3rd H),

fourth harmonic (4thH) and total energy (sum over all 4

harmonics). The last column shows the ground truth object

shape. Each image is normalised to the ‘Sum of harmonics’

total intensity (rescaling factors are shown above each image).

in keeping with the calibration tests. More complicated

images require more pixels. For example, Figs. 3(c) and

(d) show the attempts to image the letter “T” on an

8ˆ8 pixel grid. At 4 s illumination time (Fig. 3(d)),

we obtain only a very noisy image. Increasing the

illumination time to 8 s for each pattern (Fig. 3(c)),

provides a marginally better image where the letter “T”

is starting to emerge, hinting that signiﬁcantly increasing

illumination times could lead to better images. However,

this strategy would lead to impractical experiment times

that could then lead to other problems, including fatigue

for the viewer.

Adaptive ghost imaging with the human brain.

An adaptive feedback loop is employed to adjust the

projected Hadamard patterns dynamically during the

measurement process and thus improve both the imaging

speed and the image quality. The underlying principal of

this is a ‘Hadamard matrix carving’ method that is based

on the observation that when projecting Hadamard (or

any given choice of) patterns onto an object, not all

patterns will have signiﬁcant overlap with the object

and this can be used to dynamically adapt the choice of

successive projections.

In brief, patterns are taken from the Hadamard matrix

H. This matrix has columns composed of vector

Hadamard patterns, each of length equal to the total

number of pixels in the image, N, and therefore, H

has rank N. These patterns (i.e. columns taken from

H) are projected one at a time. Whenever a bucket

value is measured that is below a certain threshold, this

indicates that this speciﬁc pattern has a minimal or

zero overlap with the object. We therefore apply a ‘row

carving’ operator, R, that ‘carves’ the Hadamard matrix

by removing all rows corresponding to the non-zero

row elements of the pattern. The resulting matrix will

have a reduced rank, N{2. We then apply a ‘column

carving’ operator, C, that removes columns that do not

contribute to increasing the matrix rank. In this way

we obtain a new square, carved matrix Hc“RHC that

also has rank N{2. This process is then repeated on

H1, with additional carving being applied each time a

pattern is found with no overlap with the object, each

time reducing by a factor 2x the rank and therefore

the number of required illumination patterns. The

ﬁnal result will be a reduced Hcthat contains N{2m

patterns instead of Nwith a corresponding reduction in

measurement time. The precise value of mand therefore

of the reduction of the measurement time, depends on

the speciﬁc details of the object that is being imaged.

In general terms, sparse binary objects can lead to very

signiﬁcant gains in terms of patterns that are dropped

with a signiﬁcant decrease of measurement time, as

shown below.

Full details with a worked example of the Hadamard

carving approach are provided in the Supplementary

material.

Image reconstruction. Various approaches can be

implemented to reconstruct the ﬁnal image. As seen

above, the standard GI where the image is reconstructed

as O“řanHn, will give rather noisy images.

We can use the carving approach described above and

then reconstruct an image from O“ pHc¨HT

cq´1¨Hc¨B,

where Bindicates the vector formed by all the measured

SSVEP values (see SI for details). We can additionally

use the patterns that were eliminated as masks that

indicate where we should expect the image to have zero

intensity. This ‘carved ghost imaging’ (CGI) approach

leads to signiﬁcant improvement by removing noise from

pixels outside the object.

Finally, we implemented an end-to-end deep neural

文档加载中……请稍候！
如果长时间未打开，您也可以点击刷新试试。

下载文档到电脑，查找使用更方便

10 玖币 0人已下载

立即下载

摘要：

ComputationalimagingwiththehumanbrainG.Wang1,D.Faccio1SchoolofPhysics&Astronomy,UniversityofGlasgow,G128QQGlasgow,UK(Dated:October10,2022)Brain-computerinterfaces(BCIs)areenablingarangeofnewpossibilitiesandroutesforaug-mentinghumancapability.Here,weproposeBCIsasaroutetowardsformsofcomputation,i.e.c...

展开>> 收起<<

Computational imaging with the human brain G. Wang1 D. Faccio1 School of Physics Astronomy University of Glasgow G12 8QQ Glasgow UK.pdf

共12页,预览3页

还剩页未读，继续阅读

声明：本站为文档C2C交易模式，即用户上传的文档直接被用户下载，本站只是中间服务平台，本站所有文档下载所得的收益归上传人(含作者)所有。玖贝云文库仅提供信息存储空间，仅对用户上传内容的表现方式做保护处理，对上载内容本身不做任何修改或编辑。若文档所含内容侵犯了您的版权或隐私，请立即通知玖贝云文库，我们立即给予删除！

Computational imaging with the human brain G. Wang1 D. Faccio1 School of Physics Astronomy University of Glasgow G12 8QQ Glasgow UK

相关推荐

开通VIP享超值会员特权

作者详情

相关内容

热门标签

举报选择: