Content-based Graph Privacy Advisor Dimitrios Stoidis Centre for Intelligent Sensing

2025-05-02 0 0 1.69MB 8 页 10玖币

侵权投诉

Content-based Graph Privacy Advisor

Dimitrios Stoidis

Centre for Intelligent Sensing

Queen Mary University of London

London, UK

dimitrios.stoidis@qmul.ac.uk

Andrea Cavallaro

Centre for Intelligent Sensing

Queen Mary University of London

London, UK

a.cavallaro@qmul.ac.uk

Abstract—People may be unaware of the privacy risks of

uploading an image online. In this paper, we present Graph

Privacy Advisor, an image privacy classiﬁer that uses scene

information and object cardinality as cues to predict whether

an image is private. Graph Privacy Advisor simpliﬁes a state-of-

the-art graph model and improves its performance by reﬁning

the relevance of the information extracted from the image. We

determine the most informative visual features to be used for the

privacy classiﬁcation task and reduce the complexity of the model

by replacing high-dimensional image feature vectors with lower-

dimensional, more effective features. We also address the problem

of biased prior information by modelling object co-occurrences

instead of the frequency of object occurrences in each class.

Index Terms—graph neural networks, privacy, image classiﬁ-

cation

I. INTRODUCTION

Sharing images on social media platforms responds to our

need for communication and self-expression. However, images

may contain personal information that puts at risk the privacy

of the photographer and the people involved. As people may

be unaware of the privacy risks associated with an image being

uploaded online [1]–[3], it is important to develop predictive

models that inform users about private information before

sharing.

Privacy-related information can be extracted from the image

itself or from its metadata (e.g. user-generated tags). Zerr et

al. [4] use SIFT features [5] in conjunction with metadata,

including the title and user-generated tags. Dynamic Multi-

Modal Fusion for Privacy Prediction (DMFP) [6] fuses fea-

tures from three modalities (object, scene and tag). A compe-

tence estimation of each modality is performed to determine

the best modality and fuse its decisions in the last stage to

produce a prediction. DMFP builds upon the Combination

model [7] that merges scene-based tags extracted from images

with convolutional networks with features of the detected

objects in the image. The Gated Fusion model [8] fuses

predictions generated by single-modal models (for objects,

scene and tags) and dynamically learns the fusion weights for

each modality.

Some of the above works extract information from user-

generated tags [4], [6]–[8]. Instead we focus on content-

based image information only, rather than on the associated

metadata. In related work, image features extracted with VGG-

16 [9] are used to extract private information from images [10].

A combination of hand-crafted and learned features derived

from convolutional layers is used in Privacy-Convolutional

Neural Network with Hierarchical features (PCNH) [11]. iPri-

vacy [12] uses multi-task learning to identify sensitive objects

in images by considering relationships between objects and

their co-occurrence with deﬁned privacy settings.

In this paper, we improve the performance of a state-of-the-

art graph-based network [13] by substituting the input informa-

tion with smaller-size feature vectors obtained by models pre-

trained on scene and object detection. We also show that scene

information combined with the cardinality of objects (i.e. the

number of objects of a speciﬁc category) are salient content-

based visual features for the image privacy prediction task.

Furthermore, we address potentially biased prior information

between objects and privacy classes by encoding the co-

occurrences of objects.

II. RELATED WORK

In this section, we discuss image privacy datasets and graph-

based learning methods for image classiﬁcation.

A. Image privacy datasets and annotation

There exist four main image privacy datasets, namely Pi-

cAlert [4], Visual Privacy (VISPR) [14], Image Privacy Dataset

(IPD) [13] and PrivacyAlert [8]. PicAlert [4] has 37,535

images that were posted on Flickr from January to April 20101.

In PicAlert, most private images contain people, which induces

a bias towards the considered private objects (63% of private

images contain persons). VISPR [14] (22,167 images from

Flickr) includes images from natural everyday scenes, with

background and foreground clutter, and images with textual

content such as ID cards, bank account details, email content,

and licence plates. IPD [13] (38,525 images) combines Pi-

cAlert [4] and 6,392 private images from VISPR [14] (13,910

private images and 24,615 public images). PrivacyAlert [8]

contains 6,800 images that were posted between 2011 and

2021 (83% dating after 2015, thus more representative of

recent trends in image sharing).

Annotations for privacy datasets are notoriously difﬁcult to

obtain due to the inherent subjectivity of this labelling process.

For PicAlert, participants were asked to judge whether an

image was considered in the private sphere of the photographer

The code is available at https://github.com/smartcameras/GPA

1Note that 9,365 images have been deleted after the original publication

due to copyright issues.

arXiv:2210.11169v2 [cs.CV] 13 Nov 2022

TABLE I

PRIVACY DATASETS AND THEIR ANNOTATION.

Dataset Ref. # Images Labels Annotation

?PicAlert [4] 37,535 private, public, und.

“Private are photos which have to do with the private sphere (like self portraits, family,

friends, your home) or contain objects that you would not share with the entire world

(like a private email). The rest are public. In case no decision can be made, the picture

should be marked as undecidable”

VISPR [14] 22,167 private, public EU data Privacy [15], US Privacy Act [16], Social network rules [17]

?IPD [13] 38,525 private, public, und. see PicAlert and VISPR

?PrivacyAlert [8] 6,800

†clearly private, “Assume you have taken these photos, and you are about to upload them on your

clearly public, favourite social network [...] tell us whether these images are either private or

private, public public in nature. Assume that the people in the photos are those that you know ”

KEY – und.: undecidable. VISPR: Visual Privacy dataset [14], IPD: Image Privacy Dataset [13], ?: full dataset not available,

†: classes are merged into two classes (private and public).

(e.g. selﬁes, family, friends and home) (see Tab. I). Possible

labels are private,public and undecidable (if no decision could

be made on the image label). For VISPR, the annotations are

based on 68 attributes compiled from the EU Data Protection

Directive 95/46/EC [15], the US Privacy Act 1974 [16], social

network platform rules [17] and additional attributes after

manual inspection of the images [14]. Private images contain

at least one out of 32 privacy attributes related to personal life,

health, documents, visited locations, Internet conversations,

and automobiles. For PrivacyAlert, the images are annotated

through the Mechanical Turk2crowd-sourcing platform, and

the annotators are asked to classify the images into 4 classes

(clearly private,private,public and clearly public). The qual-

ity of the annotations is monitored using an attention checker

set that discards annotators who failed to provide the expected

response. The four-class annotations are then grouped into

binary labels that combine clearly private with private and

clearly public with public. PrivacyAlert provides binary labels

for each image and VISPR provides privacy attributes that are

classiﬁed as private or public. In PicAlert, 17% of the dataset

contains multiple ternary annotations for each image where

annotators’ agreement needs to be computed. Thus, labels

can be decided depending on the desired level of privacy, for

instance, labelling an image as private in case of annotation

disagreement.

B. Graph-based models for image classiﬁcation

Graph-based methods have recently been introduced for

privacy classiﬁcation [13], [18]. Graph-based networks model

information as nodes whose relationship is deﬁned through

edges. The representation of each node is updated, propagating

the information through the edges. The initialisation of graphs

is often referred to as prior knowledge represented as an

adjacency matrix.

Prior knowledge structured as a knowledge graph can

improve image classiﬁcation performance [19]. The Graph

Search Neural Network (GSNN) [19] incorporates prior

knowledge into Graph Neural Networks (GNN) [20] to solve

a multi-task vision classiﬁcation problem (see Tab. II). GSNN

is based on the Gated Graph Neural Network (GGNN) [21],

reducing computational requirements and observing the ﬂow

of information through the propagation model. GGNN uses

2https://www.mturk.com/

TABLE II

MAIN COMPONENTS OF THE ARCHITECTURE OF GRAPH-BASED METHODS.

Model Ref. Architecture Task

GGNN [21] GRU+GNN representation learning

GSNN [19] GGNN image classiﬁcation

GRM [23] GGNN+GAT relationship recognition

GIP [13] GGNN+GAT image privacy classiﬁcation

DRAG [18] GCN image privacy classiﬁcation

KEY – GRU: Gated Recurrent Unit [22]; GNN: Graph Neural Network [20]; GAT:

Graph Attention Networks [24]; GCN: Graph Convolutional Network [25].

the Gated Recurrent Unit (GRU) [22] to update the hidden

state of each node with information from the neighbouring

nodes. GGNN is a differential recurrent neural network that

operates on graph data representations, iteratively propagating

the relationships to learn node-level and graph-level repre-

sentations. The Graph Reasoning Model (GRM) [23] uses

objects and interactions between people to predict social rela-

tionships. The graph model weighs the predicted relationships

with a graph attention mechanism based on Graph Attention

Networks (GAT) [24]. GRM uses prior knowledge on social

relationships, co-occurrences and objects in the scene as a

structured graph. Interactions between people of interest and

contextual objects are modelled by GGNN [21], where nodes

are initialised with the corresponding semantic regions. The

model learns about relevant objects that carry relevant task-

information.

Graph Image Privacy (GIP) [13] replaces the social rela-

tionship nodes with two nodes representing the privacy classes

(private, public). Dynamic Region-Aware Graph Convolutional

Network (DRAG) [18] adaptively models the correlation be-

tween important regions of the image (including objects) using

a self-attention mechanism. DRAG [18] is a Graph Convo-

lutional Network (GCN) [25] that learns the relationships

among speciﬁc regions in the image without the use of object

recognition.

III. METHOD

In this section, we present the content-based features, the

prior information to initialise Graph Privacy Advisor (GPA)

as well as the graph-based learning and privacy classiﬁcation.

A. Features

Cardinality can affect the prediction of the privacy class,

especially considering the person category. An image is more

文档加载中……请稍候！
如果长时间未打开，您也可以点击刷新试试。

下载文档到电脑，查找使用更方便

10 玖币 0人已下载

立即下载

摘要：

Content-basedGraphPrivacyAdvisorDimitriosStoidisCentreforIntelligentSensingQueenMaryUniversityofLondonLondon,UKdimitrios.stoidis@qmul.ac.ukAndreaCavallaroCentreforIntelligentSensingQueenMaryUniversityofLondonLondon,UKa.cavallaro@qmul.ac.ukAbstractPeoplemaybeunawareoftheprivacyrisksofuploadinganimag...

展开>> 收起<<

Content-based Graph Privacy Advisor Dimitrios Stoidis Centre for Intelligent Sensing.pdf

共8页,预览2页

还剩页未读，继续阅读

声明：本站为文档C2C交易模式，即用户上传的文档直接被用户下载，本站只是中间服务平台，本站所有文档下载所得的收益归上传人(含作者)所有。玖贝云文库仅提供信息存储空间，仅对用户上传内容的表现方式做保护处理，对上载内容本身不做任何修改或编辑。若文档所含内容侵犯了您的版权或隐私，请立即通知玖贝云文库，我们立即给予删除！

Content-based Graph Privacy Advisor Dimitrios Stoidis Centre for Intelligent Sensing

相关推荐

开通VIP享超值会员特权

作者详情

相关内容

热门标签

举报选择: