An Embarrassingly Simple Approach for Intellectual Property Rights Protection on Recurrent Neural Networks Zhi Qin Tan and Hao Shan Wong and Chee Seng Chan_2

2025-04-30 0 0 704.14KB 13 页 10玖币

侵权投诉

An Embarrassingly Simple Approach for Intellectual Property Rights

Protection on Recurrent Neural Networks

Zhi Qin Tan and Hao Shan Wong and Chee Seng Chan

CISiP, Universiti Malaya, Malaysia

zhiqin1998@hotmail.com; haoshanw@gmail.com; cs.chan@um.edu.my

Abstract

Capitalise on deep learning models, offering

Natural Language Processing (NLP) solutions

as a part of the Machine Learning as a Ser-

vice (MLaaS) has generated handsome rev-

enues. At the same time, it is known that the

creation of these lucrative deep models is non-

trivial. Therefore, protecting these inventions’

intellectual property rights (IPR) from being

abused, stolen and plagiarized is vital. This

paper proposes a practical approach for the

IPR protection on recurrent neural networks

(RNN) without all the bells and whistles of

existing IPR solutions. Particularly, we intro-

duce the Gatekeeper concept that resembles

the recurrent nature in RNN architecture to em-

bed keys. Also, we design the model training

scheme in a way such that the protected RNN

model will retain its original performance iff

a genuine key is presented. Extensive exper-

iments showed that our protection scheme is

robust and effective against ambiguity and re-

moval attacks in both white-box and black-box

protection schemes on different RNN variants.

Code is available at https://github.

com/zhiqin1998/RecurrentIPR.

1 Introduction

The global Machine Learning as a Service (MLaaS)

industry with deep neural network (DNN) as the

underlying component had generated a handsome

USD 13.95 billion revenue in 2020 and is expected

to reach USD 302.66 billion by 2030, witnessing

a Compound Annual Growth Rate (CAGR)

36.2% from 2021 to 2030 (Market Research Future,

2022). At the same time, it is also an evident fact

that building a successful DNN model is a non-

trivial task - often requires huge investment of time,

resources and budgets to research and subsequently

commercialize them. As such, the creation of such

DNN models should be well protected to prevent

The mean annual growth rate of an investment over a

speciﬁed period of time longer than one year.

Genuine Key

This is a very

awesome book.

Hello, world!

Positive sentiment

Bonjour le monde!

Text Classification Output

Machine Translation Output

Counterfeit Key

Negative sentiment

Bonne nuit mon ami.

Text Classification Output

Machine Translation Output

Signature

h0RNN

h1RNN

…

Machine Translation Input

Text Classification Input

RNN

h1RNN

…

This is a very

awesome book.

Hello, world!

Machine Translation Input

Text Classification Input

Figure 1: Overview of our proposed IPR protection

scheme in white/black box settings. When a counter-

feit key is presented, the RNN model performance will

deteriorate, defeating the purpose of an infringement.

them from being replicated, redistributed or shared

by illegal parties.

At the time of writing, there are already various

DNN models protection schemes (Uchida et al.,

2017;Rouhani et al.,2018;Chen et al.,2019;Adi

et al.,2018;Zhang et al.,2018;Le Merrer et al.,

2020;Guo and Potkonjak,2018;Fan et al.,2022;

Ong et al.,2021). In general, efforts to enforce IP

protection on DNN can be categorized into two

groups: i) white-box (feature based) protection

which embeds a watermark into the internal pa-

rameters of a DNN model (i.e. model weights)

(Uchida et al.,2017;Chen et al.,2019;Rouhani

et al.,2018); and ii) black-box (trigger set based)

protection which relies on speciﬁc input-output

behaviour of the model through trigger sets (adver-

sarial sample with speciﬁc labels) (Adi et al.,2018;

Zhang et al.,2018;Le Merrer et al.,2020;Guo

and Potkonjak,2018). There are also protection

schemes that utilize both white-box and black-box

methods (Fan et al.,2022;Ong et al.,2021).

For the veriﬁcation process, typically it involves

ﬁrst remotely querying a suspicious online model

through API calls and observe the model output

(black-box). If the model output exhibits a similar

behaviour as to the owner embedded settings, it

arXiv:2210.00743v2 [cs.CL] 4 Oct 2022

will be used as early evidence to identify a suspect.

From here, the owner can appoint authorized law

enforcement to request access to the suspicious

model internal parameters to extract the embedded

watermark (white-box), where the enforcer will

examine and provide a ﬁnal verdict.

1.1 Problem Statement

Recurrent Neural Network (RNN) has been widely

used in various Natural Language Processing

(NLP) applications such as text classiﬁcation, ma-

chine translation, question answering etc. Given its

importance, however, from our understanding, the

IPR protection for RNN is yet to exist so far. This

is somewhat surprising as the NLP market, a part

of the MLaaS industry, is anticipated to grow at a

signiﬁcant CAGR of 20.2% during the forecast pe-

riod from 2021-2030. That is to say, the market is

expected to reach USD 63 billion by 2030 (Market

Research Future,2022).

1.2 Contributions

The contributions of our work are twofold:

We put forth a simple and generalized RNN

ownership protection technique, namely the

Gatekeeper concept (Eqn. 1), that utilizes

the endowment of RNN variant’s cell gate to

control the ﬂow of hidden states, depending

on the presented key (see Fig. 3);

Extensive experimental results show that

our proposed ownership veriﬁcation (both in

white-box and black-box settings) is effective

and robust against removal and ambiguity at-

tacks (see Table 4) and at the same time, with-

out affecting the model’s overall performance

on its original tasks (see Table 2).

The proposed IPR protection framework is il-

lustrated in Fig. 1. In our work, the RNN perfor-

mance is highly dependent on the availability of a

genuine key. That is to say, if a counterfeit key is

presented, the model performance will deteriorate

immediately from its original version. As a result,

it will defeat the purpose of an infringement as a

poor performance model is deemed proﬁtless in a

competitive MLaaS market.

2 Related Work

Uchida et al. (2017) were the ﬁrst to propose white-

box protection to embed watermarks into CNN by

imposing a regularization term on the weights pa-

rameters. However, the method is limited to one

will need to access the internal parameters of the

model in question to extract the embedded water-

mark for veriﬁcation purposes. Therefore, Quan

et al. (2021), Adi et al. (2018) and Le Merrer et al.

(2020) proposed to protect DNN models by training

with classiﬁcation labels of adversarial examples

in a trigger set so that ownership can be veriﬁed re-

motely through API calls without the need to access

the model weights (black-box). In both black-box

and white-box settings, Guo and Potkonjak (2018);

Chen et al. (2019) and Rouhani et al. (2018) demon-

strated how to embed watermarks (or ﬁngerprints)

that are robust to various types of attacks such as

model ﬁne-tuning, model pruning and watermark

overwriting. Recently, Fan et al. (2022) and Jie

et al. (2020) proposed passport-based veriﬁcation

schemes to improve the robustness against ambi-

guity attacks. Ong et al. (2021) also proposed a

complete IP protection framework for Generative

Adversarial Network (GAN) by imposing an ad-

ditional regularization term on all GAN variants.

Other than that, Rathi et al. (2022) demonstrated

how to generate adversarial examples by adding

noise to the input of a speech-to-text RNN model in

black-box setting. Finally, He et al. (2022) also pro-

posed a protection method designed for language

generation API by performing lexical modiﬁcation

to the original inputs in the black-box setting.

To the best of our knowledge, the closest work

to ours is Lim et al. (2022), applied on image cap-

tioning domain where a secret key is embedded

into the RNN decoder for functionality-preserving.

Although it looks similar to our idea, our proposed

Gatekeeper concept is a gate control approach

rather than element-wise operation on the hidden

states. That is to say, the embedded key in Lim et al.

(2022) is generated by converting a string into a

vector; while in our work, the embedded key is a

sequence of data similar to the input data. Further-

more, the key embedding operation in Lim et al.

(2022) method is a simple element-wise addition

or multiplication between the ﬁxed aforementioned

vector and the RNN’s hidden state. Technically, it

is equivalent to applying the same shift or scale on

the hidden state at each time step. In contrast, our

proposed method adopts both the RNN weights and

embedded key to calculate an activation recurrently

before performing the matrix multiplication on the

hidden states at each time step (see Sec. 3.1).

  

tanh













































tanh

(a) LSTM cell with Gatekeeper

tanh



























(b) GRU cell with Gatekeeper

Figure 2: Our proposed method in two major RNN variants: (a) LSTM; and (b) GRU. Solid lines denote the

original RNN operation for each cell type. Dotted red lines delineate the proposed Gatekeeper, which embeds a

key recurrently with a new gate control manner, but without introducing extra weight parameters. Best viewed in

colour.

Far and foremost, all the existing works are only

applicable on either CNN or GAN in the image

domain, else a single work in the image-captioning

that partially included RNN and two others that

only work on either speech-to-text tasks or lan-

guage generation API in the black-box setting. The

lack of protection for RNN might be due to the

difference in RNNs application domain as com-

pared to CNNs and GANs. For example, Uchida

et al. (2017) method could not be applied directly to

RNNs due to the signiﬁcant differences in both the

input and output of RNNs as compared to CNNs.

Speciﬁcally, the input to RNNs is a sequence of vec-

tors with variable length; while the output of RNNs

can be either a ﬁnal output vector or a sequence of

output vectors, depending on the underlying task

(i.e. text classiﬁcation or machine translation).

3 RNN Ownership Protection

Our idea for RNN models ownership protection is

to take advantage of its existing recurrent property

(sequence based), so that the information (hidden

states) passed between timesteps will be affected

when a counterfeit key is presented. Next, we will

illustrate how to implement the Gatekeeper concept

to RNN cells, and then followed by how to verify

the ownership via a new and complete ownership

veriﬁcation scheme. Note that, the Gatekeeper

concept uses a key

which is a sequence of vectors

similar to the input data

(herein, the key will be

a sequence of word embeddings. Please refer to

Appx. A.3 for more details). Therefore, naturally,

our key

will have varying timesteps length such

that ktis the key value at timestep t.

We will demonstrate the proposed framework on

two main RNN variants, namely LSTM (Hochre-

iter and Schmidhuber,1997) and GRU (Cho et al.,

2014) and their respective bidirectional variants.

However, one can easily apply it to other RNN vari-

ants such as Multiplicative LSTM (Krause et al.,

2017) and Peephole LSTM (Gers et al.,2002), etc.

since the implementation is generic.

3.1 Gatekeeper

As to the original design of RNN model, the

choices and amount of information to be carried

forward to the subsequent cells is decided by differ-

ent combination of gates, depending on the RNN

types. Inspired by this, we proposed the Gate-

keeper - a concept which learns to control the ﬂow

of hidden states, depending on the provided key

(e.g. genuine key or counterfeit key). Technically,

our Gatekeeper,gktis formulated as follows:

gkt=σ(Wikkt+bik +Whkhk

t−1+bhk)(1)

t=gkthx

t, cx

t=gktcx

t(for LSTM) (2)

where

denotes sigmoid operation,



is matrix

multiplication,

is the key value at timestep

t−1

is the previous hidden state of the key,

and

t(for LSTM) are the hidden state of the input, x.

One of the key points of our Gatekeeper is it does

not add weight parameters to the protected RNN

models as we chose to employ the original weights

of a RNN to calculate the value of

gkt

. That is,

for LSTM cell, we use

and

(Hochreiter and

Schmidhuber,1997) while for GRU cell, we use

and

(Cho et al.,2014) as

and

, respec-

tively. Note that the hidden state of a key at the

next time step is calculated using the original RNN

operation such that

t=R(kt, hk

t−1)

where

rep-

resents the operation of a RNN cell. Fig. 2outlines

文档加载中……请稍候！
如果长时间未打开，您也可以点击刷新试试。

下载文档到电脑，查找使用更方便

10 玖币 0人已下载

立即下载

摘要：

AnEmbarrassinglySimpleApproachforIntellectualPropertyRightsProtectiononRecurrentNeuralNetworksZhiQinTanandHaoShanWongandCheeSengChanCISiP,UniversitiMalaya,Malaysiazhiqin1998@hotmail.com;haoshanw@gmail.com;cs.chan@um.edu.myAbstractCapitaliseondeeplearningmodels,offeringNaturalLanguageProcessing(NLP)s...

展开>> 收起<<

An Embarrassingly Simple Approach for Intellectual Property Rights Protection on Recurrent Neural Networks Zhi Qin Tan and Hao Shan Wong and Chee Seng Chan_2.pdf

共13页,预览3页

还剩页未读，继续阅读

声明：本站为文档C2C交易模式，即用户上传的文档直接被用户下载，本站只是中间服务平台，本站所有文档下载所得的收益归上传人(含作者)所有。玖贝云文库仅提供信息存储空间，仅对用户上传内容的表现方式做保护处理，对上载内容本身不做任何修改或编辑。若文档所含内容侵犯了您的版权或隐私，请立即通知玖贝云文库，我们立即给予删除！

An Embarrassingly Simple Approach for Intellectual Property Rights Protection on Recurrent Neural Networks Zhi Qin Tan and Hao Shan Wong and Chee Seng Chan_2

相关推荐

开通VIP享超值会员特权

作者详情

相关内容

热门标签

举报选择: