Denoising Enhanced Distantly Supervised Ultraﬁne Entity Typing Yue Zhang Hongliang Fei Ping Li Cognitive Computing Lab_2

2025-05-06 0 0 353.63KB 10 页 10玖币

侵权投诉

Denoising Enhanced Distantly Supervised Ultraﬁne Entity Typing

Yue Zhang, Hongliang Fei, Ping Li

Cognitive Computing Lab

Baidu Research

10900 NE 8th St. Bellevue, WA 98004, USA

{yuezhang030, feihongliang0, pingli98}@gmail.com

Abstract

Recently, the task of distantly supervised (DS)

ultra-ﬁne entity typing has received signiﬁcant

attention. However, DS data is noisy and often

suffers from missing or wrong labeling issues

resulting in low precision and low recall. This

paper proposes a novel ultra-ﬁne entity typ-

ing model with denoising capability. Specif-

ically, we build a noise model to estimate the

unknown labeling noise distribution over input

contexts and noisy type labels. With the noise

model, more trustworthy labels can be recov-

ered by subtracting the estimated noise from

the input. Furthermore, we propose an entity

typing model, which adopts a bi-encoder archi-

tecture, is trained on the denoised data. Finally,

the noise model and entity typing model are

trained iteratively to enhance each other. We

conduct extensive experiments on the Ultra-

Fine entity typing dataset as well as OntoNotes

dataset and demonstrate that our approach sig-

niﬁcantly outperforms other baseline methods.

1 Introduction

Entity typing is the task of identifying speciﬁc se-

mantic types of entity mentions in given contexts.

Recently, more and more research has focused on

ultra-ﬁne entity typing (Choi et al.,2018;Onoe

and Durrett,2019;Dai et al.,2021). Comparing to

traditional entity typing tasks (Ren et al.,2016a,b;

Xu and Barbosa,2018;Ling and Weld,2012;Yosef

et al.,2013;Abhishek et al.,2017;Shimaoka et al.,

2017;Xin et al.,2018), the type set in ultra-ﬁne

entity typing is not restricted by KB schema, but

includes a vast number of free-form types.

To automatically annotate the large-scale ultra-

ﬁne entity typing data, Choi et al. (2018) utilized

different sources for distant supervision (DS), in-

cluding: 1) entity linking, where they mine entity

mentions that were linked to Wikipedia in HTML,

and extract relevant types from their encyclopedic

deﬁnitions, and 2) head words, where they auto-

matically extracted nominal head words from raw

text as types. However, distant supervision often

suffers from the low-precision and low-recall prob-

lems (Ren et al.,2016b), where recall can suffer

from KB or Wikipedia incompleteness, and preci-

sion can suffer when the selected types do not ﬁt

the context.

Instance DS label

: On her ﬁrst match on

grass at the AEGON Interna-

tional in Eastbourne, Lisicki lost

to [

Samantha Stosur

] in the ﬁrst

round.

actor, ath-

lete, person

: [

The ﬁlm

] was adapted

by Hugh Walpole, Howard Es-

tabrook and Lenore J. Coffee

from the Dickens novel, and di-

rected by George Cukor.

ﬁlm,

movie,

show,

art, enter-

tainment,

creation

Table 1: Examples selected from the Ultra-Fine Entity

Typing dataset in Choi et al. (2018).

Table 1shows two examples from these

datasets (Choi et al.,2018) to illustrate the chal-

lenges in automatic annotation using distant super-

vision. Sentence

is incorrectly annotated as

actor

through entity linking, which is beyond the

given context. Sentence

shows that simply treat-

ing the head word

film

as the type label, while

correct in this case, but misses many other valid

types: movie,show,art, etc.

To address the noisy labeling problem in dis-

tantly supervised entity typing, researchers devoted

much effort to denoising. Xiong et al. (2019) learns

the hierarchical correlations between different

types by injecting type co-occurrence Graph. Onoe

et al. (2021) considers box embedding, which is

more robust to data noise. While these methods im-

plicitly learn to denoise data noise, it is difﬁcult for

arXiv:2210.09599v1 [cs.CL] 18 Oct 2022

humans to interpret their denoising capacity. Onoe

and Durrett (2019) proposed an explicit denois-

ing method, where they learn a ﬁltering function

and a relabeling function to denoise DS data and

then train an entity typing model on the denoised

DS dataset. However, they only utilized a small

scale gold data to learn the ﬁltering and relabeling

function. Besides, their model did not model the

dependency between context and entity phrases.

In this paper, we aim to develop an explicit de-

noising method for distantly supervised ultra-ﬁne

entity typing. Our framework mainly consists of

two modules: a noise modeling component and an

entity typing model. The noise model estimates the

unknown labeling noise distribution over input con-

texts and observed (noisy) type labels. However,

noise modeling is challenging because the noise

information in the DS data is often unavailable,

and noise can vary with different distant labeling

techniques. To model the noise, we perturb the

small-scale gold-labeled dataset’s labels to mimic

the DS’s noise. Additionally, we utilize the

norm regularization on the large-scale DS data to

pursue the sparseness of labeling noise. Our noise

model conditions on the input context sentence and

its noisy labels to measure the underlying noise,

where the denoised data can be recovered from DS

data by subtracting the noise. For the entity typing

model, we adopt a bi-encoder architecture to match

input context and type phrases and train the entity

typing model on gold labeled and denoised data.

Finally, we design an iterative training (Tanaka

et al.,2018;Xie et al.,2020) procedure to train the

noise model and entity typing model iteratively to

enhance each other.

We summarize our contributions as follows:

(i) We propose a denoising enhanced ultra-ﬁne en-

tity typing model under the distant supervised set-

ting, including noise modeling and entity typing

modeling. Unlike previous denoising work (Onoe

and Durrett,2019) to ﬁlter low-quality samples, our

noise model directly measures underlying labeling

noise, regardless of DS techniques.

(ii) Onoe and Durrett (2019) learns a relabel func-

tion to directly relabel samples, while, we model

the labeling noise. iii) We evaluate our model

on both the Ultra-Fine entity typing (UFET) and

OntoNotes datasets, which are benchmarks for dis-

tantly supervised ultra-ﬁne entity typing and ﬁne-

grained entity typing tasks. We show that our

model can effectively denoise the DS data and

learn a superior entity typing model through de-

tailed comparison, analysis, and case study.

2 Related Works

2.1 Ultra-Fine Entity Typing

The ultra-ﬁne entity typing task was ﬁrst proposed

by Choi et al. (2018). They considered a multi-

task objective, where they divide labels into three

bins (general, ﬁne, and ultra-ﬁne), and update la-

bels only in a bin containing at least one positive

label. To further reduce the distant supervision

noise, Xiong et al. (2019) introduces a graph prop-

agation layer to impose a label-relational bias on

entity typing models to implicitly capture type de-

pendencies. Onoe et al. (2021) uses box embed-

ding to capture latent type hierarchies, which is

more robust to the labeling noise comparing to vec-

tor embedding. Dai et al. (2021) proposes to obtain

more weakly supervised training data by prompting

weak labels from language models. Zhang et al.

(2022) leverages retrieval augmentation to resolve

the distant supervision noise.

Among the previous works, Onoe and Durrett

(2019) is the most similar to ours, where the ﬁlter-

ing function is used to discard useless instances,

and relabeling function is used to relabel an in-

stance. Through ﬁltering and relabeling, Onoe and

Durrett (2019) explicitly denoise the distant super-

vision data. However, their denoising procedure

is trained only on a small-scale gold-labeled data,

while ignoring the large-scale data with distant su-

pervision labels. In addition, our denoising method

directly models the underlying label noise instead

of brutally ﬁltering all the samples with partial

wrong labels.

2.2 Learning from Noisy Labeled Datasets

We brieﬂy review the broad techniques for learning

from noisy labeled datasets. Traditionally, regu-

larization is an efﬁcient method to deal with the

issue of DNNs easily ﬁtting noisy labels, includ-

ing weight decay, and dropout. Besides, a few

studies achieve noise-robust classiﬁcation using

noise-tolerant loss functions, such as mean square

error and mean absolute error (Ghosh et al.,2017).

Recently, self-training (Xie et al.,2020) ﬁrst uses

labeled data to train a good teacher model, then

uses the teacher model to label unlabeled data, and

ﬁnally uses the labeled data and unlabeled data to

jointly train a student model. Furthermore, various

文档加载中……请稍候！
如果长时间未打开，您也可以点击刷新试试。

下载文档到电脑，查找使用更方便

10 玖币 0人已下载

立即下载

摘要：

DenoisingEnhancedDistantlySupervisedUltraneEntityTypingYueZhang,HongliangFei,PingLiCognitiveComputingLabBaiduResearch10900NE8thSt.Bellevue,WA98004,USA{yuezhang030,feihongliang0,pingli98}@gmail.comAbstractRecently,thetaskofdistantlysupervised(DS)ultra-neentitytypinghasreceivedsignicantattention.Ho...

展开>> 收起<<

Denoising Enhanced Distantly Supervised Ultraﬁne Entity Typing Yue Zhang Hongliang Fei Ping Li Cognitive Computing Lab_2.pdf

共10页,预览2页

还剩页未读，继续阅读

声明：本站为文档C2C交易模式，即用户上传的文档直接被用户下载，本站只是中间服务平台，本站所有文档下载所得的收益归上传人(含作者)所有。玖贝云文库仅提供信息存储空间，仅对用户上传内容的表现方式做保护处理，对上载内容本身不做任何修改或编辑。若文档所含内容侵犯了您的版权或隐私，请立即通知玖贝云文库，我们立即给予删除！

Denoising Enhanced Distantly Supervised Ultraﬁne Entity Typing Yue Zhang Hongliang Fei Ping Li Cognitive Computing Lab_2

相关推荐

开通VIP享超值会员特权

作者详情

相关内容

热门标签

举报选择: