Revealing Unobservables by Deep Learning Generative Element Extraction Networks GEEN Yingyao HuYang Liu and Jiaxiong Yao

2025-04-29 0 0 735.49KB 20 页 10玖币

侵权投诉

Revealing Unobservables by Deep Learning:

Generative Element Extraction Networks (GEEN)

Yingyao Hu∗Yang Liu†

, and Jiaxiong Yao†

October 5, 2022

Abstract

Latent variable models are crucial in scientiﬁc research, where a key variable,

such as eﬀort, ability, and belief, is unobserved in the sample but needs to be

identiﬁed. This paper proposes a novel method for estimating realizations of a

latent variable X∗in a random sample that contains its multiple measurements.

With the key assumption that the measurements are independent conditional

on X∗, we provide suﬃcient conditions under which realizations of X∗in the

sample are locally unique in a class of deviations, which allows us to identify

realizations of X∗. To the best of our knowledge, this paper is the ﬁrst to

provide such identiﬁcation in observation. We then use the Kullback–Leibler

distance between the two probability densities with and without the conditional

independence as the loss function to train a Generative Element Extraction

Networks (GEEN) that maps from the observed measurements to realizations

of X∗in the sample. The simulation results imply that this proposed estimator

works quite well and the estimated values are highly correlated with realizations

of X∗. Our estimator can be applied to a large class of latent variable models

and we expect it will change how people deal with latent variables.

1 Introduction

Unobservables play a crucial role in scientiﬁc research because empirical researchers

often encounter a discrepancy between what is described in a model and what is

∗Johns Hopkins University, Department of Economics, Johns Hopkins University, Wyman Park

Building 544E, 3400 N. Charles Street, Baltimore, MD 21218, (email: yhu@jhu.edu).

†International Monetary Fund, 700 19th St NW, Washington DC 20431 (e-mail: yliu10@imf.org

and jyao@imf.org).

arXiv:2210.01300v1 [stat.ML] 4 Oct 2022

observed in the data. A typical example is the so-called hidden Markov models, where a

series of latent variables are observed with errors in multiple periods under conditional

independence assumptions. While there is a huge literature on the estimation of the

model with latent variables (e.g.,Aigner et al. (1984); Bishop (1998)), this paper focuses

on the estimation of realizations of the latent variable, which are not observed anywhere

in the data. Suppose that the ideal data for the estimation of a model is an i.i.d. sample

of (X1, X2, ..., Xk, X∗)1and that the researcher only observed (X1, X2, ..., Xk) in the

sample. Generally, we consider Xj, j = 1...k, as multiple measurements of X∗. Under

conditional independence assumptions, this paper provides a deep learning method to

extract the common element X∗from multiple observables (X1, X2, ..., Xk). We build

a Generative Element Extraction Networks (GEEN) to reveal realizations or draws of

X∗to achieve a complete sample of (X1, X2, ..., Xk, X∗) in the sense that the generated

draws are observationally equivalent to the true values in the sample.

This paper is diﬀerent from the imputation method because the latent variable is

not observed anywhere in the sample and needs to be identiﬁed. For example, true

earnings of households are not observed anywhere in household survey data, but are

of great interest to know. By contrast, imputation requires at least some observations

of the underlying variable.

Researchers have already applied deep generative models for data imputation.

Yoon, Jordon, and Schaar (2018) creatively use the Generative Adversarial Impu-

tation Nets (GAIN) to provide an imputation method, in which missing values are

estimated so that they are observationally equivalent to the observed values from

the GAIN’s perspective. Yoon, Jordon, and Schaar (2018) creatively use the Gen-

erative Adversarial Imputation Nets (GAIN) to provide an imputation method, in

which missing values are estimated so that they are observationally equivalent to the

observed values from the GAIN’s perspective.Li, Jiang, and Marlin (2019) also pro-

pose a GAN-based (Goodfellow et al. (2014)) framework for learning from complex,

high-dimensional incomplete data to impute missing data. Mattei and Frellsen (2019)

introduce the missing data importance-weighted autoencoder for training a deep la-

tent variable model to handle missing-at-random data. Nazabal et al. (2020) present a

general framework for Variational Autoencoders (VAEs) (Kingma and Welling (2013))

that eﬀectively incorporates incomplete data and heterogenous observations. Muzel-

1We use capital letters to stand for a random variable and lower case letters to stand for the

realization of a random variable. For example, fV(v) stands for the probability density function of

random variable Vwith realization argument v, and fV|U(v|u) denote the conditional density of V

on U.

lec et al. (2020) leverage optimal transport to deﬁne a loss function for missing value

imputation. Yoon and Sull (2020) propose a novel Generative Adversarial Multiple Im-

putation Network (GAMIN) for highly missing Data. In this literature, latent spaces

are used to represent high-dimensional observations, but are not identiﬁable because

their latent spaces may vary with parameter initialization. In addition, all the missing

data models require true values to be partially observed.

However, relatively little research has focused on estimating realizations of latent

variables, which are unobserved, or completely missing. In the economics literature,

Kalman ﬁlter and structural vector autoregressions have often been used to estimate

the realizations of latent variables, such as potential output (Kuttner, 1994), natural

rate of interest (Laubach and Williams, 2003; Holston, Laubach, and Williams, 2017),

and natural rate of unemployment (King and Morley, 2007), but the literature makes

parametric assumptions about the dynamics of latent variables and thus belongs to

the estimation of models with latent variables.

In our setting, we argue that the conditional independence restrictions imply the

local identiﬁcation of the true values. That allows us to provide an estimator in the

continuous case. Our method is nonparametric in the sense that we do not assume

the distribution of the variables belong to a parametric family as in the widely-used

VAEs (Kingma and Welling, 2013), which use the so-called Evidence Lower Bound

(ELBO) to provide a tractable unbiased Monte Carlo estimator. The VAEs focus on

the estimation of a parametric model. In this paper, we focus on the estimation of the

true values in each observation in the sample without imposing a parametric structure

on the distributions.

Our loss function is a distance between two nonparametric density functions with

and without the conditional independence. Such a distance is based on a powerful

nonparametric identiﬁcation result in the measurement error literature Hu and Schen-

nach (2008). (See Hu (2017) and Schennach (2020) for a review.) It shows that the

joint distribution of a latent variable and its measurements is uniquely determined

by the joint distribution of the observed measurements under a key conditional in-

dependence assumption, together with other technical restrictions. To measure the

distance between two density functions, we choose the Kullback–Leibler divergence

(Kullback and Leibler, 1951), which plays a leading role in machine learning and neu-

roscience (P´erez-Cruz, 2008). A large literature has studied the estimation of the

Kullback–Leibler divergence (Darbellay and Vajda, 1999; Moreno, Ho, and Vasconce-

los, 2003; Wang, Kulkarni, and Verd´u, 2005; Lee and Park, 2006; Wang, Kulkarni, and

Verd´u, 2006; Nguyen, Wainwright, and Jordan, 2010; Nowozin, Cseke, and Tomioka,

2016; Belghazi et al., 2018). We use a combination of a deep neural network and ker-

nel density estimators to generate density functions with and without the conditional

independence and then compute their divergence.

In this paper, we make a further argument that the nonparametric identiﬁcation

of the latent variable distribution implies that the true values in the sample are locally

separable in the continuous case. To the best of our knowledge, this paper is the ﬁrst

to provide such identiﬁcation in observation. We expect such identiﬁcation will change

how researchers deal with latent variables and make our GEEN broadly applicable.

This paper is organized as follows. Section 2 provides the identiﬁcation arguments.

Section 3 describe the neural network and the algorithm. The Monte Carlo simulations

are provided in Section 4. Section 5 summarizes the paper. Given the page limit, we

put in the Online Appendix a high-level application in the estimation of ﬁxed eﬀects

in panel data models.

2 From identiﬁcation in distribution to identiﬁcation in ob-

servation

We assume that a researcher observe the distribution of {X1, X2, ..., Xk}from a ran-

dom sample. Putting the estimation of the population distribution fX1,X2,...,Xkfrom

the random sample aside, we face a key identiﬁcation challenge: How to determine the

distribution fX1,X2,...,Xk,X∗from the observed distribution fX1,X2,...,Xk? Here we use a

general nonparametric identiﬁcation result in the measurement error literature. We

assume

Assumption 1. There exists a random variable X∗with support X∗such that

fX1,X2,...,Xk,X∗

=fX1|X∗fX2|X∗×... ×fXk|X∗fX∗

We may consider the observables (X1, X2, ..., Xk) as measurements of X∗. Here

we use Hu and Schennach (2008) to show the uniqueness of f(X1, X2, ..., Xk, X∗). We

assume three of the kmeasurements are informative enough for the results in Hu and

Schennach (2008). We assume

Assumption 2. The joint distribution of (X1, X2, ..., Xk, X∗)with k≥3admits

a bounded density with respect to the product measure of some dominating measure

deﬁned on their supports. All marginal and conditional densities are also bounded.

文档加载中……请稍候！
如果长时间未打开，您也可以点击刷新试试。

下载文档到电脑，查找使用更方便

10 玖币 0人已下载

立即下载

摘要：

RevealingUnobservablesbyDeepLearning:GenerativeElementExtractionNetworks(GEEN)YingyaoHu*YangLiu,andJiaxiongYaoOctober5,2022AbstractLatentvariablemodelsarecrucialinscienticresearch,whereakeyvariable,suchaseort,ability,andbelief,isunobservedinthesamplebutneedstobeidentied.Thispaperproposesanovelm...

展开>> 收起<<

Revealing Unobservables by Deep Learning Generative Element Extraction Networks GEEN Yingyao HuYang Liu and Jiaxiong Yao.pdf

共20页,预览4页

还剩页未读，继续阅读

声明：本站为文档C2C交易模式，即用户上传的文档直接被用户下载，本站只是中间服务平台，本站所有文档下载所得的收益归上传人(含作者)所有。玖贝云文库仅提供信息存储空间，仅对用户上传内容的表现方式做保护处理，对上载内容本身不做任何修改或编辑。若文档所含内容侵犯了您的版权或隐私，请立即通知玖贝云文库，我们立即给予删除！

Revealing Unobservables by Deep Learning Generative Element Extraction Networks GEEN Yingyao HuYang Liu and Jiaxiong Yao

相关推荐

开通VIP享超值会员特权

作者详情

相关内容

热门标签

举报选择: