The Information Bottleneck Principle in Corporate Hierarchies Cameron Gordon

2025-05-06 0 0 775.25KB 7 页 10玖币

侵权投诉

The Information Bottleneck Principle in Corporate

Hierarchies

Cameron Gordon

Australian Institute for Machine Learning

University of Adelaide

cameron.gordon@adelaide.edu.au

Abstract

The hierarchical nature of corporate information processing is a topic of great

interest in economic and management literature. Firms are characterised by a need

to make complex decisions, often aggregating partial and uncertain information,

which greatly exceeds the attention capacity of constituent individuals. However,

the efﬁcient transmission of these signals is still not fully understood. Recently, the

information bottleneck principle has emerged as a powerful tool for understanding

the transmission of relevant information through intermediate levels in a hierar-

chical structure. In this paper we note that the information bottleneck principle

may similarly be applied directly to corporate hierarchies. In doing so we provide

a bridge between organisation theory and that of rapidly expanding work in deep

neural networks (DNNs), including the use of skip connections as a means of more

efﬁcient transmission of information in hierarchical organisations.

1 Introduction

The corporate structure of ﬁrms has been the subject of economic enquiry since Coase (1937)

[

;

]. In particular, the structures required for information processing and communication within

ﬁrms has received much interest from economists, organisational theorists, institutional researchers,

cyberneticists, and sociologists [

;

]. Firms must acquire, process, and act

upon information in order to effect a goal or strategic commercial objectives, often under conditions

of ambiguity, partial observability, and uncertainty [30; 18]. This requires the coordination of many

individuals whose bounded information capacity is substantially lower than that of an organisation as

a whole [

;

]. Employees are typically arranged in a hierarchy with lower levels reporting

to higher and so on up until the board [

;

]. While decision-making and delegation may

occur at all levels of an organisation, a formal decision-maker (e.g. a board member or executive)

may be separated by several layers of hierarchy from the original source of information relevant to

their decision [

;

]. Indeed, as Simon (1997) notes "information most important to top managers

comes mainly from external sources" [

]. Market statistics, supply-chain information, changes to

government policy, media reports, customer interactions, and sales ﬁgures feature among the external

information relevant to decision-making. This aggregated set of information in most cases far exceeds

the attention limit of any one individual to process; hence, the question of how to efﬁciently process,

transmit, and ﬁlter decision-relevant information is of critical importance for ﬁrms [14; 30; 9].

The transfer of relevant information within a ﬁrm may be seen principally as a question of signal

transmission. The well-developed theory of the information bottleneck principle provides a means

of analysing the communication of relevant information that exists between two signals potentially

involving multiple stages of processing [

]. Furthermore, this theory has recently emerged as a

powerful tool for understanding the information processing that occurs in the similarly hierarchical

structure of deep neural networks (DNNs) [

;

]. While the hierarchical similarity between DNNs

Information-Theoretic Principles in Cognitive Systems Workshop at the 36th Conference on Neural Information

Processing Systems (NeurIPS 2022).

arXiv:2210.14861v1 [cs.SI] 25 Oct 2022

and corporate hierarchies has been noted sporadically within the literature (e.g. [

]), the information

theoretic and organisational design consequences of this connection have received little examination.

The primary contribution of this paper is to provide a bridge between organisational theory and

rapidly expanding work in deep neural networks and information bottleneck theory. We additionally

explore a common component of deep neural networks (skip connections) and consequences for

efﬁcient corporate information processing. We view Helbing et al.

[14]

as the prior literature most

related to our own. The authors investigate information ﬂow in hierarchical networks and highlight

the importance of side-channels and shortcut connections for efﬁcient information transmission,

however they do not extend this to the information bottleneck principle or neural architectures.

2 The Information Bottleneck Principle

The information bottleneck principle introduced in [

] describes conditions for a compressed

intermediate representation between two random variable signals (an input and an output), by

reference to the mutual information shared between these signals and the representation [

;

The principle states that a compressed intermediate representation should contain the minimum

information with respect to the input while remaining predictive of the output [

]. Consider two

random variables Xand Y. The entropy of a random variable Xis a measure of its uncertainty [3]:

H(X) = −X

x∈X

p(x) log2(p(x)).(1)

The mutual information between two random variables

I(X;Y)

symmetrically describes the decrease

in the uncertainty in Xthat occurs given that Yis known (and vice-versa) [3]:

I(X;Y) = H(X)−H(X|Y) = H(Y)−H(Y|X).(2)

Take an intermediate representation

between an input

and output

. Then

I(X;ˆ

describes

the information that is preserved between the input and the representation; and

I(Y;ˆ

that between

representation and the output. Using a control parameter

, the information bottleneck principle

states that the compressed representation should minimise the Lagrangian [33; 34; 27]:

L[p(ˆx|x)] = I(X;ˆ

X)−βI(Y;ˆ

X),(3)

where

p(ˆx|x)

is a mapping for all

x∈X

satisfying the probability constraint

ˆx∈ˆ

p(ˆx)=1

. The

parameter βcontrols the rate of compression and may be viewed as a constraint on transmission.

The information bottleneck principle has been applied widely to problems of information processing

in decision science [

], psychology [

], and prediction [

]. Importantly, it has been used to analyse

information ﬂows in deep feedforward neural networks [

;

]. A deep feedforward neural

network involves a hierarchical processing of layers in which the output of each layer feeds directly

to the next. Typically this is written as a composition of functions

f(x) = fk(fk−1(f . . . (f1(x))))

where xis an input, fi(x)is a function applied at layer i, and kis the number of layers [11].

Tishby and Zaslavsky

[33]

describe two properties that enable the information bottleneck principle to

be applied to DNNs. The ﬁrst is the data processing inequality, a fundamental result in information

theory [

]. Brieﬂy stated, for a Markov chain

X→Y→Z

it must be that

I(X;Y)≥I(X;Z)

That is, that mutual information is non-increasing through processing. This may be seen visually by

observing the sets of information shared between intermediate representations (see Figure 1). Within

a DNN information ﬂow can be represented as a Markov chain

X→T1→T2→. . . Tk→Y

where

represents the

ith

hidden layer. The bottleneck representation implies that at convergence each

layer will be the minimal representation required to predict the next layer subject to network capacity.

Successive neural network layers hence reduce the mutual information shared between the input and

output. The second property (invariance of mutual information to invertible transformations) enables

the principle to hold for different re-parameterizations and intermediate structural forms [38].

文档加载中……请稍候！
如果长时间未打开，您也可以点击刷新试试。

下载文档到电脑，查找使用更方便

10 玖币 0人已下载

立即下载

摘要：

TheInformationBottleneckPrincipleinCorporateHierarchiesCameronGordonAustralianInstituteforMachineLearningUniversityofAdelaidecameron.gordon@adelaide.edu.auAbstractThehierarchicalnatureofcorporateinformationprocessingisatopicofgreatinterestineconomicandmanagementliterature.Firmsarecharacterisedbyanee...

展开>> 收起<<

The Information Bottleneck Principle in Corporate Hierarchies Cameron Gordon.pdf

共7页,预览2页

还剩页未读，继续阅读

声明：本站为文档C2C交易模式，即用户上传的文档直接被用户下载，本站只是中间服务平台，本站所有文档下载所得的收益归上传人(含作者)所有。玖贝云文库仅提供信息存储空间，仅对用户上传内容的表现方式做保护处理，对上载内容本身不做任何修改或编辑。若文档所含内容侵犯了您的版权或隐私，请立即通知玖贝云文库，我们立即给予删除！

The Information Bottleneck Principle in Corporate Hierarchies Cameron Gordon

相关推荐

开通VIP享超值会员特权

作者详情

相关内容

热门标签

举报选择: