
H1(X) = −ZpX(x)ln(pX(x))dx (4)
The properties of discrete and differential entropy are similar. The differences are that the discrete entropy is invariant
under variable changes and the continuous entropy is not necessarily so, furthermore the continuous entropy can take
negative values.
2.3 Entropy as a measure of uncertainty
According to Shannon (1948) [26], an uncertainty measure H(pX(x1), pX(x2), ..., pX(xn)) must satisfy:
1. Hmust be continuous on pX(xi), with i= 1, ..., n.
2. If pX(xi) = 1
n,Hmust be monotone increasing as a function of n.
3.
If an option is split into two successive options, the original
H
must be the weighted sum of the individual
values of H.
Shannon showed that a measure that satisfies all these properties is 2 multiplied by any positive constant (the constant
just sets the unit of measure). Among the properties that make it a good uncertainty choice are
1. H(X)=0if and only if all but one of pX(xi)are zero.
2.
When
pX(xi) = 1
n
i.e. when the discrete probability distribution is constant,
H(X)
is maximum and equal to
log(n).
3. H(X, Y )≤H(X) + H(Y)
, where equality holds if and only if
X
and
Y
are statistically independent i.e.
p(xi, yj) = p(xi)p(yj).
4. Any change towards the equalization of the probabilities pX(xi), increases H.
5. H(X, Y ) = H(X) + H(Y|X)
. So the uncertainty of the joint event
(Y|X)
is the uncertainty of
X
plus the
uncertainty of Ywhen Xis known.
6. H(Y)≥H(Y|X)
which implies that the uncertainty of
Y
is never increased by knowledge of
X
. Decreases,
unless Xand Yare independent, in which case it doesn’t change.
2.4 Comparison between Entropy and Variance
Ebrahimi et al. (1999) [
27
] showed that entropy can be related to higher order moments of a distribution, thus it can
offer a better characterization of
pX(x)
because it uses more information about the probability distribution than the
variance (which is only related to the second moment of a probability distribution).
The entropy measures the disparity of the density
pX(x)
of the uniform distribution. That is, it measures uncertainty in
the sense of using
pX(x)
instead of the uniform distribution [
23
]. While the variance measures an average of distances
from the mean of the probability distribution. According to Ebrahimi et al. [
27
], both measures reflect concentration,
but use different metrics. The variance measures the concentration around the mean and the entropy measures the
density diffusion regardless of the location of the concentration. Statistically speaking, entropy is not a mean-centered
measure, but takes into account the entire empirical distribution without concentrating on a specific moment. This
way you can take into account the entire distribution of returns without focusing on one particular [
28
]. The discrete
entropy is positive and invariant under transformations, but the variance is not. In the continuous case neither the
entropy nor the variance are invariant under one-to-one transformations [
11
] [
23
]. According to Pele et al. (2017) [
29
]
entropy is strongly related to the tails of the distribution, this feature is important for distributions with heavy tails or
with an infinite second-order moment, where the variance is obsolete. Furthermore, the entropy can be estimated for
any distribution, without prior knowledge of its functional form. These authors found that heavy–tailed distributions
generate low entropy levels, while light–tailed distributions generate high entropy values.
2.5 Investment Portfolios
A portfolio or investment portfolio is simply a collection of assets. They are characterized by the value invested in each
asset. Let wibe the fraction invested in asset iwith i= 1,2, ..., n, the required constraint is that
3