

Author: Archer Evan Park Il Memming Pillow Jonathan W.
Publisher: MDPI
E-ISSN: 1099-4300|15|5|1738-1755
ISSN: 1099-4300
Source: Entropy, Vol.15, Iss.5, 2013-05, pp. : 1738-1755
Disclaimer: Any content in publications that violate the sovereignty, the constitution or regulations of the PRC is not accepted or approved by CNPIEC.
Abstract
Mutual information (MI) quantifies the statistical dependency between a pair of random variables, and plays a central role in the analysis of engineering and biological systems. Estimation of MI is difficult due to its dependence on an entire joint distribution, which is difficult to estimate from samples. Here we discuss several regularized estimators for MI that employ priors based on the Dirichlet distribution. First, we discuss three “quasi-Bayesian” estimators that result from linear combinations of Bayesian estimates for conditional and marginal entropies. We show that these estimators are not in fact Bayesian, and do not arise from a well-defined posterior distribution and may in fact be negative. Second, we show that a fully Bayesian MI estimator proposed by Hutter (2002), which relies on a fixed Dirichlet prior, exhibits strong prior dependence and has large bias for small datasets. Third, we formulate a novel Bayesian estimator using a mixture-of-Dirichlets prior, with mixing weights designed to produce an approximately flat prior over MI. We examine the performance of these estimators with a variety of simulated datasets and show that, surprisingly, quasi-Bayesian estimators generally outperform our Bayesian estimator. We discuss outstanding challenges for MI estimation and suggest promising avenues for future research.
Related content


Statistical Information: A Bayesian Perspective
By Stern Rafael B. Pereira Carlos A. de B.
Entropy, Vol. 14, Iss. 11, 2012-11 ,pp. :




A Bayesian Predictive Discriminant Analysis with Screened Data
By Kim Hea-Jung
Entropy, Vol. 17, Iss. 9, 2015-09 ,pp. :


A Bayesian Approach to the Balancing of Statistical Economic Data
Entropy, Vol. 16, Iss. 3, 2014-02 ,pp. :