Differential entropy of a Gaussian mixture

Computes the differential entropy of a Gaussian mixture. The quadratic (order-2) Renyi entropy \(H_2(g) = -\log \int g(x)^2 \, dx\) is available in closed form, because \(\int g^2\) is a finite sum of Gaussian-density evaluations. Shannon entropy has no closed form for a mixture (the integrand carries the logarithm of a sum) and is estimated by Monte Carlo, reported with its standard error and an analytic upper bound that brackets it from above.

Usage

gmm_entropy(g, order = c("renyi2", "shannon"), n_mc = 5000L, seed = NULL)

Arguments

g

A gmm (or gmm_fit) object.

order

"renyi2" (closed-form quadratic Renyi entropy, the default) or "shannon" (Monte-Carlo estimate with an analytic upper bound).

n_mc

Number of Monte Carlo samples for order = "shannon".

seed

Optional integer seed for the Monte Carlo draw.

Value

For order = "renyi2", a numeric scalar. For order = "shannon", a list with components mc (the estimate), mc_se (its standard error), upper_bound (the analytic upper bound), and n_mc.

See also

gmm_divergence(), gmm_kld()

Other diagnostics: bic_aic(), ess_summary(), ess_trace(), gmm_anneal_path(), gmm_conditional_entropy(), gmm_evidence(), gmm_fit_ensemble(), gmm_fit_quality(), gmm_independence_graph(), gmm_mutual_information(), hellinger_mc(), kld_trace(), proxy_functional_ci()

Examples

g <- gmm(weights = c(0.5, 0.5),
         means = list(c(-2, 0), c(2, 0)),
         covariances = list(diag(2), diag(2)))
gmm_entropy(g)
#> [1] 3.206021
gmm_entropy(g, order = "shannon", n_mc = 2000L, seed = 1L)
#> $mc
#> [1] 3.433183
#> 
#> $mc_se
#> [1] 0.02004944
#> 
#> $upper_bound
#> [1] 3.531024
#> 
#> $n_mc
#> [1] 2000
#>