1Blurb::
2Calculate the Kullback-Leibler Divergence between prior and posterior
3
4Description::
5The Kullback-Leibler (KL) Divergence, also called the relative entropy,
6provides a measure of the difference between two probability distributions.
7By specifying \c kl_divergence, the KL Divergence between the posterior
8\f$f(\boldsymbol{\theta} | \textbf{y}^{Data})\f$ and the prior
9\f$f(\boldsymbol{\theta})\f$ parameter distributions is calculated such that
10
11\f[
12D_{KL} = \int f(\boldsymbol{\theta} | \textbf{y}^{Data} ) \log
13\frac{ f(\boldsymbol{\theta} | \textbf{y}^{Data}) }{ f(\boldsymbol{\theta}) }
14d\boldsymbol{\theta}
15\f]
16
17This quantity can be interpreted as the amount of information gained about the
18parameters during the Bayesian update.
19
20<b> Expected Output </b>
21
22If \c kl_divergence is specified, the calculated value will be reported to the
23screen at the end of the calibration, following the sample statistics of the
24response functions. Example output is given below.
25
26<b> Additional Discussion </b>
27
28The quantity calculated is a \f$k\f$-nearest neighbor approximation of the
29possibly multi-dimensional integral given above. Therefore, some applications
30whose true KL Divergence is quite close to zero may report a negative KL
31Divergence.
32
33Examples::
34Below is a \c method block of a Dakota input file that indicates the calculation
35of the KL Divergence
36
37\verbatim
38method,
39       bayes_calibration queso
40          dram
41          seed = 34785
42          chain_samples = 1000
43          posterior_stats kl_divergence
44\endverbatim
45
46The calculated KL approximation is indicated in the screen output by
47"Information gained from prior to posterior" as shown below
48
49\verbatim
50Sample moment statistics for each response function:
51                            Mean           Std Dev          Skewness          Kurtosis
52least_sq_term_1  3.9982462078e-01  4.7683816550e-04 -2.3448518080e+00  7.7381497770e+00
53
54Information gained from prior to posterior = 1.0066819600e+01
55
56<<<<< Iterator bayes_calibration completed.
57<<<<< Environment execution completed.
58\endverbatim
59