1Blurb:: 2Calculate the Kullback-Leibler Divergence between prior and posterior 3 4Description:: 5The Kullback-Leibler (KL) Divergence, also called the relative entropy, 6provides a measure of the difference between two probability distributions. 7By specifying \c kl_divergence, the KL Divergence between the posterior 8\f$f(\boldsymbol{\theta} | \textbf{y}^{Data})\f$ and the prior 9\f$f(\boldsymbol{\theta})\f$ parameter distributions is calculated such that 10 11\f[ 12D_{KL} = \int f(\boldsymbol{\theta} | \textbf{y}^{Data} ) \log 13\frac{ f(\boldsymbol{\theta} | \textbf{y}^{Data}) }{ f(\boldsymbol{\theta}) } 14d\boldsymbol{\theta} 15\f] 16 17This quantity can be interpreted as the amount of information gained about the 18parameters during the Bayesian update. 19 20<b> Expected Output </b> 21 22If \c kl_divergence is specified, the calculated value will be reported to the 23screen at the end of the calibration, following the sample statistics of the 24response functions. Example output is given below. 25 26<b> Additional Discussion </b> 27 28The quantity calculated is a \f$k\f$-nearest neighbor approximation of the 29possibly multi-dimensional integral given above. Therefore, some applications 30whose true KL Divergence is quite close to zero may report a negative KL 31Divergence. 32 33Examples:: 34Below is a \c method block of a Dakota input file that indicates the calculation 35of the KL Divergence 36 37\verbatim 38method, 39 bayes_calibration queso 40 dram 41 seed = 34785 42 chain_samples = 1000 43 posterior_stats kl_divergence 44\endverbatim 45 46The calculated KL approximation is indicated in the screen output by 47"Information gained from prior to posterior" as shown below 48 49\verbatim 50Sample moment statistics for each response function: 51 Mean Std Dev Skewness Kurtosis 52least_sq_term_1 3.9982462078e-01 4.7683816550e-04 -2.3448518080e+00 7.7381497770e+00 53 54Information gained from prior to posterior = 1.0066819600e+01 55 56<<<<< Iterator bayes_calibration completed. 57<<<<< Environment execution completed. 58\endverbatim 59