The principle of minimum relative entropy

Antoine van de Ven

Relative entropy described by the Kullback-Leibler divergence is uniquely derived from minimal axioms and different uses and interpretations are given. Shannon entropy is a special case of relative entropy with an extra minus sign. An increasing Shannon entropy then means a decreasing relative entropy, in which the difference compared to the uniform distribution decreases. Note that relative entropy can also be used when an equilibrium or most probable distribution is not equal to the uniform distribution, so it is more general. The principle of minimum relative entropy is also derived, proposed and explained. It not only can replace the principle of maximum Shannon entropy, but can also be used to update probabilities and beliefs. Given a prior probability distribution that represents prior beliefs, the principle then describes how to find the new updated probability distribution and beliefs, given the prior and new information and constraints. It is more general and can handle more than Bayes' theorem, which can be derived from it as a special case. Different examples are given to explain the different concepts and applications.