On Boltzmann, Logical, and Shannon Entropy

David Ellerman
University of Ljubljana
Ljubljana, Slovenia

My new book, New Foundations for Information Theory, lays the new foundations on the notion of information as being defined in terms of distinctions, differences, and distinguishability. The natural measure (in the sense of measure theory) is the notion of the logical entropy of a partition, the quantitative measure of how much a partition distinguishes, which is directly interpreted as the probability in two independent draws from the universe set of getting a distinction of the partition (elements in different blocks). Shannon entropy is not defined as a measure on a set, but all the Shannon formulas for simple, joint, conditional, and mutual entropy are obtained are the result of a non-linear dit-to-bit transform from the corresponding measure-theoretic formulas for logical entropy. Shannon entropy is literally a transform of logical entropy. There is no physics of thermodynamic entropy in the book, but it is argued that there is no conceptual connection between the Boltzmann entropy formula and Shannon entropy; it is only a particularly tractable numerical approximation using the first two terms in the infinite series formula for ln(n!). Von Neumann’s joking suggestion to Shannon was a very back idea leading to decades of confusion. Moreover, a whole cultish subfield called “MaxEntropy” as developed out of Edwin Jaynes’ work about finding the “best” probability distribution in light of given information constraints—particularly when the uniform distribution is not available. We show that maximizing logical entropy, not Shannon entropy, subject to the constraints gives the probability distribution that is the closest to the uniform distribution in terms of the usual notion of (Euclidean) distance.