In deep learning, the natural data set is regarded as a probability distribution defined on the manifold. The main purpose of learning is to learn the structure of the data manifold and obtain the representation of the probability distribution.
Many learning algorithms learn probability distributions based on existing observation samples. Observed samples only provide partial knowledge about the unknown distribution, so the solution is not unique. We need to define various energies to optimize these energies while satisfying the constraints of observation. The most common optimization framework ismaximum entropy principle. Entropy measures the uncertainty of a random variable. When entropy is maximum, the random variable is the most uncertain. The principle of maximum entropy is to select the probability distribution that conforms to this knowledge and has the largest entropy value when you only have partial knowledge of the unknown distribution.
Optimal transmission provides another theoretical framework. Given a space, all probability distributions on it form an infinite-dimensional space. For any probability distribution, the optimal transmission theory defines the distance between them, such as the Wasserstein distance, and then defines the Riemannian metric, parallel movement, and covariant differential, which provides us with theoretical tools for optimization. As a function defined on, entropy is a geodesic convex function, so the Hessian of entropy can be used as another Riemannian metric. In many applications, the entropy metric and the Wasserstein metric are equivalent to each other. Current deep learning is mainly based on entropy measurement, which can theoretically be explained and improved using the optimal transmission framework.
Under normal circumstances, the framework of entropy optimization is easy to understand and calculate, and using entropy as regularization energy can also improve the speed of optimal transmission operations. However, for precision applications, the optimal transmission theory provides strict accuracy and stability guarantees. For example, in the generative model, the theoretical explanation of mode collapse comes from the regularity theory of the Monge-Ampere equation, and the algorithm design to avoid mode collapse also comes from the geometric theory of optimal transmission. As another example, given the probability measure that two supports are separated from each other, the Wasserstein distance is more accurate than the KL divergence of relative entropy.
Here, we first use the principle of maximum entropy to deduce that the distribution with the largest entropy for a given expectation and variance must be a Gaussian distribution; then we use optimal transport theory to prove that on a Riemannian manifold, the gradient flow of entropy is a heat flow in the usual sense, and the distribution with the largest unconstrained entropy is a uniform distribution, which shows the equivalence of the two theories; then we examine the Wasserstein distance between Gaussian distributions, and finally extend it to the Wasserstein distance between Gaussian processes.
maximum entropy principle
Here we use the principle of maximum entropy to derive some classic results in statistics and information theory.
1. Define a probability distribution on an interval. The distribution with the largest entropy must be a uniform distribution. We maximize entropic energy,
has restrictions
Using the Lagrange multiplier method, the energy is transformed into:
We directly seek the variation and get
From this we get a constant.
2. If the probability distribution is defined in , and has finite expectation value, then the distribution with the largest entropy is the exponential distribution. In the same way, we define energy
Calculate the variation directly
From this, we get. From the conditions of unit area and finite expectation, we get, and, therefore, .
3. If the probability distribution is defined in , with mathematical expectation and variance , then the distribution with the largest entropy is the Gaussian distribution. In the same way, we define energy
Calculate the variation directly
From this, we get. From the conditions of unit area and finite expectation,
Introduction to Optimal Transport
Here we introduce the basic theory of optimal transmission. Given a probability measure in the space, and a probability measure in the space, the mapping satisfies the condition:
We call this a measure-preserving mapping, denoted as . The transmission cost function is defined as, representing the cost of transmitting unit mass from point to point.
a. Monge problemIt is to minimize the following transmission cost in all measure-preserving mappings,
The solution to the Monge problem is called the Optimal Transport Map.
b. The Kantorovich dual problem is equivalent to the Monge problem: examine all function duals, satisfying
We maximize the Kantorovich energy:
This is called the Kantorovich potential function. We define c-transformation:
Then the Kantorovich dual problem is equivalent to
We can construct a sequence to maximize the energy above. In this way, the c-convexity of the optimal Kantorovich potential energy function is obtained.
c. Brenier questionAssuming that the transmission cost and Kantorovich potential functions are sufficiently smooth, from the definition of the c-transform, we obtain the differential equation:
Assume, , we get
This shows that under these conditions, the optimal transport map is the gradient of some convex function, called the Brenier potential function. So our Brenier problem: find a convex function , satisfies. Assuming that the density function satisfies:, then the Brenier problem is equivalent to solving the followingMonge-Ampere equation.By, we get:
The solution to the Monge-Ampere equation gives the optimal transmission map, and the transmission cost of the optimal transmission map gives the so-calledWasserstein distance，
We examine the space composed of all probability measures with finite second-order momentum. Under the Wasserstein distance, it is an infinite-dimensional distance space. The geodesic in our definition satisfies the condition
McCann panThe display expression of the geodesic is given: given , is the corresponding Brenier potential energy function, then the connected geodesic is
entropy flow
We examine entropy flow from the perspective of optimal transport and prove that Wasserstein entropy flow is heat flow, which shows the consistency between the maximum entropy principle and optimal transport theory.
a. Brenier-Benamou problemAnother equivalent description of geodesics is given from the perspective of fluid mechanics. Examining the flow field in space, the density function is and the flow velocity field is. Then there is the law of conservation of mass, and we getcontinuity equation
Then, the complete description of the flow field is , and the Benamou-Brenier problem is to minimize the kinetic energy of the entire flow field to find the geodesic:
Assuming that the kinetic energy of the flow field is minimized, for any and no divergence fields, then
So it is also the admissible velocity field in the Benier-Benamous problem, from which we have:
From this we get
This means that is the gradient field of some function, . The Benamou-Brenier problem can be boiled down to:
b. Otto variation and entropy flow. Given two geodesics with the same starting point, their tangent vectors at points are their inner products (Riemannian metrics):
Given a probability measure, its entropy is defined as
Examining a path, let’s find the derivative:
From the continuity equation, it is assumed that, therefore
At the same time
we get
This means that the Wasserstein gradient of entropy is equal to, brought into the continuity equation
this meansThe Wasserstein gradient flow of entropy is equivalent to the heat flow! Substituting this in, we get that the rate of entropy dissipation is equal to
Let time tend to infinity, then it becomes a uniform distribution. The entropy flow derived from this optimal transport is consistent with the classical heat flow.
c. KL divergence Similarly, if we consider energy defined in space
Here is a smooth function that satisfies the normalization condition
The Wasserstein gradient flow thus obtained is the classic linear Fokker-Planck equation:
From the above equation, we get that at that time, . From this we get an explanation of energy:
That is, the relative Kullback information about , also known as Kullback-Leibler divergence. This explains the optimal transmission andKL divergencerelationship.
Optimal transmission of Gaussian distribution
Gaussian distribution is the most commonly used distribution, and the optimal transmission between Gaussian distributions has a closed formula, which brings great convenience to calculations. Given a Gaussian distribution, its density function with respect to the Lebesgue measure is
Here is a vector, which is a symmetric positive definite matrix. The space composed of all measures in the space is denoted as . is a total geodesic submanifold in space, it can be proved that
Here are all the constructed submanifolds. The Wasserstein metric can be expressed as
It can be proved that the 2-Wasserstein distance between two Gaussian distributions is
Given a Gaussian distribution and, we define the matrix
This defines the matrix
Then is the geodesic connecting and.
Optimal transmission distance of Gaussian process
Below we generalize the Gaussian distribution to a Gaussian process.高斯分布（Gaussian Distribution）是定义在空间的随机变量，高斯随机变量的每次采样是中的一个点；高斯过程（Gaussian Process）是定义在函数空间的随机变量，高斯过程的每次采样都是一个定义在上的函数。
Given a family of random variables, such that it is a Gaussian distribution restricted to any finite set, where is the indicator set, is called aGaussian process(Gauss Process). A Gaussian process is completely determined by the expectation function and the covariance function:
Here is a symmetric, positive semidefinite matrix. Let be defined in the space of upper-integrable functions. The covariance function defines an integral operator, defined as follows:
is called the covariance operator. Let be the orthonormal basis of , then the trace of is defined as:
Given two Gaussian processes, , and the covariance operators induced by , and are . , on upsampling, we get the Gaussian distribution defined on and , the Wasserstein distance of the Gaussian process is defined as the Wasserstein distance between,
Here are the standard measurements in .
The 2-Wasserstein distance of the Gaussian process (GP) can be approximated by the Wasserstein distance of the Gaussian distribution (GD). Let be the orthonormal basis, we define and,
The 2-Wasserstein distance between Gaussian distributions converges to the Gaussian process, the 2-Wasserstein distance between,
Summary
Recently, Mr. Qiu Chengtong and Dr. Gu co-authored "Computational Conformal Geometry-Theory", which has just been published:
Conformal geometry is rooted in basic mathematics and is the intersection of many fields: Riemann surface theory, complex analysis, differential geometry, algebraic topology, geometric partial differential equations, algebraic curves, etc. Computational conformal geometry is also related to computational geometry, digital geometry, and numerical partial differential equations in computer science. The birth of this discipline is due to the vigorous rise of 3D technology, especially 3D scanning technology (such as phase translation technology based on structured light), computer graphics technology (such as surface parameterization, texture mapping technology), computer vision technology (such as surface registration and registration, The rapid development of facial expression capture has made it impossible for traditional Euclidean geometry and linear algebra methods to solve the profound problems raised in these fields. The field of engineering and medical care must systematically introduce the ideas and methods of modern differential geometry and topology to develop rigorous and practical calculation methods. Computational conformal geometry responds to the call of the times and promotes the development of science and technology based on first principles.
Everyone is welcome to read and spread. At the same time, due to Dr. Gu's limited knowledge and negligence in his work, there will be some imprecision and errors in the book. Readers are welcome to provide valuable criticism!
The QR code of "Computational Conformal Geometry - Theory".
Machine-translated from the Chinese original 最大熵原理和最优传输 (published 2020-06-08). WeChat original ↗