In the spring semester of 2020, the epidemic was raging and the current situation was turbulent, with New York becoming the center of the storm. Education around the world has shifted to an online model, and I taught a course on "Optimal Transmission Theory and Algorithms" under quarantine. In this special historical period, everyone can get rid of the distractions of the world, calm down, cultivate knowledge, review life, and refine thoughts. The course has just ended, and I am recalling the journey of learning this theory over the past 25 years, which is full of twists and turns and mixed emotions. I have great admiration for Mr. Qiu's far-sightedness and broad-mindedness; I am sincerely grateful for Mr. Qiu's perseverance and tireless teaching. This article is written here, maybe it will be of benefit to young students.
Statistics vs. Geometry
In 1995, I had just arrived at Harvard University. Of course, I was ignorant in terms of knowledge and dealing with the world, but I was also very passionate about my skills and skills. In addition to studying computer science courses at Harvard University, I also attended Mr. Qiu's topology course, and at the same time studied Professor Berthold Horn's "Machine Vision" course at the MIT Artificial Intelligence Laboratory. Professor Horn is world-famous for his "shape from shading" method. He personally taught in class how to use partial differential equations to describe the imaging process and solve the geometry from the image by solving hyperbolic equations. He also proposed an extended Gaussian mapping method that uses Gaussian curvature to represent convex surfaces. This method is actually equivalent to the Minkowski problem in differential geometry. By studying Professor Horn's courses, I felt that the research methods of computer vision at that time were similar to the classical physics methods, using differential equations to describe natural phenomena and obtaining answers through analytical methods. In order to solve the Minkowski problem, I asked Mr. Qiu for advice. Mr. Qiu was very excited and gave me his classic paper, in which he proved the existence and uniqueness of the solution to the Minkowski problem in any dimension, and pointed me to the classic articles and works of Alexndrov and Pogorelov. During many discussions, Mr. Qiu taught me the algorithm for solving the Monge-Ampere equation. This should be the first algorithm Mr. Qiu taught me.
The theory of the Monge-Ampère equation was very difficult. At that time, I asked my senior brothers and sisters for help. The senior fellow students in the Department of Mathematics all believe that this is a very profound and valuable direction that is worth pursuing with all our efforts. Seniors in the Computer Science Department believe that even if the Minkowski problem can be solved, this method can only express convex surfaces and its scope of application is too narrow, so it is not worth the trouble. I soon realized that there was a huge difference in the values of basic scientists and engineering academics. Mathematicians such as Mr. Qiu pursue the best and pure truth of nature and beautiful geometric structures throughout their lives, but rarely consider the recent applications in human society; applied engineering scholars pursue practical and usable technologies rather than long-term and mysterious theories. Over the past decades, through personal practice, I have not only experienced the strong spiritual enjoyment brought by natural structures, and the eternity and harmony that transcend the world; I have also experienced the huge shock brought by technological progress, and the strong sense of control over materials. The doctoral students we train each have their own value standards and aesthetic tendencies. Many students with a background in mathematics have a strong pursuit of practical technology in the post-doctoral period; many students who return to university from industry are intolerable to the mediocre life that lacks intellectual depth, and have a sincere fascination for profound and abstract theories.
Computer vision research methods in 1995 relied heavily on geometric and partial differential equation methods, but at that time Professor Mumford and Brother Zhu Songchun criticized Horn's research methods. The energy functions in Horn's method (and other shape-from-X methods at the time) were artificially set manually or defined based on physical principles and could not reflect the complexity of the shape and appearance of objects in the real world, thus questioning the practicality of this method. They proposed that statistical methods should be used to study vision, and they used the maximum entropy method to automatically "learn" the structure and form of the energy function from the statistical features in real images. This was the first time they proposed using data to determine the form of the model, and truly systematically used statistics to lay a theoretical framework for the development of computer vision. At that time, few people in the field of vision had such forward-looking ideas. Brother Zhu Songchun earnestly persuaded me to give up the idea of differential geometry and go do research with him using statistical methods. When he talked about the statistical vision theoretical framework they were creating, his eyes lit up and his enthusiasm was uncontrollable, "This is a gold mine"! Brother Zhu's strategic thinking transcends the times and has gone through twists and turns in its development process. When machine learning was in its infancy, Mr. Shen Xiangyang confidently predicted in many private conversations that machine learning would become popular, and he was also very forward-looking. With the development of the Internet, the accumulation of massive data has become a reality, hardware computing power has increased rapidly, and statistical learning methods have finally taken off. Twenty-five years later, in the field of computer vision, the analytical methods of partial differential equations have been increasingly marginalized, and statistical learning and deep learning methods have almost dominated the world.
spiral of history
However, the pattern of historical development is always negation of negation, and all disciplines are spiraling upward. In recent years, deep learning methods have developed rapidly and become popular, sweeping across almost all fields of computer science. In the past, people worked hard to describe a natural physical phenomenon and established partial differential equations over several generations. Countless mathematicians worked hard to prove the existence, uniqueness, regularity, and stability of the solution. However, under the current deep learning framework, we can regard partial differential operators as reversible mappings in a specific function space, that is, there is a causal relationship between the initial value condition and solution, and then we can use deep neural networks to approximate this reversible mapping. This so-called end-to-end learning omits the process of human intellectual processing and deep understanding. The advancement of technology may bring about the degradation of the intelligence of ordinary human members.
From a basic theoretical level, various knowledge and data patterns are represented as probability distributions under the deep learning framework. The core goal of deep learning is to learn various probability distributions. Therefore, deep learning is essentially optimizing in an infinite-dimensional space composed of all possible probability distributions. For example, traditional maximum likelihood estimation, maximum posterior probability, maximum entropy principle, etc., all boil down to the optimization problem of probability space. Then, we need to establish a topology and calculus for this infinite-dimensional space composed of probability distributions. More rigorously, it is assumed to be a bottom space, which can be a subset of Euclidean space, or a Riemannian manifold, or an abstract distance space, etc. The space composed of all possible probability measures on it is. We need the topology of the definition, so that we can talk about the convergence of a family of probability measures, such as how a discrete probability measure obtained by a calculation method converges to the true solution; we need the distance in the definition, so that we can measure the similarities and differences between two probability distributions, for example, the discriminator in GAN calculates the distance between the generated probability and the true probability; we need the geodesic in the definition, so that we can predict a statistic The behavior of the system, for example, under normal circumstances, entropy will naturally increase, then the probability distribution at all times constitutes a trajectory in space, and the usual thermodynamic diffusion corresponds to a geodesic in ; we need to define the absolute differential, so that we can optimize in , for example, the maximum likelihood method with Gaussian noise is essentially to minimize the Wasserstein energy of entropy regularization in space.
The above discussion means that in order to lay a theoretical foundation for deep learning, we need to study all probability component spaces from the perspective of Riemannian geometry, clarify basic concepts such as geodesics, Riemannian metrics, parallel movements, and absolute differentials in probability spaces, and develop geometric variation methods in probability spaces. From this perspective, the development of computer vision statistical methods returns to the framework of infinite-dimensional Riemannian geometry. The Riemannian geometry of probability space is the optimal transmission theory, and one of the core partial equations of the optimal transmission theory is the Monge-Ampere equation. The first algorithm Mr. Qiu taught me 25 years ago was to solve the Monge-Ampere equation! History once again proves the forward-looking nature of basic mathematics. Mr. Qiu has taught us many times: The laws of nature have nothing to do with human economics and politics. The real progress of human civilization is rooted in a deep understanding of the natural structure.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Machine-translated from the Chinese original 丘成桐先生教给我的第一个算法 (published 2020-05-22). WeChat original ↗