and Epic GamesUnreal Engine 5Stunning the global gaming industry, itsNanitevirtual micropolygon geometry technology andLumenDynamic global illumination technology has brought a leap forward in the industry. The emergence of Nanite virtual geometry technology means that film and television-quality artworks composed of billions of polygons can be imported directly into Unreal Engine. Nanite geometry can be streamed and scaled in real time, so there is no need to consider polygon count budgets, polygon memory budgets, or draw count budgets. This greatly liberates digital artists. In the future, they only need to focus on artistic inspiration without being hampered by technology.
Figure 1. Unreal Engine 5 demo: Lumen in the Land of Nanite.
Epic Game claims that Nanite technology can support scene rendering with 16 billion polygons. Obviously, the GPU memory cannot accommodate so much geometric data anyway, so Nanite must use a technology similar to virtual memory. Many graphics colleagues speculate that Nanite virtual polygon geometry technology is likely to be a combination of virtual texture (Virtual Texture) and geometric image (Geometry Image). After the previous article, Unreal Engine 5 technical analysis: the idea of geometric images, was published, many students asked about related computer graphics concepts and algorithms. Here we further give an intuitive and detailed explanation. More details can be found in undergraduate and graduate graphics courses.
Basic graphics concepts
Figure 2. Geometry and texture.
Surfaces are generally represented by triangular meshes and texture images. The triangular mesh represents the geometric topology information of the surface, and the texture image gives information such as the color and material of the surface. The process by which we map a triangular mesh onto a planar area is called surface parameterization, and fitting a texture image onto a surface is called texture mapping. Figure 2 shows a triangular mesh and texture image of a face surface; Figure 3 is the rendering result of the data in Figure 2.
Figure 3. Geometry plus texture rendering effect.
In order to improve real-time rendering speed and reduce memory storage requirements, we prepared triangular meshes with different resolutions for the same geometric model, and decided the resolution rate used according to the Level of Details requirements. As shown in Figure 4, if the geometry is far away from the camera, we use a low-resolution model; conversely, if the surface is very close to the camera, we use a high-resolution model. Triangular meshes of different LODs can use the same texture image, so these meshes have the same parameterization.
Figure 4. LOD representation of geometric surfaces, from low poly to high poly.
Figure 5 shows that during the rendering process, the resolution rate of the surface can be dynamically increased for the parts that can be seen by the lens, and the resolution rate of the surface can be dynamically reduced for the parts that cannot be seen by the lens. For example, if we get close to the head of the Buddha statue, the body part of the Buddha statue can be divided into very rough triangulations; if we look at the angel statue from the right side, the triangulation on the right side will be much finer than on the left side.
Figure 5. LOD representation of the triangular mesh. The local resolution rate changes dynamically according to the viewpoint.
Similarly, texture images also have different resolution rates. We downsample a high-definition texture image to obtain a series of texture images, which are called mipmap pyramids of texture images. If the screen rendering the picture has a high resolution and the surface is very close to the camera, we need to use a high-resolution texture image, otherwise we use a texture image from the upper level of the pyramid.
Figure 5. mipmap pyramid of texture image.
As shown in Figure 6, on the contrary, given a low-module triangular mesh, we can subdivision the mesh according to certain rules to obtain a more refined triangular mesh. After multiple subdivisions, the triangular mesh converges to a smooth spline surface almost everywhere. The rules of triangular mesh subdivision can be written into the tessellation shader and automatically completed by the hardware.
Figure 6. Subdivision of the triangular mesh.
Texture images can store more information. For example, each pixel can record color, normal vector, material, metal, background light, scattering, highlight, transparency, geometric height, geometric displacement and many other information to describe the details of the object surface. The low-poly model plus the normal vector map generated by the high-poly model can improve rendering quality and speed.
Figure 7. Normal vector texture.
In games, normal vector maps are often used to increase the geometric details of human face surfaces, especially wrinkles. As shown in Figures 8 and 9, after the modeler's sculpture is completed, the character's skin surface is relatively smooth. After applying the normal vector map, the wrinkle effect appears and the skin is more textured.
Figure 8. Left frame, no normal map is used; right frame, normal map is applied.
Figure 9. Normal texture used to describe facial wrinkles (by Alander Alexei).
virtual texture technology
For games with high quality and complex scenes, the texture image data is very large and cannot be stored in the GPU's memory. In this case, virtual texture technology (Virtual Texture) must be used. For example, the game scene is set in a mountainous area. We have aerial terrain geometry data, as shown in Figure 10, and texture data of different resolutions, such as close-up photos. We establish virtual coordinates for the terrain surface in our minds, and merge all the photos into a huge texture image that covers the entire game world. This texture image that covers the universe is called a virtual texture.
Figure 10. Geometric data of the terrain.
The UV coordinates of the virtual texture are called virtual coordinates, or virtual address space. Every moment the player travels through the mountains corresponds to a certain point in the virtual address space. The game system tracks the player's movements in real time to know his or her virtual coordinates at any time. Because the virtual texture is too huge, we cut it into many small pieces, and each small piece is called a page, as shown in Figure 11. At any time, the system only retains a few pages close to the player's position in the GPU memory, and other pages are still placed on the hard disk. When players wander in the mountains, the system determines missing pages in real time and eliminates expired and useless pages according to certain rules (such as Least Recently Used). This algorithm for dynamically updating pages in GPU memory is very similar to the traditional virtual memory algorithm.
Figure 11. The virtual address space of the terrain texture, each square represents a page.
In modern game engines, each frame of the screen is rendered twice. For the first rendering, the system calculates the virtual coordinates of each pixel. The virtual coordinates determine the corresponding page number. At the same time, the texture mipmap level is determined based on occlusion and distance. Then, the system searches for the corresponding physical coordinates from the virtual coordinates (page, level). If a page is missing, the system searches the hard disk and transfers it to the GPU memory. Let’s take ID Company’s game Rage as an example. Figure 12 is the rendering picture that the player sees, and Figure 13 is the picture drawn by the system for the first time. Here, each square corresponds to a page, so that the virtual texture coordinates corresponding to each pixel are obtained. Figure 14 shows the active pages currently stored in GPU memory.
Figure 12. Game screen as seen by players, ID Company, Rage.
Figure 13. The page corresponding to the virtual texture coordinates. Each square corresponds to a page, ID company, Rage.
Figure 14. Currently active pages stored in GPU memory, physical memory 32 by 32 pages, ID Company, Rage.
Figure 15 briefly gives a schematic diagram of virtual texture. The mapping from virtual address to physical address is stored in a lookup table (Page Table) in the GPU. This table is generally a quadtree structure and is also expressed as a texture image. Maintaining this table and dynamically updating physical textures are the core tasks of the virtual texture system.
Figure 15. Schematic diagram of the virtual texture system (Mayer, Scheiblauer and Mayer).
Once the pages required to render the current scene are in GPU memory, the system performs a second draw. The pixel shader can find the physical coordinates from the virtual texture coordinates by querying the Page Table to obtain the corresponding texture pixels, and then filter and fuse them to obtain the color value of the current pixel. Of course, the algorithm of the real virtual texture system is very complex, and the division of labor between software and hardware is also very sophisticated. This system has a decisive impact on the quality of the entire game.
Figure 16. Demonstration of Graphine virtual textures (by Al Josha Demeulemeester).
Figure 16 shows the virtual texture technology of Granite SDK 5.0. A catamaran is flying over the mountains and the sea. The system calculates the virtual texture coordinates, obtains the corresponding page number and level of detail, and schedules the GPU memory to render in real time and obtain realistic rendering effects.
geometric image method
Virtual memory technology allows artists to no longer worry about the storage capacity of textures. Texture images can be streamed (paging) and scaled (mipmap) in real time, and game designers do not have to consider the memory budget and number of drawings of textures. Here, if we replace "texture" with "geometry", we get Epic Game's advertising slogan about Nanite virtual geometry technology.
So far, the triangular grid representation of geometry and the texture image representation are still distinct, and the texture image is still attached to the triangular grid. Although virtual texture technology has greatly liberated texture design for artists, it still does not help with geometry. For high-resolution geometric surfaces at the film and television level, the system is still unable to bear it. So, why can't we virtualize geometric data as well? For example, we put huge terrain data on the hard disk, save only the currently active pages in the GPU Cache, and ensure the player experience through real-time scheduling? In theory, there are no obstacles to this line of thinking.
First, we parameterize the terrain geometry surface as a whole, as shown in Figure 11, covering the entire virtual address space and covering the entire virtual texture image; then, we cut the entire virtual address space into pages and randomly schedule them into the GPU memory. However, here we encounter a bottleneck: the irregularity of the triangular mesh. As shown in Figure 17, the combined structure of the face surface is very irregular. We need to store the position information of the vertices and also the connection relationships between the vertices. How to express such a complex combined structure in the GPU memory page and randomly access the vertex information from the combined structure is a very difficult problem.
Figure 17. Irregularities of triangular mesh.
A simple solution is to regularize the triangular grid, that is, resample the surface with regular grid points on the parameter domain, and use a two-dimensional matrix to represent the surface, which isgeometric imageThe original intention of idea [1].
Figure 18. Schematic diagram of geometric image.
As shown in Figure 18, we conformally parameterized the King David statue (top left) to a planar rectangle (top right), regularly sampled, and converted the triangular grid into a geometric image (middle left) and a normal vector texture (middle right). The red, green, and blue color values of each pixel in the geometric image represent the three-dimensional position of the corresponding vertex, and the regular grid points (the row and column numbers of the pixels) give the connection relationship between the vertices. The surface represented by the geometric image is shown in the lower row. The left frame is the low-poly surface obtained by downsampling, and the right frame is the high-poly surface represented by the original geometric image. In this way, the LOD of the geometric image and the mipmap of the texture image are unified.
In the virtual address space, geometric images overlap with various texture images, are divided into the same pages, advance and retreat at the same time during the memory scheduling process, stream at the same time, and scale (mipmap) synchronously during the drawing process.
This mechanism reduces bandwidth consumption and GPU memory consumption, improves the locality of geometry and texture, and is more conducive to visibility culling. At the same time, the intermediate calculation results can be stored as a virtual texture, improving the synchronization of GPU task processing and simplifying the drawing process. Of course, using tessellate shader and procedure decal, we can dynamically add more geometry and textures.
The simultaneous virtualization of geometry and textures allows game designers and artists to no longer worry about the storage capacity, memory budget, and number of drawings of geometry and textures, greatly liberating their artistic nature. From this, we believe that Nanite virtual micropolygon geometry technology should be similar to the fusion of virtual texture technology and geometric image technology.
Summary
Virtualizing geometric data and allowing artists to roam freely in the virtual address space is indeed an epoch-making progress. This technology requires global parameterization of the geometry of complex scenes into a unified virtual address space, while minimizing the distortion caused by this mapping. This requires the theory of differential geometry to ensure. As shown in Figure 19, we map the Happy Buddha to a flat disk. The parameterization of the left frame keeps the local shape of the surface unchanged, and the mapping of the right frame keeps the area element of the surface unchanged. The algorithm of conformal transformation is applicable to any topological surface, and the algorithm of area-preserving transformation is applicable to manifolds of any dimension. Computational conformal geometry [2] lays the theoretical foundation for conformal transformation algorithms, and optimal transmission theory lays the foundation for area-preserving transformation. Therefore, the theory of virtual geometric images is complete and the algorithm is relatively mature. The industry is beginning to promote applications on a large scale.
Figure 19. Conformal and area-preserving parameterization.
We expect that virtual geometry and texture technology will promote the further development of the three-dimensional industry, such as game animation, VR, AR and medical imaging. We hope that more young people will no longer indulge in games, but will become strongly curious about the mathematical principles and engineering technologies behind them, study differential geometry and graphics algorithms, give full play to their talents, and make more profound and substantial contributions to society!
Figure 20. Unreal Engine 5 demo: Lumen in the Land of Nanite.
【1】 X. Gu, S. Gortler and H. Hoppe, Geometry Images, SIGGRAPH 2002, http://hhoppe.com/proj/gim/
【2】Gu Xianfeng, Qiu Chengtong, "Computational Conformal Geometry-Theory", Higher Education Press, 2020
Machine-translated from the Chinese original 虚幻引擎5技术解析（2）虚拟纹理与几何图像 (published 2020-05-17). WeChat original ↗