| blob | 8aceafc0b8ce7145cabb953bdc470e094be5f994 |
1 # 3D Rendering
3 *See also [3D modeling](3d_modeling.md).*
5 In [computer graphics](graphics.md) 3D rendering is the process of computing images which represent a projected view of 3D objects through a virtual camera.
7 There are many methods and [algorithms](algorithm.md) for doing so differing in many aspects such as computation complexity, implementation complexity, realism of the result, representation of the 3D data, limitations of viewing and so on. If you are just interested in the [realtime](realtime.md) 3D rendering used in [gaymes](game.md) nowadays, you are probably interested in [GPU](gpu.md)-accelerated 3D [rasterization](rasterization.md) with [APIs](api.md) such as [OpenGL](opengl.md) and [Vulkan](vulkan.md).
9 [LRS](lrs.md) has a simple 3D rendering library called [small3dlib](small3dlib.md).
11 ## Methods
13 As most existing 3D "[frameworks](framework.md)" are [harmful](harmful.md), a [LRS](lrs.md) programmer is likely to write his own 3D rendering system that suits his program best, therefore we should list some common methods of achieving 3D. Besides that, it's just pretty interesting to see what there is in the store.
15 A very important realization of a graphics programmer is that **3D rendering is to a great extent about [faking](cheating.md)** (especially the mainstream realtime 3D) -- it is an endeavor that seeks to produce something that looks somehow familiar to HUMAN sight specifically and so even though the methods are mathematical, the endeavor is really an [art](art.md) in the end, not dissimilar to that of a magician who searches for "smoke and mirrors" [hacks](hacking.md) to produce illusions for the audience. Reality is infinitely complex, we use nothing else but [approximations](approximation.md) and neglecting that rely on assumptions about human sight such as "60 FPS looks like smooth movement to human eye", "infrared spectrum is invisible", "humans can't tell a mirror reflection is a bit off", "inner corners are usually darker than flat surfaces", "no shadow is completely black because light scatters in the atmosphere" etc. Really 3D graphics is nothing but searching for what looks [good enough](good_enough.md), and deciding this relies on a SUBJECTIVE judgement of a human (and sometimes every individual). In theory -- if we had infinitely powerful computers -- we would just program in a few lines of electromagnetic equations and run the precise simulation of light propagating in 3D environment to produce an absolutely realistic result, but though some methods try to come close to said approach, we simply won't ever have infinitely powerful computers. For this we have to resort to a bit more [ugly](ugly.md) approach of identifying specific notable real-life phenomena individually (for example [caustics](caustic.md), [Fresnel](fresnel.md), mirror reflections, refractions, [subsurface scattering](subsurface_scattering.md), metallicity, [noise](noise.md), [motion blur](motion_blur.md) and myriads of others) and addressing each one individually with special treatment, many times correcting and masking our imperfections (e.g. applying [antialiasing](antialiasing.md) because we dared to use a simplified model of light sampling, applying texture filtering because we dared to only use finite amount of memory for our data, applying [postprocessing](postprocessing.md) etc.).
17 **Rendering spectrum**: The book *Real-Time Rendering* mentions that methods for 3D rendering can be seen as lying on a spectrum, one extreme of which is *appearance reproduction* and the other *physics simulation*. Methods closer to trying to imitate the appearance try to simply focus on imitating the look of an object on the monitor that the actual 3D object would have in real life, without being concerned with *how* that look arises in real life (i.e. closer to the "faking" approach mentioned above) -- these may e.g. use image data such as photographs; these methods may rely on lightfields, photo [textures](texture.md) etc. The physics simulation methods try to replicate the behavior of light in real life -- their main goal is to solve the **[rendering equation](rendering_equation.md)**, still only more or less [approximately](approximation.md) -- and so, through internally imitating the same processes, come to similar visual results that arise in real world: these methods rely on creating 3D geometry (e.g. that made of triangles or voxels), computing light reflections and [global illumination](global_illumination.md). This is often easier to program but more computationally demanding. Most methods lie somewhere in between these two extremes: for example [billboards](billboard.md) and [particle systems](particle_system.md) may use a texture to represent an object while at the same time using 3D quads (very simple 3D models) to correctly deform the textures by perspective and solve their visibility. The classic polygonal 3D models are also usually somewhere in between: the 3D geometry and [shading](shading.md) are trying to simulate the physics, but e.g. a photo texture mapped on such 3D model is the opposite appearance-based approach ([PBR](pbr.md) further tries to shift the use of textures more towards the *physics simulation* end).
19 With this said, let's now take a look at possible **classifications** of 3D rendering methods. As seen, there are many ways:
21 - by **order**:
22 - **object order**: The method iterates on objects and draws object by object, one after another. This results in pixels being drawn to "random" places on the screen and possibly already drawn pixels being [overdrawn](overdraw.md) with new pixels (though this can be further reduced). Typically requires a [frame buffer](frame_buffer.md) and [double buffering](double_buffering.md), often also [z-buffer](z_buffer.md) (or [sorting](sorting.md)), i.e. requires a lot of memory. This method is also a bit ugly but typically also faster than the alternative, so it is prevailing nowadays.
23 - **image order**: The method iterates on screen pixels, typically going pixel by pixel from left to right, top to bottom, deciding the color of each pixel independently. May be easier to program and require less memory (no frame buffer is needed, see e.g. [frameless rendering](frameless.md)), however though [parallelism](parallelism.md) is applicable here (many pixels may potentially be independently computed in parallel, speeding up rendering), the algorithms used (e.g. [path tracing](path_tracing.md)) often have to expensively simulate light behavior and so performance is still an issue.
24 - by **speed**:
25 - **[realtime](realtime.md)**: Able to render at interactive [FPS](fps.md), typically used in [games](game.md) etc.
26 - **[offline](offline.md)**: Spends a lot of time (even many minutes) on rendering each frame with the goal to produce output of extreme quality, typically used to render 3D movies etc.
27 - by **relative limitation**:
28 - **primitive/"pseudo3D"/2.5D/...**: Older methods that produce 3D views but had great limitations e.g. in camera degrees of freedom or possible environment geometry that was usually limited to a "2D sector map" (see e.g. [Doom](doom.md)).
29 - **full/"true" 3D**: The "new" way of 3D rendering that allows freely rotating camera, arbitrary 3D geometry etc. Though this still has limitations (as any computer approximation of reality), many people just call this the "true" 3D.
30 - by **approach** (sides of above mentioned rendering spectrum):
31 - **appearance based**: Focuses on achieving desired appearance by any means necessary, faking, "cheating", not trying to stay physically correct. This is typically faster.
32 - **[physics](physics.md) simulation** (see also [physically based rendering](pbr.md)): Focuses on simulating the underlying physics of reality with high correctness so that we also get a very realistic result.
33 - by **main method/algorithm** (see also the table below):
34 - **rasterization**: Appearance based object order methods further based on a relatively simple algorithm capable of drawing (rasterizing) a simple geometric shape (such as a triangle) which we then use to draw the whole complex 3D scene (composed of great many of triangles).
35 - **ray casting/tracing**: Physics simulation image order methods further based on tracing paths of light in a manner that's closer to reality.
36 - ...
37 - by **3D data** ([vector](vector.md) vs [raster](raster.md) classification applies here just as in 2D graphics):
38 - **triangle meshes** (vector, and other boundary representations)
39 - **[voxels](voxel.md)** (raster, and potentially other volumetric representations)
40 - **[point clouds](point_cloud.md)**
41 - **[heightmaps](heightmap.md)**
42 - **[implicit surfaces](implicit_surface.md)**
43 - **smooth [surfaces](surface.md)** (e.g. [NURBS](nurbs.md))
44 - **2D sectors** (e.g. [Doom](doom.md)'s [BSP](bsp.md) "pseudo 3D" rendering)
45 - ...
46 - by **[hardware](hw.md)**:
47 - **[software rendering](sw_rendering.md)**: Rendering only with [CPU](cpu.md). This is typically slower as a CPU mostly performs sequential computation, eliminating the possible parallelism optimization, however the approach is more [KISS](kiss.md) and [portable](portablity.md).
48 - **[GPU](gpu.md) accelerated**: Making use of specialized graphics rendering hardware (GPU) that typically uses heavy parallelism to immensely speed up rendering. While this is the mainstream, extremely fast way of rendering, it is also greatly [bloated](bloat.md) while often being an [overkill](overkill.md) that greatly complicates programming and makes programs less [portable](portability.md), less [future proof](future_proof.md) etc.
49 - by **realism** of output:
50 - **[photorealistic](photorealism.md)**
51 - **stylized**, flat [shaded](shading.md), [wireframe](wireframe.md), ...
52 - ...
53 - **[hybrids](hybrid.md)**: Methods may be combined and/or lie in between different extremes, for example we may see a rasterizer 3D renderer that uses ray tracing to add detail (shadows, reflections, ...) to the scene, we may see renderers that allow triangle meshes as well as voxels etc. { One nice hybrid looking engine is e.g. [Chasm: The Rift](chasm_the_rift.md). ~drummyfish }
54 - ...
56 Finally a table of some common 3D rendering methods follows, including the most simple, most advanced and some unconventional ones. Note that here we talk about methods and techniques rather than algorithms, i.e. general approaches that are often modified and combined into a specific rendering algorithm. For example the traditional triangle rasterization is sometimes combined with raytracing to add e.g. realistic reflections. The methods may also be further enriched with features such as [texturing](texture.md), [antialiasing](antialiasing.md) and so on. The table below should help you choose the base 3D rendering method for your specific program.
58 The methods may be tagged with the following:
60 - *2.5D*: primitive 3D, often called [pseudo 3D](pseudo3d.md) or fake 3D, having significant limitations e.g. in degrees of freedom of the camera
61 - *off*: slow method usually used for offline (non-realtime) rendering (even though they indeed may run in real time e.g. with the help of powerful GPUs)
62 - *IO* vs *OO*: [image order](image_order.md) (rendering by pixels) vs [object order](object_order.md) (rendering by objects)
64 | method | notes |
65 |------------------------------------------|------------------------------------|
66 |[3D raycasting](raycasting.md) |*IO off*, shoots rays from camera |
67 |[2D raycasting](raycasting.md) |*IO 2.5D*, e.g. [Wolf3D](wolf3D.md) |
68 |[AI](ai.md) image synthesis | "just let AI magic do it" |
69 |[beamtracing](beamtracing.md) |*IO off* |
70 |[billboarding](billboard.md) |*OO* |
71 |[BSP rendering](bsp.md) |*2.5D*, e.g. [Doom](doom.md) |
72 |[conetracing](conetracing.md) |*IO off* |
73 |"[dungeon crawler](dungeon_crawler.md)" |*OO 2.5D*, e.g. Eye of the Beholder |
74 |edge list, scanline, span rasterization |*IO*, e.g. [Quake](quake.md) 1 |
75 |ellipsoid rasterization |*OO*, e.g. Ecstatica |
76 |flat-shaded 1 point perspective |*OO 2.5D*, e.g. Skyroads |
77 |reverse raytracing (photon tracing) |*OO off*, inefficient |
78 |[image based rendering](ibr.md) | generally using images as 3D data |
79 |[light fields](light_field.md) |image-based, similar to holography |
80 |[mode 7](mode7.md) |*IO 2.5D*, e.g. F-Zero |
81 |[parallax scrolling](parallax.md) |*2.5D*, very primitive |
82 |[pathtracing](pathtracing.md) |*IO off*, Monte Carlo, high realism |
83 |[portal rendering](portal_rendering.md) |*2.5D*, e.g. [Duke3D](duke3d.md) |
84 |prerendered view angles |*2.5D*, e.g. Iridion II (GBA) |
85 |[raymarching](raymaching.md) |*IO off*, e.g. with [SDFs](sdf.md) |
86 |[raytracing](raytracing.md) |*IO off*, recursive 3D raycasting |
87 |segmented road |*OO 2.5D*, e.g. Outrun |
88 |[shear warp rednering](shear_warp.md) |*IO*, volumetric |
89 |[splatting](splatting.md) |*OO*, rendering with 2D blobs |
90 |[texture slicing](texture_slicing.md) |*OO*, volumetric, layering textures |
91 |[triangle rasterization](rasterization.md)|*OO*, traditional in GPUs |
92 |[voxel space rendering](voxel_space.md) |*OO 2.5D*, e.g. Comanche |
93 |[wireframe rendering](wireframe.md) |*OO*, just lines |
95 TODO: Rescue On Fractalus!
97 TODO: find out how build engine/slab6 voxel rendering worked and possibly add it here (from http://advsys.net/ken/voxlap.htm seems to be based on raycasting)
99 TODO: VoxelQuest has some innovative voxel rendering, check it out (https://www.voxelquest.com/news/how-does-voxel-quest-work-now-august-2015-update)
101 ## 3D Rendering Basics For Nubs
103 If you're a complete noob and are asking what the essence of 3D is or just how to render simple 3Dish pictures for your game without needing a [PhD](phd.md), here's the very basics. Yes, you can use some 3D engine such as [Godot](godot.md) that has all the 3D rendering preprogrammed, but you'll surrender to [bloat](bloat.md), you won't really know what's going on and your ability to tinker with the rendering or optimizing it will be basically zero... AND you'll miss on all the [fun](fun.md) :) So here we just foreshadow some concepts you should start with if you want to program your own 3D rendering.
105 The absolute basic thing in 3D is probably **[perspective](perspective.md)**, or the concept which says that "things further away look smaller". This is basically the number one thing you need to know and with which you can make simple 3D pictures, even though there are many more effects and concepts that "make pictures look 3D" and which you can potentially study later (lighting, shadows, [focus and blur](depth_of_field.md), [stereoscopy](stereo.md), [parallax](parallax.md), visibility/obstruction etc.). { It's probably possible to make something akin "3D" even without perspective, just with [orthographic](ortho.md) projection, but that's just getting to details now. Let's just suppose we need perspective. ~drummyfish }
107 If you don't have rotating camera and other fancy things, perspective is actually mathematically very simple, you basically just **divide the object's size by its distance from the viewer**, i.e. its Z coordinate (you may divide by some multiple of Z coordinate, e.g. by 2 * Z to get different [field of view](fov.md)) -- the further away it is, the bigger number its size gets divided by so the smaller it becomes. This "dividing by distance" ultimately applies to all distances, so in the end even the details on the object get scaled according to their individual distance, but as a first approximation you may just consider scaling objects as a whole. Just keep in mind you should only draw objects whose Z coordinate is above some threshold (usually called a *near plane*) so that you don't divide by 0! With this "dividing by distance" trick you can make an extremely simple "3Dish" renderer that just draws [sprites](sprite.md) on the screen and scales them according to the perspective rules (e.g. some space simulator where the sprites are balls representing planets). There is one more thing you'll need to handle: **[visibility](visibility.md)**, i.e. nearer objects have to cover the further away objects -- you can do this by simply [sorting](sorting.md) the objects by distance and drawing them back-to-front ([painter's algorithm](painters_algorithm.md)).
109 Here is some "simple" [C](c.md) code that demonstrates perspective and draws a basic animated wireframe cuboid as ASCII in terminal:
111 ```
112 #include <stdio.h>
114 #define SCREEN_W 50 // ASCII screen width
115 #define SCREEN_H 22 // ASCII screen height
116 #define LINE_POINTS 64 // how many points for drawing a line
117 #define FOV 8 // affects "field of view"
118 #define FRAMES 30 // how many animation frames to draw
120 char screen[SCREEN_W * SCREEN_H];
122 void showScreen(void)
123 {
124 for (int y = 0; y < SCREEN_H; ++y)
125 {
126 for (int x = 0; x < SCREEN_W; ++x)
127 putchar(screen[y * SCREEN_W + x]);
129 putchar('\n');
130 }
131 }
133 void clearScreen(void)
134 {
135 for (int i = 0; i < SCREEN_W * SCREEN_H; ++i)
136 screen[i] = ' ';
137 }
139 // Draws point to 2D ASCII screen, [0,0] means center.
140 int drawPoint2D(int x, int y, char c)
141 {
142 x = SCREEN_W / 2 + x;
143 y = SCREEN_H / 2 + y;
145 if (x >= 0 && x < SCREEN_W && y >= 0 && y <= SCREEN_H)
146 screen[y * SCREEN_W + x] = c;
147 }
149 // Divides coord. by distance taking "FOV" into account => perspective.
150 int perspective(int coord, int distance)
151 {
152 return (FOV * coord) / distance;
153 }
155 void drawPoint3D(int x, int y, int z, char c)
156 {
157 if (z <= 0)
158 return; // at or beyond camera, don't draw
160 drawPoint2D(perspective(x,z),perspective(y,z),c);
161 }
163 int interpolate(int a, int b, int n)
164 {
165 return a + ((b - a) * n) / LINE_POINTS;
166 }
168 void drawLine3D(int x1, int y1, int z1, int x2, int y2, int z2, char c)
169 {
170 for (int i = 0; i < LINE_POINTS; ++i) // draw a few points to form a line
171 drawPoint3D(interpolate(x1,x2,i),interpolate(y1,y2,i),interpolate(z1,z2,i),c);
172 }
174 int main(void)
175 {
176 int shiftX, shiftY, shiftZ;
178 #define N 12 // side length
179 #define C '*'
181 // cuboid points:
182 // X Y Z
183 #define PA -2 * N + shiftX, N + shiftY, N + shiftZ
184 #define PB 2 * N + shiftX, N + shiftY, N + shiftZ
185 #define PC 2 * N + shiftX, N + shiftY, 2 * N + shiftZ
186 #define PD -2 * N + shiftX, N + shiftY, 2 * N + shiftZ
187 #define PE -2 * N + shiftX, -N + shiftY, N + shiftZ
188 #define PF 2 * N + shiftX, -N + shiftY, N + shiftZ
189 #define PG 2 * N + shiftX, -N + shiftY, 2 * N + shiftZ
190 #define PH -2 * N + shiftX, -N + shiftY, 2 * N + shiftZ
192 for (int i = 0; i < FRAMES; ++i) // render animation
193 {
194 clearScreen();
196 shiftX = -N + (i * 4 * N) / FRAMES; // animate
197 shiftY = -N / 3 + (i * N) / FRAMES;
198 shiftZ = 0;
200 // bottom:
201 drawLine3D(PA,PB,C); drawLine3D(PB,PC,C); drawLine3D(PC,PD,C); drawLine3D(PD,PA,C);
203 // top:
204 drawLine3D(PE,PF,C); drawLine3D(PF,PG,C); drawLine3D(PG,PH,C); drawLine3D(PH,PE,C);
206 // sides:
207 drawLine3D(PA,PE,C); drawLine3D(PB,PF,C); drawLine3D(PC,PG,C); drawLine3D(PD,PH,C);
209 drawPoint3D(PA,'A'); drawPoint3D(PB,'B'); // corners
210 drawPoint3D(PC,'C'); drawPoint3D(PD,'D');
211 drawPoint3D(PE,'E'); drawPoint3D(PF,'F');
212 drawPoint3D(PG,'G'); drawPoint3D(PH,'H');
214 showScreen();
216 puts("press key to animate");
217 getchar();
218 }
220 return 0;
221 }
222 ```
224 One frame of the animation should look like this:
226 ```
227 E*******************************F
228 * * *** *
229 * ** *** *
230 * H***************G* *
231 * * * *
232 * * * *
233 * * * *
234 * * * *
235 * * * *
236 * * * *
237 * D***************C *
238 * ** *** *
239 * * * *
240 * * ** *
241 *** * * *
242 A*******************************B
244 press key to animate
245 ```
247 **PRO TIP**: It will also help if you learn a bit about [photography](photo.md) because 3D usually tries to simulate [cameras](camera.md) and 3D programmers adopt many terms and concepts from photography. At least learn the very basics such as [focal length](focal_length.md), [pinhole camera](pinhole_camera.md), the "exposure triangle" (shutter speed, [aperture](aperture.md), [ISO](iso.md)) etc. You should know how focal length is related to [FOV](fov.md), what the "f number" means, how to use exposure settings to increase or decrease things like [motion blur](motion_blur.md) and [depth of field](depth_of_field.md), what [HDR](hdr.md) means etc.
249 ## Mainstream Realtime 3D
251 You may have come here just to learn about the typical realtime 3D rendering used in today's [games](game.md) because aside from research and niche areas this kind of 3D is what we normally deal with in practice. This is what this section is about.
253 These days "game 3D" means a [GPU](gpu.md) accelerated 3D [rasterization](rasterization.md) done with rendering [APIs](api.md) such as [OpenGL](opengl.md), [Vulkan](vulkan.md), [Direct3D](d3d.md) or [Metal](metal.md) (the last two being [proprietary](proprietary.md) and therefore [shit](shit.md)) and higher level engines above them, e.g. [Godot](godot.md), [OpenSceneGraph](osg.md) etc. The methods seem to be evolving to some kind of rasterization/[pathtracing](pathtracing.md) hybrid, but rasterization is still the basis.
255 This mainstream rendering uses an [object order](object_order.md) approach (it blits 3D objects onto the screen rather than determining each pixel's color separately) and works on the principle of **triangle rasterization**, i.e. 3D models are composed of triangles (or higher polygons which are however eventually broken down into triangles) and these triangles are projected onto the screen according to the position of the virtual camera and laws of [perspective](perspective.md). Projecting the triangles means finding the 2D screen coordinates of each of the triangle's three vertices -- once we have thee coordinates, we draw (rasterize) the triangle to the screen just as a "normal" 2D triangle (well, with some asterisks).
257 Additionally things such as [z-buffering](z_buffer.md) (for determining correct overlap of triangles) and [double buffering](double_buffering.md) are used, which makes this approach very memory ([RAM](ram.md)/[VRAM](vram.md)) expensive -- of course mainstream computers have more than enough memory but smaller computers (e.g. [embedded](embedded.md)) may suffer and be unable to handle this kind of rendering. Thankfully it is possible to adapt and imitate this kind of rendering even on "small" computers -- even those that don't have a GPU, i.e. with pure [software rendering](sw_rendering.md). For this we e.g. replace z-buffering with [painter's algorithm](painters_algorithm.md) (triangle sorting), drop features like [perspective correction](perspective_correction.md), [MIP mapping](mip_mapping.md) etc. (of course quality of the output will go down).
259 Also additionally there's a lot of [bloat](bloat.md) added in such as complex [screen space](screen_space.md) shaders, [pathtracing](pathtracing.md) (popularly known as *raytracing*), [megatexturing](megatexturing.md), [shadow rendering](shadow.md), [postprocessing](postprocessing.md), [compute shaders](compute_shader.md) etc. This may make it difficult to get into "modern" 3D rendering. Remember to [keep it simple](kiss.md).
261 On PCs the whole rendering process is hardware-accelerated with a [GPU](gpu.md) (graphics card). GPU is a special hardware capable of performing many operations in [parallel](parallelism.md) (as opposed to a [CPU](cpu.md) which mostly computes sequentially with low level of parallelism) -- this is ideal for graphics because we can for example perform mapping and drawing of many triangles at once, greatly increasing the speed of rendering ([FPS](fps.md)). However this hugely increases the [complexity](complexity.md) of the whole rendering system, we have to have a special [API](api.md) and [drivers](driver.md) for communication with the GPU and we have to upload data (3D models, textures, ...) to the GPU before we want to render them. [Debugging](debugging.md) gets a lot more difficult. So again, this is [bloat](bloat.md), consider avoiding GPUs.
263 GPUs nowadays are no longer just focusing on graphics, but are kind of a general device that can be used for more than just 3D rendering (e.g. [crypto](crypto.md) mining, training [AI](ai.md) etc.) and can no longer even perform 3D rendering completely by themselves -- for this they have to be programmed. I.e. if we want to use a GPU for rendering, not only do we need a GPU but also some extra code. This code is provided by "systems" such as [OpenGL](opengl.md) or [Vulkan](vulkan.md) which consist of an [API](api.md) (an [interface](interface.md) we use from a [programming language](programming_language.md)) and the underlying implementation in a form of a [driver](driver.md) (e.g. [Mesa3D](mesa3d.md)). Any such rendering system has its own architecture and details of how it works, so we have to study it a bit if we want to use it.
265 The important part of a system such as OpenGL is its **rendering [pipeline](pipeline.md)**. Pipeline is the "path" through which data go through the rendering process. Each rendering system and even potentially each of its version may have a slightly different pipeline (but generally all mainstream pipelines somehow achieve rasterizing triangles, the difference is in details of how they achieve it). The pipeline consists of **stages** that follow one after another (e.g. the mentioned mapping of vertices and drawing of triangles constitute separate stages). A very important fact is that some (not all) of these stages are programmable with so called **[shaders](shader.md)**. A shader is a program written in a special language (e.g. [GLSL](glsl.md) for OpenGL) running on the GPU that processes the data in some stage of the pipeline (therefore we distinguish different types of shaders based on at which part of the pipeline they reside). In early GPUs stages were not programmable but they became so as to give a greater flexibility -- shaders allow us to implement all kinds of effects that would otherwise be impossible.
267 Let's see what a typical pipeline might look like, similarly to something we might see e.g. in OpenGL. We normally simulate such a pipeline also in [software renderers](sw_rendering.md). Note that the details such as the coordinate system [handedness](handedness.md) and presence, order, naming or programmability of different stages will differ in any particular pipeline, this is just one possible scenario:
269 1. Vertex data (e.g. 3D [model space](model_space.md) coordinates of triangle vertices of a 3D model) are taken from a vertex buffer (a GPU memory to which the data have been uploaded).
270 2. **Stage: [vertex shader](vertex_shader.md)**: Each vertex is processed with a vertex shader, i.e. one vertex goes into the shader and one vertex (processed) goes out. Here the shader typically maps the vertex 3D coordinates to the screen 2D coordinates (or [normalized device coordinates](ndc.md)) by:
271 - multiplying the vertex by a [model matrix](model_matrix.md) (transforms from [model space](model_space.md) to [world space](world_space.md), i.e. applies the model move/rotate/scale operations)
272 - multiplying by [view matrix](view_matrix.md) (transforms from [world space](world_space.md) to [camera space](camera_space.md), i.e. takes into account camera position and rotation)
273 - multiplying by [projection matrix](projection_matrix.md) (applies perspective, transforms from [camera space](camera_space.md) to [screen space](screen_space.md) in [homogeneous coordinates](homogeneous_coordinates.md))
274 3. Possible optional stages that follow are [tessellation](tessellation.md) and geometry processing ([tessellation shaders](tessellation_shader.md) and [geometry shader](geometry_shader.md)). These offer possibility of advanced vertex processing (e.g. generation of extra vertices which vertex shaders are unable to do).
275 4. **Stage: vertex post processing**: Usually not programmable (no shaders here). Here the GPU does things such as [clipping](clipping.md) (handling vertices outside the screen space), primitive assembly and [perspective divide](perspective_divide.md) (transforming from [homogeneous coordinates](homogeneous coordinates.md) to traditional [cartesian coordinates](cartesian_coordinates.md)).
276 5. **Stage: [rasterization](rasterization.md)**: Usually not programmable, the GPU here turns triangles into actual [pixels](pixel.md) (or [fragments](fragment.md)), possibly applying [backface culling](backface_culling.md), [perspective correction](perspective_correction.md) and things like [stencil test](stencil_test.md) and [depth test](dept_test.md) (even though if fragment shaders are allowed to modify depth, this may be postpones to later).
277 6. **Stage: pixel/fragment processing**: Each pixel (fragment) produced by rasterization is processed here by a [pixel/fragment shader](fragment_shader.md). The shader is passed the pixel/fragment along with its coordinates, depth and possibly other attributes, and outputs a processed pixel/fragment with a specific color. Typically here we perform [shading](shading.md) and [texturing](texture.md) (pixel/fragment shaders can access texture data which are again stored in texture buffers on the GPU).
278 7. Now the pixels are written to the output buffer which will be shown on screen. This can potentially be preceded by other operations such as depth tests, as mentioned above.
280 ### Complete Fucking Detailed Example Of Rendering A 3D Model By Hand
282 WORK IN PROGRESS
284 { This turned out to be long as hell, sowwy. ~drummyfish }
286 This is an example of how two very simple 3D models would be rendered using the traditional triangle rasterization pipeline. Note that this is VERY simplified, it's just to give you an idea of the whole process, BUT if you understand this you will basically get an understanding of it all.
288 Keep in mind this all can be done just with [fixed point](fixed_point.md), [floating point](float.md) is NOT required.
290 First we need to say what conventions we'll stick to:
292 - We'll be using ROW VECTORS, i.e. we'll be writing vectors like [x,y,z]. Some people rather use column vectors, which then means their matrices are also transposed and they do multiplication in opposite direction etcetc. Watch out about this, it's quite important to know which convention you're using, because e.g. matrix multiplication is non-commutative (i.e. with matrices A * B does NOT generally equal B * A) and the order you need to multiply in depends on this convention, so be careful.
293 - Counterclockwise triangles are front facing, clockwise ones are back facing (invisible).
294 - We'll be using [LEFT](left_right.md) HANDED coordinate systems, i.e X axis goes right, Y goes up, Z goes forward (right handed system would be the same except Z would go the opposite way -- backwards). Watch out: some systems (for example OpenGL) use the other one. I.e. our coordinate system looks like this:
296 ```
297 Y ^ _
298 | _/| Z
299 | _/
300 |_/
301 '-------> X
302 ```
304 Now let's have a simple **3D model data** of a quad. Quad is basically just a square made of four vertices and two triangles, it will look like this:
306 *quadModel*:
308 v3________v2
309 | _/ |
310 | _/ |
311 | _/ |
312 |/_______|
313 v0 v1
315 In a computer this is represented with two arrays: vertices and triangles. Our vertices here are (notices all Z coordinates are zero, i.e. it is a 3D model but it's flat):
317 *quadVertices*:
319 ```
320 v0 = [-1, -1, 0]
321 v1 = [ 1, -1, 0]
322 v2 = [ 1, 1, 0]
323 v3 = [-1, 1, 0]
324 ```
326 And our triangles are (they are indices to the vertex array, i.e. each triangle says which three vertices from the above array to connect to get the triangle):
328 *quadTriangles*:
330 ```
331 t0 = [0,1,2]
332 t1 = [0,2,3]
333 ```
335 Note the triangles here (from our point of view) go counterclockwise -- this is called *winding* and is usually important because of so called **backface culling** -- the order of vertices basically determines which is the front side of the triangle and which is the back side, and rendering systems often just draw the front sides for efficiency (back faces are understood to be on the inside of objects and invisible).
337 Now the vertex coordinates of the model above are in so called **model space** -- these are the coordinates that are stored in the 3D model's file, it's the model's "default" state of coordinates. The concept of different spaces is also important because 3D rendering is a lot about just transforming coordinates between different spaces ("frames of reference"). We'll see that the coordinates in model space will later on be transformed to world space, view space, clip space, screen space etc.
339 OK, so next let's have 2 actual **3D model instances** that use the above defined 3D model data. Notice the difference between 3D model DATA and 3D model INSTANCE: instance is simply one concrete, specific model that has its own place in the global 3D world (world space) and will be rendered, while data is just numbers in memory representing some 3D geometry etc. There can be several instances of the same 3D data (just like in [OOP](oop.md) there can be multiple instances/objects of a class); this is very efficient because there can be just one 3D data (like a model of a car) and then many instances of it (like many cars in a virtual city). Our two model instances will be called *quad0* and *quad1*.
341 Each model instance has its own **transformation**. Transformation says where the model is placed, how it's rotated, how it's scaled and so on -- different 3D engines may offer different kind of transformations, some may support things like flips, skews, non-uniform scaling, but usually at least three basic transforms are supported: translation (AKA offset, shift, position), rotation and scale. The transforms can be combined, e.g. a model can be shifted, rotated and scaled at the same time. Here we'll just rotate *quad0* by 45 degrees (pi/4 radians) around Y (vertical) axis and translate *quad1* one unit to the right and 2 back:
343 *quad0*:
345 - translation = [0,0,0]
346 - rotation = [0,pi/4,0]
348 *quad1*:
350 - translation = [1,0,2]
351 - rotation = [0,0,0]
353 So now we have two model instances in our world. For rendering we'll also need a **camera** -- the virtual window to our world. Camera is also a 3D object: it doesn't have 3D model data associated but it does have a transformation so that we can, naturally, adjust the view we want to render. Camera also has additional properties like field of view (FOV), aspect ratio (we'll just consider 1:1 here), near and far distances and so on. Our camera will just be shifted two units back (so that it can see the first quad that stays at position [0,0,0]):
355 *camera*:
357 - translation = [0,0,-2]
358 - rotation = [0,0,0]
360 It is important to mention the **near and far planes**. Imagine a camera placed at some point in space, looking in certain direction: the volume of space that it sees creates a kind of infinitely long pyramid (whose "steepness" depends on the camera field of view) with its tip at the point where the camera resides. Now for the purpose of rendering we define two planes, perpendicular to the camera's viewing direction, that are defined by the distance from the camera: the near plane (not surprisingly the one that's the nearer of the two) and the far plane. For example let's say our camera will have the near plane 1 unit in front of it and the far plane 5 units in front of it. These planes will CUT OFF anything that's in front and beyond them, so that only things that are BETWEEN the two planes will be rendered (you can notice this in games with render distance -- if this is not cleverly masked, you'll see things in the distance suddenly cut off from the view). These two planes will therefore CUT OFF the viewing pyramid so that now it's a six sided, finite-volume shape that looks like a box with the front side smaller than the back side. This is called the **view frustum**. Nothing outside this frustum will be rendered -- things will basically be sliced by the sides of this frustum.
362 You may ask WHY do we establish this frustum? Can't we just leave the near and far planes out and render "everything"? Well, firstly it's obvious that having a far cutoff view distance can help performance if you have a very complex model, but this is not the main reason why we have near and far planes. We basically have them for mathematical convenience -- as we'll see for example, perspective mapping means roughly "dividing by distance from camera" and if something was to be exactly where the camera is, we'd be dividing by zero! Attempting to render things that are just very near or on the back side of the camera would also do very nasty stuff. So that's why we have the near plane. In theory we might kind of get away with not having a strict far plane but it basically creates a nice finite-volume that will e.g. allow us to nicely map depth values for the z-buffer. Don't worry if this doesn't make much sense, this is just to say there ARE good reasons for this stuff.
364 Now let's summarize what we have with this top-down view of our world (the coordinates here are now *world space*):
366 ```
367 Z
368 :
369 - - - - - - - 3 + - - - - - - - - far plane
370 :
371 :
372 2 +-----*------ quad2
373 :
374 :
375 1 +
376 : quad1
377 0 : _/
378 .|.....|.....*/....|.....|. X
379 -2 -1 _/: 1 2
380 . / : .
381 - - - - -'.=====:=====.'- - - - - near plane
382 '. : .'
383 '. : .'
384 -2 '*' camera
385 :
386 :
387 ```
389 NOW actually let's see how to in fact render this. The big picture overview is this:
391 1. Get the model instances from model space to world space, i.e. transform their vertex coordinates according to each instance's transformation.
392 2. Get the model instances from world space to view space (AKA camera space). View space is the coordinate system of the camera in which the camera sits in the origin (poinr [0,0,0]) and is looking straight forward along the positive Z axis direction.
393 3. Get the model instances from view space to clip space. This applies perspective (deforms objects so that the further away ones become smaller as by distance) and transform everything in the view frustum to a nice cube of fixed size and with walls aligned with principal axes (unlike view frustum itself).
394 4. Clip everything outside the clip space, i.e. throw away everything outside the box we've got. If some things (triangles) are partially in and partially out, we CLIP them, i.e. we literally cut them into several parts and throw away the parts that aren't in (some simpler renderers just do simpler stuff like throw away anything that sticks outside or just force push the vertices inside but it will look a bit awkward).
395 5. Get everything from clip space into screen space, i.e. to actual pixel coordinates of the screen we are rendering to.
396 6. Rasterize the triangles of the models between the points we have mapped to the screen now, i.e. literally fill all the triangle pixels so that we get the final image.
398 As seen this involves doing many transformations between different spaces. We do these using **[linear algebra](linear_algebra.md)**, i.e. with **[vectors](vector.md)** and **[matrices](matrix.md)**. Key things here are:
400 - We can represent all the transformations that we need using matrices.
401 - Every such transformation (translation, rotation, scale, ...) can be represented with one 4x4 matrix.
402 - To apply a transformation to a model we simply multiply each of its vertices with the transformation matrix. So if we want to rotate a model by 30 degrees, we make a matrix that represents this rotation and just multiply all the model's vertices and it's done. Pretty elegant!
403 - AMAZING stuff: any number of these transformations combined in ANY order can still be represented just by a single 4x4 matrix! You just take the matrix of each of the transformations, multiply them together and you get a matrix that just does all these transformation at once. Duuuude what? Yeah that's right, this is extremely awesome, isn't it? We can basically create a single matrix that combines in it all the above mentioned rendering steps and it will just do everything. This is not only elegant but also very efficient (instead of just moving, rotating and scaling points there and back many times we simply perform ONE matrix multiplication for each vertex and that's it).
405 You HAVE TO learn how to multiply vector with matrix and matrix with matrix (it's not hard) else you will understand nothing now.
407 BIG BRAIN MOMENT: **[homogeneous coordinates](homogeneous_coordinates.md)**. Please DO NOT ragequit, it looks complicated as hell (it is a little bit) but it makes sense in the end, OK? We have to learn what homogeneous coordinates are because we need them to be able to do all the awesome matrix stuff described above. In essence: in 3D space we can perform linear transformations with 3x3 matrices -- linear operations are for example scaling and rotation, BUT some, most importantly translation (shifting and object, which we absolutely NEED), are not linear (but rather [affine](affine.md)) so they cannot be performed by a 3x3 matrix. But it turns out that if we use special kind of coordinates, we CAN do affine 3D transformations with 4x4 matrices, OK? These special coordinates are homogeneous coordinates, and they simply add one extra coordinate, *w*, to the original *x*, *y* and *z*, while it holds that that multiplying all the *x*, *y*, *z* and *w* components by the same number does nothing with the point they represent. Let's show it like this:
409 If we have a 3D point [1,2,3], in homogeneous coordinates we can represent it as [1,2,3,1] or [2,4,6,2] or [3,6,9,3] and so on. That's easy no? So we will ONLY add an additional 1 at the end of our vertex coordinates and that's basically it.
411 Let's start doing this now!
413 Firstly let us transform *quad0* from model space to world space. For this we construct so called **model matrix** based on the transformation that the model instance has. Our *quad0* is just rotated by pi/4 radians and for this the matrix will look like this (you don't have to know why, you usually just look up the format of the matrix somewhere, but you can derive it, it's EZ):
415 *quad0* model matrix:
417 ```
418 | cos(A) 0 sin(A) 0| |0.7 0 0.7 0|
419 Mm0 = | 0 1 0 0| ~= |0 1 0 0|
420 |-sin(A) 0 cos(A) 0| |-0.7 0 0.7 0|
421 | 0 0 0 1| |0 0 0 1|
422 ```
424 Let's see if this works, we'll try to multiply the first model vertex with this matrix (notice we add 1 at the end of the vertex, to convert it to homogeneous coordinates):
426 ```
427 |0.7 0 0.7 0|
428 |0 1 0 0|
429 |-0.7 0 0.7 0|
430 |0 0 0 1|
432 v0 * Mm0 = [-1, -1, 0, 1] [-0.7 -1 -0.7 1] <-- result
433 ```
435 So from [-1,-1,0] we got [-0.7,-1,-0.7] -- looking at the top-down view picture above this seem pretty correct (look at the coordinates of the first vertex). Try it also for the rest of the vertices. Now for the model matrix of *quad1* (again, just look up what translation matrix looks like):
437 *quad1* model matrix:
439 ```
440 |1 0 0 0|
441 Mm1 = |0 1 0 0|
442 |0 0 1 0|
443 |1 0 2 1|
444 ```
446 Here you can even see that multiplying a vector by this will just add 1 to *x* and 2 to *z*, right? Again, try it.
448 NEXT, the **view matrix** (matrix that will transform everything so that it's "in front of the camera") will basically just do the opposite transformation of that which the camera has. Imagine if you shift a camera 1 unit to the right -- that's as if the camera stands still and everything shifts 1 unit to the left. Pretty logical. So our view matrix looks like this (notice it just pushes everything by 2 to the front):
450 *view matrix*:
452 ```
453 |1 0 0 0|
454 Mv = |0 1 0 0|
455 |0 0 1 0|
456 |0 0 2 1|
457 ```
459 Then we'll need to apply perspective and get everything to the clip space. This will be done with so called **projection matrix** which will in essence make the *x* and *y* distances be divided by the *z* distance so that further away things will shrink and appear smaller. You can derive the view matrix, its values depend on the field of view, near and far plane etc., here we'll just copy paste numbers into a "template" for the projection matrix, so here it is:
461 *projection matrix* (*n* = near plane distance, *f* = far plane distance, *r* = right projection plane distance = 1, *t* = top projection plane distance = 1):
463 ```
464 |n/r 0 0 0| |1 0 0 0|
465 Mp = |0 n/t 0 0| = |0 1 0 0|
466 |0 0 (f+n)/(f-n) 1| |0 0 3/2 1|
467 |0 0 -2*f*n/(f-n) 0| |0 0 -5/2 0|
468 ```
470 This matrix will basically make the points so that their *w* coordinates will be such that when in the end we divide all components by it (to convert them back from [homo](gay.md)geneous coordinates), we'll get the effect of the perspective (it's basically the "dividing by distance from the camera" that perspective does). That is what the homogeneous coordinates allow us to do. To visually demonstrate this, here is a small picture of how it reshapes the view frustum into the clip space box (it kind of "squishes" all in the back of the frustum pyramid and also squeezes everything to shape it into that little box of the clipping space, notice how the further away objects became closer together -- that's the perspective):
472 ```
473 ___________________ far plane _____________
474 \ A B C / | ABC |
475 \ / | |
476 \ D E F / | D E F |
477 \ / | |
478 \G H I/ |G H I|
479 \_______/ near plane |_____________|
480 : : : :
481 : : : screen :
482 : : : :
483 * camera
484 ```
486 At this point we have the matrices of the individual transforms, but as we've said, we can combine them into a single matrix. First let's combine the view matrix and projection matrix into a single view-projection matrix by multiplying the two matrices (WATCH OUT: the order of multiplication matters here! It defines in which order the transformations are applied):
488 *view-projection matrix*:
490 ```
491 |1 0 0 0|
492 Mvp = Mv * Mp = |0 1 0 0|
493 |0 0 3/2 1|
494 |0 0 1/2 2|
495 ```
497 The rendering will begin with *quad0*, we'll combine its model matrix and the view-projection matrix into a single uber matrix that will just do the whole transformation for this model instance:
499 *quad0 model-view-projection matrix*:
501 ```
502 |0.7 0 21/20 0.7|
503 Mm0vp = Mm0 * Mvp = |0 1 0 0 |
504 |-0.7 0 21/20 0.7|
505 |0 0 1/2 2 |
506 ```
508 Now we'll just transform all of the model's vertices by multiplying with this matrix, and then we'll convert back from the homogeneous coordinates to "normal" coordinates by dividing all components by *w* (AKA "perspective divide") like this:
510 ```
511 v0: [-1, -1, 0, 1] (matrix multiplication) => [-0.7, -1, -0.55, 1.3] (w divide) => [-0.53, -0.76, -0.43]
512 v1: [ 1, -1, 0, 1] (matrix multiplication) => [ 0.7, -1, 1.55, 2.7] (w divide) => [ 0.26, -0.37, 0.57]
513 v2: [ 1, 1, 0, 1] (matrix multiplication) => [ 0.7, 1, 1.55, 2.7] (w divide) => [ 0.26, 0.37, 0.57]
514 v3: [-1, 1, 0, 1] (matrix multiplication) => [-0.7, 1, -0.55, 1.3] (w divide) => [-0.53, 0.76, -0.43]
515 ```
517 And let's also do this for *quad1*.
519 *quad1 model-view-projection matrix*:
521 ```
522 |1 0 0 0|
523 Mm1vp = Mm1 * Mvp = |0 1 0 0|
524 |0 0 3/2 1|
525 |0 0 7/2 4|
526 ```
528 and
530 ```
531 v0: [-1, -1, 0, 1] (matrix multiplication) => [-1, -1, 3.5, 4] (w divide) => [-0.25, -0.25, 0.87]
532 v1: [ 1, -1, 0, 1] (matrix multiplication) => [ 1, -1, 3.5, 4] (w divide) => [ 0.25, -0.25, 0.87]
533 v2: [ 1, 1, 0, 1] (matrix multiplication) => [ 1, 1, 3.5, 4] (w divide) => [ 0.25, 0.25, 0.87]
534 v3: [-1, 1, 0, 1] (matrix multiplication) => [-1, 1, 3.5, 4] (w divide) => [-0.25, 0.25, 0.87]
535 ```
537 Hmmm mkay let's draw the transformed points to an X/Y grid:
539 ```
540 Y
541 |
542 [-1,1]|_______________:_______________[1,-1]
543 | |
544 | v3 +--.... |
545 | | '''---+ v2 |
546 | | .'| |
547 | |v3 +-----:--|+ v2 |
548 | | | .' ..|| |
549 --| | | :..' || |--
550 | | | '' || |
551 | |v0 +'-------|+ v1 |
552 | | : | |
553 | |.' ...---+ v1 |
554 | v0 +--'''' |
555 |_______________________________|___X
556 [-1,-1] : [1,-1]
557 ```
559 HOLY SHIT IT'S 3D!!1! [Magic](magic.md)! In the front we see *quad0*, rotated slightly around the vertical (Y) axis, behind it is *quad1*, non-rotated but smaller because it's further away. This looks very, very good! We're almost there.
561 Also notice that the points -- now nicely projected onto a 2D X/Y plane -- still have 3 coordinates, i.e. they retain the Z coordinate which now holds their depth, or distance from the camera projection plane kind of. This depth is now in range from -1 (near plane) to 1 (far plane). The depth will be important in actually drawing pixels, to decide which are more in the front and therefore visible (this is the problem of *visibility*). The depth value can also be used for cool effects like the distance fog and so on.
563 The work up until now -- i.e. transforming the vertices with matrices -- is what **vertex shaders** do. Now comes the **rasterization** part -- here we literally draw triangles (as in individual pixels) between the points we have now mapped on the screen. In systems such as OpenGL This is usually done automatically, you don't have to care about rasterization, however you will have to write the algorithm if you're writing e.g. your own [software renderer](sw_rendering.md). Triangle rasterization isn't trivial, it has to be not only efficient but also deal with things such as not leaving holes between adjacent triangles, interpolating triangle attributes and so on. We won't dive deeper into this, let's just suppose we have a rasterization algorithm now. For example rasterizing the first triangle of *quad0* may look like this:
565 ```
566 _______________________________
567 | |
568 | +--.... |
569 | | '''---# |
570 | | .## |
571 | | +-----####+ |
572 | | | .#####| |
573 | | | :######| |
574 | | | ########| |
575 | | ##########+ |
576 | | ########### |
577 | | ############ |
578 | ####### |
579 |_______________________________|
580 ```
582 During this rasterization process the Z coordinates of the mapped vertices are important -- the rasterizer [interpolates](interpolation.md) the depth at the vertices, so it knows the depth of each pixel it creates. This depth is written into so called **z-buffer** (AKA depth buffer) -- basically an off-screen buffer that stores one numeric value (the depth) for each pixel. Now when let's say the first triangle of *quad1* starts to be rasterized, the algorithm compares the rasterized pixel's depth to that stored on the same position in the z-buffer -- if the z-buffer value is small, the new pixel mustn't be drawn because it's covered by a previous drawn pixel already (here probably that of the triangle shown in the picture).
584 So the rasterization algorithm just shits out individual pixels and hands them over to the **fragment shader** (AKA pixel shader). Fragment is a program that just takes a pixel and says what color it should have (basically) -- this is called **[shading](shading.md)**. For this the rasterizer also hands over additional info to the fragment shader which may include: the X/Y coordinates of the pixel, its interpolated depth (that used in z-buffer), vertex normals, ID of the model and triangle and, very importantly, the **[barycentric coordinates](barycentric.md)**. These are three-component coordinates that say where exactly the pixel is in the triangle. These are used mainly for **texturing**, i.e. if the model we're rendering has a texture map (so called UV map) and a bitmap image (texture), the fragment shader will use the UV map and barycentric coords to compute the exact pixel of the texture that the rasterized pixel falls onto AND this will be the pixel's color. Well, not yet actually, there are more things such as **lighting**, i.e. determining what brightness the pixel should have depending on how the triangle is angled towards scene lights (for which we need the normals), how far away from them it is, what colors the lights have etcetc. And this is not nearly all, there are TONS and tons of other things, for example the interpolation done by rasterizer has to do **perspective correction** (linearly interpolating in screen space looks awkward), then there is texture filtering to prevent [aliasing](aliasing.md) (see e.g. [mipmapping](mipmap.md), transparency, effects like bump mapping, environment mapping, screen space effects, stencil buffer etcetc. -- you can read whole books about this. That's beyond the scope of this humble tutorial -- in simple renderers you can get away with ignoring a lot of this stuff, you can just draw triangles filled with constant color, or even just draw lines to get a wireframe renderer, all is up to you. But you can see it is a bit [bloated](bloat.md) if everything is to be done correctly -- don't forget there also exist other ways of rendering, see for example [raytracing](raytracing.md) which is kind of easier.
586 ## See Also
588 - [3d modeling](3d_model.md)
589 - [software rendering](sw_rendering.md)
590 - [autostereogram](autostereogram.md)