How do Graphics Cards Work? Exploring GPU Architecture

How many calculations do you think your graphics card performs every second while running video games with incredibly realistic graphics? Maybe 100 million? Well, 100 million calculations a second is what’s required to run Mario 64 from 1996. We need more power. Maybe 100 billion calculations a second? Well, then you would have a computer that could run Minecraft back in 2011. In order to run the most realistic video games such as Cyberpunk 2077 you need a graphics card that can perform around 36 trillion calculations a second. This is an unimaginably large number, so let’s take a second to try to conceptualize it. Imagine doing a long multiplication problem once every second. Now let’s say everyone on the planet does a similar type of calculation but with different numbers. To reach the equivalent computational power of this graphics card and its 36 trillion calculations a second we would need about 4,400 Earths filled with people, all working together and completing one calculation each every second. It’s rather mind boggling to think that a device can manage all these calculations, so in this video we’ll see how graphics cards work in two parts. First, we’ll open up this graphics card and explore the different components inside, as well as the physical design and architecture of the GPU or graphics processing unit. Second, we’ll explore the computational architecture and see how GPUs process mountains of data, and why they’re ideal for running video game graphics, Bitcoin mining, neural networks and AI. So, stick around and let’s jump right in. This video is sponsored by Micron which manufactures the graphics memory inside this graphics card. Before we dive into all the parts of the GPU, let’s first understand the differences between GPUs and CPUs. Inside this graphics card, the Graphics Processing Unit or GPU has over 10,000 cores. However, when we look at the CPU or Central Processing Unit that’s mounted to the motherboard, we find an integrated circuit or chip with only 24 cores. So, which one is more powerful? 10 thousand is a lot more than 24, so you would think the GPU is more powerful, however, it’s more complicated than that. A useful analogy is to think of a GPU as a massive cargo ship and a CPU as a jumbo jet airplane. The amount of cargo capacity is the amount of calculations and data that can be processed, and the speed of the ship or airplane is the rate at which how quickly those calculations and data are being processed. Essentially, it’s a trade-off between a massive number of calculations that are executed at a slower rate versus a few calculations that can be performed at a much faster rate. Another key difference is that airplanes are a lot more flexible since they can carry passengers, packages, or containers and can take off and land at any one of tens of thousands of airports. Likewise CPUs are flexible in that they can run a variety of programs and instructions. However, giant cargo ships carry only containers with bulk contents inside and are limited to traveling between ports. Similarly, GPUs are a lot less flexible than CPUs and can only run simple instructions like basic arithmetic. Additionally GPUs can’t run operating systems or interface with input devices or networks. This analogy isn’t perfect, but it helps to answer the question of “which is faster, a CPU or a GPU? ”. Essentially if you want to perform a set of calculations across mountains of data, then a GPU will be faster at completing the task. However, if you have a lot less data that needs to be evaluated quickly than a CPU will be faster. Furthermore, if you need to run an operating system or support network connections and a wide range of different applications and hardware, then you’ll want a CPU. We’re planning a separate video on CPU architecture, so make sure to subscribe so you don’t miss it, but let’s now dive into this graphics card and see how it works. In the center

of this graphics card is the printed circuit board or PCB, with all the various components mounted on it, [Animator Note: Highlight and list out the various parts that will be covered. ] and we’ll start by exploring the brains which is the graphics processing unit or GPU. When we open it up, we find a large chip or die named GA102 built from 28. 3 billion transistors. The majority of the area of the chip is taken up by the processing cores which have a hierarchical organization. Specifically, the chip is divided into 7 Graphics Processing Clusters or GPCs, and within each processing cluster are 12 streaming multiprocessors or SMs. Next, inside each of these streaming multiprocessors are 4 warps and 1 ray tracing core, and then, inside each warp are 32 Cuda or shading cores and 1 tensor core. Across the entire GPU are 10752 CUDA cores, 336 Tensor Cores, and 84 Ray Tracing Cores. These three types of cores execute all the calculations of the GPU, and each has a different function. CUDA cores can be thought of as simple binary calculators with an addition button, a multiply button and a few others, and are used the most when running video games. Tensor cores are matrix multiplication and addition calculators and are used for geometric transformations and working with neural networks and AI. And ray tracing cores are the largest but the fewest and are used to execute ray tracing algorithms. Now that we understand the computational resources inside this chip, one rather interesting fact is that the 3080, 3090, 3080 ti, and 3090 ti graphics cards all use the same GA102 chip design for their GPU. This might be counterintuitive because they have different prices and were released in different years, but it’s true. So, why is this? Well, during the manufacturing process sometimes patterning errors, dust particles, or other manufacturing issues cause damage and create defective areas of the circuit. Instead of throwing out the entire chip because of a small defect, engineers find the defective region and permanently isolate and deactivate the nearby circuitry. By having a GPU with a highly repetitive design, a small defect in one core only damages that particular streaming multiprocessor circuit and doesn’t affect the other areas of the chip. As a result, these chips are tested and categorized or binned according to the number of defects. The 3090ti graphics cards have flawless GA102 chips with all 10752 CUDA cores working properly, the 3090 has 10,496 cores working, the 3080ti has 10,240 and the 3080 has 8704 CUDA cores working, which is equivalent to having 16 damaged and deactivated streaming multiprocessors. Additionally, different graphics cards differ by their maximum clock speed and the quantity and generation of graphics memory that supports the GPU, which we’ll explore in a little bit. Because we’ve been focusing on the physical architecture of this GA102 GPU chip, let’s zoom into one of these CUDA cores and see what it looks like. Inside this simple calculator is a layout of approximately 410 thousand transistors. This section of 50 thousand transistors performs the operation of A times B plus C which is called fused multiply and add or FMA and is the most common operation performed by graphics cards. Half of the CUDA cores execute FMA using 32-bit floating-point numbers, which is essentially scientific notation, and the other half of the cores use either 32-bit integers or 32-bit floating point numbers. Other sections of this core accommodate negative numbers and perform other simple functions like bit-shifting and bit masking as well as collecting and queueing the incoming instructions and operands, and then accumulating and outputting the results. As a result, this single core is just a simple calculator with a limited number of functions. This calculator completes one multiply and one add

operation each clock cycle and therefore with this 3090 graphics cards and its 10496 cores and 1. 7 gigahertz clock, we get 35. 6 trillion calculations a second. However, if you’re wondering how the GPU handles more complicated operations like division, square root, and trigonometric functions, well, these calculator operations are performed by the special function units which are far fewer as only 4 of them can be found in each streaming multiprocessor. Now that we have an understanding of what’s inside a single core, let’s zoom out and take a look at the other sections of the GA102 chip. Around the edge we find 12 graphics memory controllers, the NVLink Controllers and the PCIe interface. On the bottom is a 6-megabyte Level 2 SRAM Memory Cache, and here’s the Gigathread Engine which manages all the graphics processing clusters and streaming multiprocessors inside. Now that we’ve explored this GA102 GPU’s physical architecture, let’s zoom out and take a look at the other parts inside the graphics card. On this side are the various ports for the displays to be plugged into, on the other side is the incoming 12 Volt power connector, and then here are the PCIe pins that plug into the motherboard. On the PCB, the majority of the smaller components constitute the voltage regulator module which takes the incoming 12 volts and converts it to one point one volts and supplies hundreds of watts of power to the GPU. Because all this power heats up the GPU, most of the weight of the graphics card is in the form of a heat sink with 4 heat pipes that carry heat from the GPU and memory chips to the radiator fins where fans then help to remove the heat. Perhaps some of the most important components, aside from the GPU, are the 24 gigabytes of graphics memory chips which are technically called GDDR6X SDRAM and were manufactured by Micron which is the sponsor of this video. Whenever you start up a video game or wait for a loading screen, the time it takes to load is mostly spent moving all the 3D models of a particular scene or environment from the solid-state drive into these graphics memory chips. As mentioned earlier, the GPU has a small amount of data storage in its 6-megabyte shared Level 2 cache which can hold the equivalent of about this much of the video game’s environment. Therefore in order to render a video game, different chunks of scene are continuously being transferred between the graphics memory and the GPU. Because the cores are constantly performing tens of trillions of calculations a second, GPUs are data hungry machines and need to be continuously fed terabytes upon terabytes of data, and thus these graphics memory chips are designed kind of like multiple cranes loading a cargo ship at the same time. Specifically, these 24 chips transfer a combined 384 bits at a time, which is called the bus width and the total data that can be transferred, or the bandwidth is about 1. 15 terabytes a second. In contrast the sticks of DRAM that support the CPU only have a 64-bit bus width and a maximum bandwidth closer to 64 gigabytes a second. One rather interesting thing is that you may think that computers only work using binary ones and zeros. However, in order to increase data transfer rates, GDDR6X and the latest graphics memory, GDDR7 send and receive data across the bus wires using multiple voltage levels beyond just 0 and 1. For example, GDDR7 uses 3 different encoding schemes to combine binary bits into ternary digits or PAM-3 symbols with voltages of 0, 1, and negative 1. Here’s the encoding scheme on how 3 binary bits are encoded into 2 ternary digits and this scheme is combined with an 11 bit to 7 ternary digit encoding scheme resulting in sending 276 binary bits using only 176 ternary digits. The previous generation, GDDR6X, which is the memory in this 3090 graphics card, used a different encoding scheme, called PAM-4, to send 2 bits of data using 4 different voltage levels, however, engineers and the graphics memory industry agreed to switch to PAM-3 for future

generations of graphics chips in order to reduce encoder complexity, improve the signal to noise ratio, and improve power efficiency. Micron delivers consistent innovation to push the boundaries on how much data can be transferred every second and to design cutting edge memory chips. Another advancement by Micron is the development of HBM, or the high bandwidth memory, that surrounds AI chips. HBM is built from stacks of DRAM memory chips and uses TSVs or through silicon vias, to connect this stack into a single chip, essentially forming a cube of AI memory. For the latest generation of high bandwidth memory, which is HBM3E, a single cube can have up to 24 to 36 gigabytes of memory, thus yielding 192 gigabytes of high-speed memory around the AI chip. Next time you buy an AI accelerator system, make sure it uses Micron’s HBM3E which uses 30% less power than the competitive products. However, unless you’re building an AI data center, you’re likely not in the market to buy one of these systems which cost between 25 to 40 thousand dollars and are on backorder for a few years. If you’re curious about high bandwidth memory, or Micron’s next generation of graphics memory take a look at one of these links in the description. Alternatively, if designing the next generation of memory chips interests you, Micron is always looking for talented scientists and engineers to help innovate on cutting edge chips and you can find out more about working for Micron using this link. Now that we’ve explored many of the physical components inside this graphics card and GPU, let’s next explore the computational architecture and see how applications like video game graphics and bitcoin mining run what’s called “embarrassingly” parallel operations. Although it may sound like a silly name, embarrassingly parallel is actually a technical classification of computer problems where little or no effort is needed to divide the problem into parallel tasks, and video game rendering and bitcoin mining easily fall into this category. Essentially, GPUs solve embarrassingly parallel problems using a principle called SIMD, which stands for single instruction multiple data where the same instructions or steps are repeated across thousands to millions of different numbers. Let’s see an example of how SIMD or single instruction multiple data is used to create this 3D video game environment. As you may know already, this cowboy hat on the table is composed of approximately 28 thousand triangles built by connecting together around 14,000 vertices, each with X, Y, and Z coordinates. These vertex coordinates are built using a coordinate system called model space with the origin of 0,0,0 being at the center of the hat. To build a 3D world we place hundreds of objects, each with their own model space into the world environment and, in order for the camera to be able to tell where each object is relative to other objects, we have to convert or transform all the vertices from each separate model space into the shared world coordinate system or world space. So, as an example, how do we convert the 14 thousand vertices of the cowboy hat from model space into world space? Well, we use a single instruction which adds the position of the origin of the hat in world space to the corresponding X, Y, and Z coordinate of a single vertex in model space. Next we copy this instruction to multiple data, which is all the remaining X, Y, and Z coordinates of the other thousands of vertices that are used to build the hat. Next, we do the same for the table and the rest of the hundreds of other objects in the scene, each time using the same instructions but with the different objects’ coordinates in world space, and each objects’ thousands of vertices in model space. As a result, all the vertices and triangles of all the objects are converted to a common world space coordinate system and the camera can now determine which objects are in front and which are behind. This example illustrates the power of SIMD or single instruction multiple data and how a single instruction is applied to 5,629 different objects with a total of 8. 3 million vertices within the scene resulting

in 25 million addition calculations. The key to SIMD and embarrassingly parallel programs is that every one of these millions of calculations has no dependency on any other calculation, and thus all these calculations can be distributed to the thousands of cores of the GPU and completed in parallel with one another. It's important to note that vertex transformation from model space to world space is just one of the first steps of a rather complicated video game graphics rendering pipeline and we have a separate video that delves deeper into each of these other steps. Also, we skipped over the transformations for the rotation and scale of each object, but factoring in these values is a similar process that requires additional SIMD calculations. Now that we have a simple understanding of SIMD, let’s discuss how this computational architecture matches up with the physical architecture. Essentially, each instruction is completed by a thread and this thread is matched to a single CUDA core. Threads are bundled into groups of 32 called warps, and the same sequence of instructions is issued to all the threads in a warp. Next warps are grouped into thread blocks which are handled by the streaming multiprocessor. And then finally thread blocks are grouped into grids, which are computed across the overall GPU. All these computations are managed or scheduled by the Gigathread Engine, which efficiently maps thread blocks to the available streaming multiprocessors. One important distinction is that within SIMD architecture, all 32 threads in a warp follow the same instructions and are in lockstep with each other, kind of like a phalanx of soldiers moving together. This lock step execution applied to GPUs up until around 2016. However, newer GPUs follow a SIMT architecture or single instruction multiple threads. The difference between SIMD and SIMT is that while both send the same set of instructions to each thread, with SIMT, the individual threads don’t need to be in lockstep with each other and can progress at different rates. In technical jargon, each thread is given its own program counter. Additionally, with SIMT all the threads within a streaming multiprocessor use a shared 128 kilobyte L1 cache and thus data that’s output by one thread can be subsequently used by a separate thread. This improvement from SIMD to SIMT allows for more flexibility when encountering warp divergence via data-dependent conditional branching and easier reconvergence for the threads to reach the barrier synchronization. Essentially newer architectures of GPUs are more flexible and efficient especially when encountering branches in code. One additional note is that although you may think that the term warp is derived from warp drives, it actually comes from weaving and specifically the Jacquard Loom. This loom from 1804 used programmable punch cards to select specific threads out of a set to weave together intricate patterns. As fascinating as looms are, let’s move on. The final topics we’ll explore are bitcoin mining, tensor cores and neural networks. But first we’d like to ask you to ‘like’ this video, write a quick comment below, share it with a colleague, friend or on social media, and subscribe if you haven’t already. The dream of Branch Education is to make free and accessible, visually engaging educational videos that dive deeply into a variety topics on science, engineering, and how technology works, and then to combine multiple videos into an entirely free engineering curriculum for high school and college students. Taking a few seconds to like, subscribe, and comment below helps us a ton! Additionally, we have a Patreon page with AMAs and behind the scenes footage, and, if you find what we do useful, we would appreciate any support. Thank you. So now that we’ve explored how single instruction multiple threads is used in video games, let’s briefly discuss why GPUs were initially used for mining bitcoin. We’re not going to get too far into the algorithm behind the blockchain and will save it for a separate episode, but essentially, to create a block on the blockchain, the SHA-256 hashing algorithm is run on a set of data that includes transactions, a time stamp, additional data, and a random number called a nonce. After feeding these values through

the SHA-256 hashing algorithm a random 256-bit value is output. You can kind of think of this algorithm as a lottery ticket generator where you can’t pick the lottery number, but based on the input data, the SHA-256 algorithm generates a random lottery ticket number. Therefore, if you change the nonce value and keep the rest of the transaction data the same, you’ll generate a new random lottery ticket number. The winner of this bitcoin mining lottery is the first randomly generated lottery number to have the first 80 bits all zeroes, while the rest of the 176 values don’t matter and once a winning bitcoin lottery ticket is found, the reward is 3 bitcoin and the lottery resets with a new set of transactions and input values. So, why were graphics cards used? Well, GPUs ran thousands of iterations of the SHA-256 algorithm with the same transactions, timestamp, other data, but, with different nonce values. As a result, a graphics card like this one could generate around 95 million SHA-256 hashes or 95 million randomly numbered lottery tickets every second, and hopefully one of those lottery numbers would have the first 80 digits as all zeros. However, nowadays computers filled with ASICs or application specific integrated circuits perform 250 trillion hashes a second or the equivalent of 2600 graphics cards, thereby making graphics cards look like a spoon when mining bitcoin next to an excavator that is an asic mining computer. Let’s next discuss the design of the tensor cores. It’ll take multiple full-length videos to cover generative AI, and neural networks, so we’ll focus on the exact matrix math that tensor cores solve. Essentially, tensor cores take three matrices and multiply the first two, add in the third and then output the result. Let’s look at one value of the output. This value is equal to the sum of values of the first row of the first matrix multiplied by the values from the first column of the second matrix, and then the corresponding value of the third matrix is added in. Because all the values of the 3 input matrices are ready at the same time, the tensor cores complete all of the matrix multiplication and addition calculations concurrently. Neural Networks and generative AI require trillions to quadrillions of matrix multiplication and addition operations and typically uses much larger matrices. Finally, there are Ray Tracing Cores which we explored in a separate video that’s already been released. That’s pretty much it for graphics cards. We’re thankful to all our Patreon and YouTube Membership Sponsors for supporting our videos. If you want to financially support our work, you can find the links in the description below. This is Branch Education, and we create 3D animations that dive deeply into the technology that drives our modern world. Watch another Branch video by clicking one of these cards or click here to subscribe. Thanks for watching to the end!