Async Rust Demystified: Next-Level Performance Many Will Miss

There are a lot of things about async Rust and Tokio that are frequently misunderstood, so let's talk about them. We have this simple program here, an asynchronous main function annotated with the Tokio main macro, and then we have one asynchronous function get page. It takes a URL as a parameter, and it makes a request to that URL. It doesn't actually do anything with it, but that's not relevant for what we're going to talk about today. The important part is that it makes a request to the URL. This is an asynchronous get function, request get. It's going to immediately await the future returned by request get. And then in the main function, we have two invocations of get page. In this example, we're going to immediately await the future after it's returned. And so, effectively, we're going to execute both of these requests in serial. Let's run this to confirm that that's the case, and we can see that's actually what's happening. We're waiting for the first request to finish before we even kick off the second request. So, we're not really getting any benefit from the async nature of this program yet. But what's actually going on under the hood here? When we annotate the main function with the Tokio main macro, what's actually happening under the hood is that we're creating a new multi-threaded Tokio runtime. And all the contents of our main function is being wrapped in an async block, which gets passed to the block on method of the runtime. That's what's going on under the hood. Ultimately, we wind up with this synchronous main function here. Now, when you create a multi-threaded Tokio runtime, you get what's called worker threads. You get a pool of worker threads. And by default, the number of worker threads is going to be equal to the number of logical CPU cores on the system. That is a very sensible default. I've never actually had to change it, but if you do want to change it, you can use the worker threads method of the Tokio runtime builder. Now, we don't usually create our Tokio runtime with the builder. So, if you are using the Tokio main macro, you can use the worker threads parameter of the Tokio main macro. So, we're setting the number of worker threads to two. It's going to be two for the remainder of this video, but the behavior of this program has not changed. If we run it again, we can still see we're still executing these requests in serial. Now, what if we want to kick off the second request before we get the response to the first request? Well, for that, we can use the Tokio join macro. In this example, instead of immediately awaiting the future returned by get page, we're saving that future to a variable A for the first request and variable B for the second request, and we're passing both of those to the Tokio join macro. What Tokio join is going to do is suspend the current thread until all of the futures are resolved. going to run this and see what happens. You can see now, we actually initiate that second request before we get the response from the first request back. What's going on here? Well, you might be tempted to think, "Oh, it's dispatching these get page requests to the worker threads. " But that's not actually what's happening. This is the first punchline. Right now, as written, this is a single-threaded application, and the requests are being executed concurrently. There's no parallelism happening yet. What's actually happening is the first call to get page is happening. We're hitting this await on the future returned by get, and that's telling Tokio that's yielding to the Tokio scheduler saying, "Hey, I have no more work to do right now. You can use this thread for something else in the meantime. " The Tok- Tokio scheduler is then going to the second invocation of get page and executing that. That's hitting the await and yielding to the Tokio scheduler, and now we're waiting on both requests at the same time. One thread. Now, what happens if for whatever reason, we have a CPU-bound or blocking operation in get page? I'm going to switch to a synchronous version of the get function. So, Ureq is another HTTP client library that has a synchronous version of get, and that's what I'm using here. Nothing else in the program has changed. We'll run this, and we can see we've devolved. We've gone back to executing the requests in serial. Um that's because this is single-threaded. Now, I've been talking about Tokio worker threads this entire time, but so far, we haven't dispatched any work to the Tokio worker threads. How do you do that? There's a function Tokio spawn. And Tokio spawn accepts a future, and it will dispatch that asynchronous operation to a worker thread. We've added two calls to Tokio spawn, and we send them the future returned by get page. And we've gone back to the asynchronous version of the get function. Let's run this and see what happens. We initiate the second request without waiting for the response to the first request. Okay, but that's the same result we got even without using the Tokio worker threads. So, how do I prove that we're actually dispatching work to the Tokio worker threads? Well, let's switch back to the synchronous version of the get function. We switched from the asynchronous get to the synchronous get. What's going to happen here? Now, previously, that would cause us to go back to waiting for the first response before executing the second request. Now, we know these are being dispatched to the Tokio worker threads because these are blocking operations, and they can't yield control to something else to use the thread, right? These two requests are now happening in parallel because they're being dispatched to the Tokio worker threads.

Now, what if we want to spice things up a little more? We're going to add a third call to get page. We're not sending it to Tokio spawn. We're not creating a Tokio task out of it. We're still using the synchronous version of the get function, still two worker threads. What's going to happen here? Let's run this and find out. We're waiting for all three responses at the same time. How does this happen? We only have two worker threads, right? Well, the punchline here is that we still have the main thread to work with. There's still a main thread that our logic in the main function is executing on outside of the worker threads. So, even though we only have two worker threads, we have three threads that we can use to do some work. That surprising to some people. Now, what if we add a third invocation or a fourth invocation to get page inside of another call to Tokio spawn. So, we're making three Tokio tasks, still two worker threads, and we still have that fourth call to get page that's going to be executed on the main thread. Still the synchronous version of the get function. Let's run this and see what happens. As you might suspect, we actually have to wait for that one of those requests, one of those three initial requests to come back before we can initiate that fourth request. And then ultimately, the last three requests complete as well. So, we have kind of a bottleneck here. We're limited to three parallel operations. But what if we switch back to the asynchronous version of the get function? Let's run that. Now, we can kick off all four requests and wait for them all at the same time. This is concurrency and parallelism used in tandem. And this is super super powerful. Right? So, again, only two operating system, well, three operating system threads total, two worker threads, and we can execute or wait for kick off four requests and wait for them in parallel. To drive this message home, we can expand this and say we're going to create 500 requests, each of which is a Tokio task. We're still using the asynchronous version of the get function. Now, I'm not going to run this because I don't want Hacker News to think that we're DDoS-ing them, but theoretically, this should be able to kick off all 500 requests and wait for all the responses at the same time, even though we only have two operating system threads. That is incredibly powerful. This avoids spinning up and tearing down operating system threads. There is a lot of overhead in spinning up and tearing down operating system threads. But what if we go back again to the synchronous version of the get function? In this case, it's never yielding to the scheduler, so we only have two worker threads. We can only execute two requests in parallel. They're not yielding while the request is pending, so we can't start executing the other request. This one is going to take a long time. And Tokio tasks are designed for asynchronous operations that have downtime and can yield control back to the scheduler such that the thread can be used for other things. What do you do in this situation? Well, in this situation, the best approach is to use spawn_blocking. Spawn_blocking is a Tokio function that takes a synchronous closure, not an asynchronous closure. We've actually converted get page to a synchronous function. There's really no point in it being asynchronous if it doesn't ever await a future. It's synchronous, and we're passing the synchronous closure to spawn_blocking. This is going to create a brand new thread for every blocking operation. That's what you want so that you don't saturate your worker threads with blocking operations and block your task pipeline. Um by default, the most threads it will ever create for blocking operations is 512. I believe that's configurable, but this is the canonical approach for CPU-bound blocking operations. Folks, I hope this has given you a new perspective on async Rust and Tokio. If it's helped you in some way, let me know down in the comments. Give the video a like. We'd love to have you as a subscriber. If you like this video, definitely check out one of these other videos. Thanks for watching, and we'll see you in the next one.