How I Reduced AWS Latency 200x (From 7ms to 35μs)

When you are in the stage of designing your AWS architecture, you need to be aware of the latencies you get when you place your components in different regions or availability zones, as well as the associated data transfer costs. We'll start with placing EC2 instances in two different VPCs in different regions. We’ll use VPC peering and measure the latency between them. That's usually the highest level of availability you can get. Then we'll place EC2 instances in the same region but different availability zones; this is by far the most common setup when you're trying to achieve HA. Then we'll compare that latency with the latency between VMs in the same availability zone. And as a bonus, if you really want to achieve ultra-low latency—for example, for HFT trading—you would create a placement group of type cluster and place your VMs there. You can also use shared placement groups and place VMs from different accounts. For example, you can colocate your VMs with a crypto exchange on AWS. And of course, since in that case the VMs would use the same hardware, there is a higher chance of failures. So it's a tradeoff between latency, availability, and cost. Now, to run this test, I have created a simple C++ application using raw sockets and SBE as a protocol for encoding/decoding my messages. It’s frequently used in trading, and it’s very fast—faster than Protobuf. On the server side, I use Prometheus to measure duration and throughput and push that data to a remote Thanos server. You can find the source code in my public repo. For all of those VMs, I'm using m7a. medium except for the Prometheus agent, and I have another EC2 instance in a different VPC in the us-east-2 region. If you're interested in how I run most of my tests, I have a full EKS course here on YouTube, but in this test, I use plain EC2 instances. Initially, I just wanted to compare latency improvements when you place your VMs in the same placement group, but then I decided to cover other use cases as well. In this test, I'll just set the client to send 1,000 messages per second and measure latency. In the latency graph, I use a logarithmic scale because cross-region latency is so much higher that it causes all other lines to blend together otherwise. I also have network usage for the server, which I get using Node Exporter, and on the right bottom graph, I basically use that metric and calculate how much it would cost. For example, data transfer costs between different availability zones—as well as between regions in one region like us-east-1 and us-east-2—would cost you 0. 01 cents in one direction, so 2 cents for a typical request-response. And as you can see, there are no data transfer charges for the same AZ and placement group. Those cross-AZ data transfer fees can be huge if you run distributed messaging systems like Kafka or other databases that replicate data. So it's fine for stateless applications to be deployed in different availability zones, since most of them do not replicate data, but it can be very expensive otherwise. And of course, we have latency, which is around 7 ms between the closest regions, us-east-1 and us-east-2, just because of the distance. Then we have cross-AZ VMs with much lower latency of around 300 microseconds. Then EC2 instances deployed in the same AZ with around 100 microseconds. And finally, the best and lowest latency you can achieve by placing those VMs in the same placement group—you can get as low as 35 microseconds. Just be aware of the tradeoffs. The best example is to place your trading bot as close as possible to your exchange and use shared placement groups if possible.