Netflix Engineering: DB Migration to Aurora results in 75% speedup

Netflix has just made its databases up to 75% faster, and the secret behind it is AWS Aurora. For years, Netflix had been running self-managed Postgres on EC2 instances. This meant that their infrastructure team had to manage their 1. Scaling 2. Fault tolerance 3. Database upgrades These processes were often manual, and during load spikes it would take time for Postgres to scale up, leading to slow response times. So the team decided to move to Amazon Aurora, which gave the managed fault tolerance and managed database upgrades. The biggest benefit here is that Aurora is different from the standard Postgres implementation. Typically, Postgres uses something called “Shared Buffers”, which is like an internal cache for Postgres that stores data and index pages. There's also a second layer of cache, which is the OS-page cache. This is managed by the file system and can have duplicate entries with the shared buffer. When the request comes to a Postgres instance, it first checks in the shared buffer, and if there's a cache miss, it checks in the OS cache before finally reading from disk. Aurora gets rid of the second layer of caching. It allocates around 75% of all memory to the shared buffer cache, and in case there is a cache miss, it directly queries the data store. This means that most read and write operations can be served from memory, resulting in lower costs and faster response times. As a result of the migration, the Netflix engineering team observed that the latency went down by 75% and the Postgres management cost went down by 28%. I've linked all the relevant papers and blogs in the description. Thanks for watching. Cheers.