How OpenAI scaled Postgres to manage 800 million users

This is how OpenAI has scaled Postgres to manage 800 million users on ChatGPT. OpenAI uses a very simple Postgres architecture with a single primary instance, and many read replicas spread across the globe. This architecture easily scales out horizontally. But the architecture has two major drawbacks. One is that if there is a write storm, where many write operations are sent from the application server, the single WRITE instance becomes the bottleneck. OpenAI mitigated this by scaling up the WRITE instance and moving the write heavy workloads to a new system using cosmosDB. The second drawback is that this system has write amplification, where a single write request has to be copied onto all the replicas, effectively converting 1 request into 50. Here the applications were asked to do “lazy writes”, where multiple write operations are collected in batch and then sent to the database as a single write operation. To improve on fault tolerance, the primary instance has a ready to take over instance that replaces the primary instance in case of any failure. OpenAI also used connection pooling, which limits the number of connections that can be created with the database by presetting a pool of connections and managing any incoming request with the pool. This allows multiple clients to use the same connection, while reducing the time required to set up a new connection for a new request. They also used request-collapsing in their caches. Here, when a client asks for a key, the key is stored in cache and the value is a promise / future. When another client asks for the same data, the cache entry is used to return the future. When the data is retrieved from the database, all of the waiting clients receive the same future, resulting in an immediate response. This avoids making multiple requests to the database for the same key. To scale read replicas effectively, OpenAI moved away from simple WAL to a hierarchical WAL structure, where the primary sends write events to some read replicas, which further propagate the events to the remaining read replicas. This increases the operational complexity of the system but results in faster transmission of all WRITE operations. Overall, these operations have been well-studied in distributed systems. An example of this is Memcached, which powers the entire social graph of Meta. You can view its detailed research paper analysis in the description. Thanks for watching. Cheers!