Your API Can’t Handle Real-World Integrations

Let's say you build a nice REST API product, but then you release it and you find out your customers can't integrate it with their own systems. One customer needs a CRM ID, another needs a warehouse reference, and someone asks whether you can just add one more field for some external system that they use and before you know it, your schema becomes a huge mess. Now, in this video, I'll show you a much cleaner approach that won't require you turning your API into a junk drawer. It's very simple to build. The Stripe API does it this way. And I think more APIs should add this feature because it makes them way more usable. Now, what I've learned over the years is that with ideas like this, it's not just important that you know they exist, but that you also understand the design trade-offs behind them. And that's exactly what I go deep into in software design mastery. This is not just another course with patterns and principles. It's a comprehensive system for thinking about design and trade-offs. If you want to be part of that, join the waiting list at iron. codes/mastery. The link is in the description. Now, I'm

starting with a simple working fast API app. So, this is the main file. I'm using SQL alchemy database for this fast API app and it's very basic. It just has a few concepts. So, there is a customer, there is a product, and there are orders. And this is what the API allows us to work with. And as you can see, I have a declarative base just like you're supposed to do with SQL Alchemy. And then I create these different classes. So customer has ID, email, name, there are orders, uh there are products that are also mapped to orders. And an order itself is again linked to a customer and a product and has some other data as well. This is a pretty basic setup. And by the way, the full code example as usual, you can find in the code example git repository link is also in the description of the video. Now, next to these models, there is a couple of other modules as well like the routes which is the most important one where we have the different endpoints like posting to the customer's endpoint to create a new customer. There is an endpoint for getting the customer by ID. You can patch a customer. Uh you can do the same thing for products and orders as well. So let's start this API using uorn. And now I can send a simple request to create a new customer like so. And now we have this customer with ID one. And then I can use the API in different ways. For example, here I'm retrieving that same customer from the database that I just created. Now once

this app starts integrating with other systems then new requirements are going to appear very quickly. Maybe one of your clients has a CRM and they want their own customer ID that they store along with the customers in your API or some accounting system wants a contact reference or a warehouse tool wants an order reference. A migration script wants to remember some old legacy identifier. And these fields are really important but they're not really part of the core domain model of your API. You can't know in advance what all these fields are going to be because they're going to be dependent on the clients that use your API. These are really integration specific. Now, some APIs offer custom fields for this and then it's often implemented like you need to specify what custom fields there are. These things get their own ID and then you can read or write these custom fields that are then attached to different resources in your API. But there is actually a simpler way to do it and that's by using a really great data structure called JSON. The idea is and this is also how Stripe sort of does it is that you have a meta data or custom data field that is a JSON format that your customer can basically set in any way that they want. But you don't just want to add a JSON field and leave it at that because then it's not really useful because if you want to update let's say one value in that JSON structure, you need to pass in the complete JSON data into the request in order not to delete everything that's in there. So we need to implement it in a specific way and that's also how it's done in Stripe. And you can actually add this to your API as

well. So we want let's call it custom data to behave like a normal field in the API. Client should be able to send it on create request. They should be able to update it and pass request and they should also be able to remove individual keys without replacing the whole object. So they should be able to send a key value pair to add or update the value. They should be able to send a key with a null value which will unset it. And they should also be able to send an empty object to basically clear all of the custom data. And combined this gives you a very flexible and predictable behavior that works very well for APIs. So how do we add this

here? Well, there's a very logical place in the database models actually, which is that we have this declarative base because obviously we want this custom data to be available for all of the different resources for customers, for products, and for orders. And we want to avoid that we copy this all over the place. So, what you can do here is actually rely on inheritance. Even though I'm personally more of a fan of composition, in this case I think inheritance is a good solution because it's already built into SQL alchemy. So what we'll do is we will add this custom data field to the declarative base. So when you do that, this is actually what

that looks like. So base now has a custom data field. It's a JSON type. We won't allow it to be null because we always want it to be an object. So that's what we have. But now the only thing we need to do is of course define the behavior of changing this custom data object. Now you could add that to the routes but again you'd have to copy paste that behavior all over the place and actually there is a very easy way to add it to declarative base in SQL alchemy. The first thing that you can do is define a function that applies the data to the custom data. So and then we'll call this from somewhere else. So this gets a dictionary with the new data and there we basically have to work with the three different cases. The first case is very simple that if the patch is the empty dictionary then self. custom data is simply going to be the empty dictionary and we're done. We can return. So this is basically a way of clearing the custom data. The next step is that we can go through the key value pairs in the patch and apply them one by one. So first let's store the custom data in a temporary variable. And if that doesn't happen to exist, let's make it the empty object so we don't run into any issues. We always know that there is an object. So for key the value in the items of patch and then now we can handle the two remaining values. So if the value is none that means that we want to unset the key. So we do current dot pop the key and default if the key doesn't exist we simply ignore it. Else that means the key has a value. So in that case current key equals the value. So we simply set the value. And then finally once we've done this we set the custom data to the current object. So that this is also handled correctly if this is the empty dictionary. Now maybe you could do this directly on the custom data but I kind of like using a temporary variable here. It feels a bit safer. But now the magic is that we can have a method apply patch and this gets data and this actually overrides a standard method in the declarative base. So this is going to be called automatically whenever we update the object. And here what we're going to do is we're going to loop over the items in data and then we can simply handle the cases here. So if the value is none and the key is not custom data, we can simply ignore this case. And if the key is custom data, well, in that case, we're going to call our custom data patch method. And we're simply going to pass the data like so. Else we can use set enter to set the key to the value. So this basically means the standard behavior of SQL alchemy. And then the only thing

left to do is that in our schemas for creating and reading customers and so on, we have to make sure that there is a custom data field. So here's how you could do that. It's very simple. We simply define custom data as part of our schema by simply adding it there to the classes. So in this case, I created a custom data schema and then whenever I create a customer, I make a subclass of that. So we can also set the custom data whenever we create a customer. And I did the same thing for reading the customers. And you can also add it for other things like creating products for example. So adding the custom data here is actually very little work. So now that we have this, it's really easy to add custom data to our resources. For example, here I'm creating a new customer where I set an actual customer ID in the CRM that I pass as custom data. And now my customer has this as part of the object. And if I want to add more custom data, I can actually simply pass it by setting a key value pair in the custom data like so. And as you can see, it currently does something weird, which is that it puts custom data inside the custom data. And that's because I made a mistake. Namely, when I apply the catch, I shouldn't pass all the data here. I should actually pass the value like so. Let's try this again. I've deleted the database and I'm creating the customer one more time. And now let's add another custom data field. As you can see, now it has the desired behavior. So now it sets our billing ID, but it doesn't delete the CRM custom ID, which is really neat. We can also remove a custom field value by simply setting the value to null. Like so. As you can see, we get the customer back. It keeps the CRM customer ID, but it has deleted billing ID. And finally, if we want to clear all custom data, we simply set it to the empty object, and that also works. So overall, I think

this is a really useful pattern you can add to your API and it's actually really simple to do, especially if you're using something like SQL Alchemy. The only thing you need to do is basically add this piece of code to your declarative base. And anything that inherits from this is going to have custom data with the behavior that I demonstrated in this video. And of course, you need to make sure that custom data is part of your schemas, but that's like a pretty easy thing to add. So in my opinion more APIs

should add this feature because this is really useful but only use this for the right thing. So good uses for this are external ids that you want to couple with integration specific references small optional flags or tags uh data that is uh useful context but not core domain structure. I wouldn't use this to store important business logic or data that you constantly need to query or search through. uh things that really should be first class fields in your API. Now, a good rule of thumb is that if the field is part of the meaning of the model, make it a real column. If it's just custom context, then it fits great into a custom data object like this. And by the way, if you enjoy these types of practical design breakdowns where we go beyond just making something work and look at why design is good or bad, like this video, subscribe to the channel. It helps me out a lot. In

summary, what makes this feature nice is that it's not just storing JSON, but that we slightly modify the behavior so that it's really easy to use. And by making it part of the declarative base, it's really easy to add to the various resources in your API. But I'd like to hear what you think. Have you added this type of metadata, custom data thing to your APIs? Have you seen other APIs implement this feature? Let me know in the comments. Now, I haven't talked much about how to actually organize the business logic in this API, like the database, the routes, anything else your API needs. Watch this video next to learn more about how to properly design it using a really cool pattern called ports and adapters. Thanks for watching and see you next