This Design Pattern Scares Me To Death

This is a video about the specification design pattern with a big warning. It's extreme overengineering. I wasn't even sure whether I should actually post this video. The reason I'm doing it is because, well, there are a couple of really interesting design ideas that I just don't want to keep from you. But frankly, when I look at the final version of the code, it scares the [ __ ] out of me. It's love craftsion. Is that a word? That should be a word. When you first look at it, you go like, but then it's also kind of interesting and then it sucks you in and then bam, you're done. We're going deep today. Higher order functions, decorators, thunder method, overrides, generics, the whole shebang. It's going to be insane, but really good. This video is sponsored by SER API. More about them later. Now

what is the specification? But let me first show you a code example. I have a user class with a bunch of different settings. And there's a function that checks whether this user can access some API. As you can see, the logic is kind of convoluted. There is an age, whether the user is active, is it banned, what country the user is in. Now, you can maybe refactor this a bit to turn a lot of these things into guard clauses, right? But it's still kind of complicated business logic. And you can imagine that this is maybe something that regularly changes, maybe because the company changes uh the policy or something like that. And perhaps you have very similar kind of business logic in other places as well like can you access some kind of report which you know looks kind of like it but is slightly different or maybe you have another function that checks whether you can access the CLI which again is slightly different from the first version. Now maybe this is intended maybe not who knows. The problem with this type of business logic is that well here this example is of course very basic but this would normally be spread out over your entire codebase and mixed with other types of business logic and whenever you want those rules to change it means you have to update this in many different places and you don't have a clear overview of what those rules actually are so today I'm going to show you the specification pattern which fixes this by turning these complicated if conditions into reusable building blocks and ultimately this will let you turn all of this code into data and I'll show you an example of how that works at the end of this video. Now this specification pattern doesn't come from the gang of four book it's from domain driven design and at the core the idea is to encapsulate business rules into small specifications and then you can combine these specifications using boolean logic and then you can reuse that everywhere. Now if you have a classical object-oriented language, you would then basically implement this with lots of small classes and each class represents one of these business rules. You can find a bit more information about this pattern on Wikipedia. Uh so you can see there is like a UML diagram where we have various types of specifications. Uh there are some code example which is extremely abstract. So, and when I read this the first time, I had a hard time understanding why you would actually need this because what's the advantage, right? You can just write the same logic in an if condition. So, why would you have all of these classes to wrap around that? So, that's what I'm trying to address in today's video. There's even a Python example in Wikipedia that adds all of these classes without really showing why you would want to do that. The only actual example that's given on the Wikipedia page is in my opinion something pretty basic that you can also just capture in a regular if condition. So in this video I hope I can take this a few steps further. Now the goal that

we have is that we want to have rules that are reusable and composable. And obviously like I already said the example here is pretty basic. uh but I do think in Python we can do much better than this objectoriented example from Wikipedia. In Python we can use functions in much better way. So we can build a small functional framework for these rules and we can rely on other features like uh decorators and generics and cool stuff like that we have access to in Python. So the first step

that I want to take is to move these conditions out of this if statement into a sort of uh let's call that a predicate. Basically a predicate is a function that returns true or false and then we can later on combine these functions so that we get more complicated logic. So for example here you see a bit of code that could be turned into a predicate. So that would be then a simple function called let's say is admin. That's a user. It returns a boolean value and this returns whether the user is an admin. So that's a function returning true or false. That could be a predicate. rule. And we can do this for other types of rules as well. For example, is active like so. or this check whether the account is over 30 days. So these are some basic example of rules that we can pull out of these functions. Now I can directly call these functions as part of an if statement, right? I can replace uh this with a call to this function. So that would basically look like this. But the problem with this approach is that we now create the specification of the logic and we call it at the same time. That's the whole issue with these if statements. So what we actually need to do is find a way to specify the logic here without already evaluating it. So that means we need to somehow combine these functions and then only afterwards we should then execute them after they've been combined and then we can separate the specification which is the definition of how all these functions look like and how they're combined with actually using them. But of course if we don't evaluate the functions we don't have a boolean result. So we need a way to combine these functions logically without already evaluating them. So what we can do is create a small class. Let's call it the predicate that contains a function and that overloads a few basic logical operators and, or, and not. So, I'm going to create a new file here. Let's call that rules where we're going to encode this logic into. So, what I would like here is a class. Let's call that a predicate. And for now, I'll change this back later. I'm going to remove the user class and put it in here. Later on, I'll put it back in the main file. So my predicate is basically a wrapper around a function. So that needs an initializer that gets a function which is a collable and this function gets a user and returns a boolean value like so. and I store this function inside the instance and from typing we're going to import gable like so. So this gives me a predicted class that's basically a wrapper around function. And then what we can do so that we can still call it is override the call upon the method. This user lower case obviously returns a loop and that calls this function with the particular user. So it's really now a completely empty wrapper around this function. If you call the predicate then it's simply going to call the function. But what you can do now is since we are in a class we can override the logical operators. So and it's an example. So that gets an other predicate like so. And that will also return a new predicate. And this will return predicate object that is wrapped around a function lambda function that gets a user and that returns oh actually I don't need return self and other. So basically we generate a new function here that when we call it it's going to call this and logically combine the results. Now similarly I can define or and not invert. So now this predicate class allows me to specify the conditions without already executing them which is what we want in the specification pattern. So we can specify it and then when we need it we can actually run the function. So what you can do now is take uh these functions and actually uh from rules I'm going to need to import user and predicates because we won't need these. We can now specify that is admin is a predicate and that gets a lambda function of a user where we check whether the user is an admin like so that we delete that and same thing for is active and finally account older than 30 is also a predicate where we check that user account age is over 30 like so and I delete that. And now we can start composing the rules. For example, I could say that well a rule is that the user should be an admin or it should be active and account older than 30. And here I'm not executing the logic. I'm purely combining these specifications. So this is exactly the power of the specification pattern. So we have a predicate which specifies what a rule looks like. We can then define these rules and we can combine them without actually executing them. So let me just clean up this example a bit and remove this. So uh we have a user and what I can then do is print rule and rule actually this is now a function that we can call with a user. So I'm simply going to pass the user that I created here before. This is a sample user. So when I run this code, I get outcome true because well this particular user is not an admin but the user is active and the account age is 35. And if I do 25 for example, what the result is surely going to be false. So we've now created a very simple implementation of the specification pattern using functions where uh we have a predicate class that allows us to combine functions compose them logically without already calling them. So that gives me a rule system and then I can have a user print the rule. Before I clean this up and make these predicates much simpler, let me quickly talk about something related. getting consistent data from web scraping APIs because just like business rules, API responses can be surprisingly inconsistent. If you ever try to scrape data from Google or other search engines, you know how fragile that can be. One request times out, another gets blocked or the HTML changes and your scraper breaks. That is exactly the problem SER API solves. SER API gives you access to Google, YouTube, and many other search engines through an easy to use, reliable API. In a few lines of code, you can extract live search result data and use it in your application, train your AI models, or send it to a spreadsheet. They handle proxies, captures, and retries automatically and return well ststructured JSON, so you can focus on using the data, not fighting the network. It's really easy to use. Here's an example in Python of how to access the Google search API using the SER API package that's available for free. Now I just get the API key from an environment variable that I set in M files. That's why I'm doing load. N. Then I create a client, pass the API key, and then I run client. arch. I provide the query. I provide the target engine. In this case, that's Google search. And I also provide my location. And then I simply get back JSON data in a dictionary like structure that I can work with directly in Python like so. So this prints the first organic result. Let's run this and see what happens. As you can see, we get a dictionary here with all the information that we need. Or how about searching YouTube for content about the solid design principles in Python? It works almost exactly the same way except of course we change the engine to YouTube and then we can print the video results. And here we go. We have as a first result my video on Uncle Bob's solid principles. If you want to make your data pipelines more reliable, check out SER API. The link is in description or you can scan the QR code on screen. Now back to the video. The next step is

making all of this a bit easier. If you have to write predicate and lambda functions all the time is kind of annoying. So let's define a small decorator. So from fun tools I'm going to import wraps because that's the best practice for defining decorators in Python. And then what I'm going to do I'm going to scroll down here and I'm going to define a decorator predicate and this is going to get a callable that gets a user and returns a bool. Actually maybe we should introduce a type alias for that. Let's say we have a predicate function which is a callable that gets a user and that returns a bull like so. And then we can also use that here which is nice in our decorator like so. And what this decorator is going to do, this will return a predicate because this is going to do the wrapping for us. So we use wraps around the function and then we have the wrapper. So this is the function that uh gets a user returns a bool and this calls the function with the user. And then finally, we're going to return a predicate wrapping around the wrapper like so. And now instead of having to write like this, I can go back to my function. So for example, this admin user returns a bool like so. And then I can remove this. And I'm going to import this predicate decorator. And I can do it like this. So now I have my function again defined in the way that I like it. It's much easier to read than uh this stuff. And now it turns into a predicate. And we should get exactly the same result as before. And we do. So this is much nicer, but there's still a couple of problems. We're kind of hard coding the user type here in our uh rules and we're also sort of mixing the idea of what is a rule and what is a predicate because for example I would like to do something like this. But of course that's not possible because all of these predicates they only accept a single user object and not any other arguments. So I need to refactor this code a bit so that it can also do stuff like this because then we can truly encode any types of rules with all sorts of different arguments. So the

first step is that we go into our rules system and then we generalize these predicates and wrappers so that they can work basically with any type. Not just users but with any type. Because if you look at the predicate and the decorator, it doesn't really care that this is a user. This can basically be any type T, right? As long as it's something uh that the function accepts as an argument, we're good to go. So, what I'm going to do now is let's move user back to the main file where I think it also belongs like so. So, this one no longer going to import here because that's defined here. And of course we will need to import the data class again otherwise that doesn't work. And now what we can do in the rule system is that a predicate function is something generic. So it gets a type t and it's basically a function that gets a t and returns a boolean. The predicate class should also be generic and this should then of course get a predicate function of type t. And now of course we can also change this. So let's call this some object. This is going to be of type t. And we're going to do the same thing here. So this is a predicate of type t. This as well. This is already getting fancy, right? Like so. And we do that here as well like so. And our decorator is the same. So this also gets a type a predicate function of type t. This needs to be generic. And this also returns something generic. And then this also gets an object of type t. And now the type issues are gone. And we turn this into nice generic code. Let's take a look at our main file. Did this break anything? So, of course, here you see some issues because it's not clear now what type of predicate this is. Uh, this works as expected. So, uh, Python's type system can basically infer that this is a predicate with a user used as a generic type. But let's take these other ones and also turn them into more modern predicates using the decorator. So, is active. Let's say user returns a boolean view like so. And then this one we're going to delete because we don't need that anymore. And same thing for the accounts age. And for now I'm simply keeping it like this. Uh 30. There we go. And we delete this. then let's for now keep it like this. And just to make sure that this still works as expected. And [clears throat] it seems like it

does. So now we have our generic version. The next step is that I want to have a bit more flexibility in how I define these rules. So basically what I want is that this rule is able to accept other arguments than just a user. So let's just assume for example I would like to write something like this right and then here we use that as an argument. Now currently this doesn't work because the predicate accepts only a function that gets only a user. So we need to make this a bit more generic. So what we're going to do is that instead of the decorator returning directly a predicate function we can make it a rule factory. So that's basically a let's also rename this to rule in that case. So it gets a uh rule definition. A rule definition can something be a bit more generic so that it also accepts this kind of thing. So let's say we have a type rule definition which is a callable that accepts any type of argument. Now, ideally, I would like to be able to specify here that this should uh define a list of arguments and then the last one should be of type T. Unfortunately, that's not possible in Python's typing system. So, I'm going to leave it at that and accept the limitation of this. And then the final type that we need is that we will have a predicate factory of type T. And this is a callable that gets some arguments and that will return a predicate of type T. And then the rule decorator which was before the predicate decorator will return a predicate factory. Are you getting confused yet? I can understand if you are but give me a minute and then it'll all come together. So we have our rule decorator. This should now get a rule definition so that it accepts any function as long and that's something that we'll have to assume that the last argument of the function is going to be of type t. So our wrapper is going to have arguments type any. We don't really uh specify these at the moment. So I'm going to need to import this like so. And there is keyword arguments as well. And my wrapper will now return a predicate. And then at the end we are going to return the wrapper so that the type matches what we expect here. And inside the wrapper so this is now where we create the predicate and we're going to give it a function a lambda function that gets an object. So this is the thing of type T and it's going to call that other function where we have the arguments. The last argument is the object and then we have the keyword arguments. So this is not perfect because if you now define the object as a keyword argument is not exactly going to work but in most common cases this is going to be fine. It's a limitation. It's not perfect. Uh we have to work around it in some way. So this is our new rule decorator and now we need to change a few things in the main file as well. So of course we need that new decorator. So I'm going to add it here rule here as well. The only thing we need to fix now is that rule returns a function that creates a predicate. So instead of calling it like this, we now need to call everything like functions. But what we can do now is because these are now function calls, I can also change this into a function like so. And if we run this, you see that this still works. Even though at the moment, unfortunately, my uh type annotation system no longer recognizes this. I think we're just going way too far here. But I warned you, right? This was going to be blatant overengineering. Now the nice thing about this approach is that we have a rules management system with predicates and a handy decorator and we have the main file where we actually specify the domain. So we have the user and we have the rules that apply to uh this particular domain and we combine them in a specification without actually executing the logic. So this line here doesn't execute any logic. It doesn't call these uh functions directly. It simply calls the wrapper that generates the predicate which will only be uh called when we do this and that is basically the idea of the specification pattern. So we can specify the logic here without executing it which only happens here. And now that we have this

we can basically build out this application. So I still have my data class. So this is a slightly more complete example where I've coded all of these original rules including uh the account holder them uh from which country uh a user is the credit score etc. And then I can create these kinds of structures now uh which are the specification of a particular condition right and then this is again a function that I can call in other functions if I want to. So for example for reporting I can check for a list of users whether they adhere to these rules or same for a CLI tool or wherever you want to use it.

Now the interesting thing and I think this is where we move from blatant overengineering to something that is maybe actually useful is that because we make the specification here it shouldn't be that hard to actually turn this into data. I mean we already have all of the rules and the logic. So this is something you can actually specify somewhere else and in order to not make this video too long. I'm not going to code this from scratch, but I'm going to quickly show you how you could do this. So the first thing that you need to do if you want to load it from let's say a config file is that you need to keep track of which rules are actually available. So in order to support that I've created a rules registry. So this is a dictionary that maps name to a predicate factory. And in my rule decorator, I do exactly the same as what I had before. I just add a single line where I also store this function in my rules dictionary. And then what I can do is I can write a function that loads a rule from a configuration file. Let's say a JSON file for example, where you can specify uh conditions like this with uh arguments and you can specify the logical relation between them. I have a more complete sample file here. So this is just a JSON file with a particular set of rules. So we have uh is active or we have account older with the argument 30 or from country Netherlands or Belgium or the credit score above 650 and so on. So you can build out this JSON structure without having to uh touch the code at all. And then I have a function that basically loads this JSON files. That's what happens here. And then it translates these conditions and logical combinations into the actual predicates. So here I'm getting the factory. I call the function with the argument. So this is going to return the predicate, the function that takes a user and returns true or false. And then I'm appending it to my list of predicates. And then depending on what the logic is, whether that's and or I'm uh combining them in a different way. And then I return that as a result. And then in the main file, what I can do is I can create a rule by loading it from a config file. And then I can actually execute that rule. And then when I run this particular example, well, it runs a bunch of rules including the ones that it loaded from the config file. So that means that with this change, you can now move parts of this business logic into uh JSON or a YAML file or a database. You could let your product owners tweak uh thresholds and conditions in the user interface. You can reuse the same rules in code and in config and all thanks to this specification pattern where everything still composes nicely and stays testable. Now, in my opinion, like I

already said in the beginning of the video, this is like nextderee overengineering and I happily accept your criticism in the comment section because it's all worth it to me. I mean, this ended up being such a cool design, and I didn't want to keep it from you. I mean, seriously, this predicate class that allows for logical composition, the decorator that generates these predicates, the generic version that works for any domain type, and finally, how this allows you to turn your business logic into data that you can specify in a JSON file or in a user interface. It's insane. But I'd like to know what you think. Did you learn something interesting today? Did this give you some ideas for your own projects? Do you think I'm completely out of my mind? Let me know in the comments. Like I said, it's very unlikely that you will actually use this, but this whole combination of predicates, thunder, method, overrides, generics, decorators, it's just a really cool design. Now, if you want to explore more software design patterns, maybe not as overengineered as this one, check out my design patterns playlist right here. Thanks for watching and see you next