The ins and outs of context managers and try-finally in Python

a big part of writing good code is developing good habits that prevent you from making easy mistakes context managers and try finally are tools to help you do that the point of both of these is to run some cleanup code even if an exception occurs did you open a file then you better close it did you open a socket create a temp file then you better close and delete it or did you acquire a lock then you better release it context managers help us ensure that all that cleanup happens automatically even if there's an exception if we weren't worried about exceptions we might just open the file use the file and then close the file but if an exception is thrown here then the Clos line never runs and the file stays open until the whole program exits this can also be an issue if we put in early return or a break or continue in this middle section that prevents us from ever reaching the close but the file object returned by open here is a context manager which means that the object itself knows that it needs to do some cleanup and python lets you ensure that cleanup happens using a width statement like this or as it's more commonly written like this did you open a file the width statement automatically calls close did you open a socket create a temp file the width statement automatically calls close and deletes the file or did you acquire a lock again the width statement automatically calls release if you want to do multiple of these things like opening two files you can even combine them into a single WID statement and python will ensure all the cleanup still happens you can even use the built-in context Libs exit stack if you have a variable or unknown number of context managers that you want to use I pretty much never need this but maybe you will but starting simpler the important part for now is to understand when the cleanup code runs we'll get to how python knows what cleanup to call for an object later the rule is that python runs the cleanup code whenever execution leaves the indented block I happen to know that for a file the cleanup code is just calling the close method so let's do some surgery here and modify the close function so we can see when it runs by adding a print the function takes an object's old close function and creates a new one that just prints closing and then calls the old one it sets this new close as the object's close and then returns it back let's put that to use in this example if there's no exception close runs when we run off the end of the block after the last statement in the block and before the first statement outside the block so as expected we see before width then inside width then the closing run so we see closing and then after withth on the other hand let's see what happened if there's an exception here python saves the exception runs the close then continues to raise the exception this time we'll see before width then inside width then the exception will raise causing the closing to run so we'll see closing and then the exception will be caught and we'll see as expected we see before inside closing and then Cod exception similarly if there's a return break or continue that would cause execution to leave the WID block the cleanup code runs on the way out meaning after the statement that causes exiting the block but before the next statement that would otherwise Run next keep in mind this can mean some weird execution flow like running code after the return statement in a function but before control flow returns to the caller in this case we'll see before then inside then returning then closing then the stuff after the return even if you're already familiar with using withd statements you may not have ever thought about how weird this actually is we literally have a return statement and we see that it's executed because we see returning and then we have code that's running between that and the first thing that we see after the return anyway one way or another when you leave the wi block python will run your cleanup code this is essentially how a try finally works as well in this case we can achieve the same effect as using the width statement by putting this file close call in the finally block the main difference here is that I had to know to call file. close as opposed to the width statement letting the file object itself decide just like with the width

statement whenever control flow leaves the try finally the finally part runs on the way out if your try also has except or else Clauses the finally runs after any of those in this case we'd see before try then raise the exception and so we would catch it and see Cod exception then the finally would run we'd see closing and then after try and just like with the width statement keep in mind that finally code can run after a return statement which means the finally Branch can actually interrupt replace or prevent a return in this case we start to return inside but before we actually return to the caller the finally runs so we close the file and then we return from our finally this new return actually overrides the previous one so this one is just completely lost and the function returns finally and if you liked that one this is an infinite Loop it never returns even though there's just a plain return right here this is because although we start to return something the finally runs on the way out and that hits a continue which then starts at the beginning of the loop so that return when we try to return inside is just completely forgotten about it's lost so I hope you're starting to get the idea that maybe returning continuing or Breaking Inside of finally isn't such a great idea just to make matters worse if you use a return break or continue inside a finally that will discard any inflight exception just like a bare except pass would so it's generally not recommended to return break or continue inside a finally in this case we tried to raise a value error there's no accept Handler in site but because we returned from our finally it's gone it's just gone although by following these best practices you're not normally going to run into these issues in general the semantics of trif finally are somewhat simpler than context managers in fact the semantics of the width statement are actually defined in terms of Tri finally so this little width statement is actually semantically equivalent to all of this extra stuff using Tri finally don't worry I'll put a Time code so you can find this part later so although Tri finally is somewhat simpler semantically than context managers because TR finally puts the burden of remembering how to do the cleanup on the user of an object you should prefer to use a withd statement over a try finally if you can now most programs aren't going to be opening that many files so unless you're opening tons of files in a loop it's unlikely that you're going to hit any kind of limit so it's more of a hygiene thing for files more than anything however that's just files for a lot of your cleanup code you can't just ignore it and expect everything to be fine forgetting to delete a temporary file could mean leaving an unwanted file behind for your user and forgetting to release a lock could cause your program to freeze while another thread is waiting to acquire that lock and in general cleanup code can be pretty much anything so other parts of your code may just depend on it running now on to the how of context managers how does python know what cleanup code to run when you exit a wi Block it's all part of the definition of the context manager a context manager is defined by just two methods enter and exit if you have these two methods then you are a context manager and you can be used with a width statement the enter method runs at the beginning of the width statement and you can grab its return value here you can actually return anything you want from enter so if I returned 42 here then CM would be 42 but the way it reads in English with expression as name it really looks like name should be whatever the expression evaluates to for that reason it's kind of an unspoken rule that enter should return self anyway the purpose of enter is to enter the context that the object defines don't let this fool you though many context managers do nothing when you enter the context except maybe check that the exit hasn't already been called for instance a call to open opens the file immediately even if you don't use a width statement it's enter just checks the file isn't already closed contrast this with a lock which actually does work when you enter the context if acquires the lock both ways of doing it are common just do whichever one makes the most sense for your situation successfully returning from the enter function is what defines having entered the WID block and yes that means that if you failed to enter because you're enter function through an exception then the exit never gets called this makes sense because if you never successfully acquired the resource then you can't be expected to release it but once you've entered then python ensures that if you exit then the

corresponding exit function will be called on the way out the exit function is where you put your cleanup code your file close your lock release the signature of exit though is a little bit wonky python gives you information about whether exit is occurring because of an exception or not the parameters are the exceptions type the actual exception and the trace back if you got to the end of the WID block normally you'll see all three of these arguments are none the wonky part is that if you have an exception the type of the exception can be accessed like this and the trace back so those arguments are completely redundant but what if someone tries to raise an exception class instead of an instance could the exception be none but the type be value error you can raise an exception type instead of an instance but no python will instantiate the exception instance before calling your exit function so the type and Trace back really are redundant so what's the purpose of providing this exception data most of the time you just want to do nothing with it if you do nothing with the exception it will automatically continue to propagate after the exit function Ends don't try to manually re Rae it only raise an exception inside an exit function to indicate that the exit itself has failed like if you went to delete a temp file you created but receive a permissions error when trying to delete it because someone did a recursive pseudo mod on the directory while you weren't looking that would be a failure in exit itself on rare occasions though you want to suppress the exception from being raised just like try accept finally can use the accept Clause to deal with or ignore certain exceptions you can prevent certain exceptions from propagating outside a with block by returning a truthy value from your exit function the simplest example of this is the suppress context manager from the built-in context lib let's take a look at its source pretty simple it just takes in some exception types and then during cleanup if it sees one of them it returns true to suppress it other than using this built-in suppress I can't remember the last time I actually wanted to suppress an exception in a context manager though most often I just want to let exceptions propagate so I hope you agree that most context managers are extremely simple enter does nothing but return self and exit just calls some other cleanup function let this be an invitation to you to write your own simple context managers if you have a class that you need to open do something then close begin do something then end start do something then stop activate do something then deactivate anytime you have a pair of functions like this consider creating a Contex manager to ensure that proper cleanup is impossible to forget even if you aren't the author of the code or don't want to modify a class you can still write a free function that uses someone else's code with a context manager say I have a library I'm using that has a begin end style API but doesn't Implement any enter or exit methods I can still create a function that calls begin and returns a context manager whose exit calls end if you need the return value of the original begin call this would be a case where returning something other than self in the enter could be useful here we return the result of the original begin call so I can pretend that the original begin call return to context manager and use it like this instead of begin use end we just have a single with begin see my video on python ising the M GUI library for how I personally did this with the open source piim guui library in that Library it was particularly important to use context managers because in mgui a forgotten end call Will outright Crasher program at this point I should at the very least mention the context manager decorator from the buil-in context Li module this decorator does some magic to build enter and exit methods using a generator function that you provide the way it works is you write a generator with a single yield the stuff up to the yield is called in the enter method which Returns the yielded value then the stuff after the yield is called in the exit method it looks nice and clean until you realize that generators aren't magically immune from exceptions so in order to ensure the code after the yield actually runs you need to wrap the yield in a try finally which to me kind of defeats the purpose of using this method yeah it's still just a single function which is nice and it's shorter definitely shorter than defining a function and the class with enter and exit but I still prefer the class way it's really up to you if you prefer this approach some people love it but I pretty much never use this we should also talk about using context managers for maintaining some kind of actual context I don't know about you but I'm not sure what kind of context text opening a file really entails I think of

it much more like cleanup not context to me entering a context sounds like setting up some variables to be used within the scope and exiting a context sounds like unsetting or resetting those variables to their old values these types of context managers are often used for temporarily setting Global variables this can be achieved as follows when we enter the context we grab the original value of some Global and set it to a new value and then and in exit we put it back if you have a bunch of values that you might be setting this is often wrapped up into a global context object and you make a copy of the global context before making modifications this is how the built-in decimal modules local context works there's a global context object for the decimal Library well technically it's a thread local but we only have one thread the local context function copies the current context and sets the copy to the active Global context this way we can do a high Precision calculation in a local context if we have a particularly sensitive function then the global context gets set back to whatever it was when we exit the width block bonus points if you understand why this actually computes Pi in any case you can see the first call to Pi has less Precision than the second one and finally we need to talk about the limitations of context managers and tri finally although both of these do their best to ensure that your cleanup code runs you can't depend on them absolutely firstly your python executable is at the mercy of your operating system and your OS can kill python without letting The Interpreter do any cleanup if you call cy. exit1 you'll get a nice system exit exception raised and all the cleanup will run as expected on the way up but if you call Os doore exit one there's no exception raised context manager exits do not have a chance to run finally blocks do not run add exit handlers do not run it just calls the underscore exit function from C which the posix standard at least guarantees will close file handles but other than that do not pass go do not collect $200 the program is done but even if you're shutting python down normally using cy. exit or hitting control C to send a keyboard interrupt it's still possible that your exit won't run properly if you just so happen to interrupt execution right here in the enter after the lock is acquired but before the enter returns or right here in the exit after exit starts but before the lock is released this is a pretty unlikely thing to happen in a real program but you can force it just by doing it in a loop we'll have a while true that just continually locks and unlocks this is an infinite Loop that will interrupt with a keyboard interrupt as the interrupt travels up we'll hit this finally where we can check whether or not the lock is unlocked most of the time when I interrupt the program we can see the lock was correctly not locked when the program exits but some of the time we see that it was still locked like here there's nothing particularly wrong with not releasing a lock before the program exits I'm just using this as an example so you can see that even in a semi-normal exit case you can't truly depend on python to run your cleanup code this can actually be prevented but not within python itself if you write a custom C extension module then you can ensure that your enter and exit functions don't get interrupted which is why things like the built-in lock need to be implemented in C if I use a real lock here instead of my custom wrapper you'll find that the lock is always released on exit for exactly this reason it's written in C and just doesn't give control back to The Interpreter in the middle of enter or exit anyway that's all I've got thanks to my patrons and donors I'll see you in the next one