Unit Testing Framework for n8n: Preview from the Prague Meetup (Sept 2024)

all right hey everybody hope everyone here is doing good hey my name is Liam mcgargle I uh am from Philadelphia in the United States and I came all the way here to Prague to show you guys what I've been working on inside of naden uh before we get started I want to take a quick survey of the room uh Bart helped me along a little bit here um but uh does anyone here consider them themselves power users in naden like you're a real big Advanced user I know there's at least two over here what about in your program at Choice make. com any of those no power users I think you guys probably selling yourself short so most of us are familiar with the workflow based automation right though okay so uh who here relies on those workflows for like their work their personal lives right one person two people okay so of those three people in here uh does has anyone ever had a uh mistake take down a production workflow no never and give you a little bit of a headache find mistake well listen mistakes are part of the process this is a very normal thing to happen and it's definitely not unique to naden is such a great tool the accessibility of it being uh free and open source to get started into it the support behind it is great um and it's even beautiful uh I feel like it's the perfect mix of being easy to use and having the technical superpowers if you do want to go in and become an advanced user there's 's just one thing missing for me which is unit testing so unit testing is what developers made in the coding world to check all these little mistakes before they make it to production so it enables you to make a bunch of little tests that run automatically when you save your workflow project every series Dev project uses unit testing for example nadn runs hundreds of unit tests uh before every version release to check all the little logic they have in there to make sure we have a nice reliable experience uh so I thought why not bring that unit testing inside of nadn so here's my simple proof of concept um on the bottom here is what you would normally have as your workflow so this is using the naden trigger form which gives you a for those who don't know about you uh naden it gives you a guy UI with a form that when you submit it brings the data in through here and then just has some um contact information parsing and this will be just whatever your lead automation adding to the system stuff is over here what really matters here is this testing stuff I put on top it's a bit of a mess right now I won't spend too much time explaining it but essentially what is happening here is there's an external workflow for each one of these tests and then to run these tests that external workflow is calling the beginning of these sending in a certain amount of test data in this case emails and phone numbers to check the logic there and then the result going back to it is checking whether it was a uh past or a failed test and then these workflows up here because you can have multiple triggers in n8n this runs whenever this workflow is saved or activated which is a vanilla built into n8n and this will run those two external workflows so whenever this workflow is saved or activated it will run those tests and make sure all of our logic is still working as expected so this is one of those external workflows for that test again this doesn't matter that much for what I'm doing but uh I'm passing in an array of phone numbers here all the different for formats that I want to test uh you can just see it's all different styles there it goes out splits it out do all the stuff to get it ready to send it this right here the HTTP request is what actually is sending it off to that test workflow and then down here at add expected results I'm saying this is what I want the output of that test to be if anything comes back that doesn't exactly match this it's sending it as a failed test and then uh we'll get a notification that it failed and it will turn off the workflow for us and uh as you guys can probably tell that looks like it works but it's pretty clunky I doubt anyone here would want to use it and it definitely creates way more problems than it solves uh but I was still super excited about this the ual testing part about it worked it checked the logic it did it automatically and it gave me uh the information I really wanted to have uh so I started the brainstorming process to try to improve it so here is my uh original list of requirements and the sketch that I had going into this um essentially what I wanted was a test trigger node that could go in and it could encapsulate all of the test runs you want to have in your tests so all the different test cases can go right here no external

workflow required and then there's a test evaluation node where you would have all of your test um evaluations so you could see what was a pass or a fail I also wanted a fail Branch so whenever anything did fail you can set up notifications or whatever you needed and then a cleanup branch which is really important for nadn because there's so many different connections that we can make sometimes we'll be making something externally or internally that we need to clean up and delete after so the list of uh technical challenges to overcome was pretty large on this one uh specifically the important stuff like um starting and stopping on the right node which since not all of us are familiar with nadn what would normally happen here uh on this workflow if we ran this test it would go every opportunity it can past so if this test ran it wouldn't know to magically go up here and stop it would continue along to all the other stuff like if you're adding a Salesforce contact it would go and it would do that and break your workflow even more so that is one of the big tech technical challenges to stop also and uh that a single button press can run all of the work the tests on the workflow a button to hide all so you can imagine if you add a bunch of tests to the workflow it can get pretty cluttered and uh can make it harder to edit going forward but after a few sleepless nights later uh here is the Prototype that I have working uh it is a bit nicer looking than the proof of concept I think you would all agree uh it has pretty much everything uh that we laid out in that original brainstorm let me show you how it works so when we start with the input node here we have the test ID which this needs to be the same in the test uh in the trigger and the evaluation node it's how it knows where to start and stop we can add all the test cases here like this we can add up as many as we want that can be either in key value pairs like this uh or in Json if that's how you prefer to work like me and then these test run names I'm putting in is optional but it is super helpful because you can add dozens of tests in here and if something fails then you'll know exactly it failed you'll know what to go and fix pretty quickly and then you can probably tell from this example what we're going in and testing with this is just a simple phone number formatting step for us style phone numbers it's taking whatever input it is and it is giving me what I like to see in my phone number and then in the evaluation step oh and I have the include other fields disabled which again for those that don't know about naden you can have steps along the way that clears the data in your workflow this is a little more advanced for those so I won't sick on it but essentially you can pretty much lose data that goes through and you need to add an extra node to get that data back um so that's just something that helps out uh I'm just showing that I am removing that data and here I use this like an if node like a normal uh control flow everything that evaluates to True will be a pass test everything that evaluates to false will be a fail test and if we run it there from the workflow we can see that the cleanup branch has all of the tests in here all those four phone numbers we put in uh the tests whether they pass or fail go in the cleanup Branch so we can clean up everything from every test and we see we actually have a failed test here and we can actually if we're not using the cleanup Branch like we aren't here we can disable it right there and then uh we'll just reconnect that node and then we go back in here we can see it ignores all the past tests if we don't need it and we can just deal with all the failed tests and you can notice that metadata from the test trigger node came back even though we scrubbed all that data away this is something I wanted to make sure we had so we can have really good error messages that we knew what the test was and what the inputs were because uh as you'll see in a second here is an example of an error message we can make of when something fails so since we make sure we always have that metadata back at that last step we have as much information as we could need to immediately know what failed and what didn't and you could mark this up with links to the exact workflow run and all that fancy stuff and now that uh we know kind of how to set it up we can move on to seeing how it will run so you can run it from the workflow just on the regular play button like I showed already before without the changes I made to the nadn code base that would start from one of the other triggers not with the matching

ID we can use this new run unit test button which will run all the tests in the workflow as we can see from the executions here both of those tests ran at the same time uh it will also run on activation by default when we activate it you can see those two uh tests ran again um and it will also run on Save on an active workflow by default if you wanted to change these settings you can go into one of the evaluation nodes and you can change the settings by evaluation node so each test can be different it's just in the additional fields we can even turn on Save on inactive workflow which we can see that ran there with the inactive workflow so you guys might be thinking running the tests every single time uh you save probably seems like Overkill like that's a lot of test running if you're saving a lot um and I would agree I wasn't originally going to add that but when I was testing this testing the tests I was noticing that I was catching a lot of edge cases and errors that I wasn't expecting to have while implementing the test and this actually calls to an existing um method of development called test driven development where you set your original end goal in tests and then you iterate on your development until you pass all your tests and that will leave you with a product that has less errors overall that's already tested to your capabilities you can take a halfway approach too where you set up your tests first and uh because of that I set up the save on inactive workflow and also an error notification so if it fails it can just pop up in red and you know you failed something those would likely only be on while you're actively developing but um it's easy to turn off and uh gives an easy way to test as you go so here is an example of a bigger workflow hiding and showing I probably should have had this clip stay on a little bit longer but I was um meeting the size requirements for Google Slides uh this is actually the workflow that inspired this project after we delivered this workflow to a client uh there was a small change made to it with a very small syntax error that took down the whole workflow in production and I thought uh if I coded this I would have written a unit test to cover this and so that's where it all started so I thought it was a good uh symbolically a good workflow to show with a bigger more complex workflow showing all of these um being hidden so we can see we still have a nice beautiful canvas to work with if we want to add to this um so this is the result of two weeks of working at night so it's not exactly a polished final product yet it is just a prototype so I have a small list of bugs to fix and a very large list of things I want to add to it um I think this is the perfect stage to share it to a room of people though so I'd love to hear from people uh about what they think of it and which direction I should go in terms of continuing to develop so it's super easy to try it out if you're a developer um my GitHub Fork is public you can take a peek at the code you can reach out about uh contributing I'd love help especially with the view specific stuff um and if you're just a regular user or if you're a Dev and just want to try it out uh my Docker image is public the only thing you have to do in any Docker command just substitute out the n8n URL with my Docker image and it will seamlessly transition and change between just don't use it for anything production yet because like I said it's still a prototype use it at your own risk uh in production um and please give it a try reach out with any feedback back any of you who haven't used naden here's your call to action go use naden with or without my Docker command this is all you need to paste in your terminal and you're up and running completely free so that's all I have for you guys I'd love to answer any questions if anyone has questions about that I know I dumped a lot on you pretty quickly yes sir mostly I'm truly like mindblowing stuff just seeing how to use base like will like freak things and tweak things is a great thing about you know the sours available but I'm curious like two questions one is this something you would have done differently and how you kind of architected or thought about it now that you've built it with the kind of hindsight building it out oh yeah maybe it's that's a great question and to touch first I meant to thank naden for this because as opposed to zapier and make which I love competition I've used both of those products before you can't do this with zapier orake you can do this with naden and it is a great opportunity for us I can go and share this to everybody in the world and they

can use this instead of nad's official image and that is open and they get excited about it like that is amazing um in terms of what I do it differently in the future I have been thinking about this for a pretty long time probably 3 to five months before I actually started on it so I've been whiteboarding it for a long time um I was going to do a workflow based approach with similar to that proof of concept but with a couple UI buttons added in I decided not to go that rout and to do it inside of their so we could test specific logic node by node or multiple nodes you could put a test node in evaluation node to cover the entire workflow if you decided so answer to that I wouldn't have done it differently but I still have a big list of stuff to do that will add a bunch of other stuff that I didn't expect anybody else yeah uh why wouldn't you just create a one to one copy of the flow and run test the data through it that was one of my can you repeat the question for the repord so the question was why don't I make a copy of the uh workflow and use that person and write a bunch of thousands of lines of code to make this work I was going to do that and I was going to have three separate um instances one development essentially one staging and one production like they do in the coding world and everything gets tested and thoroughly messed with in the production I mean in the Stag uh the reason why I didn't do that is because to test you could put even on that fairly small but more complex workflow I showed you can put 100 of these unit test nodes and test every single piece of logic a thousand times with testing an NAD regularly vanilla without modification even if I had a copy of the workflow you can only do start to end if that makes sense and to do start to end and testing every Edge case with every phone number start to end it's a bit clunky and a bit much and it gets pretty difficult if that makes sense great question that's that was my first idea yeah it creates a lot of unnecessary processing as well exactly I would have to do some crazy GitHub actions yeah all right anything else yes sir I'm curious Liam like for some of um like I don't know if you you've done these unit tests now for some of your cant work but how much would you estimate how much more overhead in effort is it to be shipping you know with this now here's the workflow now here's a unit tested prob ready workl no and that's a great idea because um that's kind of a uh I don't know if trop's the right word something in the development industry where uh things that are done cheap and fast aren't unit tested and then they hire other developers after and things break as soon as you try to touch anything so it would definitely add development time I would estimate maybe one and a half times if you're going to try to get everything possible that you can do but then for the customers that would add you know a lot of value in that they can go in and edit something small which naden is great for that someone really technical can go and create a workflow they can go change a couple things around with the UI and be pretty sure that the logic is locked down if they mess something up they'll be notified about it so uh that is essentially that's essentially to answer the question too I was actually surprised with how much better it made me work with it I didn't use this in anything in production yet I finished this on the plane on the way here like I said I've only been working on it uh for a couple weeks um so uh I expect it won't be as more than as expected because um it has you think of the edge cases up essentially yeah so maybe quick question Y what would this mean for person that would Implement their own custom nodes so there is yeah so the question was what would do be difference if someone wanted to use their own custom nodes or Community nodes there would be absolutely no difference uh the as long as you use this Docker image that has this built in this is exactly the same as everything else with NN the inputs and outputs of regular nodes the logic of that hasn't been changed at all and this is actually so compatible you could a current production workflow that you have or a current production instance that you have you could just swap out the docker image and it will function the same exact way it'll boot up Acts exactly like if nadn was just updated so you could actually move back and forth between the unit test and not which the way I'm going to use that is do the same thing with the uh Dev um staging and production I will have a lot of work being done automatically in production and on my Dev server that will be pushed automatically with uh cicd tools which is a whole different topic uh won't have this built in it will be a fully vanilla thing but everything will have been tested before it gets pushed there that makes sense but it will work with

custom okay um I have two questions to um and updates pretty frequently once a week or so um is that are you able to keep up with that yes so the question was uh nen updates a lot am I able to keep up with it um anden does update a lot I went through two updates just in the two weeks of having to go and fix code uh conflicts but I will be able to keep up with it pretty easily because nen is built well it's built very modular like a modern code coding project should be which essentially means everything's broken up I file really well all I need to do to release this is one Docker command on my computer as long as um code doesn't conflict from the N project to mine which means where I added logic as long as they didn't edit something that happens to be in the location where I added code which I said happened twice already um it's then I can just do it automatically with no extra work for the rest of it's very separated ex exactly so it's for the most part separated so for most releases it it'll be as easy as me sending a reminder pressing this and then you can use my new one which will be justce updated and then when it's uh has merging conflicts that's not that big of a deal those are pretty easy to fix I just have to make sure both changes will still work that stuff usually isn't that bad but it's a little bit more work and I'm a busy guy so sometimes this will be a couple versions behind mind um second question is um so you you showed how you can run the tests when you save your workflows or um but n then workflows don't only depend on their internal logic but also on external apis right and those change too yes so are you planning on having like scheduled tests or yes so um I know I have been going through stuff pretty quickly I think in here I might have that might fall under the many more ideas nice but that uh falls under here has another note to start testing another workflow and you can set up a crown trigger a schedule trigger to run the tests perfect but that is definitely a uh that could be useful for our internal tools as well because we do sometimes not notice when API updates yeah and that's one of the things about the cleanup jobs is we can actually test um external apis and AI nodes like for stuff like prompt injection we canest test it on a schedule see if it's within the thresholds we want it to be and then get automatic notifications that can be customized on that fail Branch to really know exactly what's going on with your external tools one of the big things I want to work on that I'm excited about is um test credentials uh there's a lot of tools like um Salesforce Zoho uh there's lots of tools that have production and testing credentials even stripe uh twio tons of stuff so I'm going to add in there it's on my list anyway uh if you have a credential with the same name but with like hyphen test or something it will automatically substitute that out if you're running a test workflow I haven't really ironed that out completely yet but I think that would be pretty cool cuz then you know if you have a whole team working and then you have stuff on sales ports popping on and off that's not exactly ideal um even if it's only for a short period of time that wouldn't really fly in a big company uh so I'm excited about the test credentials because that's something important for me I answer your questions definitely Perfect all right anybody else all right perfect pretty impressive thank you can noce a careers URL I showed ear ER listen I'm loving it hear in Europe I would uh I'd consider it um thank you guys so much for the time I am so happy I made it here speak and talk to everybody and I'm glad I uh committed to doing this so that I had the crunch time to do this um at night so thank you guys so much again if you uh need my contact info it's right there and if you want a card I have a bunch in my pocket so come and see you later thank you awesome next