Logs in n8n by Omar Ajoue

thank you everyone for joining it's really a pleasure spending some time with you talking about nhan and what we are have been working on so um well let me share my screen i have prepared a small presentation for us to go through so um if you're having any issues seeing this please let me know and okay so well first off who i am so as harsh already introduced my name is omar i am one of any 10 senior software engineers i was born in brazil i moved to berlin to work for natan recently i just got to berlin last december and well as a senior software engineer i work in a few interesting projects on na10 the scalability part which we had some conversations about previously already logging and now working on some interesting projects as well uh related to the future of node building on an a10 more than the technical part so some interesting stuff that we'll be coming to so those are my contact information in case you want to connect feel free please so without further ado let's go into what are locks so just to set the level

here um well systems usually logs are away and messages that a system communicates with you to help you identify what is actually happening under the hood so for instance you can be using logs for tracking events that are happening logging transactions that might be relevant to your system or to your business logic maybe for audits um tracking requests that are happening to your system so you can for instance count user usage and perhaps block access or check for malicious actions so logs are basically a way of the system communicating what is happening okay so um blocks can be written and generated

in a big variety of formats um some of the most common like delimited with text in case like you have a tabular structure of data uh tab delete delimited messages which is like csv files um just plain text human readable messages one of one line uh per instruction json files csv uh that i already mentioned parquet which is a format that is used by some of the apache products for parsing and collection of data as well so blogs can be generated in a variety of formats and the format that you're using logs actually the chosen implementation will arrive based on the use case that you'll be used right so in nhn specifically how do we generate the logs so we decided to use a json format for a few um interesting points that it provides first uh well nhan already uses this json extensively so it's something that most of our users if are not familiar with will probably super quickly get in touch with and understand how it works and understand the structure so it easily allows you to show and see data in variety formats lists of stuff complex objects nested information so this is one of the reason which we choose json and also because it's becoming more and more popular for logs as more um it's more and more apps are using it choosing logs for this uh so this is the reason why we chose uh json as performance for nhm logs so um i'll go through some of the default settings in na10 so one important point is that by default to make an a10 super lightweight and not taking up take too much uh disk space locks are turned off by default so this is like to make it non-breakable as possible like no braking changes so you can turn those on and we'll go through that in a second unless you change the location the logs will be saved inside your dot and a hand folder in a subfolder called logs and it can also do what we call log rotation which means um the fiber of the information will grow up to a certain size when it reaches that size uh and it can automatically rename it and create a new one with the hot data so you're always having some sort of archive to go back to if you need um by default again anything uses 16 megabytes per file and saves up to 100 files so you have a very good history of information with this okay so now that we know like how anytime generates logs and where you're saved um how are those log messages organized

what can i expect from the log messages that napan generates we created a simple structure that it's easier to read so if you're not familiar with json objects that would be super familiar to you we use the level which is one of those five options error warn info verbals and debug and we'll go through each one of them those in a second uh the message itself which is a usually a human friendly message uh we do our best to make to write messages that can be understood by anyone so we're writing plain text messages that help you understand what was happening and there's also an interesting object which we call metadata which contains the time stamp so basically when that action happened the file and function where an a10 was running at that time and this really helps a lot when we need to debug something if we have an issue with anything and also you can add additional data like for instance the id of the workflow um the execution id at the moment this the user session id and session id in this case i'm not saying like uh understanding who is the user but actually simply a connection between the browser and nhn so that information this information can flow so it's not actually personalizing someone it's just it's impersonated data it's sorry it's not impersonated data

and so now what are the log levels um so error is the first and like most critical one which means anything is reporting some faulty behavior which means something went wrong those messages are always locked um usually if you don't change any of the enhanced settings you will see those in your terminal and we'll go through the settings in a moment but those are always visible for you second level is born which means this is an important message but not necessarily an error this could be something like hey there's a library that was deprecated there is information about um some node that needs updating for instance so those kinds of messages that are important that you need your attention are used as form and for general information like oh anytime started then it happens listening to poor texts like that i don't know web hooks have reached some sort of information that is more um it's not an error it's not a debug information it's more like a general information overall you have the verbose which is actually an a10 telling you everything it's doing so i got to remember that logs are messages that the developer adds to the code so whenever we as developers or engineers or you as a contributor decide to hey this is an important step that i want to check what's happening you can naturally add those informations and we'll talk about that too so verbose is a way of telling you that we reached a certain point maybe a checkpoint that is interesting that helps you understand what is happening and the difference between verbose and debug is that debug also contains more data about the current state so verbals is like more an overview of where m810 is going so where it's heading and debug is really pinpointing with a bit more information saying hey i'm specifically in this node and i'm taking this action the operation is xyz for instance i'm calling a certain resource this is a way that helps you really understand exactly in detail what was happening at that moment okay um

um now that we said about it how do we set up logging for na10 so basically you would need to look at two different settings the first one is the lower level it means what do you want any hand to output these are like cumulative so if you set error only errors will be shown but if you set one and it can go output errors and warm so as a consequence if you set debug and a camera will output all messages to you so this is one easy way of letting you know like uh i want the most verbals one it's actually like the default and the second most important setting is the output so usually it's just console unless you change it which means uh information just being output your terminal you can set it to a file or both so those are the three or three options currently available we would love to get more options in the future but there are a few consequences to this which are adding more dependencies to anything making the nhm docker image or the app size bigger and naturally with external dependencies we have problems with security concerns so we decided for this moment to simply use file and console which are super standard and if you need to use an external system you can set up your nav to write information to a file and there are applications that allow you to leave information from a file and send to an external system like loggly or log stash foreign and a few letters also um beyond those two environment those two settings that i mentioned you can also set the file counts which i mentioned in the beginning 100 files so you can change this number up or down according to your needs the maximum size for each file which is 16 megabytes by default you can increase or decrease that we chose 16 megabytes because it's a file that is long enough to store a good amount of data that you can read through and understand what's happening uh in a good period of time but at the same time not so huge that you can that you would be unable to open it with like a regular text editor and finally the files location so where do you want the files to be stored as i mentioned uh they will be stored inside the nhm home folder which is like the dot n a pen folder inside the a sub folder called logs and the files will be called anagan. log and when rotated and again dash one that's true that stream um some more information about login you

can always go to the documentation this contains the most upshifting information i'll leave this qr code here for a moment so if you want um you can have a look there but it's just a anytime documentation it's super easy to find if you just type blog or login in the search and you'll easily find this link and one important thing that is there that i will not be covering now is that this page also covers if you're interested in the contributing to anytime how do you use the anytime tools to add log messages i mean to create more log messages if you see hey i'm trying to diagnose something to debug a specific situation there are no log messages available at this time so you can easily add more love messages make contributions when they can as you know our source code is always open and contributions are super well

also well let's touch a little bit about the logging best practices so as i mentioned logs are not saved by default they are just output to your console so my recommendation is to set and again to save those two files so you can easily rely on those files in the future whenever you need to debug something or find a situation this is super useful for us like if you're recording a an error on nam those log files can be super useful in those kind of situations um we do not expose any sensitive information of the log messages that we added so far did not expose any sensitive information like credentials or so uh they just tried to really pinpoint what was happening what the what are the resources and operations that were happening so uh it should be safe to have those files although i would really recommend if you're filing a report please send those files privately as this is the best way nationally so uh if this space is not an issue i would really recommend you use the default level it's really the one that contains the most information which is like the purpose of logs but if you have a space in constraints or some sort of restriction definitely you can set this level over or use a debug but set um the number of files to be small you know uh one also one another important thing is that well those are files so they're stored um in the file system and the hard disk and if you're running an extended a container let's say and you update an a10 so you spin up another container with an update you lose the information in the previous container so this is one of the reasons you should be using volumes and this is described and um in some examples on how to animate them to actually map the data over from inside your container to your host machine so that you don't lose this data in case the container changes you know um so also if you're unfamiliar with this please let us know uh there are examples for those in the documentation we can help you with this um and also like now that you have the files in your file system it's easy to install external tools like demons that will upload those files of those logs to external tools like log stash locally and cloud watch like these are super standard items and remember logs are your friends they will help you in debugging and finding any some useful information in a lot of cases and we are adding more and more logs every time so that any end is each time more helpful and understanding what's going on and helping us build pinpoint issues so if you have any questions please feel

free to share um i hope it wasn't too quick and well thank you