Machine Learning for Log Analysis Explained by @dankornas

hello there my name is Daniel cornis or you could just call me Dan for short and I am a machine learning engineer and today I want to talk about how you choose machine learning for log analysis and when I mean logs I don't mean trade logs I mean server logs or logs are generated from various products so let's get right into it so in regards to logs let's talk about a little bit about what these logs are see logs are generated by various products whether it's a server or base stations or any kind of basically electronical products some kind of logs are being generated and with the help of those logs various developers and testers can use that information to test debug find problems fix problems and so on and so in a specific example that I have previously worked on using BTS or base station information for analyzing or determining what problems may possibly

occur see base stations are these boxes that basically help us to communicate with our cell phones to the Internet so we can make calls go on Tick Tock or any of these various whole things that we could do on our cell phone this is all done with the help of Base stations our phone connects to that base station and information is transmitted back and forth and so as developers and testers we have to analyze what is going on that base station and so with the help of these logs we could analyze these various problems however the big problem with these logs is that hundreds and thousands of various logs are being generated every second and as testers especially we do not have the human capacity to analyze everything single specific detail or statistic we can only get a handful of the most important statistics and be able to analyze that and hopefully we don't miss anything else that we didn't check so the

solution for this problem would be to use machine learning and so with the health of machine learning like any kind of computer it can analyze all the statistics with a matter of seconds effortlessly and additionally not only tell us whether or not there's a problem but also explain to us what could be causing the problem in the future and so in the specific project that I have been working on in the previous years we use machine learning to analyze those statistical problems with the help of classical machine learning nothing really too complex or anything complicated we used machine learning specifically xgboost to analyze these various statistics we use historical data from previous historical tests from previous months to train our machine learning model to determine which set of logs showed correct behavior and did not show correct behavior and so with that we're able to create a classification model that told us okay something's wrong in this set of logs you need to check it out as a test and so we also went one step further uh with the help of an additional Library called shap which basically gives us an explanation of which features within the model uh steered the decision of the model to go either true or false or correct or incorrect and so with the help of that Library we were able to determine which statistics persuaded the model to determine that the outcome of these logs were incorrect or correct and so with that additional information we were able to analyze more specifically more thoroughly the various statistics or logs that were being generated and be able to find various bugs and problems faster before they went out into production because if any problems were found on production on the client side of course that costs us a lot more money to be able to analyze and fix those problems so it is possible to use machine learning for analyzing logs it's a lot of fun it's a very specific niche not the typical analyzing images or texts this is very specific text or numerical values but it is possible and so if you want to enhance your knowledge in this area or try to better understand how to construct these type of projects models from beginning to end analyzing the problem determining which model to use moving into production I highly recommend the book designing a machine Learning Systems it's an O'Reilly book very helpful very well explained of the various steps and processes that you need to go within the machine learning life cycle to be able to launch your machine learning products uh into production so I highly recommend using that book and so that's it if you want to check out more information that I post on the internet be sure to check me out on social media everywhere I am d-a-n-k-o-r-n-a-s Dan cornis that is my social media tag you can find me on YouTube Twitter Instagram Facebook LinkedIn wherever you can so thanks for listening and see you later