Low Code AutoML UI in Microsoft Fabric Data Science

hey everyone welcome back this is fabric espresso this YouTube series directly coming from Microsoft fabric product group and today we are going to focus on not data engineering but on the data science part because Misha is joining us as she join us twice or three times to talk about Automated machine learning and the recent enhan M recent updates Misha thanks for joining us yeah thanks for having me so could you recap what are the items you are working on and what are the key new changes that you have introduced in automated machine learning yeah so a few months ago we had released code first automl and really that was an integration that we had done with an open source library that came out of Microsoft research a few years ago called Flamel and what we did a few months ago is we released it we deeply integrated it into the fabric product and so as you're creating different automl trials you'd see that all of your different iterations trials would be tracked in your machine learning experiments and models and so our most recent release what we're announcing at ignite will be an evolution of that right now that you have all the code first apis what we're layering on is a low code UI which allows data scientist analysts any kind of user to come into Fabric and start getting started with building machine learning models uh through automl and so very exciting release um it's a very you know simple easy to use UI and so we're really interested and really excited to see more users start coming into machine learning CL awesome so the last time you presented Automated machine learning automl that was related to the code experience meaning that from we needed to code basically the experiments to find the best model and right now the key change is that there is just a UI I assume that's similar to the UI is available in Azure machine learning so basically I can select the group of algorithms so the objective regression classification that stuff yeah exactly so um as part of our low code experience it's really simple to get started right all you have to do is pick your lake house then you tell us what machine learning task you want and so in this case if you're say picking like a classification example it'll walk you through a few standard set of prompts in terms of you know the modes that you might want to run like are you looking for a prototype more performant mode that maybe runs a little bit longer and um you know experiments with more different kinds of models um and so you know just through a few clicks you can configure how you want your autom trial to run and what that's really cool is it'll spit back a notebook that templatized a lot of the low code not um experience behind Flamel and so you don't have to worry about knowing all the intricacies of you know every different API and how to run it it'll generate a template so you can go back and customize it you can inspect how the auto trial is created and so you know a real big focus on transparency there uh as you run your automl notebook and the templated thing that's been created for you you'll see all the different trials all the iterations captured for you all the models you know with our MLF flow integration will capture things like the parameters the metrics the model files and so very kind of simple easy to use way of getting started with your ml experiences so just through the UI I will get a code that is well written and with all the integration with ML flow included yes exactly I would love to ask about the context of this functionality who is the end user that you anticipate will get the benefit out of the automated mlu site yeah that's a great question so I think you know as we were doing the development of automl you know one of the things that we found was it's something that data scientists really want be able to use right a lot of times they're coming into their machine learning flows these projects and you know they just want to be able to do a benchmark or something quickly to prototype you know what the results might be and so it's a great way for data science to be able to do that to be able to you know just rapidly build machine learning models but one of the other things that we found is that we had a lot of users that were analysts or uh maybe people who are a little familiar with python but they're not experts but they also wanted to start coming into the data science workflow to also you know start playing around with data science flows start getting their feet wet with machine learning models and so it's a really great way for us to be able to let these users um access to fabric and data science capabilities and so our you know our goal is really to simplify some of these workflows and make it such that all of these different kinds of users whether you're Advanced data scientist or um an analyst just getting started with machine learning that you're able to come in and start building these kinds of capabilities I want to ask if the uh the UI extension that you are uh releasing

and nouncing is that compatible fully compatible with automl from code experience yes exactly so um as you go through the automl UI you'll see that we'll generate a templated notebook for you and this is you know you could have written this notebook yourself you could have gone through all the code and all the apis that are available on the docs and you could have written this notebook um but what the automl UI does is it really simplify some of that where you don't need to worry about all the different apis that you should be able to call and how to configure and what settings to invoke um a lot of that has been templatized and kind of automated for you so you don't need to worry about that um so all you have to really worry about is just running that notebook um if you want to do more customizations you definitely can and that's a core reason why we're giving users access to the notebook and the code that would be needed to create that automl trial and we really wanted to make sure that this whole process is transparent that you have control over it you can um configure all of the different things but at the end of the day you do have that notebook so you're able to reproduce your aom M trial and even look at how that was generated Misha can you share what are the scenarios when we should consider using automl yeah so automl supports a variety of tasks everything from binary classification multiclass classification regression forecasting and so you know we get a lot of different customer scenarios that we hear about forecasting definitely comes up a lot so do things like classification and like one example that I really like to use is uh let's say I wanted to break predict customer turn right the customers that might be leaving the bank or a particular organization in this case I might be working with a binary variable of you know one if the customer exits zero if they stay and what I'm trying to predict is the customers that are likely to leave and so here what I can use the autom mlui for if I'm a data scientist is to help me quickly get started with building a model that might help me predict some of these things right and so here what I would do is I'd pick my machine learning task which is classification I pick the mode so if I want just quick result I might pick a prototype mode and they'll just go through a few different um configurations right do I want to run it using Spark Run it with pandas and you know provide my final model files and um experiment names and then once I run that notebook that's created you'll see all the different results right but again a very easy way for you to say hey you know this is a machine learning problem that I want to be able to solve this is the task that I need to be able to do and then all I have to do is go through a the set of steps to get that final answer so um you know a lot of different tasks other scenarios we hear about are things like forecasting regression but um you know there's a lot that you can do and configure within this yes and our documentation is having all the list of there are dozens of algorithms my favorite one time serious forecasting arax Auto regression moving average with explanatory variables yeah lots of different models yeah so yeah there's a lot of different model types and what's really cool is um our code for experiences supported both single node models right like things like psych it learn XG boost also supported spark base models right or snap smell models so these are distributed models that work across multiple modes and so the low code UI gives you an option if you want to run it in parallel with pandas or with spark and so you know that's another great configuration that you can use to help you know tune your automl trial yeah distributed model training that's awesome I can't wait to see the demo let's go yeah let's take a look Auto Mount and fabric simplifies building and deploying machine learning models with minimal effort so let's take a look at how we can create an automail trial from a fabric experiment item first I'll explore and select from my available lake houses once I've chosen a lake house I can then select the files or tables within it to use for my automl trial next I'll select my automl task for this scenario I want to build a turn classification model after that I'll then select my automl mode since I want quick and Speedy results here I'll opt for the Prototype mode now I'll select my prediction column the exited field finally I can select how I want to run my auto trial using spark after reviewing my selections the automl wizard generates a customized Notebook based on My Chosen parameters this notebook not only allows me to further refine the model but also provides reproducible workflow for future use last once I run my automl notebook I can easily track the best performing model from my automl trial this is powered by mlflow integration ensuring that each model logs the key metrics parameters and model files necessary for reproducibility I can also inspect and compare all the other models that were explored during this trial with ease with these tools at hand autom melon fabric streamlines the entire process making it easier than ever to build track and optimize machine learning models Misha thanks for doing this amazing demo

now I want to ask what's the release stage of this functionality and what's the plan to make it generally available and here at the cave this we need to explain it generally available means that there is a SLA coming from Microsoft Azure and we recommend you can use it for production workloads for the preview features we really recommend you to try it but at the same time there is no recommendation to use it for production scenarios MH yeah so the auto feature is already available um both the code first and low code UI and so you can look through Microsoft docs and you'll find Links of how you can access low code automl experience um all you need is a capacity where you can run spark the other thing is we don't have a ga time frame just yet this is just coming into public preview um but what we would love is for feedback in terms of the features you'd like to see what you like what you don't like and you know you can for some of this feedback through ideas. fabric. com we'd love to hear um what you have to say good awesome so just leaving the link ideasfabric microsoft. com for double resonance with you and Micha thanks a lot for joining us today and for sharing the great news because that will for sure simplify getting started with Automated machine learning and for those who are watching us please remember to leave that comment and like button subscribe the channel and until the next time happy exploring being taking the role of data scientist and training your model just with leveraging Automated machine Learning Without coding thanks aome thanks so