How to implement Logistic Regression from scratch with Python

welcome to another lesson of machine learning from scratch by assembly ai in this video we're going to learn about logistic regression if you watched our previous lesson on linear regression you will remember that this is the equation that we use to find a best fitting line on our data set with logistic regression what we're trying to do is to create probabilities instead of a specific value and to do that what we're going to do is to put in our value inside the sigmoid function that looks like this and then get a probability distribution that will be between 0 and 1. and this is how the sigmoid function is calculated so our equation goes from this to this and to make it easier we also call this equation with an h so in the coming pages you might see it depicted as this so don't just don't get confused with logistic regression instead of using the mean squared error we're using something called cross entropy so it's a little bit more complicated than what we have time for this video so if you want to learn about how cross entropy works other than this equation that you see in front of you can go ahead and check that now again we do the same thing for logistic regression we need to use gradient descent and we need to calculate the gradient of this error function in terms of the weight and the bias and this is how it looks like again very simple we just jump to the end so that we can implement this in our code very quickly again what gradient descent is here what we see is a graph that shows us given a parameter value a weight value what the error is and this is their relationship gradient descent tells us given at any point of where we are in this graph which direction to go to minimize the error once we know the direction that we need to go to through the gradients what we do is we update the parameters so the weight and the bias by subtracting from the current value of the weight uh the gradient multiplied by the learning rate and again to quickly recap what the learning rate is tells us how fast to approach in this direction or how fast to go in this direction that gradient tells us gradient descent tells us to go so if you go too slow then you will make progress very slowly and you will not be able to you might make it to your minima in time it might take too long but if your learning rate is too high then you might keep missing your minima by jumping around but if you choose a good learning rate very likely you will eventually reach a point where your error is minimum so to recap in logistic regression very similar to linear regression what we do is at first we initialize the weights and the bias and then we predict or estimate a result using this equation then we calculate the error and on top of this error or using this error we use gradient descent to figure out in which direction to change our weights and biases also including our learning rate and then we repeat this process number of times that the model creator told us to do so and during testing given a data point we put the values that we get from the data point inside this equation and then calculating after we calculate this probability value we choose a label that is based on the highest probability so let's start implementing it now all right so let's start with the implementation it's actually going to be quite similar to linear regression but let's do it from the beginning it's not going to take that long anyways so at first of course i need an initialization function here we need to pass the learning rate again and a number of iterations we can again say like hundred thousand uh the learning rate could again be point zero one and we need uh learning rate will be learning rate we need the number of iterations oops number of iterations is going to be number of iterations and again we need the weights for now will be none and we need the bias again then i need another fit function and a predict function so i will pass the x and the y to the fit function let's remember what we did for logistic regression so initialize the weights and wires as zero so we can do it again if you remember we first need to get the number of samples and the number of features and that's that we're going to get from x shape and then i'm going to create the or initialize the weights or at least assign zeros as weights that's going to be

features need to import numpy of course and the bias just as zero next thing that we need to do predict the results using this calculation well this is basically um what we did with linear regression so weights times the x values plus the bias put into a sigmoid function so that's that can be what we're doing i'll already write the iterations so we need to do it for number of iterations or a number of iterations times this whole part here and i will calculate the predictions yeah numpy dot of x that we pass here and self weights plus the bias uh maybe i can call this linear predictions and then the actual predictions will be a sigmoid function and linear predictions inside it so i need to create the sigmoid function so let's do that maybe let's say sigmoid what i need to pass it are some values so let's refresh our memory what the sigmoid function look like 1 over 1 plus e over minus x so let's see what we can do 1 over 1 plus exponential minus x and that's all we need to do actually to calculate the sigmoid so we'll do it here and then we call it here and now we have the predictions after the next thing that we want to do is to calculate the gradients and that's exactly the same as last time so let's refresh our memory all right 1 over the number of samples times uh the dot product we're going to get of x but we need to get the transpose of x to be able to do this dot product because otherwise the dimensions don't work and the difference between the predictions and the actual y so that's dw and to calculate the gradient for the bias we do one over a number of samples again times let's see uh for that we're not going to use a dot product we're going to use the summation of predictions minus y and that's it and to do the updates i think we're going to do the same thing yeah exactly so self weights equals to self weights minus learning rate times d dw and self bias the new self bias is going to be self bias minus the learning rate times db so once this is done this means that our logistic regression algorithm has run and is trained next thing that we want to do is to get predictions so we need to pass it a set again let's see how we did it here for the inference time given a data point puts the values from the data point into the equation so we can just copy and paste this actually i call this y pred for now uh if we do the calculations here all right next thing choose the label based on the probability okay so what we have in the predictions is the probability of course if we take a look at what the sigmoid the results of the sigma function looks like uh if we're going to have values changing from zero to one basically so um what we're gonna say is if it's between zero and zero point five it's going to be zero the label will be zero if it's going to be between 0 and 5 0. 5 and 1 it's going to be another label so let's do that here for every y in y predicted the result should be 0 if y is smaller than 0. 5 else it should be 1. maybe let's say smaller or equal to 0. 5

uh put it in the wrong place and this will be the class predictions and then we can return this and to say four of course so for every y and y predictions we assign them the value of 0 if they if the probability is probability value is lower than 0. 5 or equal to 0. 5 otherwise it's going to be 1 and then we return this all right so this is all with logistic regression if you watch the linear regression video you might realize that they're actually quite similar the only difference is the addition of the sigmoid function and the calculation of the class labels after the probability is calculated and oh it looks like i wrote it down in the wrong file i'll just copy cut and paste it here in the logistic regression and in the train file i'm going to see how accurate this logistic regression algorithm is for that i am loading the breast cancer data set from scikit-learn data sets and i need to import of course logistic regression here import logistic regression all right i'm going to create an instance of it and we can keep the original values for now this is going to be a classifier and then i'm going to fit this classifier with the x train in y train and then we get the predictions predict with x test values and this will be y pred all right i want to report on its accuracy 2 of course so let's make a function that will calculate the accuracy for us so it will be y print and y tests and what i wanted to do is to tell me how many times y print is equal to y test and divide this number by the number of values in y test and then it can return this number and we'll have the accuracy so let's print the accuracy and see what we get uh we also need to calculate it of course let's do that here accuracy y predictions and y tests all right let's see hopefully i didn't make any typos this time oh i did line 16 self features oh no it has to be number of features all right uh so what we have here is we have an accuracy of 0. 89 which is pretty good but what we can do is to change the learning rate and see if we're going to get any better results so our current learning rate is let's see 0. 001 so what if i do 0. 01 let's see what happens then okay we have a better accuracy so it looks like this algorithm is this handmade homemade algorithm is working quite well uh again you can find all of this code in our github repository the link is in the description don't forget to ask your questions in the comments section me and patrick will be happy to answer them but i will see you in the next lesson