Credit Risk Model Building Project | Session 4 | Venkat Reddy AI Classes

Hi, this is Wenut. We just turned into wanket ready AI classes. This video is part of a series. Complete the previous videos in this playlist. Before you start this video, the complete playlist information, the material and the code file information is given in the video description below. So until now we have discussed we started with uh the kind of objective in a credit risk model excluding some of the accounts observation point performance window by deciding the performance window by doing the vintage analysis and then deciding the bad definition by using roll rate or waterfall analysis. Most of the times this happens in Excel. This also happens in Excel. And then variable selection. Variable selection has multiple steps. One of the important step is information uh value calculation and then the other steps uh are multiolinarity by using vif and then rank ordering and then uh p value. After that we built the model. The model has to be logistic regression and the model this is known as pd model. It will give you probability of default and from that PD model we calculate the score. So scorecard is done. What is a scorecard? It will take the customer details and then finally it will give us the customer score. Then we will do the model validation. Here also we did a model validation. So that is ML model validation by using your confusion metrics and accuracy. Now we will see specific to scorecard what are the other validation metrics that are used because scorecard is all about uh separating good from bad or separating bad from good. So this separation power has to be pretty good not just the accuracy. So there is a different way to check this. There are three validation measures that are very commonly used. One is rank ordering of the scorec card. score card. What do you mean by that? If I take the bad rate here and score here, finally the model score as the score is increasing, let's say score is low here, medium and high. As the score is increasing, let's say from 300 to 400 to 500 to 600 to let's say the score is going up. As higher and higher, what should happen to bad rate in each of the bin? Bad rate — should increase or decrease. — Increase or decrease? Think about it once again. It can happen both. — No. — Bad rate. — It should decrease. — It should decrease when the score is low. Bad will be higher or lower. — Uh if it increase, it should be lower. — So credit score is let's say 300. Is a bad customer or good customer? It's a good — It's a bad bad customer. No, credit score is 900. Bad customer or good customer? Good customer. — Very good customer. — Is there any other choice? No matter what model you're building, low credit score, low score means customer is bad. High score means good. To avoid that machine learning confusion only the score has been created. Isn't it simple? Low score means bad customer. High score means good customer. No doubt about it. So if I see the bad rate customers who are having very low score 300 350 or 32 that means what should be the bad rate? If I take thousand customers who are in the range of 300 to 400 score what should be the bad rate in that — 30% 30%. — Higher bad rate or lower bad rate — lower. — So you will have higher bad rate. — As the score increases the bad rate keeps on reducing. If you take a very high score what should be the bad rate? If you take high very high score the bad rate should be very low. So this is known as rank ordering of the score. As you go from worst to best the bad rate keeps on decreasing. If you see it from the good rate point of view. When the score is very less percentage of good will be very less. As the score is increasing the percentage of good will be increasing. Typically people tend to see only one of these which is bad rate. When the score is less bad will be very high. When the score is high when it gets higher and higher the bad rate will be lower and lower. So your score should rank order the bad rate easily or bad rate should monotonically decrease. Bad rate should slowly decrease and when the score is very high battery should be almost zero. In fact this is like a visual inspection. This is known as rank ordering of the bad rate. A better way or a more stronger way typically rank ordering is checked by using KS pulmono skogro spinro statistic or KS statistic. So what does KS give us? KS gives us a number and it tells us how well the model here the model is the scorecard. How well the scorecard is separating bad from good. In fact KS will tell you the separation power of the model.

model is decided by KS. Now how does it give us the separation power of the model? So to calculate the KS value the calculations go like this. You start from the least score each bucket. How many are good customers? How many are bad customers? We keep on growing the score to the highest score from smallest value. How many good customers are there? How many bad are there? Now tell me as the score is increasing what should happen to the good rate? — Increase. — Score is increasing the good rate should increase. As the score is increasing, what should happen to the bad rate? — Decrease. — It should decrease. Isn't it? That is what is happening here. So how KS is calculated? We will take the cumulative good cumulative bad in each bin. And then we take the difference of cumulative good and cumulative bad. This is And whichever is the maximum that is where the maximum separation happens that is known as KS. KS is maximum of cumulative percentage bad versus cumulative percentage good into 100. So when you're comparing two models if you have higher KS that means that model has a better separation power. So KS is the measure that is used for deciding how good is the model. That is one of the most widely used model validation measure. So model validation teams in credit risk they use this K. So one of the percentage is cumulative good cumulative bad both of them will end up like 100%. But good will be very less in the beginning slowly they will go to 100 bad will be very high in the beginning but slowly they will converge finally the separation between that. So the KS should give you a higher separation. So the industry standard is if KS is less than 20 it's a pure poor separation. uh the model is not acceptable. Generally the model will get rejected if KS is less than 20. 20 to 30 it's weak separation. The model needs improvement. That means you may have to change some variables to make the model better. The boundary case this one model may not be accepted. But if the model is let's say uh having 5 to 7 years old and if the business thinks that okay model development we will do it later on then with some inspection that is used. KS 30 to 40 acceptable. This is minimum that is I think some of the companies say 35 is the minimum limit. Some companies said 40 is the minimum limit for the model to be accepted. This is like that final test for any model where KS has to be very good and then uh if it is above 40 usually 40 is a very safe limit. KS above 40 it is a strong model that is accepted above 50 very strong excellent discrimination between good and bad there's a good separation. So what are the triggers for model revalidation? Let's say at the time of development your KS was 45 everybody was happy and your model got approved developed in 2024 January now you reach 20 25 January what we have seen is KS has dropped more than 25% or 10 points in absolute that means KS from 45 it has dropped to what 33 so what will happen from the development which is developed in what January 2024 by the time within one year the model has dropped from 45 to 33. In that case, we will go for model revalidation or we will be first calling it as a red flag. We have to do extra analysis. We will be doing deep dive investigation. Variable by variable. We will do the investigation and find out what went wrong. That is one scenario. Let us suppose it hasn't dropped much in Jan 2025. Let's say from 45 it has gone to 43. That is not considered as a very big deal. Again, next year it will get approval. So now 20 26th January this month again we will do the revalidation from 43 it has dropped 15% or more than five points from the previous validation it has dropped more than five points. Now usually that five points is a very big deal. So if it is going to let's say 37 from 43 it has gone to 37 then again we will call it as a red flag. So there are two red flags. Where is a red flag? Compared to development time, the model should not go down more than 10 points. Compared to last year validation, it should not go down more than what points. Compared to last year, — five points. — More than five points. So every year, every scorecard that we build, they go through model validation cycles. Separate dedicated teams will be there. Every scorecard has a scorecard owner. That scorecard owner has to submit all the details. What was the development time accuracy? What is the what was the accuracy last revalidation? again? in this way validation etc. So once this red flag is raised then there will be discussions there will be analysis around it. There will be separate variable by variable analysis which is known as characteristic analysis that characteristic analysis will tell us which variable has caused that issue or whether the population itself has changed. Who knows? So those kind of analysis we will do. I will also explain you how that is done. But as of now we will try to do the KS calculation. Typically KS calculation also happens mostly in Excel only because all these are percentages. All these are graphs which are much more comfortable inside our Excel to draw and present. So let's go to our uh files from where you have this KS calculation. This is the one

that I want you to open. So you have this link. Let me paste this link once again in the chat window for your benefit. That is a place where all the files are mentioned. I pasted the link and we will go to KS calculation. I will now download this is the file that I'm talking about. There are two scorecards in it. We will do the case calculation for both of them. Again, you don't really need to memorize these formulas because the formulas uh even if we just copy paste the same formulas from a different file, it will be the it will be giving us the numbers. In reality, you may not see 0 to,000 as the score. In reality, the score starts from maybe 300 to 900. That is the regular range that you will see. Now, you have percentage of good, percentage of bad and all of this. Let me just redo this calculation. Percentage of good is good divided by this one. Let me lock this. After that, percentage of bad is this divided by this. Cumulative good. Can somebody tell me how cumulative good is done? This and the second one is last one plus current one. Do you know what is the cumulative good? How cumulative good is calculated? — Cumulative adding the previous one. — Yes, previous I will not say previous. I will say until that point. — So current percentage of good is six. So 6 plus 1. 5 whatever was the number plus 0. 6 six like 1. 5 cumulative good 7 this is 9. 8 which is 2. 2 + 7. 6 six like that cumulative good cumulative means at the end of the day you will have 100% of good covered here same way first it will be the same and then this plus the previous number if you keep on do like maximum bad is like the way that you interpret is bad 80% of the bad are caught by 500 range remaining 20% are found in here other way to see it is like almost uh only 30% or 25% of the good are below 600 remaining all are in the above range That is how it should be, isn't it? A lot of good should be in the high score ranges. A lot of bad should be in the low score ranges. The KS is simply cumulative bad minus cumulative good. You take all of those and the maximum value of it. Where is the maximum separation? That indicates the KS value. So 0. 67 into 100 KS is 66. Is it accepted the model? — Yes, — the model is very well accepted. This is called good separation. I would like to calculate this one. All of you give it a try and tell me what is the KS value for this particular scorecard. Calculate the second scorecard. First of all, I had to put a summation of all these values here. percentage of good. Similarly, I'll get percentage of bad. However, that has to be this one. Cumulative percentage of good. Similarly, cumulative percentage of bad as is percentage of bad minus percentage of good. Max of this into 100. How much is that? 30. Is that what you got all of you? We cannot say 0. 30. It has to be multiplied by your 100. Okay, maximum of all of that into 100 that is what the memor is KS is 30. So KS30 according to our limits if we go to limits if it is a new model it may not be accepted but if it is a couple of years back model KS of 30 is still somewhat said okay you are on the boundary like it'll be put under some scrutiny that is how KS is calculated so every year if you build the model in 2023 J we will see the KS we will check 2024 Jan what is the KS Yes. Is there a significant drop? No. Go ahead. 20 25 Jan. Is there a significant drop from here? No. But when you compare to the first development, this is the development one. When you compare to development time, is there a 10 points drop? Two things. Is there a fivepoint drop from here? No. Is there a 10point drop from there? No. 20 26 J. Is there a five point drop from here? No. Is there a 10point drop from the development

time? Yes. So that means compared to previous year, not a big deal. But compared to development there is a 10point drop. That means this model has to go for further analysis. It may recalibration or maybe redevelopment also. That means from scratch we may have to scrap that model. Mostly what happens is first four five years it may not go for scrapping and totally redeveloping. In reality the credit risk patterns they don't change very dynamically. Four years back also if a customer has too many loans those customers end up as defaulters. After four five years also customers has too many loans they will end up like defaulters. Do you agree? The kind of patterns that are associated with bad customers, good customers, do they change drastically year on year or these patterns are kind of permanent most often? Think about it. Similar — yeah mostly similar. So if a model is performing very well at the time of development, they will definitely have a good life. I have seen models that are pretty good for 6 years, 7 years, 8 years. In some cases especially in commercial portfolios even for 10 years some models are used they show good separation power. There is one more metric that is used. KS will tell you how good is the scorecard. separation power of the model? model. Now there are two things here. One is the model. The other one is the population itself. What if the population itself has changed? One is population or the data. Let's call it as population. On that you built a model. Now earlier we check the KS will tell whether the model is still good or not. This was 2024 uh 2023 Jan. From 2023 Jan to 2024 Jan the model is still good. Again you check the KS 2025 Jan. Based on KS the model still shows good separation. But we can't be happy just like that because the model was built on certain population. What is the population from 2023 Jan to 2024 Jan? The population itself has changed. The population that we use for building the model is no more the same population. For example, for if you take the variable called income in 2023 Jan, we found that around 30% of the people have income less than 10,000. from 10K to 40K. Remaining 40% of the people have income more than 40K. Understood? 30% have less than 10,000 income. 30% of the people have income from 10k to 40k. 40% of the people have greater than 40k. That is what the development time scenario was. But after 1 year we found that something happened. We don't know what happened. Something happened. A lot of population shift. People have totally changed drastically changed. What happened? There is nobody with less than 10,000 income anymore. There is nobody less than 30,000 income. Everybody has more than 30,000 income now. Don't you think? With respect to that variable income, the population has changed. Do you agree? — Yes. — That is one way of seeing. The other way of seeing is that is with respect to one variable. What if with respect to score itself? Let's say we have seen this distribution in the score from 300 to 400 there are 10,000 people. Let's say if you take one lakh population 10,000 are from 300 to 400 range. 400 to 500 range 20,000 people are there. 500 to 600 range there are 35,000 people are there. So total it is around 65 right now. 600 to 700 around 15,000 people are there. 700 to 800 how many is left? 10 plus — 30. How many is left out of one lakh? — So you have this as the 50 plus — 30k. — How much is left? — 20% — 20k. This was at the time of development. When did we develop this model? You were using the example of 2023 Jan and you waited one year. This is out of time which is like 2024 Jan. Once you do the scoring for people at 20 24 Jan you found that score 300 to 400 there are 20,000 people. Score 400 to 500 there are 30,000 people. Score 500 to 600 there are around 30,000 people. again score 600 to 700 or let me say 20,000 around 5,000 are here are you getting earlier what percent were there here out of whole population there were 300 to 400 range there were 10% of the people but now how what percent are here out of 1 lakh 20,000 are here that means 20% are here from 400 to 500 range there were 20% earlier but now they are 30% earlier there were 35 5% here. Now there are only 20% here. Earlier there are how much? 15% are here. There are how many? 20 + 30 + 30. It's almost like what we have reached 70 which is like 15 + 15. Let's say 15% are here. Earlier there were 20% here which means 15% are here. So there is some shift

that has happened. Earlier it was largely towards the high score. A lot of people were here. Now they have shifted from high score to low score. Earlier in these two buckets hardly 30% were there. Now 50% are present in these two buckets. Are you with me? — So sir, so this will happen only when we change the duration, right? When we — not change anything, we have built the model in 2023. Okay. — You applied. Okay. — Yes. — Now this year we are at 2024. Okay. Now you have to make a decision whether you want to apply the same model or not. — Yes sir. But here I have a couple of questions. The data will be till 2023 22 December and away. Right. Okay. But in real time scenario, how frequent that the existing data might change. Every one year you do the validation every one year. Let's say you have built the model in 2023 January. Okay. By 2023 January, the model was ready. You have implemented the model. You will use the same model for one year. Okay. H — next year you have received fresh applications for the loan or the — so that means so the data right the test data or the training data will be different for both — the train data will be the same you are not rebuilding the model — okay — if you're rebuilding then you are considering the data once again are we rebuilding the model here no — I'm using the same model but when I'm asking question hey can I use this model first of Mhm. — Then if somebody tells me if the model is working as well as the last year then go ahead use it yes or no. Last year the model was giving me let's say 90% accuracy. In terms of accuracy instead of accuracy we will say let's say KS. Last year it was having 45 as KS. This year also it is giving 43 as KS. Can I use the model on the new data that last one year data if I consider KS is again 43. Can I use the model? — Yes. But there's another question we had to ask. You have built the model on a population with this distribution. What was the distribution? There were only 30% of the customers who were from the range of 300 to 500 score. Now if this distribution changes that means there is a problem with the population. Population itself would have changed. The model that you have built during the development time you have built the data on a set of customers who are mostly good. Yes or no? — Yes. But right now the last one year customers you are getting are mostly bad. Yes or no? — Yeah. — So that means the model development population is not same as the population that you are seeing. So there's a shift in the population or there is a population drift that has happened then you may not want to go ahead proceed with this model. You may want to do some analysis and find out what has happened. Do you agree? Because we cannot rebuild the model every year. If that is the case then it will be too much a hectic job. A large bank hardly may build 10 to 30 models every year but they handle 300 scorecards every year. So building all these 300 scorecards will not be possible. So what they do they check two things whether still the model has good separation power using KS and whether the population still remains the same. How do you check same? How do you quantify how much is the shift? If the model has shift if the population has shifted by 1% I don't care. 10%, okay, no problem. But if the population has shifted by 25%. Now that is a big drift. So that is done by using population stability index. What does it do? It will try to see your development time. Pop population is let's say population A. Your current population is population B or vice versa. We will try to check how much it has changed from development time to current time. population stability index measures how much the current population or the score distribution has shifted compared to the development type. So it is used for this model monitoring to detect the data drift whether there is any drift that has happened in the data that means they have drifted towards good or towards bad and then we will try to decide okay can I recalibrate the model and see the PSI once again population stability index if it is working well I'll go ahead even after recalibration even after some tweaks the model is still showing the drift population stability index is still very high then we may have to go for redevelopment so I'm not saying that once we build the model we will never redevelop it we tend to utilize the same model check a KS value to see the separation power, check PSA population drift. If both of them are giving a go ahead, use the model. If one of them is stopping you, go for partial redevelopment or calibration. If both of them give up and say that the model has deteriorated a lot, then the redevelopment. So if the PSA value is less than 10%, there's not much significant shift, the model is stable, we go ahead. If it is 0. 1 to 0. 25, moderate shift, we monitor it carefully. Earlier the model uh monitoring cycle was one year. You tend to hear the risk manager will be given an instruction instruction. Check it even after 6 months. Don't wait for one

year. See what will be the shift after 6 months. You have to monitor it carefully. Extra monitoring is done. That will be the final recommendation. If it is greater than 0. 25 25% significant shift, rebuild the model or major review needed. Sometimes full redevelopment of the model is recommended. So during the development time there are 8% of the people in this range 9% 12% these are all the distributions during the development time we are calling that as Jan 2023 after 1 year these are the values that we got so overall here also roughly 10 to 12% 8 to 12 here also roughly 8 to 12 so I don't see a huge drift here so even if you calculate psi so what should be ideally the value of a minus b here ideally for ideal scenario when there is not How much of shift — 0%. — Roughly it should be always near to zero and a by b also this also should be near to zero and then when you multiply this that will you will get the PSI value and finally that has to be near to zero. If it is less than 10% less than 0. 1 then you will say that is not a very big deal. Is this clear? What is the difference between KS and PSA? What are the two different things they are doing? Tell me what does KS tell you? KS talks about — the obvious the separation power — separation power of the model. Does it talk about any drift in the population? — No. — What happened to population previous population versus current population last year compared to this year population? Does as have the ability to talk about the population? — No. It completely focuses on the model. What PSI talk about what it does? It will check — drift — the population. Drift in the current model — not model current population — compared to previous — previous or development. — Yeah. Now — does it tell me the separation power of the model PSI? — No. — How well the model is separating? — So there are two different things. One is you are building a model on the population. Which one will talk about the model? population? Model KS. — KS it will tell me how good was the model compared to the previous one. Population will be taken care by — PSI. So you have to make sure the model is also working well compared to last year. Population is also not much drifted. Then you can go ahead reuse the same model this year also. Now we will go and calculate how PSI is calculated. Let us go to the supporting file. There is a file called PSI calculation. There are two scorecards. Scorecard one, scorecard 2. We will download this file. If you see these formulas, most of them sound very much familiar. The information value formula, KS formula, PSA formula, the overall percentages that we calculate, they are much similar. The numbers etc. So let us see. So the percentage of a during the development time in this band there are 8%. 8. 9 12. 7 etc. But during the current scenario there are how many customers were there in our development sample? the development sample? — One lakh. One lakh actually. — No that was the example that I gave. But here how many are there? Development time. uh 49,000 49,180 the recent 2023 or let's say development was 23 for us and the recent let's say 24 during 2024 how many customers have been there you developed the model of 49,000 customers there are 95,000 customers right now you are in a doubt whether these 95,000 are behaving similar to the previous guys or not so roughly it looks similar so let us see a minus b there you go and then log of A minus B or A / B. There you go. And these two values multiplied. You can see that this is the number. Summation of all this will give you the drift in the population. 2. 7% drifted. Beyond what point you will say there is a significant drift beyond what? Beyond — how many point? — Beyond 25% there is a significant drift here 2. 7% which is considered as no significant drift. No significant drift instead of percentages let us see in the numbers then it will be easier 0. 027 027 which is less than 0. 1 it will

say no significant drift. If we do the same exercise, I can just pretty much copy paste everything here and change the value one value which is this one. That's one scorecard two. Is there a significant drift in scorecard two population that was used for building scorecard 2? Is there any significant drift? — Yes. — So here there is a significant drift suggested for model re reccalibration or rebuilding from scratch. What do you mean by rebuilding from scratch? You may have to start from variable selection or even before that everything so that uh 3 to 5 months time you may have to spend for recalibration. Maybe 15 days to 20 days are sufficient for recalibration. I'll try to explain what exactly is recalibration. Redevelopment is anyway there's nothing to discuss about it. What is redevelopment? I said the model must be redeveloped. What does that mean? — Again we need to come from scratch like — there is nothing much explanation isn't it? It's not new for us like you have developed the model in June 2023 or January 2023. I think if you're releasing the model in January 2023 maybe you might have started in June 2022. So whatever you have done in June 2022 the way that you started with the initial discussions objective exclusions and then the overall uh observation window performance window your bad definition segmentation variable selection model building all of the things that you have done you have to do all of them once again but what is recalibration let us discuss now final step in which scenario like model recalibration happens in which scenario model rebuilding happens this happens through lots and lots of discussions I have worked for significant amount of time in model validation team, model building team. So I have seen this whole cycle and I found that model validation team is the one that get lot of exposure to different type of models, different type of issues and different type of discussions and what are the business decisions that are made to make these models work or what are the tweaks that uh data scientists do small tweaks so that this model can be again brought back to at least the limits that we are expecting. So what exactly we will do whether if we are doing recalibration it is also known as re-estimation of coefficients re-estimation of coicient versus redevelopment. So mainly it is driven by two matrix or rank ordering is also seen but KS takes care of rank ordering. KS is the model still separating goods from bads. PSI is a current population similar to development population or not. Looking at these two now this is the table that will tell us if your model is stable that means you are having KS which is close to development time that means your model is still strong more than 30 KS or even though there's a drop there's a very small drop PSI is very low population stability PSI low means population is very stable yes or no PSI has to be a lower value or higher value — low it has to be lower value so PSI is good KS is good the model still separates good from bads population is similar there is no problem with cranking is correct everything is fine. You don't really need recalibration or reestimation. You can just like if at all you see any shift apart from these two if you see that there is a small shift then you can do this. But usually you do not do anything you just go ahead with the same model. Generally we don't take any action. We just get an approval. The risk head the product head or scorecard owner will go ahead and present it and then we will give it to the team that is scoring uh in the front end. That devops team will be using the same model. Now let us see the issues. One there is red flag the other one is this amber or orange. KS is a stable that means model KS stable. What does that indicate? KS is stable indicates that the model separation — separation power is still good. Is that a good sign for us initially? — Yes. — That means the model is doing good. No problem with the model. But PSI is high. — It means that the population has changed drastically. So in that case you do the partial redevelopment or you do the characteristic analysis. What do you mean by characteristic analysis? Population has changed. That means there must be some variables that may have changed. What you can do is since the model is doing still good, you can change some variables. The variables that have changed. Let us suppose during the development in Jan 2023 you have used 25 variables. during the validation in let's say 2024 it has validated very well but Janu 2025 is when we got the issue because within one year you don't see drastic changes these 25 variables you don't expect there is a shift or there is a problem with all these 25 variables maybe out of these 25 variables 22 variables are almost same

three variables have shifted drastically three variables from here to here these three variables one example that I gave earlier was let's say income. A lot of people were having high income or low income here. Now they have shifted to high income ranges. A lot of people were having a lot of late payments here but now they have uh improved last two years a lot of people are paying on time. Is that possible? Like there are some changes or maybe suddenly in between pandemic has come. Don't you think pandemic will have such a big impact? — Yes. — Suddenly interest rates have increased. If interest rates increase what happens? The defaults will increase or decrease? — Increase. If interest rates are increasing, you will have defaults increasing. If late payments like policy changes, we made a policy change that your late payment earlier was very less now it is high. If policy changes happens, those kind of things happen. Then you may see there are some variables that are taking a very big impact. So you will do characteristic analysis. So each one of them are attributes or characters they called. In the character analysis you'll find literally you will find population. Earlier we did for whole population. You'll find the PSI for every character or every variable and you will find the variables that have shifted drastically. We can talk to business and then uh we keep everything else same. We don't redevelop from the beginning with bad definition etc. We keep everything else same. Instead of those three variables, we will try to use the next best three variables during the development time. We started with 400 features. Finally, we choose the top 25 only. 26, 27, 28. We ignored them because they are not as powerful as these ones. where information value was less or something that we have seen. So that can be considered and we can partially redevelop the model. That is option one. Partially redevelop the model after doing the characteristic analysis. Or what we can do is we can use the same features which is 25 features. Maybe during the development of Jan 2023, you have come up with one PD model which has y equal to e power beta plus beta 1 x1 and so on plus beta kx divided by 1 + e power beta plus beta. These are what called what? Betas are known as coefficients. — Coefficients in a sense these are all the weightages of x1, x2 and xk. Do you agree? — Yes. So you can change those weightages that means that is known as re-estimation. What do you mean by re-estimation? I will use the same 25 features. Jan 2023 you were using data until Jan 20 or December 2023 roughly. Okay? Or 2022 sorry. Now instead of this data what I will do I will use the same variables but right now I'm in Jan 2025. What I will do while I'm — December 24 — until December 24 if my performance window is 18 months what I'll do I will take this 18 months same variable same 25 variables I will build the model on this recent data do you think the coefficients will be exactly same as this or they will change — they'll change — they will change so that is reestimation that is also known as re-estimation so either you can go for partial development I think first instead of partial development first reestimation it tried after reestimation you will Calculate KS, you will calculate PSI. If both of them are in your limits, you go ahead approve this model. Otherwise, if this is also not working, go for character analysis and find out which are the characters, which are the features that are changing a lot. Then you try to change only those characters and see whether KS and PSI are falling in your limit. Go ahead. Usually what happens is we don't expect these kind of tests fail. Most of the times KS and PSI are stable. 90% of the times it works. If you go to the details that I gave you out of 300 models when you are validating do you think 50 60 models will be giving you the trouble. Usually what happens let us go to that slide where I was talking typically how many models will be there rebuilt and all that. Typically you handle 300 scorecards out of them tell me typically 10 to 30 only 10% of them — will go for either recalibration or rebuilding. remaining 90% of them will be somehow managed and they will not be showing a very bad issue or anything mostly they will go ahead for the approvals because they are following in the limits that's the reality most of the times so you first do the reestimation if there is something wrong if it is working go ahead and then you do the caristic analysis and see what are the characters that are changing and try to replace them that is partial redevelopment but both of them have failed that means the model has been built five six years back but while building the model there were different economic scenarios now the economic scenarios have changed due to certain reason the model has totally lost KS is too low huge drop PSI is very high population also changed model also not separating well then uh you have to go for full reddevelopment redevelopment versus re-estimation or calibration have you understood the difference

recalibration which one is faster recalibration or redevelopment recal Re-calibration takes around 1 month time or even less than that but redevelopment may take to five four five months or 6 months time. Redevelopment is like you are starting from scratch. You redefine observation window, performance window, redo, exploited data analysis, variable selection. Are we doing all of that in reestimation? Are you doing data exploration, validation, cleaning? Are you doing variable selection? Are you doing multiple initiality checking and all that? — No. You're directly taking them and trying to because you have done all of that. You're just checking whether the weightage must have been changed. Earlier there was a lot of weightage given to X1. Less weight is given to XK. Maybe it may get swapped or it may get adjusted. So how do you get to know which character has changed? If you want to know whether income has drastically changed or not, what you can do is during the development time in January 2023 and this is a recent time. Recent is let's say January 2025. So this range of income these many people are there. 9% are here, 13% are here, 17% are here. Whereas recent times less income guys have gone. So most of them have shifted to higher incomes. Now how do you know whether this there has been a significant shift in the population or not? You check this like you use the same PSI formula to see within income how much is the shift and if it is 36% what does that indicate? — There is a significant shift — significant shift in the income. This is known as character analysis which will tell us are there any characters are there any features that have changed drastically. Maybe that particular feature we will drop and we will try to see the next best alternative and then we will try to partially redevelop the model. Partially redevelop. Are you with me everyone? Shall we do this exercise? — Yes sir. — But before that any quick questions is it confusing? Are you following me all of you? As I told you as a data scientist as a credit risk model building guy you will not be doing all this end to end alone. You understood the process right here and there are so many other nuance small steps that it is almost impossible for me to cover also I'm giving you the overall picture what happens but everywhere there will be a lot of documentation that will happen do you know that isn't it at the end of everything there will be lots and lots of documentation discussions will happen there will be few things only some companies will be following some other companies may have some other rules let's say if you say model building is done for us but for a client we have to submit a document where all the features that were used each variable name EDW name that documentation Everybody have their own different style of maintaining it. How do we know like when you are comparing with development sample type where are these uh data points stored like once you create these tables where are all those stored. So those kind of things are kind of depending on the company depending on the team they may change. So that is why you will see that for scorecard building you will see minimum four five teams working hand in hand. There are some teams that will work on data pooling, data gathering, creating table views etc. And there are times some teams that will work on scorecard model development. Model development doing the exploited data analysis and then logistic regression, scorecard scaling. In fact, I have seen some uh companies will keep the scorecard scaling team separately. And then model validation. Definitely this is a different team that works on KS, VSSI, out of time validation, stability, stress testing team. Basically, they're testing like whether the model is working well or not. And then finally like maybe these are the teams. Some of them will be in US, India. Strategies based on these models. What should be the cut off point? who should be approved, who should be rejected. When somebody's rejected, you have to send them the reason why their application has been rejected and then documentation. There uh there will be a team that will talk to regulatory people and communicate with the model development guys and all that like one two three four five different teams may work in scorecard development. Okay. And again on the onore you will have the risk manager and a couple of guys here. the it these guys will come into the picture right at the end. Once the model is finalized they will deploy the model and they will put it on server or wherever it is needed. We will go to one last exercise which is character analysis to find out whether a particular variable has changed drastically or not. You go and download this character analysis file recent let's call it as 2025 just remove this recent number I'll update it let me just delete this and rewrite You can also do it with me. This divide by overall number. Earlier we used to call it as good and bad for a different purpose. But this is development time and recent time. Development time, reset time.

Let us see from development to recent. That's a big change. Now logarithm of that a / b and then characteristic index just like psi. So this will be this multiplied by this one summation of all this 36% is a admissible drift or significant uh drift in this particular variable. This is a what? Significant drift. — Yes, significant drift. — Significant drift in this character. You may want to change that. recommend it to business. Similarly, let us see in the number of loans. — 4. 8%. — 4. 8. Let us see that. What I'll do is I'll just copy paste the same. Probably it may not work because the pins are different. So, I'll copy paste until here. There you go. And this one has to be right here. Drag it further. And this one must be here. Largely you will have pre-written macros or pre-written tables for this. You will not be really doing what we are doing here to have everything well set. All that you need to do is just verify and make sure that nothing is wrong. It is insignificant. Typically this is how a model validation document typically looks like. Annual account acquisition score renewal. renewal model approval package. This is the one that we have to fill from the model management or the model validation team. This is known as map document date etc etc. What is the revenue number of accounts? If you have any sub products you'll fill them. Development sample information. How many goods were there? How many bads rejects were there? There's some intermediates which is like not that good, not that bad. Usually they are uh not considered and then uh scorecard scaling information validation uh stability uh you will write KS statistic value critical value or some other values and then uh cut off values from the development time to this time all these values will be filled and then uh bad rates total 90dp different values are failed. Finally in the comments we will write please comment if the patterns have observed indicator contain any package are the variance forecast what are the causes uh is there any problem. So we tend to write KS is stable no problem. PSI Or if we see that KS there is a drop we are recommending it for recalibration. PSI is changing partial redevelopment. KSVS huge drop we are recommending it for uh the model redevelopment and then the approvals will happen. Score specialist and uh country manager will sign this off. — Sir we have to submit this to the clients. We have — no not you. uh there will be separate uh people maybe usually people with little experience they will take inputs from us they will be filling this — okay — because there is a lot of information related to business which uh we may not be having access to all right — yes — those are all the steps if you see in a credit risk model building and for a single person to understand end to end process and do end to- end process that may not be possible in reality out of these five or six teams you may be part one of the team usually I train you to become the part of model building team. You generally tend to say that I have built logistic regression model. I have uh looked at the accuracy I have taken care of class imbalance. I made sure that model is giving good accuracy etc etc. That is what we generally get trained into. Uh we can also check the other aspects as well. So if you see in this uh whole uh grid disk model building process we have discussed multiple points. We started with the objective setting. In fact, even before that how the team structure and all that happens uh from the overall management point of view also we have seen the portfolio size point of view and then we went on to discuss windage analysis, role rate analysis, weight of evidence, information value, model building, KS, PSI and model recalibration and character analysis. These are the analysis that uh will give you a good idea on what happens and you might have realized that it's not the Python coding that is done maximum number of times. It is the Excel adoc analysis that you will be doing very frequently. Even today if you go to most of the companies, Excel adoc analysis is the one that you will be presenting in your day-to-day life because most of the code is already written. You just need to change some values in that. But mostly the Excel add analysis is the one that you will be doing on a daily basis. So that concludes our discussion on credit risk model building. Continue with the next video in the playlist. We are covering everything

step by step. If you have any questions or the comments, please post them in the comments window below.