Introduction to model ensembling

So far, we've been trying to choose between two different models, namely logistic regression and random forests. However, you can actually use a process called ensembling to combine multiple models. The goal of ensembling is to produce a combined model, known as an ensemble, that is more accurate than any of the individual models. The process for ensembling is simple: For a regression problem, you calculate the average of the predictions made by the individual regressors and use that as your prediction. For a classification problem, you can either average the predicted probabilities output by the classifiers, or you can let the classifiers vote on which class to predict. We'll see examples of this below. The idea behind ensembling is that if you have a collection of individually imperfect models, the "one-off" errors made by each model are probably not going to be made by the rest of the models. Thus, the errors will be discarded, or at least reduced, when ensembling the models. Another way of saying this is that ensembling produces better predictions because the ensemble has a lower variance than any of the individual models. In this chapter, we'll ensemble our classification models two different ways, and then we'll tune the ensemble to try to achieve even better performance.