What is AdaBoost, and how does it work?

AdaBoost or Adaptive Boosting is an ensemble technique using which multiple decision trees can be ensembled to solve a regression or classification problem. In our previous video, we discussed bagged decision trees. In this article, we will discuss what AdaBoost is and how an AdaBoost Classifier can be used to solve a classification problem.

Let’s say we are reading the Pima Indians Diabetes dataset. The dataset contains various predictor variables such as the number of pregnancies the patient has had, the BMI, insulin level, age, etc. A machine learning model can learn from the dataset and predict whether the patient has diabetes based on these predictor variables.

Now, let’s say the training dataset D has n number of records. In AdaBoost, we assign equal weights to each record initially. We create a new dataset, say D1. And records are added to the new dataset D1 based on their weights. In other words, a record with a higher weight has a better probability of getting added to the dataset D1. Please note that initially, we assign equal weights to each record. So, initially, all the records are equally likely to be present in D1.

Now, we create a model M1 with a stump or one-level decision tree and train the model with D1. After training M1, we test the model with all the records of the training dataset that contains n records. Some records of the dataset will be incorrectly classified, and some records will be correctly classified by M1.

Now, we will increase the weights of the records that are incorrectly classified by M1. And we will decrease the weights of the records that are correctly classified by M1.

Next, we will create another dataset D2 based on the new weights of the records in D. So, records with higher weights will have a better probability to be added in D2. Now, we will create another model M2 with a stump and train M2 with D2. We will then test M2 with D and adjust the weights of all the records in D based on whether the record is correctly or incorrectly classified by M2.

We can repeat this process for a specific number of models. In other words, we can repeat the process and create a specific number of models.

Now, we can test all the models with the test set and combine the predictions. If it is a classification problem, we can consider the prediction with the highest number of voting. And if it is a regression problem, then we can take an average of all the predictions. This is how the AdaBoost algorithm works.

AdaBoost Classifier using sklearn in Python

We can use the following Python code to implement an AdaBoost Classifier using sklearn…