A hands-on explanation of Gradient Boosting Regression
Introduction
One of the most powerful ways of training models is to train multiple models and aggregate their predictions. This is the main concept of Ensemble Learning. While many flavours of Ensemble Learning exist, some of the most powerful algorithms and Boosting Algorithms. In my previous article, I broke down one of the most popular Boosting Algorithms; Adaptive Boosting. Today, I want to talk about its equally powerful twin; Gradient Boosting.
Boosting & Adaptive Boosting vs Gradient Boosting
Boosting refers to any Ensemble Method that can combine several weak learners(a predictor with poor accuracy) to make a strong learner(a predictor with high accuracy). The idea behind boosting is to train models sequentially, each trying to correct its predecessor.
An Overview Of Adaptive Boosting
In Adaptive Boosting, the main idea occurs with the model assigning a certain weight to each instance, and training a weak learner. Based on the predictor’s performance, it gets assigned its own separate weight based on a weighted error rate. The higher the accuracy of the predictor, the higher its weight, and the more “say” it will have on the final prediction.
Once the predictor has made predictions, AdaBoost looks at the misclassified instances, and boosts their instance weights. After normalising the instance weights so that they all equate to 1, a new predictor is trained and the process is repeated until a desirable output is reached, or a threshold is reached.
The final classification is done by taking a weighted vote. In other words, if we were predicting heart disease on a patient, and 60 stumps predicted 1 and 40 predicted 0, but the predictors in the 0 class had a higher cumulative weight(i.e the predictors had more “say”), then the final prediction would be 0.
Gradient Boosting
In contrast to Adaptive Boosting, instead of sequentially boosting misclassified instance weights, Gradient Boosting actually make predictions on the predecessors residuals. Woah, hold it. What?
Ok, so let’s break down the model’s steps:
- The first thing Gradient Boosting does is that is starts of with a Dummy Estimator. Basically, it calculates the mean value of the target values and makes initial predictions. Using the predictions, it calculates the difference between the predicted value and the actual value. This is called the residuals.
- Next, instead of training a new estimator on the data to predict the target, it trains an estimator to predict the residuals of the first predictor. This predictor is usually a Decision Tree with certain limits, such as the maximum amount of leaf nodes allowed. If multiple instances’ residuals are in the same leaf node, it takes their average and uses that as the leaf node’s value.
- Next, to make predictions, for each instance, it adds the base estimator’s value onto the Decision Tree’s predicted residual value of the instance to make a new prediction. It then calculates the residuals again between the predicted and actual value.
- This process is repeated until a certain threshold is reached or the residual difference is very small.
- To make a prediction for an unseen instance, it gives the instance to each and very decision tree made, sums their predictions and adds the base estimator’s value.
Learning Rate
An important hyperparameter to take note of here is the learning rate. This actually scales the contribution of each tree, so essentially increasing bias in exchange for a lower variance. So at step 3 and 4, the predicted value is actually multiplied by a learning rate to achieve better generalisation on unseen data.
A hands-on example of Gradient Boosting Regression with Python & Scikit-Learn
Some of the concepts might still be unfamiliar in your mind, so, in order to learn, one must apply! Let’s build a Gradient Boosting Regressor to predict house prices using the infamous Boston Housing Dataset. Without further ado, let’s get started!
import pandas as pd
import numpy as npfrom sklearn.metrics import mean_squared_error
from sklearn.model_selection import KFold
from sklearn.datasets import load_bostonfrom sklearn.tree import DecisionTreeRegressor
from sklearn.ensemble import GradientBoostingRegressor
Ok, so we do some basic imports, along with our dataset, which is conveniently builtin to scikit-learn, and Kfold cross validation, for splitting our data into a train set and validation set. We also import the DecisionTreeRegressor as well as the GradientBoostingRegressor
df = pd.DataFrame(load_boston()['data'],columns=load_boston()['feature_names'])
df['y'] = load_boston()['target']
df.head(5)Here, we just convert our data into a DataFrame for convenience
X,y = df.drop('y',axis=1),df['y']kf = KFold(n_splits=5,random_state=42,shuffle=True)for train_index,val_index in kf.split(X):
X_train,X_val = X.iloc[train_index],X.iloc[val_index],
y_train,y_val = y.iloc[train_index],y.iloc[val_index]
Here, we initialise our features and our target, and use 5 Fold cross validation to split our dataset into a training set and a validation set.
Before I go ahead and implement scikit-learn’s GradientBoostingRegressor, I would like to make a custom one of my own, just to help illustrate the concepts I wrote about earlier.
First, we create our initial predictions to be just the average of the training label values and assign our learning rate to be 0.1:
base = [y_train.mean()] * len(y_train)
learning_rate = 0.1Then, we calculate the residuals and get the MSE:
residuals_1 = base - y_train
mean_squared_error(y_val, base[len(y_val)])OUT:
71.92521322606885
Well, not so bad, considering we just predicted the mean value the whole time!
After that, we create our first tree and train it on the residuals. Again we will get the MSE of our predictions on the validation set:
predictions_dtree_1 = base + learning_rate * dtree_1.predict(X_train)
mean_squared_error(y_train,predictions_dtree_1)OUT:
70.90445609876541
Ok, so a slight improvement, already showing the power of Gradient Boosting! Again, we get the residuals:
residuals_2 = y_train - predictions_dtree_1
dtree_2 = DecisionTreeRegressor(random_state=42)
dtree_2.fit(X_train,residuals_2)And we get the MSE by making predictions, but note how we combined the predicted value of the first predictor with the new predictor’s values:
predictions_dtree_2 = ((dtree_2.predict(X_train) * learning_rate)) + predictions_dtree_1
mean_squared_error(y_train,predictions_dtree_2)OUT:
57.43260944000001
Wow, that was a big leap indeed! Now, let’s keep training a few more predictors to see what we can achieve:
residuals_3 = y_train - predictions_dtree_2
dtree_3 = DecisionTreeRegressor(random_state=42)
dtree_3.fit(X_train,residuals_3)predictions_dtree_3 = (dtree_3.predict(X_train) * learning_rate) + predictions_dtree_2residuals_4 = y_train - predictions_dtree_3
dtree_4 = DecisionTreeRegressor(random_state=42)
dtree_4.fit(X_train,residuals_4)predictions_dtree_4 = (dtree_4.predict(X_train) * learning_rate) + predictions_dtree_3
mean_squared_error(y_train,predictions_dtree_4)OUT:43.90388561846081
So we definitely improved our score, but now it’s time for the ultimate test: the validation set!
To make a final prediction, we do the following:
initial prediction(the mean of the target values) * learning rate +
- predicted reisudal values from tree 1
- predicted reisudal values from tree 2
- predicted reisudal values from tree 3
- predicted reisudal values from tree 4
In code:
y_pred = base[:101] + learning_rate *
(dtree_1.predict(X_val)) +
(dtree_2.predict(X_val) * learning_rate) +
(dtree_3.predict(X_val) * learning_rate) +
(dtree_4.predict(X_val) * learning_rate)And the result (Drumroll please…):
mean_squared_error(y_train, y_pred)OUT:
42.32013345535233
Fantastic! Not only did it fit the training set well, it also generalised smoothly on the test set! Why? Because we used a learning rate to control each trees contribution size, making sure that the ensemble did not overfit the data.
Gradient Boosting with Scikit-Learn’s GradientBoostingRegressor
We have now manually made a basic gradient boosting algorithm, but now let us code out a Gradient Boosting Regressor using scikit-learn!
Using the same data as above:
gradient_booster = GradientBoostingRegressor(loss='ls',learning_rate=0.1)
gradient_booster.get_params()OUT:
{'alpha': 0.9,
'ccp_alpha': 0.0,
'criterion': 'friedman_mse',
'init': None,
'learning_rate': 0.1,
'loss': 'ls',
'max_depth': 3,
'max_features': None,
'max_leaf_nodes': None,
'min_impurity_decrease': 0.0,
'min_impurity_split': None,
'min_samples_leaf': 1,
'min_samples_split': 2,
'min_weight_fraction_leaf': 0.0,
'n_estimators': 100,
'n_iter_no_change': None,
'presort': 'deprecated',
'random_state': None,
'subsample': 1.0,
'tol': 0.0001,
'validation_fraction': 0.1,
'verbose': 0,
'warm_start': False}
Now, this model has a lot of parameters, so it is worth mentioning the most important ones:
learning_rate: exactly the same parameter as we have discussed about above; it scales the contribution of each tree
init: the initial estimator, which equates to the DummyEstimator by default(aka predicts the mean for everything)
max_depth: the maximum depth you want your trees to grow
n_estimators: the amount of trees you want to create
criterion: what loss function you would like to minimise for the decision trees to use when it is searching for the best feature and threshold that splits the data.
loss: the loss to use for calculating residuals(the default is “ls”, or least squares)
max_leaf_nodes: the maximum number of leaf nodes you want to have for each tree. If this number is smaller then the number of training instances, and if two or more instances are in the same leaf, then the leaf’s value will be the average of all the training instance values in that leaf.
Let’s fit our model to the dataset and get its R2 score:
gradient_booster.fit(X_train,y_train)gradient_booster.score(X_train,y_train)OUT:
0.9791009142174039
And let’s get its R2 Score on the validation set:
gradient_booster.score(X_val,y_val)OUT:
0.8847454683496595
Ok , so our algorithm is slightly overfitting. Let’s adjust the learning_rate parameter to see if we can get better results:
gradient_booster = GradientBoostingRegressor(loss='ls',learning_rate=0.25)gradient_booster.fit(X_train,y_train)gradient_booster.score(X_train,y_train)OUT:
0.994857818295815gradient_booster.score(X_val,y_val)OUT:
0.9082261292781879
This is a much better result, where we sacrificed some bias for a better variance. Finally, let’s get the ensemble’s MSE on the validation set:
predictions = gradient_booster.predict(X_val)
mean_squared_error(y_val,predictions)OUT:
6.599129139886324
Ok, so that wraps up this one, folks! I hope you enjoyed it, but my Gradient Boosting days are not over yet! I will be back, and in the next articles I will be talking about Gradient Boosting for Classification, and more importantly; the big boy: XTREME GRADIENT BOOSTING! But, for now:
