I’ve been looking at the LightGBM (light gradient boosting machine) system lately. A few weeks ago, I zapped out a binary classification demo. One morning before work, I figured I’d enhance that demo by adding a custom, program-defined confusion matrix for model evaluation.
LightGBM is a sophisticated tree-based system that can perform classification (binary and multi-class), regression, and ranking.
There are three language interfaces to LightGBM — C, Python, R. I like the relatively easy-to-use Python language API. LightGBM isn’t installed by default with the Anaconda Python distribution I use, so I installed it with the command “pip install lightgbm”.
For my demo, I used one of my standard synthetic datasets. The goal is to predict sex from age, State, income, and political leaning. The 240-item raw data looks like:
F 24 michigan 29500.00 liberal M 39 oklahoma 51200.00 moderate F 63 nebraska 75800.00 conservative M 36 michigan 44500.00 moderate F 27 nebraska 28600.00 liberal . . .
For LightGBM, it’s best to use ordinal encoding for categorical variables. I encoded the sex variable as M = 0 and F = 1. I encoded State as Michigan = 0, Nebraska = 1, Oklahoma =2. I encoded politics as conservative = 0, moderate = 1, liberal = 2.
Because LGBM is tree-based, it’s not necessary to normalize numeric data (age and income).
I split the encoded data into a 200-item set of training data and a 40-item set of test data. The resulting comma-delimited encoded data looks like:
1, 24, 0, 29500.00, 2 0, 39, 2, 51200.00, 1 1, 63, 1, 75800.00, 0 0, 36, 0, 44500.00, 1 1, 27, 1, 28600.00, 2 . . .
The key statements of my demo program are:
import numpy as np
import lightgbm as lgbm
train_x = np.loadtxt(train_file, usecols=[1,2,3,4],
delimiter=",", comments="#", dtype=np.float64)
train_y = np.loadtxt(train_file, usecols=0,
delimiter=",", comments="#", dtype=np.int64)
params = {
'objective': 'binary', # not needed this API
'boosting_type': 'gbdt', # default
'num_leaves': 31, # default
'learning_rate': 0.05, # default = 0.10
'n_estimators': 100, # default
'feature_fraction': 1.0, # default
'min_data_in_leaf': 10, # default = 20
'random_state': 0,
verbosity': -1 # default = 1
}
model = lgbm.LGBMClassifier(**params)
model.fit(train_x, train_y)
The main challenge when using LightGBM is wading through the dozens of parameters. There are 57 Learning Control Parameters (min_data_in_leaf, bagging_fraction, etc.), and the LGBMClassifier module has 19 parameters, for a total of 76 parameters to deal with. Here are the 19 model parameters:
boosting_type='gbdt', num_leaves=31, max_depth=-1, learning_rate=0.1, n_estimators=100, subsample_for_bin=200000, objective=None, class_weight=None, min_split_gain=0.0, min_child_weight=0.001, min_child_samples=20, subsample=1.0, subsample_freq=0, colsample_bytree=1.0, reg_alpha=0.0, reg_lambda=0.0, random_state=None, n_jobs=None, importance_type='split', **kwargs
Because the number of parameters is not manageable, you must rely on the default values and then try to find the handful of parameters that will create a good model. For my demo, I changed the learning rate from default 0.10 to 0.05, the random_state (from default None to an arbitrary value of 0, to get reproducible results), and the min_data_in_leaf from the default of 20 to 10. I also set verbosity to -1 to suppress most messages, but in a non-demo scenario you really want to see all system warning and error messages. The near impossibility of fully understanding all the LightGBM parameters and their interactions is one reason why I rarely use LightGBM.
Anyway, the LightGBM model predicted the 200-item training data with 0.9700 accuracy (194 out of 200 correct), and 40-item test data with 0.7250 accuracy (29 out of 40 correct). This is roughly comparable to the accuracy achieved by a neural network binary classifier. When LightGBM works, it often works very well. However, tree-based systems are highly susceptible to overfitting — although the LightGBM algorithms mitigate overfitting.
I added functions to compute and display a confusion matrix. The calling statements are:
print("Confusion matrix test data: ")
cm = confusion_matrix(model, test_x, test_y)
show_confusion(cm)
The output is:
Confusion matrix test data:
---------------
actual = 0 | 17 | 9 | 0.6538
actual = 1 | 2 | 12 | 0.8571
---------------
predicted: 0 1 0.7250
precision = 0.5714
recall = 0.8571
F1 = 0.6857
Precision and recall are specialized types of accuracy. It’s best not to overthink them. When precision and recall differ significantly, it means one class is predicted with significantly different accuracy than the other class. For the demo test data, class 1 (female) is predicted at 0.8571 accuracy which is much better than the prediction accuracy of 0.6538 for class 0 (male). The exact meanings of precision and recall are ambiguous because they depend on which of the two classes is considered “positive”, which is arbitrary.
The F1 score is the harmonic mean (average of two fractions) of precision and recall. It’s best considered as yet another type of overall accuracy.
I’ve always liked black-and-white video. It’s sort of an approximation to a binary form. Many hobbyists are experimenting with AI-generated videos. I came across a clever, short, black-and-white video by someone who uses the online name “hey anglomangler” titled “Toxoplasmosis”. The video looks like it’s made of AI-generated images that have had short motion clips created from the images. I think the video may have been created using the Stable Diffusion and Deforum tools. Very nightmarish and very creative.
Demo program:
# people_gender_lgbm.py
# predict sex from age, State, income, politics
# Anaconda3-2023.09-0 Python 3.11.5 LightGBM 4.3.0
import numpy as np
import lightgbm as lgbm
# -----------------------------------------------------------
def accuracy(model, data_x, data_y):
# simple
preds = model.predict(data_x) # all predicted values
n_correct = np.sum(preds == data_y)
result = n_correct / len(data_x)
return result
# -----------------------------------------------------------
def confusion_matrix(model, data_x, data_y):
# assumes 2 classes (binary)
cm = np.zeros((2,2), dtype=np.int64)
for i in range(len(data_x)):
x = data_x[i].reshape(1, -1) # batch it
trgt_y = data_y[i] # scalar 0 or 1
pred_y = model.predict(x) # array like [0]
pred_y = pred_y[0] # like 0
cm[trgt_y][pred_y] += 1
return cm
# -----------------------------------------------------------
def show_confusion(cm):
# assumes 2 classes (binary)
acc_0 = cm[0][0] / (cm[0][0] + cm[0][1])
acc_1 = cm[1][1] / (cm[1][0] + cm[1][1])
acc = (cm[0][0] + cm[1][1]) / np.sum(cm)
print(" ---------------")
print("actual = 0 |%5d |%5d |" % (cm[0][0], cm[0][1]),
end="")
print("%9.4f" % acc_0)
# print(" ---------------")
print("actual = 1 |%5d |%5d |" % (cm[1][0], cm[1][1]),
end="")
print("%9.4f" % acc_1)
print(" ---------------")
print("predicted: 0 1 ", end="")
print("%9.4f" % acc)
print("")
tp = cm[1][1] # "true positive"
tn = cm[0][0] # "true negative"
fp = cm[0][1] # falsely predicted as positive
fn = cm[1][0] # falsely predicted as negative
pre = tp / (tp + fp)
rec = tp / (tp + fn)
f1 = 1.0 / ( ((1.0 / pre) + (1.0 / rec)) / 2.0 )
print("precision = %8.4f" % pre)
print("recall = %8.4f" % rec)
print("F1 = %8.4f" % f1)
# -----------------------------------------------------------
def main():
# 0. get started
print("\nBegin People predict sex LightGBM ")
print("Predict sex from age, State, income, politics ")
np.random.seed(1)
# 1. load data
# sex, age, State, income, politics
print("\nLoading train and test data ")
train_file = ".\\Data\\people_train.txt"
test_file = ".\\Data\\people_test.txt"
train_x = np.loadtxt(train_file, usecols=[1,2,3,4],
delimiter=",", comments="#", dtype=np.float64)
train_y = np.loadtxt(train_file, usecols=0,
delimiter=",", comments="#", dtype=np.int64)
test_x = np.loadtxt(test_file, usecols=[1,2,3,4],
delimiter=",", comments="#", dtype=np.float64)
test_y = np.loadtxt(test_file, usecols=0,
delimiter=",", comments="#", dtype=np.int64)
np.set_printoptions(precision=0, suppress=True)
print("\nFirst few train data: ")
for i in range(5):
print(train_x[i], end="")
print(" || " + str(train_y[i]))
print(". . . ")
# 2. create and train model
print("\nCreating LightGBM binary classification model ")
params = {
'objective': 'binary', # not needed this API
'boosting_type': 'gbdt', # default
'num_leaves': 31, # default
'learning_rate': 0.05, # default = 0.10
'n_estimators': 100, # default
'feature_fraction': 1.0, # default
'min_data_in_leaf': 10, # default = 20
'random_state': 0,
'verbosity': -1 # default = 1
}
model = lgbm.LGBMClassifier(**params)
model.fit(train_x, train_y)
print("Done ")
# 3. evaluate model
acc_train = accuracy(model, train_x, train_y)
print("\nAccuracy of model on training data = %8.4f " \
% acc_train)
acc_test = accuracy(model, test_x, test_y)
print("Accuracy of model on test data = %8.4f " \
% acc_test)
print("\nConfusion matrix test data: ")
cm = confusion_matrix(model, test_x, test_y)
show_confusion(cm)
# 4. use model
print("\nPredicting sex for 35 Oklahoma $55,000 moderate ")
print("(0 = male, 1 = female) ")
x = np.array([[35, 2, 55000.00, 1]], dtype=np.float64)
y_pred = model.predict(x)
print("\nPredicted sex = " + str(y_pred[0]))
# 5. save model
import pickle
print("\nSaving model ")
pth = ".\\Models\\gender_model.pkl"
with open(pth, "wb") as f:
pickle.dump(model, f)
print("Done ")
# with open(pth, "rb") as f:
# model2 = pickle.load(f)
#
# x = np.array([[35, 2, 55000.00, 1]], dtype=np.float64)
# pred = model2.predict(x)
# print("\nPredicted sex = " + str(pred[0]))
print("\nEnd demo ")
if __name__ == "__main__":
main()
Training data:
# people_train.txt # sex (M = 0, F = 1) # age # State (Michigan = 0, Nebraska = 1, Oklahoma = 2) # income # politics (conservative = 0, moderate = 1, liberal = 2) # 1, 24, 0, 29500.00, 2 0, 39, 2, 51200.00, 1 1, 63, 1, 75800.00, 0 0, 36, 0, 44500.00, 1 1, 27, 1, 28600.00, 2 1, 50, 1, 56500.00, 1 1, 50, 2, 55000.00, 1 0, 19, 2, 32700.00, 0 1, 22, 1, 27700.00, 1 0, 39, 2, 47100.00, 2 1, 34, 0, 39400.00, 1 0, 22, 0, 33500.00, 0 1, 35, 2, 35200.00, 2 0, 33, 1, 46400.00, 1 1, 45, 1, 54100.00, 1 1, 42, 1, 50700.00, 1 0, 33, 1, 46800.00, 1 1, 25, 2, 30000.00, 1 0, 31, 1, 46400.00, 0 1, 27, 0, 32500.00, 2 1, 48, 0, 54000.00, 1 0, 64, 1, 71300.00, 2 1, 61, 1, 72400.00, 0 1, 54, 2, 61000.00, 0 1, 29, 0, 36300.00, 0 1, 50, 2, 55000.00, 1 1, 55, 2, 62500.00, 0 1, 40, 0, 52400.00, 0 1, 22, 0, 23600.00, 2 1, 68, 1, 78400.00, 0 0, 60, 0, 71700.00, 2 0, 34, 2, 46500.00, 1 0, 25, 2, 37100.00, 0 0, 31, 1, 48900.00, 1 1, 43, 2, 48000.00, 1 1, 58, 1, 65400.00, 2 0, 55, 1, 60700.00, 2 0, 43, 1, 51100.00, 1 0, 43, 2, 53200.00, 1 0, 21, 0, 37200.00, 0 1, 55, 2, 64600.00, 0 1, 64, 1, 74800.00, 0 0, 41, 0, 58800.00, 1 1, 64, 2, 72700.00, 0 0, 56, 2, 66600.00, 2 1, 31, 2, 36000.00, 1 0, 65, 2, 70100.00, 2 1, 55, 2, 64300.00, 0 0, 25, 0, 40300.00, 0 1, 46, 2, 51000.00, 1 0, 36, 0, 53500.00, 0 1, 52, 1, 58100.00, 1 1, 61, 2, 67900.00, 0 1, 57, 2, 65700.00, 0 0, 46, 1, 52600.00, 1 0, 62, 0, 66800.00, 2 1, 55, 2, 62700.00, 0 0, 22, 2, 27700.00, 1 0, 50, 0, 62900.00, 0 0, 32, 1, 41800.00, 1 0, 21, 2, 35600.00, 0 1, 44, 1, 52000.00, 1 1, 46, 1, 51700.00, 1 1, 62, 1, 69700.00, 0 1, 57, 1, 66400.00, 0 0, 67, 2, 75800.00, 2 1, 29, 0, 34300.00, 2 1, 53, 0, 60100.00, 0 0, 44, 0, 54800.00, 1 1, 46, 1, 52300.00, 1 0, 20, 1, 30100.00, 1 0, 38, 0, 53500.00, 1 1, 50, 1, 58600.00, 1 1, 33, 1, 42500.00, 1 0, 33, 1, 39300.00, 1 1, 26, 1, 40400.00, 0 1, 58, 0, 70700.00, 0 1, 43, 2, 48000.00, 1 0, 46, 0, 64400.00, 0 1, 60, 0, 71700.00, 0 0, 42, 0, 48900.00, 1 0, 56, 2, 56400.00, 2 0, 62, 1, 66300.00, 2 0, 50, 0, 64800.00, 1 1, 47, 2, 52000.00, 1 0, 67, 1, 80400.00, 2 0, 40, 2, 50400.00, 1 1, 42, 1, 48400.00, 1 1, 64, 0, 72000.00, 0 0, 47, 0, 58700.00, 2 1, 45, 1, 52800.00, 1 0, 25, 2, 40900.00, 0 1, 38, 0, 48400.00, 0 1, 55, 2, 60000.00, 1 0, 44, 0, 60600.00, 1 1, 33, 0, 41000.00, 1 1, 34, 2, 39000.00, 1 1, 27, 1, 33700.00, 2 1, 32, 1, 40700.00, 1 1, 42, 2, 47000.00, 1 0, 24, 2, 40300.00, 0 1, 42, 1, 50300.00, 1 1, 25, 2, 28000.00, 2 1, 51, 1, 58000.00, 1 0, 55, 1, 63500.00, 2 1, 44, 0, 47800.00, 2 0, 18, 0, 39800.00, 0 0, 67, 1, 71600.00, 2 1, 45, 2, 50000.00, 1 1, 48, 0, 55800.00, 1 0, 25, 1, 39000.00, 1 0, 67, 0, 78300.00, 1 1, 37, 2, 42000.00, 1 0, 32, 0, 42700.00, 1 1, 48, 0, 57000.00, 1 0, 66, 2, 75000.00, 2 1, 61, 0, 70000.00, 0 0, 58, 2, 68900.00, 1 1, 19, 0, 24000.00, 2 1, 38, 2, 43000.00, 1 0, 27, 0, 36400.00, 1 1, 42, 0, 48000.00, 1 1, 60, 0, 71300.00, 0 0, 27, 2, 34800.00, 0 1, 29, 1, 37100.00, 0 0, 43, 0, 56700.00, 1 1, 48, 0, 56700.00, 1 1, 27, 2, 29400.00, 2 0, 44, 0, 55200.00, 0 1, 23, 1, 26300.00, 2 0, 36, 1, 53000.00, 2 1, 64, 2, 72500.00, 0 1, 29, 2, 30000.00, 2 0, 33, 0, 49300.00, 1 0, 66, 1, 75000.00, 2 0, 21, 2, 34300.00, 0 1, 27, 0, 32700.00, 2 1, 29, 0, 31800.00, 2 0, 31, 0, 48600.00, 1 1, 36, 2, 41000.00, 1 1, 49, 1, 55700.00, 1 0, 28, 0, 38400.00, 0 0, 43, 2, 56600.00, 1 0, 46, 1, 58800.00, 1 1, 57, 0, 69800.00, 0 0, 52, 2, 59400.00, 1 0, 31, 2, 43500.00, 1 0, 55, 0, 62000.00, 2 1, 50, 0, 56400.00, 1 1, 48, 1, 55900.00, 1 0, 22, 2, 34500.00, 0 1, 59, 2, 66700.00, 0 1, 34, 0, 42800.00, 2 0, 64, 0, 77200.00, 2 1, 29, 2, 33500.00, 2 0, 34, 1, 43200.00, 1 0, 61, 0, 75000.00, 2 1, 64, 2, 71100.00, 0 0, 29, 0, 41300.00, 0 1, 63, 1, 70600.00, 0 0, 29, 1, 40000.00, 0 0, 51, 0, 62700.00, 1 0, 24, 2, 37700.00, 0 1, 48, 1, 57500.00, 1 1, 18, 0, 27400.00, 0 1, 18, 0, 20300.00, 2 1, 33, 1, 38200.00, 2 0, 20, 2, 34800.00, 0 1, 29, 2, 33000.00, 2 0, 44, 2, 63000.00, 0 0, 65, 2, 81800.00, 0 0, 56, 0, 63700.00, 2 0, 52, 2, 58400.00, 1 0, 29, 1, 48600.00, 0 0, 47, 1, 58900.00, 1 1, 68, 0, 72600.00, 2 1, 31, 2, 36000.00, 1 1, 61, 1, 62500.00, 2 1, 19, 1, 21500.00, 2 1, 38, 2, 43000.00, 1 0, 26, 0, 42300.00, 0 1, 61, 1, 67400.00, 0 1, 40, 0, 46500.00, 1 0, 49, 0, 65200.00, 1 1, 56, 0, 67500.00, 0 0, 48, 1, 66000.00, 1 1, 52, 0, 56300.00, 2 0, 18, 0, 29800.00, 0 0, 56, 2, 59300.00, 2 0, 52, 1, 64400.00, 1 0, 18, 1, 28600.00, 1 0, 58, 0, 66200.00, 2 0, 39, 1, 55100.00, 1 0, 46, 0, 62900.00, 1 0, 40, 1, 46200.00, 1 0, 60, 0, 72700.00, 2 1, 36, 1, 40700.00, 2 1, 44, 0, 52300.00, 1 1, 28, 0, 31300.00, 2 1, 54, 2, 62600.00, 0
Test data:
# people_test.txt # 0, 51, 0, 61200.00, 1 0, 32, 1, 46100.00, 1 1, 55, 0, 62700.00, 0 1, 25, 2, 26200.00, 2 1, 33, 2, 37300.00, 2 0, 29, 1, 46200.00, 0 1, 65, 0, 72700.00, 0 0, 43, 1, 51400.00, 1 0, 54, 1, 64800.00, 2 1, 61, 1, 72700.00, 0 1, 52, 1, 63600.00, 0 1, 30, 1, 33500.00, 2 1, 29, 0, 31400.00, 2 0, 47, 2, 59400.00, 1 1, 39, 1, 47800.00, 1 1, 47, 2, 52000.00, 1 0, 49, 0, 58600.00, 1 0, 63, 2, 67400.00, 2 0, 30, 0, 39200.00, 0 0, 61, 2, 69600.00, 2 0, 47, 2, 58700.00, 1 1, 30, 2, 34500.00, 2 0, 51, 2, 58000.00, 1 0, 24, 0, 38800.00, 1 0, 49, 0, 64500.00, 1 1, 66, 2, 74500.00, 0 0, 65, 0, 76900.00, 0 0, 46, 1, 58000.00, 0 0, 45, 2, 51800.00, 1 0, 47, 0, 63600.00, 0 0, 29, 0, 44800.00, 0 0, 57, 2, 69300.00, 2 0, 20, 0, 28700.00, 2 0, 35, 0, 43400.00, 1 0, 61, 2, 67000.00, 2 0, 31, 2, 37300.00, 1 1, 18, 0, 20800.00, 2 1, 26, 2, 29200.00, 2 0, 28, 0, 36400.00, 2 0, 59, 2, 69400.00, 2
.NET Test Automation Recipes
Software Testing
SciPy Programming Succinctly
Keras Succinctly
R Programming
Visual Studio Live
Microsoft MLADS Conference
DevIntersection Conference
Machine Learning Week
Ai4 Conference
G2E Conference
iSC West Conference
You must be logged in to post a comment.