The scikit library has several ways to perform binary classification. Weirdly, there are two different ways to use logistic regression. The LogisticRegression class uses matrix operations (L-BFGS algorithm by default) or SGD (actually SAG — Stochastic Average Gradient) to train a model. The SGDClassifier class is multipurpose and is a strangely-designed wrapper over several binary classification algorithms, one of which is logistic regression that is trained using stochastic gradient descent.

Put another way: LogisticRegression solves only logistic regression problems using one of several algorithms, including a variant of SGD. SGDClassifier solves several different types of problems, all using SGD.

I experimented with SGDClassifier. I used one of my standard datasets where the goal is to predict sex (male = 0, female = 1) from age, State, income, and political leaning. The data looks like:

 +1   0.24   1 0 0   0.2950   0 0 1
 -1   0.39   0 0 1   0.5120   0 1 0
 +1   0.63   0 1 0   0.7580   1 0 0
 -1   0.36   1 0 0   0.4450   0 1 0
. . . 

The columns are sex, age divided by 100, State (Michigan = 100, Nebraska = 010, Okahoma = 001), income divided by $100,000, and politics (conservative = 100, moderate = 010, liberal = 001). There is 200 training items and 40 test items. The data can be found at jamesmccaffrey.wordpress.com/2022/09/23/binary-classification-using-pytorch-1-12-1-on-windows-10-11/.

The key statements that create and train a logistic regression model are:

print("Creating SGD logistic regression model")
model = SGDClassifier(loss='log', penalty=None,
  alpha=0.0001, max_iter=1000, tol=0.00001, shuffle=True,
  verbose=0, random_state=1,  learning_rate='optimal',
  eta0=0.0)
model.fit(train_X, train_y)

When you specify loss=’log’ you implicitly create a logistic regression model. If you use a different loss type, you create a different model. For example, loss=’hinge’ creates a SVM (support vector machine) model. This is a very poor API design.

The SGDClassifier has a lot of parameters, many of which are somewhat confusing. For example, learning_rate=’optimal’ type ignores the initial learning rate value of eta0.

The SGDClassifier is extremely sensitive to its parameter values. It took me about 3 hours of trying different combinations of values to get reasonable results of 84.00% accuracy on the training data and 72.50% accuracy on the test data.

Note: See my example of using the scikit LogisticRegression class at https://jamesmccaffreyblog.com/2023/01/24/revisiting-binary-classification-using-scikit-logistic-regression/.

When I worked on a cruise ship, one of my favorite ports was Venice. The Venice carnival runs for two weeks in February. I think the Venice carnival costumes, with their gradient colors, are beautiful.

Demo code. Replace “lt” (less–than) with Boolean operator symbol.

# people_gender_sgd.py

# predict gender (0 = male), 1 = female) 
# from age, State, income, job-type

# uses scikit SGD version of logistic regression

# data:
# 1   0.24   1   0   0   0.2950   0   0   1
# 0   0.39   0   0   1   0.5120   0   1   0
# 1   0.27   0   1   0   0.2860   0   0   1
# . . . 

# Anaconda3-2020.02  Python 3.7.6
# Windows 10/11

import numpy as np
from sklearn.linear_model import SGDClassifier
from sklearn.metrics import precision_score
import pickle

def show_confusion(cm):
  # Confusion matrix whose i-th row and j-th column entry
  # indicates the number of samples with true label being
  # i-th class and predicted label being j-th class.

  ct_act0_pred0 = cm[0][0]  # TN
  ct_act0_pred1 = cm[0][1]  # FP wrongly predicted as pos
  ct_act1_pred0 = cm[1][0]  # FN wrongly predicted as neg 
  ct_act1_pred1 = cm[1][1]  # TP
  
  print("actual 0  | %4d %4d" % (ct_act0_pred0, ct_act0_pred1))
  print("actual 1  | %4d %4d" % (ct_act1_pred0, ct_act1_pred1))
  print("           ----------")
  print("predicted      0    1")
  
# -----------------------------------------------------------

def main():
  # 0. get ready
  print("\nBegin scikit logistic regression SGD learning ")
  np.random.seed(1)

  # 1. load data
  print("\nLoading data into memory ")
  train_file = ".\\Data\\people_train.txt"
  train_xy = np.loadtxt(train_file, usecols=range(0,9),
    delimiter="\t", comments="#",  dtype=np.float32) 
  train_X = train_xy[:,1:9]
  train_y = train_xy[:,0].astype(int)

  test_file = ".\\Data\\people_test.txt"
  test_xy = np.loadtxt(test_file, usecols=range(0,9),
    delimiter="\t", comments="#",  dtype=np.float32) 
  test_X = test_xy[:,1:9]
  test_y = test_xy[:,0].astype(int)

  print("\nTraining data:")
  print(train_X[0:4])
  print(". . . \n")
  print(train_y[0:4])
  print(". . . ")

  # 2. create model and train
  # SGDClassifier(loss='hinge', *, penalty='l2', alpha=0.0001,
  # l1_ratio=0.15, fit_intercept=True, max_iter=1000,
  # tol=0.001, shuffle=True, verbose=0, epsilon=0.1,
  # n_jobs=None, random_state=None, learning_rate='optimal',
  # eta0=0.0, power_t=0.5, early_stopping=False,
  # validation_fraction=0.1, n_iter_no_change=5,
  # class_weight=None, warm_start=False, average=False)

  # loss{'hinge', 'log_loss', 'log', 'modified_huber',
  # 'squared_hinge', 'perceptron', 'squared_error', 'huber',
  # 'epsilon_insensitive', 'squared_epsilon_insensitive'},
  # default='hinge'

  print("\nCreating SGD logistic regression model")
  model = SGDClassifier(loss='log', penalty=None,
    alpha=0.0001, max_iter=1000, tol=0.00001, shuffle=True,
    verbose=0, random_state=1,  learning_rate='optimal',
    eta0=0.0)
  model.fit(train_X, train_y)

  # 3. evaluate
  print("\nComputing model accuracy ")
  acc_train = model.score(train_X, train_y)
  print("Accuracy on training = %0.4f " % acc_train)

  acc_test = model.score(test_X, test_y)
  print("Accuracy on test = %0.4f " % acc_test)
  
  # 4. make a prediction 
  print("\nPredict age 36, Oklahoma, $50K, moderate ")
  X = np.array([[0.36, 0,0,1, 0.5000, 0,1,0]],
    dtype=np.float32)
  
  p = model.predict_proba(X) 
  p = p[0][1]  # first (only) row, second value P(1)

  print("\nPrediction prob = %0.6f " % p)
  if p  "lt" 0.5:
    print("Prediction = male ")
  else:
    print("Prediction = female ")

  # 5. save model
  # print("\nSaving trained logistic regression model ")
  # path = ".\\Models\\people_scikit_model.sav"
  # pickle.dump(model, open(path, "wb"))

  # 6. confusion matrix with labels
  y_predicteds = model.predict(test_X)
  from sklearn.metrics import confusion_matrix
  cm = confusion_matrix(test_y, y_predicteds)
  print("\nConfusion matrix custom: ")
  show_confusion(cm)
 
  print("\nEnd People logistic regression demo ")

if __name__ == "__main__":
  main()