I wrote an article titled “Regression Using scikit Kernel Ridge Regression” in the July 2023 edition of Microsoft Visual Studio Magazine. See https://visualstudiomagazine.com/articles/2023/07/06/scikit-kernel-ridge-regression.aspx.

A regression problem is one where the goal is to predict a single numeric value. For example, you might want to predict the annual income of a person based on their sex, age, State where they live and political leaning.

There are many different techniques available to create a regression model. Some common techniques, listed from less complex to more complex, are: linear regression, linear lasso regression, linear ridge regression, k-nearest neighbors regression, (plain) kernel regression, kernel ridge regression, Gaussian process regression, decision tree regression and neural network regression.

My article explains how to create and use kernel ridge regression (KRR) models using the scikit-learn KernelRidge module. Compared to other regression techniques, KRR is especially useful when there is limited training data.

The key to KRR is understanding kernel functions. A kernel function accepts two vectors and returns a value that is a measure of similarity/dissimilarity. There are many possible kernel functions and each type has several variations.

One of the most common kernel functions is the radial basis function. A common version is defined as rbf(a, b, gamma) = exp(-gamma * ||a – b||^2). The ||a – b||^2 term is the squared Euclidean distance between vectors a and b. For example:

rbf(x0, x1, 1.0) 
 = exp( -1 * (0.1-0.4)^2 + (0.5-0.3)^2 + (0.2-0.0)^2 )
 = exp( -1 * (0.09 + 0.04 + 0.04) )
 = exp(-0.17)
 = 0.8437

Creating, training, and using a scikit KRR model is easy:

# load train and test data here
print("Creating and training KRR RBF(1.0) model ")
model = KernelRidge(alpha=0.1, kernel='rbf', gamma=1.0)
model.fit(train_X, train_y)
print("Predicting income for M 34 Oklahoma moderate: ")
X = np.array([[0, 0.34, 0,0,1,  0,1,0]],
    dtype=np.float32)
pred_inc = model.predict(X)

The scikit KernelRidge module creates models that are often surprisingly effective, especially with small datasets. A technique that is closely related to, but is definitely different from kernel ridge regression, is called just kernel regression. The use of plain kernel regression is quite rare so the term “kernel regression” is often used to refer to kernel ridge regression.

Kernel ridge regression often works very well, but it’s not magic. The demo program in the article predicts a person’s annual income using kernel ridge regression. KRR is an early form of machine learning, which is a form of artificial intelligence. Here are two images I generated using the DALL-E text-to-image AI tool with a prompt of “a witch predicting a person’s annual income”.