In spite of decades of research in machine learning, there are still dozens of pieces of common, but just plain incorrect, pieces of knowledge floating about on the Internet. I came across one recently. Does a kernel ridge regression model need a bias term or not?

Let me cut to the chase and say that the answer is that kernel ridge regression models do not need a bias term. But many Internet and AI resources claim that a bias term is absolutely needed. For example, an AI told me today:

“The bias term (intercept) in a kernel ridge regression model acts as an offset to shift the regression line or hyperplane up or down. Without it, the model is geometrically constrained to pass exactly through the origin (f(x) = 0 when (x = 0), which would drastically hurt predictive accuracy.”

Wrong.

The correct logic is:

“Kernel ridge regression works well without an explicit bias term because the kernel trick implicitly maps data into a high-dimensional space where “non-centered” data is inherently handled by the flexibility of the kernel, and the similarity computations act as a localized center.

Many kernels (including the radial basis function) measure the localized similarity between points. The prediction is essentially computed as a weighted sum of similarities to the training vectors. Even if raw training data is not centered, this process simply shifts the “humps” of the kernel evaluations around, allowing the model to naturally fit the true offsets.”

The scikit KernelRidge module does not introduce a bias term into its models.

Output of a demo:

Begin demo

Generating 40 rows dummy data
Done

First three X:
[[1.0976]
 [1.4304]
 [1.2055]]

First three y:
[0.8013 0.7921 0.8992]

Creating and training SVR model
Done

Model bias = 0.6412
Model R2 score = 0.5289

Creating and training KRR model
Done

Model has no bias
Model R2 score = 0.6373

End demo

The moral of the story is that it’s prudent to be wary of AI-generated information. AI always presents information with a sense of complete confidence. Humans are conditioned to believe just about anything, from anyone or anything, when it is presented with an air of absolute confidence. (Including information in blog posts like this one).

Machine learning myths have relatively low impact. But some myths have very big consequences. One common myth that I hear, primarily from young people in their 20s and 30s, is that intelligence has no genetic component, and therefore is not inherited.

When I was a college student at UC Irvine, I worked at Disneyland in the evenings and the weekends. Everyone knows that the Park’s motto is, “The Happiest Place on Earth”. I’m going to guess that these brawling families at Disneyland are not examples of inherited intelligence at its finest, and that they are not having an especially Happy Time. And they probably had even less fun after they were all arrested.

Demo program:

# svr_bias_scikit.py

import numpy as np
from sklearn.svm import SVR
from sklearn.kernel_ridge import KernelRidge

import random
import numpy as np
random.seed(0)
np.random.seed(0)

# -----------------------------------------------------------

np.set_printoptions(precision=4, suppress=True,
  floatmode='fixed', linewidth=60)

# -----------------------------------------------------------

print("\nBegin demo ")

print("\nGenerating 40 rows dummy data ")
X = 2 * np.random.rand(40, 1)
y = np.sin(X).ravel() + \
  np.random.normal(0, 0.1, X.shape[0])
print("Done ")

print("\nFirst three X: ")
print(X[0:3])

print("\nFirst three y: ")
print(y[0:3])

gamma = 0.02

print("\nCreating and training SVR model ")
svr = SVR(gamma=gamma)
svr.fit(X, y)
print("Done ")

if hasattr(svr, "intercept_"):
  svr_bias = svr.intercept_[0]
  print("\nModel bias = %0.4f " % svr_bias)
else:
  print("\nModel has no bias ")

svr_r2 = svr.score(X, y)
print("Model R2 score = %0.4f " % svr_r2)

print("\nCreating and training KRR model ")
krr = KernelRidge(gamma=gamma)
krr.fit(X, y)
print("Done ")
if hasattr(krr, "intercept_"):
  krr_bias = krr.intercept_[0]
  print("\nModel bias = %0.4f " % krr_bias)
else:
  print("\nModel has no bias ")

krr_r2 = krr.score(X, y)
print("Model R2 score = %0.4f " % krr_r2)

print("\nEnd demo ")