Kernel ridge regression (KRR) is a technique to predict a single numeric value. KRR uses a kernel function, which compares two training item vectors and computes a measure of their similarity in order to handle complex non-linear data, and the ridge technique (aka L2 regularization) to prevent model overfitting.

If you have three xi training vectors, the predicted value for an input vector x is y’ = (w0 * k(x, x0)) + (w1 * k(x, x1)) + (w2 * k(x, x2)). The wi are the model weights, the xi are the predictors, and k() is a kernel function. Of course, you usually have a lot more than just three training items.

There are many possible kernel functions. The most common, based on my experience, is the radial basis function (RBF). There are two main versions of RBF. The one I use is rbf(v1, v2) = exp(-1 * gamma * ||v1 – v2||^2) where gamma is a free parameter, sometimes called the inverse bandwidth, and ||v1 – v2||^2 is the squared Euclidean distance between vectors v1 and v2. If v1 = v2, the RBF value is 1.0 (maximum similarity). As the difference between v1 and v2 increases, RBF decreases towards 0.0.

Training a KRR prediction model is the process of finding the values of the weights. There is one weight per training item. There are several techniques that can be used to train a KRR model. The techniques fall into two categories: 1.) closed-form techniques that use a matrix inverse, and 2.) iterative techniques that do not use a matrix inverse. This blog describes the closed-form training technique that uses a matrix inverse. The technique is fast but does not work for huge datasets.

For my demo, I used synthetic data that looks like:

-0.1660,  0.4406, -0.9998, -0.3953, -0.7065,  0.4840
 0.0776, -0.1616,  0.3704, -0.5911,  0.7562,  0.1568
-0.9452,  0.3409, -0.1654,  0.1174, -0.7192,  0.8054
. . .

The first five values on each line are predictor values. The last value is the target y value to predict. The data was generated by a 5-10-1 neural network with random weights and bias values. There are 200 training items and 40 test items.

Saving a trained KRR model is not technically difficult, but choosing a good API signature is extremely difficult. If a KRR model using the RBF kernel function was trained using N training items, the Predict() method requires N model weights, and the N training data items, and the gamma value used for the internal RBF kernel function.

A design to save a trained KRR-RBF model can save just the weights (and assume that the training data and gamma value are available), or save the weights and the training data, or save everything (training data, weights data, gamma). There are many other design possibilities, but there is no clear best design.

I put together a demo that saves just model weights, and creates a new KRR model using those saved weights. The output of the demo is:

Begin KRR with Cholesky matrix inverse training

Loading train (200) and test (40) from file
Done

First three train X:
 -0.1660  0.4406 -0.9998 -0.3953 -0.7065
  0.0776 -0.1616  0.3704 -0.5911  0.7562
 -0.9452  0.3409 -0.1654  0.1174 -0.7192

First three train y:
  0.4840
  0.1568
  0.8054

Setting RBF gamma = 0.3
Setting alpha noise =  0.005

Creating and training KRR model using Cholesky
Done

Model weights:
 -2.0218  -1.1406 . . . 1.2897

Predicting for x =
  -0.1660   0.4406  -0.9998  -0.3953  -0.7065
Predicted y = 0.4941

Saving trained model weights

Creating new model using saved weights

Using new model to predict for x =
  -0.1660   0.4406  -0.9998  -0.3953  -0.7065
Predicted y = 0.4941

End demo

The key calling statements of the demo are:

// load trainX, trainY up here

double gamma = 0.3;    // RBF param
double alpha = 0.005;  // training regularization

KRR krr = new KRR(gamma, alpha);
krr.Train(trainX, trainY);

double[] x = trainX[0];
double predY = krr.Predict(x);
Console.WriteLine("Predicted y = " + predY.ToString("F4"));

Console.WriteLine("Saving trained model weights ");
string weightsFile = 
  "..\\..\\..\.\\Weights\\model_wts.txt";
krr.SaveWeights(weightsFile);

Console.WriteLine("Creating new model using" +
  " saved weights ");
KRR krr2 = new KRR(gamma, alpha);
krr2.LoadWeights(weightsFile);
krr2.LoadTraingData(trainX, trainY); // needed to predict

double y = krr2.Predict(x);
Console.WriteLine("Predicted y = " + y.ToString("F4"));

I was surprised by the huge number of design possibilities. I expected my exploration to be simple and quick. But the exploration produced well over two dozen significantly different designs, and the coding took me over 10 hours on a Sunday afternoon and evening and into Monday early morning.

The moral of the story is that in software development, sometimes activities that should be simple are anything but simple.

Machine learning is essentially automated prediction.

Right: The Hog’s Head Pub is at several of the Universal Studios parks. The boar’s head moves and he grunts. Theme park restaurants have built-in new customers so the quick loss of novelty isn’t an issue.

The base example and data are at: https://jamesmccaffreyblog.com/2025/08/25/kernel-ridge-regression-with-cholesky-decomposition-matrix-inverse-training-using-csharp/

The save-load code is here. Replace “lt” with the Boolean less-than operator.

public void SaveWeights(string modelData)
{
  // just weights. assume train data available
  // assume RBF gamma is known
  FileStream ofs = new FileStream(modelData,
    FileMode.Create);
  StreamWriter sw = new StreamWriter(ofs);

  for (int i = 0; i "lt" this.wts.Length; ++i)
    sw.WriteLine(this.wts[i]);
  sw.Close(); ofs.Close();
}

public void LoadWeights(string modelData)
{
  FileStream fs = new FileStream(modelData,
    FileMode.Open);
  StreamReader sr = new StreamReader(fs);

  List weightsLst = new List();
  while (sr.EndOfStream == false)
    weightsLst.Add(double.Parse(sr.ReadLine()));
  this.wts = new double[weightsLst.Count];
  this.wts = weightsLst.ToArray();
  sr.Close(); fs.Close();
}

public void LoadTraingData(double[][] trainX,
  double[] trainY)
{
  this.trainX = trainX;  // by ref -- could copy
  this.trainY = trainY;  // not used this version
}