"Nadaraya-Watson Kernel Regression Using C#" in Visual Studio Magazine

I wrote an article titled “Nadaraya-Watson Kernel Regression Using C#” in the April 2025 edition of Microsoft Visual Studio Magazine. See https://visualstudiomagazine.com/articles/2025/04/18/nadaraya-watson-kernel-regression-using-c.aspx.

The goal of a machine learning regression problem is to predict a single numeric value. There are approximately a dozen common regression techniques, including ordinary linear regression, k-nearest neighbors regression, random forest regression, kernel ridge regression, and neural network regression.

Kernel regression (not to be confused with kernel ridge regression, a very common technique) is a relatively rare regression technique. There are several versions of kernel regression, and each version has several variations. The specific type of kernel regression presented in the VSM article is Nadaraya-Watson kernel regression (NW kernel regression) using a radial basis function kernel. The article presents an end-to-end demo.

A NW kernel regression model requires a kernel function which measures the similarity between two data items/vectors. The demo uses a radial basis function (RBF) kernel. The RBF kernel requires a single parameter, gamma, which is set to 6.00 in the demo. The gamma parameter must be determined by trial and error.

The output of the demo is:

Begin Nadaraya-Watson kernel regression prediction demo

Loading synthetic train (200) and test (40) data
Done

First three train X:
 -0.1660  0.4406 -0.9998 -0.3953 -0.7065
  0.0776 -0.1616  0.3704 -0.5911  0.7562
 -0.9452  0.3409 -0.1654  0.1174 -0.7192

First three train y:
  0.4840
  0.1568
  0.8054

Creating NW kernel regressor with RBF gamma = 6.00
Done

Evaluating model

MSE train = 0.0140
MSE test = 0.0405

Accuracy train (within 0.10) = 0.9850
Accuracy test (within 0.10) = 0.7500

Predicting for x =
  -0.1660   0.4406  -0.9998  -0.3953  -0.7065

Predicted y = 0.5308

End prediction demo

What is happening in NW kernel regression is that a predicted y value is the weighted average of all the target y values in the training data, where the weights are the corresponding RBF kernel function values. When a kernel value is close to 1, it means the two vectors are similar and the predicted y component is close to the corresponding target y.

The NW kernel regression technique is similar to weighted k-nearest neighbors regression, but instead of using just k training data items and Euclidean distance, NW kernel regression uses all data items and RBF similarity. Clever!

I love science fiction movies from the 1950s and 1960s. None of these films had a machine learning kernel, but many had a military colonel character. Here are three early examples from 1958 and 1959.

Left: In “Angry Red Planet” (1959), a crew of four goes to Mars but only two return. The explorers encounter Martians, a weird giant bat-rat, a man-eating plant, and an enormous amoeba. This movie generates extreme opinions. I give it a C+ grade mostly because I liked the amoeba. Near the beginning of the movie, Colonel Davis (actor Jack Haddock, in the center of this photo) plays a minor role in the command center.

Center: In “The Space Children” (1958), a group of children living on a military base become controlled by an alien being. The alien directs them to destroy a nuclear weapon being developed on the base. But the alien has good intentions because other aliens are destroying similar devices being made by the Soviet Union and China, so that no country could pull the nuclear trigger. Colonel Alan Manley (actor Richard Shannon) runs the military base. I give the movie a C grade.

Right: “War of the Colossal Beast” (1958) is a sequel to “The Amazing Colossal Man” released the previous year. Colonel Glenn Manning (actor Dean Parkin) was exposed to a plutonium bomb and got big. Most sci fi movie fans like this movie more than I do. I give it a C- grade.