I wrote an article titled “Support Vector Regression with SGD Training Using C#” in the July 2026 issue of Microsoft Visual Studio Magazine. See https://visualstudiomagazine.com/articles/2026/07/01/support-vector-regression-with-sgd-training-using-csharp.aspx.

The goal of a machine learning regression problem is to predict a single numeric value. For example, a bank might want to predict the maximum safe loan amount for a customer, based on age, account balance, annual income, and so on.

There are approximately a dozen common regression techniques. Each technique has pros and cons. A technique that sometimes produces highly accurate predictions for specific types of data is called support vector regression (SVR).

The term “support vector regression” by itself is ambiguous, because there are two types of SVR: linear SVR and kernelized SVR. Linear SVR is rarely used so the term “SVR” by itself usually means kernel SVR.

My article presents a demo of kernel SVR, trained using a variation of stochastic gradient descent (SGD) called stochastic sub-gradient descent (SSGD). Note: Even though SSGD is technically a variation of SGD, the two techniques are so similar that SSGD is often called SGD.

Support vector regression (SVR) predicts using a kernel function, usually RBF, that computes similarity between two data items. During training, some of the training items are determined to be irrelevant to some extent, and they are removed. The training items that are left are called the support vectors.

The key parts of the demo output are:

Creating SVR object
Setting RBF gamma = 0.3000
Setting epsilon = 0.007500
Setting C = 1.00
Setting lrnRate = 0.0010
Setting maxEpochs = 5000
Setting tol = 0.000100

Training SVR model using SGD
epoch =      0 MSE = 0.0430 acc = 0.1300
epoch =   1000 MSE = 0.0001 acc = 0.9850
epoch =   2000 MSE = 0.0001 acc = 0.9850
epoch =   3000 MSE = 0.0001 acc = 0.9800
epoch =   4000 MSE = 0.0001 acc = 0.9800
Done

Model alpha (weights):
 -0.9256  -0.0443  -0.0041  -0.6581  . . .   0.0049
  0.2983   0.4573  -0.0488   0.0956  . . .   0.1471
. . .
  0.0017   0.0023   0.3193   0.0063  . . .  -0.0078
 -0.0690   0.9895   . . .    0.3843

Model bias = 0.4030
Number supp vectors = 194

Train acc (within 0.10) = 0.9900
Test acc (within 0.10) = 0.9250

The first versions of support vector regression in the 1980s required quadratic programming optimization training, which is very complicated, slow, and doesn’t scale well to large datasets. The sequential minimal optimization (SMO) training algorithm was developed in the late 1990s. SMO is fast but the algorithm is complicated, and very difficult to correctly implement. The SGD (actually stochastic sub-gradient descent, SSGD) training technique presented in the article is by far the simplest training algorithm, and often works well in practice because processing one data item at a time introduces a form of implicit regularization, which helps the SVR model to predict new, previously unseen data.

Support vector regression had a brief surge of popularity in the late 1990s and early 2000s. However, data scientists realized that the closely related kernel ridge regression (KRR) has several significant advantages over SVR, and so the use of SVR declined to the point where it is not used very much today. SVR is more difficult to implement than KRR, SVR is much more difficult to tune than KRR (KRR can use true SGD, which is easier to tune than SVR sub-gradient descent), and SVR often gives slightly worse prediction accuracy than KRR (due mostly to the difficulty in parameter tuning). That said, there are some problem domains, such as biology and chemistry, where kernel SVR is often used and is highly effective.

I don’t collect things. I collect weird experiences. A few years ago, I noticed that when I have my TV closed captioning turned on, a surprising number of animals “chitter”. Now I’m always on the alert for “chittering”.

Right: “Holland” (2025) – I had hopes for this movie because it stars Nicole Kidman who is a terrific actress. Nope. A woman discovers she married a serial murderer. Huh? A racoon in an alley trash can was the highlight of the movie. Grade = C-.