Category Archives: Scikit

Ridge regression, aka linear ridge regression, is just linear regression with L2 regularization integrated. The idea is to discourage large model weights, which is a cause of model overfitting. Note: linear ridge regression is sometimes confused with kernel ridge regression, … Continue reading →

The RANSAC (random sample consensus) regression technique can be used when the training dataset is known to have outliers (typically, blatant errors). RANSAC is a meta-heuristic that can be applied to any base regression technique, but is usually applied to … Continue reading →

Bottom line: I did a short experiment to compare support vector regression (SVR) with post-training-trimmed kernel ridge regression (KRR). I used the scikit library SVR and KernelRidge modules. Because of the nearly infinite number of hyperparameter values involved, it wasn’t … Continue reading →

I’ve worked with machine learning for decades. In my opinion, one of the least understood topics is cross-validation. I’ll cut to the chase and say that using cross-validation is never a good idea. Frankly, I am astonished that cross-validation techniques … Continue reading →

I write code almost every day. Like many skills, writing code is something that must be practiced, and anyway, I enjoy writing code. One evening after work, I figured I’d run the well-known Diabetes Dataset through a support vector regression … Continue reading →

Two closely related machine learning regression techniques are kernel ridge regression (KRR) and support vector regression (SVR). Both techniques use a kernel function (usually the radial basis function, RBF) to compare two data items for similarity. Both techniques must store … Continue reading →

I try to learn something new every day. That usually means writing some computer code. Over the past few years, I have run into LARS (least angle regression) a few times, but never took the time to try and understand … Continue reading →

Bottom line: For a neural network regressor, if you have categorical predictor data, you should use standard one-hot encoding rather than the drop-first encoding that is usually needed for a linear regression model. Drop-first encoding works for a neural network, … Continue reading →

Isotonic regression is a very niche machine learning technique. The basic scenario is you have a single predictor variable (for example, the size of something) and a single target value to predict (for example, the porosity of the something), where … Continue reading →

Bottom line: I almost never use a stacking regressor model because the number of meta parameters (which base regression models, and which final model to use) and individual model parameters (such as a model learning rate) is astronomically large and … Continue reading →