Category Archives: Scikit

I recently implemented support vector regression (SVR) from scratch using the C# language, trained using stochastic sub-gradient descent (SSGD). The SVR system seemed to work well but I figured I’d compare it with the gold standard scikit-learn SVR module. Bottom … Continue reading →

Bottom line: I put together a demo of (kernelized) support vector regression that uses stochastic gradient descent (SGD) training. It works fine but training is relatively slow. Actually, the technique I used is stochastic sub-gradient descent (SSGD) but it’s common … Continue reading →

In spite of decades of research in machine learning, there are still dozens of pieces of common, but just plain incorrect, pieces of knowledge floating about on the Internet. I came across one recently. Does a kernel ridge regression model … Continue reading →

Linear ridge regression (often called just ridge regression, but not to be confused with kernel ridge regression) uses mathematical L2 regularization to prevent model weights from becoming too large, which in turn limits model overfitting. Whew! There’s a lot of … Continue reading →

In machine learning, if training data is multicollinear, the interpretability of a resulting linear model will likely be poor. Multicollinear data has two or more rows that are highly correlated mathematically. For example, if a set of training data has … Continue reading →

Bottom line: I compared kernel ridge regression (KRR) and support vector regression (SVR), using a small set of synthetic data. The results were essentially the same. Over the past few years, based on several machine learning regression projects that I … Continue reading →

There is absolutely no reason to ever use L1 regularization. Well, maybe that’s a bit too extreme of a statement. But, “I never use L1 regularization or lasso regression unless I’m forced to by a legacy system” is accurate. I … Continue reading →

I write code almost every day. Like most skills, writing code is something that must be practiced. And besides that, I enjoy writing code. I had recently implemented a version of kernel support vector regression (SVR), from-scratch, using Python. So, … Continue reading →

I came across a relatively obscure module in the scikit-learn library called IsolationForest. The module is used for anomaly detection. In a nutshell, if you apply a decision tree to a dataset, because of the way the branches are created, … Continue reading →

Bottom line: I never use a voting regressor model because the number of meta parameters (which regression models to use) and model parameters (such as a model learning rate) is astronomically large and therefore unmanageable in practice. A voting regressor … Continue reading →