Category Archives: Machine Learning

My standard from-scratch implementation of kernel ridge regression (KRR) trains using the Cholesky inverse of a RBF kernel matrix. It’s possible to train by computing a Cholesky decomposition and then using a solve method. The explicit inverse approach looks like: … Continue reading →

There are three main ways to train a quadratic regression model: 1.) using stochastic gradient descent, 2.) using left pseudo-inverse (normal equations) via Cholesky inverse, 3.) using relaxed Moore-Penrose pseudo-inverse via one of many possible inverses. I have implemented many … Continue reading →

Bottom line: I decided to implement AdaBoost.R2 regression using Extra (“Extremely Randomized”) Trees learners. Bottom line: For my demo dataset, the technique worked better than the standard architecture that uses regular decision trees as the learners. As is often the … Continue reading →

One Sunday evening, I was sitting in my living room. I decided to implement a regression system (to predict a single numeric value), using a neural network with exactly two hidden layers, from scratch, using Python with NumPy. The effort … Continue reading →

I like to write code every day. Writing code is a skill that must be practiced, and I enjoy writing code. Over the course of my career, I saw countless colleagues move from an engineering role to an engineering-management role, … Continue reading →

The goal of a machine learning regression problem is to predict a single numeric value. Quadratic regression is an enhanced form of basic linear regression. One of several ways to train a quadratic regression model is to use relaxed Moore-Penrose … Continue reading →

AdaBoost.R2 (“adaptive boosting regression, version 2”) is a technique to predict a single numeric value. AdaBoost works by creating a collection of simple decision trees. The final prediction is the average of all the tree predictions. Each tree is trained … Continue reading →

In general, when training a machine learning regression model using stochastic gradient descent, I do not use early-exit from the training loop. Any type of early-exit requires some sort of parameter that must be determined by trial and error, which … Continue reading →

Machine learning quadratic regression is a technique to predict a single numeric value. Thee are many ways to train a quadratic regression model. One training approach that is fast, reliable, and scales to large datasets — but is very complicated … Continue reading →