Comparing 4 ML Classification Techniques: Logistic Regression, Perceptron, Support Vector Machine, and Neural Networks

I wrote an article titled “Comparing 4 ML Classification Techniques: Logistic Regression, Perceptron, Support Vector Machine, and Neural Networks” in the April 2020 edition of the PureAI Web site. See https://pureai.com/articles/2020/04/10/ml-techniques.aspx.

The goal of a classification problem is to predict the value of a variable that can take on discrete values. In binary classification the goal is to predict a variable that can be one of just two possible values, for example predicting the gender of a person (male or female). In multi-class classification the goal is to predict a variable that can be three or more possible values, for example, predicting a person’s state of residence (Alabama, Alaska, . . . Wyoming). Note that a regression problem is one where the goal is to predict a numeric value, for example the annual income of a person.



I used this example to explain how logistic regression binary classification works.


There are dozens of ML classification techniques, and most of these techniques have several variations. One way to mentally organize ML classification techniques is to place each into one of three categories: math equation classification techniques, distance and probability classification techniques, and tree classification techniques.

Four common math equation techniques are logistic regression, perceptron, support vector machine, and single hidden layer neural networks. These four ML classification techniques all involve some sort of a math equation that is a sum of products of weights times predictor input values. In the article, I explain briefly how each technique works by giving a short example. Then I compare the four techniques based on nine criteria:

1.) dealing with model overfitting
2.) dealing with error, loss, and accuracy
3.) dealing linearly separable data
4.) dealing with varying magnitude numeric predictor variables
5.) dealing with non-numeric predictor variables
6.) dealing with multi-class classification problems.
7.) dealing with training and test data
8.) dealing with interpretability
9.) dealing with implementation and integration.

Learning about machine learning is difficult. Although each individual idea in machine learning classification is relatively simple and easy to understand, there are many ideas involved and the ideas interact in complicated ways. Learning some subjects can be done sequentially. For example, to understand math Calculus, you first learn algebra, then trigonometry, then limits, and so on. But there is no sequential path for mastering machine learning. You must learn in a spiral manner, revisiting topics several times until you reach a solid understanding.


“The Precious Stones” are a series of four posters painted in 1900 by Czech artist Alphons Mucha (1860 – 1939). From left: “Ruby”, “Amethyst, “Emerald”, “Topaz”. These works are classified as “Art Nouveau”.

This entry was posted in Machine Learning. Bookmark the permalink.