I present quite a few workshops on machine learning using Python. Popular workshop topics include Keras, PyTorch, and classical ML techniques (such as logistic regression and k-means clustering). All of these topics use data files, for example, training and test data for a neural network.

Whenever I present a workshop, I’m always surprised by the number of questions I get about basic data preparation. (I guess I shouldn’t be surprised, because it happens every time).

It’s well-known that data preparation for ML tasks is usually extremely tedious and time consuming. It’s not uncommon for data preparation to take 90% of the time required to create an ML prediction system. For example, in one 12-week long “Advanced Projects” effort I oversee, team members routinely use 10 of the 12 weeks for data preparation.

There’s no single roadmap for data preparation — every problem scenario is quite different. But the overall steps are usually:

1. do a preliminary analysis of the raw data
2. clean the data by dealing with missing data and outliers
3. normalize numeric predictor variable values
4. encode non-numeric predictor variable values
5. if necessary, encode the variable-to-predict values
6. if necessary, split to train and test files

Each of these steps requires far more time than you’d guess. The preliminary analysis is usually straightforward. You need to know the structure of the source data files. At a bare minimum you need to know 10 things.

Is the file encoded using ASCII, UTF-8, or something else? How many lines are there? Are there comment lines? Is there a header line? What newline character is used to terminate lines? Are lines delimited by tabs or commas or whitespace or something else? How many columns are there in each line? Is there missing data? If so, how is it indicated? Does the file contain non-printable control characters?

So, there’s no moral here except to reiterate that data preparation for machine learning is annoyingly time consuming.

The am a licensed minister and can perform weddings in the State of Washington. Brides always spend a lot of time on wedding preparations, especially the wedding dress. Left: I like this dress which eliminates the need to stand in line at the bar at the wedding reception. Center: The dress immediately calls to mind Ursula from “The Little Mermaid”. I think the groom should be concerned. Right: I like tiered wedding cakes but I’m not so sure about tiered wedding dresses.