2  Machine Learning Project Workflow

  1. Frame problem
  2. Get data
  3. Gain insights
  4. Prepare data
  5. Choose models
  6. Fine-tune and combine models
  7. Present solutions
  8. Launch, monitor and maintain system

Remember: For all tasks, automate as much as possible

2.1 Frame problem

  1. Objective and current solutions?
  2. New solutions: how to use
  3. Depend on type of problems: possible models, performance measuring
  4. Minimum needed performance
  5. Comparable problems -> Can reuse experiment, tools?
  6. List and verify assumptions (if available)

2.2 Get data

  1. List data: where to get, how much (features, instances), storage space
  2. Get and convert data if necessary
  3. Anonymize sensitive information
  4. Recheck data

2.3 Gain Insights

  1. Create a copy of the data for exploration (sampling it down to a manageable size if necessary).
  2. Create a notebook to keep a record of data exploration.
  3. Study each attribute and its characteristics:
    • Name
    • Type (categorical, int/float, bounded/unbounded, text, structured, etc.)
    • % of missing values
    • Noisiness and type of noise (stochastic, outliers, rounding errors, etc.)
    • Usefulness for the task
    • Type of distribution (Gaussian, uniform, logarithmic, etc.)
  4. For supervised learning tasks, identify the target attribute(s).
  5. Visualize the data.
  6. Study the correlations/mututal information
  7. Identify the promising transformations/feature engineering
  8. Identify extra data that would be useful.
  9. Document what we have learned.

2.4 Prepare data

  • Work on copies of the data (keep the original dataset intact).
  • Write functions for all data transformations we apply, for 3 reasons:
    — Easily prepare the data the next time we get a fresh dataset
    — Easily to apply transformations for test set/new instances once solution is live
    — Treat preparation choices as hyperparameters
  1. Clean the data
    • Fix or remove outliers (optional).
    • Fill in missing values (e.g., with zero, mean, median…) or drop their rows (or columns).
  2. Perform feature selection (optional)
    • Drop the attributes that provide no useful information for the task.
  3. Perform feature engineering, where appropriate
    • Discretize continuous features.
    • Decompose features (e.g., categorical, date/time, etc.).
    • Add promising transformations of features (e.g., log(x), sqrt(x), x2, etc.).
    • Aggregate features into promising new features.
  4. Perform feature scaling
    • Standardize or normalize features.

2.5 Choose models

If the data is very large, it might be better to sample smaller training set to train many different models in a reasonable time, but this will affect performance of complex models such as Random Forest, Neural Networks, etc.

  1. Train many quick-and-dirty models from different categories (e.g., Linear, Naive Bayes, SVM, Random Forest, Neural Networks, etc.) using standard parameters.
  2. Measure and compare their performance: For each model, use N-fold cross-validation and compute the mean and standard deviation of the performance measures.
  3. Analyze the most significant variables for each algorithm.
  4. Analyze the types of errors the models make: What data would a human have used to avoid these errors?
  5. Perform a quick round of feature selection and engineering.
  6. Perform one or two more quick iterations of the five previous steps.
  7. Shortlist the top three to five most promising models, preferring models that make different types of errors.

2.6 Fine-tune and combine models

  • Implementing on full training set.
  • Don’t tweak the model after measuring the generalization error: it would just start overfitting the test set.
  1. Fine-tune the hyperparameters using cross-validation:
    • Treat data transformation choices as hyperparameters, especially when we are not sure about them (e.g., if we are not sure whether to replace missing values with zeros or with the median value, or to just drop the rows).
    • Unless there are very few hyperparameter values to explore, prefer random search over grid search. If the training is very long, prefer a Bayesian optimization approach (e.g., using Gaussian process priors)
  2. Try ensemble methods. Combining our best models will often produce better performance than running them individually.
  3. Once we are confident about final model, measure performance on the test set to estimate the generalization error.

2.7 Present solutions

  1. Document what we have done.
  2. Create a nice presentation: Make sure to highlight the big picture first.
  3. Explain why solution achieves the business objective.
  4. Don’t forget to present interesting points noticed along the way:
    • Describe what worked and what did not.
    • List our assumptions and system’s limitations.
  5. Ensure the key findings are communicated through beautiful visualizations or easy-to-remember statements (e.g., “the median income is the number-one predictor of housing prices”).

2.8 Launch, monitor and maintain system

  1. Get the solution ready for production (plug into production data inputs, write unit tests, etc.).
  2. Write monitoring code to check the system’s live performance at regular intervals and trigger alerts when it drops:
    • Beware of slow degradation: models tend to “rot” as data evolves.
    • Measuring performance may require a human pipeline (e.g., via a crowdsourcing service).
    • Also monitor the inputs’ quality (e.g., a malfunctioning sensor sending random values, or another team’s output becoming stale). This is particularly important for online learning systems.
  3. Retrain the models on a regular basis on fresh data (automate as much as possible).