机器学习过程中的四个误区：

数据泄露；过拟合；数据采用和切分；数据质量。

　　In a recent presentation, Ben Hamnerdescribed the common pitfalls in machine learning projects he and his colleagues have observed during competitions on Kaggle.

　　The talk was titled “Machine Learning Gremlins” and was presented in February 2014 at Strata.

　　In this post we take a look at the pitfalls from Ben’s talk, what they look like and how to avoid them.

　　Machine Learning Process

　　Early in the talk, Ben presented a snap-shot of the process for working a machine learning problem end-to-end.

　　

　　Machine Learning Process

　　Taken from “Machine Learning Gremlins” by Ben Hamner

　　This snapshot included 9 steps, as follows:

Start with a business problem

Source data

Split data

Select an evaluation metric

Perform feature extraction

Model Training

Feature Selection

Model Selection

Production System

　　He commented that the process is iterative rather than linear.

　　He also commented that each step in this process can go wrong, derailing the whole project.

　　Discriminating Dogs and Cats

　　Ben presented a case study problem for building an automatic cat door that can let the cat in and keep the dog out. This was an instructive example as it touched on a number of key problems in working a data problem.

　　

　　Discriminating Dogs and Cats

　　Taken from “Machine Learning Gremlins” by Ben Hamner

　　Sample Size

　　The first great takeaway from this example was that he studied accuracy of the model against data sample size and showed that more samples correlated with greater accuracy.

　　He then added more data until accuracy leveled off. This was a great example of understanding how easy it can be get an idea of the sensitivity of your system to sample size and adjust accordingly.

　　Wrong Problem

　　The second great takeaway from this example was that the system failed, it let in all cats in the neighborhood.

　　It was a clever example highlighting the importance of understanding the constraints of the problem that needs to be solved, rather than the problem that you want to solve.

　　Pitfalls In Machine Learning Projects

　　Ben went on to discuss four common pitfalls in when working on machine learning problems.

　　Although these problems are common, he points out that they can be identified and addressed relatively easily.

　　

　　Overfitting

　　Taken from “Machine Learning Gremlins” by Ben Hamner

Data Leakage: The problem of making use of data in the model to which a production system would not have access. This is particularly common in time series problems. Can also happen with data like system id’s that may indicate a class label. Run a model and take a careful look at the attributes that contribute to the success of the model. Sanity check and consider whether it makes sense. (check out the referenced paper “Leakage in Data Mining” PDF)

Overfitting: Modeling the training data too closely such that the model also includes noise in the model. The result is poor ability to generalize. This becomes more of a problem in higher dimensions with more complex class boundaries.

Data Sampling and Splitting: Related to data leakage, you need to very careful that the train/test/validation sets are indeed independent samples. Much thought and work is required for time series problems to ensure that you can reply data to the system chronologically and validate model accuracy.

Data Quality: Check the consistency of your data. Ben gave an example of flight data where some aircraft were landing before taking off. Inconsistent, duplicate, and corrupt data needs to be identified and explicitly handled. It can directly hurt the modeling problem and ability of a model to generalize.

Summary

　　Ben’s talk “Machine Learning Gremlins” is a quick and practical talk.

　　You will get a useful crash course in the common pitfalls we are all susceptible to when working on a data problem.

　　出处：machinelearningmastery。