Most data scientists have written a scikit-learn pipeline that imputes and scales numeric columns, one-hot encodes everything else, and routes columns by dtype. It is short, reusable, and easy to trust. That is exactly the problem. To find out what it silently gets wrong, the author deliberately tried to break it.
The author generated five synthetic customer-churn datasets, each with 6,000 training rows and a 2,000-row clean holdout. Four traps were planted that a dtype-based pipeline cannot see: a numeric region code with no natural order, informative missing values in support_calls, a signup date stored as text that became nearly 1,931 one-hot columns, and a customer ID that leaked the label in the training data.
The naive pipeline scored 0.979 in 5-fold cross-validation but only 0.564 on new customers. Nothing crashed and no warning appeared. The logistic regression's strongest feature, out of 1,940, was the customer ID, which the model used to read the label off the ID and memorize per-date noise. Dropping the ID made cross-validation honest, at 0.640 versus 0.632 on holdout, but the other problems remained.
The missing-value indicator gave the largest single gain, lifting holdout AUC from 0.564 to 0.670. Only applying all four fixes together reached 0.826 on holdout, where the honest cross-validation score of 0.833 finally agreed with it. A stronger model such as gradient boosting only partly rescued the naive pipeline, scoring 0.614 on holdout.
The lesson is that a clean validation score can hide serious leakage, and automated preprocessing needs to be checked against the meaning of each column.
Source: Towards Data Science · Summarized by HeadlinesBriefing