A data scientist warns that treating text‑derived themes as raw observations corrupts causal inference. In a typical pipeline, support transcripts are converted into binary flags like “billing frustration,” with missing entries filled as zero. The resulting regression shows a significant coefficient, which product teams copy into roadmaps, unaware that the variable originates from a generative process.

Four hidden flaws arise from this practice. First, selection bias appears because only customers who left a textual trace receive a theme, reshaping the population under analysis. Second, timing errors misclassify pre‑, concurrent, or post‑treatment text, leading to classic post‑treatment bias. Third, measurement noise stems from imperfect classifiers that may perform differently across treatment groups. Fourth, the role a theme plays—confounder, mediator, or outcome—depends on the causal graph, not the column name.

The article demonstrates the problem with a synthetic retention‑offer experiment where a “bill‑shock” flag, derived from post‑treatment calls, flips the estimated effect of the offer from harmful to beneficial. Ignoring the underlying data‑generating process produces contradictory results and masks true treatment impact. Analysts must expose selection, timing, measurement, and role assumptions before trusting LLM‑generated features.