I wasted ~$4,000 in compute costs last month due to a silent broadcasting bug that likely derailed my work many times. If you're an ML practitioner, this has probably happened to you without you realizing it. A single mismatched tensor dimension can silently rewrite your loss function, gut your gradients, or poison your project, and neither PyTorch nor TensorFlow will raise an error.

Broadcasting allows elementwise math on tensors of different shapes. But the compatibility of (N, 1) and (N,) is a hidden killer. Combining them produces an (N, N) matrix that is likely not what you intended. The operation runs silently, loss goes down, gradients flow, but your model trains towards garbage.

Example 1: Your regression loss quietly optimizes for the mean, not the input. In PyTorch or TensorFlow, pred - target broadcasts to (N, N), computing target[i] - pred[j] for every pair. The "loss" becomes L = (1/N^2) * sum((t_i - p_j)^2). Every prediction converges to the batch mean of targets. Training doesn't crash, loss drops fast, but the model learns nothing about the relationship between x and y.

This is the most common version of the bug. I've seen it in production because the loss looks asymptomatic and the mean value solution can produce reasonable performance. New models and features are often blamed, but silent broadcasting is the real culprit.