When implementing EEG machine-learning pipelines, splitting data into random windows can create misleadingly optimistic results if the same subject appears in both training and testing sets. This approach fails to test true generalization to new subjects, as models may learn participant-specific patterns rather than task-related signals. The issue extends beyond EEG to any domain with repeated measurements from the same source, such as wearable sensors or behavioral data.
In this study, researchers replicated a simple EEG experiment comparing background and mental arithmetic task data using spectral features and three standard machine learning algorithms. They contrasted random-window splitting with participant-held-out evaluation, where all windows from a subject are assigned exclusively to either training or testing. The dataset used was the EEG During Mental Arithmetic Tasks (EEGMAT) from Physio Net, containing 36 recordings (18 subjects, each with background and arithmetic recordings).
After filtering and windowing, 2,134 four-second windows were extracted—1,594 background and 540 arithmetic—yielding 76 features per window via relative power in four frequency bands across 19 EEG channels. The analysis aimed to isolate the effect of evaluation methodology, not to develop new biomarkers, by using conventional features and controlling for class balance and feature discriminability. Results showed that participant-aware evaluation significantly altered performance estimates, revealing that standard random splits overestimate model generalization when subject-specific structure is present.
Source: Towards Data Science · Summarized by HeadlinesBriefing