EEG Eye State Classification and Analysis Report using Machine and Deep Learning Algorithm

Dataset Source: UCI Machine Learning Repository — EEG Eye State

Submitted to:

• Prof. Giuseppe Longo,
• Prof.ssa Roberta Siciliano

Report by:

• Andrea Manzo
• Francesco Ventimiglia
• Danilo Rodriguez
• Rohan Baidya

Table of Contents

• EEG Eye State Classification and Analysis Report using Machine and Deep Learning Algorithm
  • Table of Contents
  • Process Work Flow
• 1. Data Description Overview
  • 1.1 Dataset Citation & Source
  • 1.2 Dataset Loading
  • 1.3 Variable Classification & Electrode Positions
  • 1.4 Basic Statistics
  • 1.5 Class Distribution
• 2. Data Imputation
• 3. Data Visualization (Raw Data)
  • 3.1 Class Balance
  • 3.2 Correlation Heatmap
  • 3.3 Box Plots
  • 3.4 Histograms
  • 3.5 Violin Plots
• 4. Signal Preprocessing
  • 4.1 IQR Spike Removal (applied first, before filtering)
  • 4.2 Bandpass Filter (0.5–45 Hz) — applied after spike removal
• 5. Data Visualization (After Preprocessing)
  • 5.1 Corrected Correlation Heatmap (after preprocessing)
  • 5.2 Box Plots Comparison
  • 5.3 Histograms After Cleaning
• 6. Log-Normalization Assessment (Rejected)
  • 6.1 Before vs After — All Channels
  • 6.2 Skewness & Kurtosis Analysis
  • 6.3 Summary Statistics Before vs After
• 7. Feature Engineering
  • 7.1 Hemispheric Asymmetry
  • 7.2 Frequency Band Power Features
• 8. FFT, Spectrogram and PSD Analysis
  • 8.1 FFT Frequency Spectrum
  • 8.2 Power Spectral Density (PSD)
  • 8.3 Spectrogram Analysis
• 9. Dimensionality Reduction
  • 9.1 LDA
  • 9.2 t-SNE
  • 9.3 UMAP
  • 9.4 Clustering Evaluation
  • 9.5 Inference: Dimensionality Reduction Comparison
• 10. Machine Learning Classification
  • 10.1 Temporal Concept Drift Diagnosis
  • 10.2 Split Configurations
  • 10.3 Cross-Validation Results (5-Fold TimeSeriesSplit)
  • 10.4 Hold-Out Split Results
    • Split 70/15/15
    • Split 60/20/20
    • Split 80/10/10
  • 10.5 Walk-Forward CV (Expanding Window) — 5 Folds
  • 10.6 Sliding-Window CV (Fixed-Size Window) — 5 Folds
• 11. Deep Learning Classification
  • 11.0 Architecture Overview & Training Setup
    • Split 70/15/15
  • 11.1 LSTM
  • 11.2 CNN_LSTM
  • 11.3 EEGTransformer
  • 11.4 EEGNet
  • 11.5 PatchTST_Lite
  • 11.6 Soft-Vote Ensemble — 70/15/15
  • 11.7 DL Model Comparison — 70/15/15
    • Split 60/20/20
  • 11.6 Soft-Vote Ensemble — 60/20/20
  • 11.7 DL Model Comparison — 60/20/20
    • Split 80/10/10
  • 11.6 Soft-Vote Ensemble — 80/10/10
  • 11.7 DL Model Comparison — 80/10/10
• 12. Final Comparison and Inference
  • 12.1 Unified Model Comparison
    • Split 70/15/15
    • Split 60/20/20
    • Split 80/10/10
  • 12.2 Inference and Recommendation
    • Best Overall Model: EEGNet

Process Work Flow

1. Data Description Overview

1.1 Dataset Citation & Source

Source: UCI Machine Learning Repository — EEG Eye State

All data is from one continuous EEG measurement with the Emotiv EEG Neuroheadset. The duration of the measurement was 117 seconds. The eye state was detected via a camera during the EEG measurement and added later manually to the file after analysing the video frames. '1' indicates the eye-closed and '0' the eye-open state. All values are in chronological order with the first measured value at the top of the data.

1.2 Dataset Loading

The dataset is loaded from dataset/eeg_data_og.csv.

1.3 Variable Classification & Electrode Positions

Numerical Variables (Continuous): 14 EEG electrode channels recording voltage in micro-volts (µV). The Emotiv EPOC headset uses a modified 10-20 international system for electrode placement. Each electrode captures electrical activity from a specific cortical region.

Categorical Variable (Target):

1.4 Basic Statistics

Descriptive statistics for all 14 EEG channels (µV).

Note on Spike Artifacts: Some channels exhibit extremely large max values — orders of magnitude above the 75th percentile. These are likely electrode spike artifacts caused by momentary loss of contact, muscle movement, or impedance changes in the Emotiv headset. These extreme values will be addressed by the outlier removal step.

1.5 Class Distribution

Distribution of the target variable eyeDetection (per UCI: 0 = open, 1 = closed).

2. Data Imputation

Missing values are detected and filled using column-wise median imputation to preserve the statistical properties of each EEG channel.

Result: No missing values detected across any of the 14 EEG channels. The dataset is complete.

3. Data Visualization (Raw Data)

Visualizations of the raw EEG data before any preprocessing.

3.1 Class Balance

3.2 Correlation Heatmap

The correlation heatmap reveals linear relationships between EEG channels. Highly correlated channels may carry redundant information.

Note on spike artifacts: The raw dataset contains extreme hardware spike artifacts (e.g., AF3 max ≈ 309,231 µV, FC5 max ≈ 642,564 µV) with values 75–150× the 99th percentile. When multiple distant channels spike simultaneously (e.g., AF3 and P8 co-spike on ~82 samples), those extreme outliers dominate the Pearson calculation and produce artificial r ≈ 1.00 between electrodes that should be uncorrelated. The heatmap below is therefore computed on data winsorized at the 1st–99th percentile to expose the true inter-channel structure. The full preprocessing pipeline (IQR spike removal → bandpass filter) in Section 4 corrects this permanently.

3.3 Box Plots

Box plots highlight potential outliers beyond the 1.5x IQR whiskers.

The raw box plots are compressed by extreme spike artifacts. Below is a zoomed view clipped at the 1st–99th percentile range to reveal the actual distribution of most samples.

3.4 Histograms

Amplitude distributions per channel split by eye state.

3.5 Violin Plots

Violin plots combine box-plot summaries with kernel density estimates.

4. Signal Preprocessing

EEG signals contain artifacts from eye blinks, muscle movement, and electrode drift that must be removed before analysis. This section applies a two-stage cleaning pipeline in the correct causal order:

1. IQR spike removal first — raw hardware spike artifacts (up to 715,897 µV) are removed before filtering. Applying filtfilt to spikes first smears them to neighbouring samples via the backward pass, inflating data loss from ~9% to ~19%.

2. Bandpass filter (0.5–45 Hz) second — applied to the already spike-free signal so no artifact energy is convolved into the physiological EEG bands.

4.1 IQR Spike Removal (applied first, before filtering)

A light IQR filter (3.0x IQR, max 3 passes) removes hardware spike artifacts from the raw signal. Applying this step before filtering is critical: filtfilt convolves forward then backward, so a single spike at sample $t$ would contaminate samples $t - N$ through $t + N$ after filtering. Removing spikes first keeps those neighbouring samples clean and reduces total data loss from ~19% to ~9%.

Threshold: $Q_3 + 3.0 \times IQR$ (wider than the traditional 1.5× to preserve genuine EEG excursions while rejecting hardware glitches).

Removing 1374 spike samples (9.2%) from the raw signal before filtering. The wrong order (filter first, then IQR) would remove ~2,882 samples (19.2%) — more than double the data loss, because filtfilt spreads each spike to ~8–10 adjacent samples via its backward pass.

4.2 Bandpass Filter (0.5–45 Hz) — applied after spike removal

A 4th-order Butterworth bandpass filter (0.5–45.0 Hz) removes DC drift and high-frequency noise while preserving the physiologically relevant EEG bands (Delta through Gamma). Applied via scipy.signal.filtfilt (zero-phase, forward-backward filtering) to avoid phase distortion.

$$H(s) = \frac{1}{\sqrt{1 + \left(\frac{s}{\omega_c}\right)^{2N}}}$$

Because spikes have already been removed, filtfilt operates on a clean signal and will not spread artifact energy to adjacent samples.

Preprocessing Summary (corrected order): IQR spike removal (3.0×, 9.2% removed) → Bandpass filter (0.5–45.0 Hz). Total retained: 13,606 / 14,980 samples (90.8%).

5. Data Visualization (After Preprocessing)

Comparison of distributions before and after preprocessing (IQR spike removal → bandpass filter).

5.1 Corrected Correlation Heatmap (after preprocessing)

With spike artifacts removed, the correlation heatmap now reflects the true physiological relationships between EEG channels. The artificial r ≈ 1.00 values seen in the raw data are eliminated. Some genuine frontal correlations (e.g., AF3–AF4 ≈ 0.94) remain and are expected given the Emotiv EPOC’s common reference architecture.

5.2 Box Plots Comparison

Side-by-side box plots confirm preprocessing effectiveness. Whiskers are set to 3.0x IQR to match the cleaning threshold.

5.3 Histograms After Cleaning

6. Log-Normalization Assessment (Rejected)

Logarithmic normalization compresses the dynamic range of EEG amplitudes, reducing the impact of extreme values and making distributions more symmetric. We test log10(x - min + 1) on each channel and evaluate whether it improves distribution quality. The transformed data is not used downstream — this section documents the assessment only.

6.1 Before vs After — All Channels

The following grid shows the distribution of every EEG channel before (blue) and after (red) log-normalization.

6.2 Skewness & Kurtosis Analysis

Skewness measures distribution asymmetry (0 = perfectly symmetric). Kurtosis (excess) measures tail heaviness (0 = normal). Log-normalization should reduce both towards zero.

Result: Log-normalization improved distribution quality (reduced |skewness| + |kurtosis|) for 0/14 channels (0%).

Decision: Log-normalization REJECTED. The transform worsened distribution quality for the majority of channels. After outlier removal, the EEG distributions are already approximately symmetric. All subsequent analyses use the cleaned (non-transformed) data.

6.3 Summary Statistics Before vs After

7. Feature Engineering

Feature engineering derives new variables from raw EEG channels to capture domain-specific patterns for exploratory analysis. Note: The ML/DL pipeline in Sections 10–11 uses the raw 14 channels directly to avoid preprocessing data leakage.

7.1 Hemispheric Asymmetry

The asymmetry index $(Left - Right)$ for paired electrodes captures lateralisation differences linked to cognitive and emotional states.

Asymmetry by Eye State — do hemispheric differences change with eye state?

2/7 asymmetry features show a statistically significant difference between eye states (Welch's t-test, p < 0.05). Hemispheric asymmetry contributes partial discriminative signal.

7.2 Frequency Band Power Features

Band power features capture the relative energy in each EEG frequency band. Research shows that band powers — particularly alpha — are among the strongest predictors for eye state classification (up to 96% accuracy in papers).

$$P_{\text{band}}(t) = \frac{1}{C} \sum_{c=1}^{C} \left[x_c^{\text{band}}(t)\right]^2$$

6 band power features added. Alpha asymmetry captures the Berger effect.

The bar chart above compares mean band power between eye-open and eye-closed states. A ratio > 1.0 indicates higher power during eye closure. The alpha band (8–12 Hz) is expected to show the strongest increase when eyes are closed (Berger effect), which is the primary physiological marker exploited by the classification models.

8. FFT, Spectrogram and PSD Analysis

Frequency-domain analysis reveals the power distribution across brain wave bands: Delta (0.5-4 Hz), Theta (4-8 Hz), Alpha (8-12 Hz), Beta (12-30 Hz), and Gamma (30-64 Hz). Alpha power increases when eyes are closed (the Berger effect).

8.1 FFT Frequency Spectrum

The FFT decomposes each EEG channel into constituent frequencies.

8.2 Power Spectral Density (PSD)

Welch's method estimates the PSD for each channel. Shaded regions indicate standard EEG frequency bands.

PSD Interpretation — Berger Effect: Alpha-band power (8–12 Hz) increases when the eyes are closed, particularly in occipital electrodes (O1, O2). If the red curve (closed) shows higher power in the alpha band compared to blue (open), this confirms the dataset captures genuine physiological differences between eye states.

8.3 Spectrogram Analysis

Spectrograms show the time-frequency power distribution. Horizontal dashed lines mark band boundaries.

9. Dimensionality Reduction

Projecting high-dimensional EEG data into lower-dimensional spaces reveals clustering structure. LDA maximises class separability; t-SNE and UMAP capture non-linear manifold structure.

To improve class separation, we apply a feature-augmentation pipeline before projection: (1) IQR-based outlier removal on the feature space, (2) rolling-window statistics (mean and std, window=10), and (3) FFT magnitude features. This enriched representation captures both temporal dynamics and spectral content.

After IQR filtering on feature space: 8208 samples retained (removed 5398).

Augmented feature matrix: 116 dimensions (29 original + 29 rolling-mean + 29 rolling-std + 29 FFT).

9.1 LDA

LDA maximises the ratio of between-class to within-class variance, yielding a single discriminant for binary classification. Applied to the augmented feature space.

9.2 t-SNE

t-Distributed Stochastic Neighbor Embedding is a non-linear technique that preserves local neighbourhood structure. A subsample of 5000 points is used for computational efficiency.

9.3 UMAP

UMAP preserves both local and global structure, often producing cleaner clusters than t-SNE.

9.4 Clustering Evaluation

Clustering metrics quantify separation quality in reduced spaces.

9.5 Inference: Dimensionality Reduction Comparison

Clustering metric summary:

• Best Silhouette Score: LDA (1D) (0.1556)
• Best Davies-Bouldin Index: LDA (1D) (1.5464)
• Best Calinski-Harabasz Score: LDA (1D) (2137.80)

10. Machine Learning Classification

The ML pipeline addresses two critical issues from standard approaches: (1) temporal concept drift — the last 20% of the recording is 90%+ eyes-open, creating severe distribution shift; and (2) class imbalance — all models use class_weight='balanced' and CV-optimised decision thresholds. Primary metric: Macro-F1 (equally weights both eye states under distribution shift). All splits are chronological — no shuffling, no data leakage.

10.1 Temporal Concept Drift Diagnosis

The subject's eye-state distribution changes dramatically over the recording. Every hold-out split places the test window in the heavily open-dominant tail, which is the root cause of the accuracy paradox and low binary-F1.

Note: The last 15% of the recording is only 8.1% closed-eye. Models trained on balanced data (≈50% closed) and tested on this window face a 44.9% distribution shift. Accuracy is misleading — Macro-F1 is the honest metric.

10.2 Split Configurations

10.3 Cross-Validation Results (5-Fold TimeSeriesSplit)

5-fold time-series CV on the 70/15 training portion. Each fold trains on all preceding data, respecting temporal order. Scaling inside Pipeline prevents data leakage.

10.4 Hold-Out Split Results

Split 70/15/15

Train=9524 (56.0% closed) | CV=2041 (35.9% closed) | Test=2041 (5.1% closed) | Δ shift=50.9%

LogisticRegression: Logistic Regression models the posterior probability:

$$P(y=1 \mid \mathbf{x}) = \sigma(\mathbf{w}^T \mathbf{x} + b) = \frac{1}{1 + e^{-(\mathbf{w}^T \mathbf{x} + b)}}$$

Uses class_weight='balanced' to penalise minority-class misclassification.

Acc=0.7423 | MacroF1=0.4540 | BinaryF1=0.0573 | AUC=0.3627 | Threshold=0.53 | TrainTime=0.0s

TP=16 FP=438 FN=88 TN=1499

SVM_RBF: SVM with RBF kernel maps features into higher-dimensional space:

$$K(\mathbf{x}_i, \mathbf{x}_j) = \exp(-\gamma \|\mathbf{x}_i - \mathbf{x}_j\|^2)$$

Maximises the soft margin with class_weight='balanced'.

Acc=0.5987 | MacroF1=0.3973 | BinaryF1=0.0488 | AUC=0.3736 | Threshold=0.64 | TrainTime=38.8s

TP=21 FP=736 FN=83 TN=1201

RandomForest: Random Forest builds 200 decision trees, each trained on a bootstrapped subset:

$$\hat{y} = \text{mode}\{h_b(\mathbf{x})\}_{b=1}^{200}$$

Uses class_weight='balanced' and splits by Gini impurity.

Acc=0.6164 | MacroF1=0.4009 | BinaryF1=0.0416 | AUC=0.3984 | Threshold=0.61 | TrainTime=2.2s

TP=17 FP=696 FN=87 TN=1241

GradientBoosting: Gradient Boosting corrects residual errors sequentially:

$$F_m(\mathbf{x}) = F_{m-1}(\mathbf{x}) + \eta \cdot h_m(\mathbf{x})$$

200 boosting rounds, learning rate $\eta = 0.1$, max depth 5.

Acc=0.6781 | MacroF1=0.4316 | BinaryF1=0.0574 | AUC=0.3968 | Threshold=0.65 | TrainTime=30.4s

TP=20 FP=573 FN=84 TN=1364

XGBoost: XGBoost uses scale_pos_weight = n_neg / n_pos to handle class imbalance directly in the gradient computation, producing the highest closed-eye recall among ML models.

Acc=0.5169 | MacroF1=0.3710 | BinaryF1=0.0681 | AUC=0.4011 | Threshold=0.67 | TrainTime=0.7s

TP=36 FP=918 FN=68 TN=1019

70/15/15 — ML Test Summary (ranked by Macro-F1):

Split 60/20/20

Train=8163 (62.3% closed) | CV=2721 (34.6% closed) | Test=2722 (5.3% closed) | Δ shift=57.0%

LogisticRegression: Logistic Regression models the posterior probability:

$$P(y=1 \mid \mathbf{x}) = \sigma(\mathbf{w}^T \mathbf{x} + b) = \frac{1}{1 + e^{-(\mathbf{w}^T \mathbf{x} + b)}}$$

Uses class_weight='balanced' to penalise minority-class misclassification.

Acc=0.7439 | MacroF1=0.4812 | BinaryF1=0.1121 | AUC=0.4831 | Threshold=0.54 | TrainTime=0.0s

TP=44 FP=598 FN=99 TN=1981

SVM_RBF: SVM with RBF kernel maps features into higher-dimensional space:

$$K(\mathbf{x}_i, \mathbf{x}_j) = \exp(-\gamma \|\mathbf{x}_i - \mathbf{x}_j\|^2)$$

Maximises the soft margin with class_weight='balanced'.

Acc=0.6102 | MacroF1=0.4170 | BinaryF1=0.0814 | AUC=0.4691 | Threshold=0.71 | TrainTime=29.2s

TP=47 FP=965 FN=96 TN=1614

RandomForest: Random Forest builds 200 decision trees, each trained on a bootstrapped subset:

$$\hat{y} = \text{mode}\{h_b(\mathbf{x})\}_{b=1}^{200}$$

Uses class_weight='balanced' and splits by Gini impurity.

Acc=0.6323 | MacroF1=0.4271 | BinaryF1=0.0842 | AUC=0.4515 | Threshold=0.68 | TrainTime=1.8s

TP=46 FP=904 FN=97 TN=1675

GradientBoosting: Gradient Boosting corrects residual errors sequentially:

$$F_m(\mathbf{x}) = F_{m-1}(\mathbf{x}) + \eta \cdot h_m(\mathbf{x})$$

200 boosting rounds, learning rate $\eta = 0.1$, max depth 5.

Acc=0.6242 | MacroF1=0.4200 | BinaryF1=0.0759 | AUC=0.4339 | Threshold=0.71 | TrainTime=26.1s

TP=42 FP=922 FN=101 TN=1657

XGBoost: XGBoost uses scale_pos_weight = n_neg / n_pos to handle class imbalance directly in the gradient computation, producing the highest closed-eye recall among ML models.

Acc=0.5918 | MacroF1=0.4149 | BinaryF1=0.0931 | AUC=0.5097 | Threshold=0.81 | TrainTime=0.7s

TP=57 FP=1025 FN=86 TN=1554

60/20/20 — ML Test Summary (ranked by Macro-F1):

Split 80/10/10

Train=10884 (55.3% closed) | CV=1361 (6.7% closed) | Test=1361 (3.8% closed) | Δ shift=51.5%

LogisticRegression: Logistic Regression models the posterior probability:

$$P(y=1 \mid \mathbf{x}) = \sigma(\mathbf{w}^T \mathbf{x} + b) = \frac{1}{1 + e^{-(\mathbf{w}^T \mathbf{x} + b)}}$$

Uses class_weight='balanced' to penalise minority-class misclassification.

Acc=0.8663 | MacroF1=0.4642 | BinaryF1=0.0000 | AUC=0.2041 | Threshold=0.56 | TrainTime=0.0s

TP=0 FP=130 FN=52 TN=1179

SVM_RBF: SVM with RBF kernel maps features into higher-dimensional space:

$$K(\mathbf{x}_i, \mathbf{x}_j) = \exp(-\gamma \|\mathbf{x}_i - \mathbf{x}_j\|^2)$$

Maximises the soft margin with class_weight='balanced'.

Acc=0.9214 | MacroF1=0.4795 | BinaryF1=0.0000 | AUC=0.3734 | Threshold=0.84 | TrainTime=49.0s

TP=0 FP=55 FN=52 TN=1254

RandomForest: Random Forest builds 200 decision trees, each trained on a bootstrapped subset:

$$\hat{y} = \text{mode}\{h_b(\mathbf{x})\}_{b=1}^{200}$$

Uses class_weight='balanced' and splits by Gini impurity.

Acc=0.9030 | MacroF1=0.5030 | BinaryF1=0.0571 | AUC=0.3742 | Threshold=0.73 | TrainTime=2.4s

TP=4 FP=84 FN=48 TN=1225

GradientBoosting: Gradient Boosting corrects residual errors sequentially:

$$F_m(\mathbf{x}) = F_{m-1}(\mathbf{x}) + \eta \cdot h_m(\mathbf{x})$$

200 boosting rounds, learning rate $\eta = 0.1$, max depth 5.

Acc=0.9155 | MacroF1=0.5251 | BinaryF1=0.0945 | AUC=0.4217 | Threshold=0.79 | TrainTime=35.8s

TP=6 FP=69 FN=46 TN=1240

XGBoost: XGBoost uses scale_pos_weight = n_neg / n_pos to handle class imbalance directly in the gradient computation, producing the highest closed-eye recall among ML models.

Acc=0.9133 | MacroF1=0.5015 | BinaryF1=0.0484 | AUC=0.3974 | Threshold=0.95 | TrainTime=0.7s

TP=3 FP=69 FN=49 TN=1240

80/10/10 — ML Test Summary (ranked by Macro-F1):

10.5 Walk-Forward CV (Expanding Window) — 5 Folds

Expanding-window walk-forward CV simulates real deployment: the model always trains on all available past data before predicting the next window. Future data never leaks into training.

Fold 1 — train=6803 | val=1133 | val_closed=100.00%

LogisticRegression: Acc=1.0000 MacroF1=1.0000 AUC=nan t=0.05

SVM_RBF: Acc=1.0000 MacroF1=1.0000 AUC=nan t=0.05

RandomForest: Acc=1.0000 MacroF1=1.0000 AUC=nan t=0.05

GradientBoosting: Acc=0.9974 MacroF1=0.4993 AUC=nan t=0.05

XGBoost: Acc=0.9709 MacroF1=0.4926 AUC=nan t=0.05

Fold 2 — train=7936 | val=1133 | val_closed=41.92%

LogisticRegression: Acc=0.5631 MacroF1=0.5187 AUC=0.5059 t=0.53

SVM_RBF: Acc=0.6117 MacroF1=0.6042 AUC=0.6512 t=0.65

RandomForest: Acc=0.5922 MacroF1=0.5621 AUC=0.5905 t=0.67

GradientBoosting: Acc=0.5728 MacroF1=0.5633 AUC=0.5849 t=0.66

XGBoost: Acc=0.5737 MacroF1=0.5603 AUC=0.5851 t=0.78

Fold 3 — train=9069 | val=1133 | val_closed=0.97%

LogisticRegression: Acc=0.9868 MacroF1=0.7939 AUC=0.9927 t=0.66

SVM_RBF: Acc=0.9656 MacroF1=0.5579 AUC=0.9308 t=0.87

RandomForest: Acc=0.9947 MacroF1=0.8623 AUC=0.9952 t=0.87

GradientBoosting: Acc=0.9982 MacroF1=0.9541 AUC=0.9987 t=0.93

XGBoost: Acc=0.9232 MacroF1=0.5733 AUC=0.9665 t=0.95

Fold 4 — train=10202 | val=1133 | val_closed=60.19%

LogisticRegression: Acc=0.6328 MacroF1=0.5265 AUC=0.4801 t=0.43

SVM_RBF: Acc=0.5560 MacroF1=0.5170 AUC=0.4896 t=0.47

RandomForest: Acc=0.5402 MacroF1=0.5129 AUC=0.4911 t=0.49

GradientBoosting: Acc=0.5649 MacroF1=0.5160 AUC=0.4962 t=0.44

XGBoost: Acc=0.5772 MacroF1=0.5303 AUC=0.5206 t=0.37

Fold 5 — train=11335 | val=1133 | val_closed=8.03%

LogisticRegression: Acc=0.8853 MacroF1=0.5821 AUC=0.6395 t=0.56

SVM_RBF: Acc=0.8729 MacroF1=0.5741 AUC=0.5345 t=0.76

RandomForest: Acc=0.8650 MacroF1=0.5407 AUC=0.5356 t=0.68

GradientBoosting: Acc=0.8976 MacroF1=0.5807 AUC=0.5334 t=0.75

XGBoost: Acc=0.8817 MacroF1=0.5494 AUC=0.4580 t=0.92

Walk-Forward CV — Mean ± Std (primary: Macro-F1):

10.6 Sliding-Window CV (Fixed-Size Window) — 5 Folds

Sliding-window CV tests how well models generalise across different temporal regimes (different epochs of the recording). High fold-variance directly quantifies the severity of concept drift.

Fold 1 — train=6803 | val=1133 | val_closed=100.00%

LogisticRegression: Acc=1.0000 MacroF1=1.0000 AUC=nan

SVM_RBF: Acc=1.0000 MacroF1=1.0000 AUC=nan

RandomForest: Acc=1.0000 MacroF1=1.0000 AUC=nan

GradientBoosting: Acc=0.9974 MacroF1=0.4993 AUC=nan

XGBoost: Acc=0.9709 MacroF1=0.4926 AUC=nan

Fold 2 — train=6803 | val=1133 | val_closed=41.92%

LogisticRegression: Acc=0.5490 MacroF1=0.5150 AUC=0.5073

SVM_RBF: Acc=0.6161 MacroF1=0.6042 AUC=0.6490

RandomForest: Acc=0.5816 MacroF1=0.5593 AUC=0.5911

GradientBoosting: Acc=0.5490 MacroF1=0.5480 AUC=0.5874

XGBoost: Acc=0.5790 MacroF1=0.5548 AUC=0.5836

Fold 3 — train=6803 | val=1133 | val_closed=0.97%

LogisticRegression: Acc=0.9550 MacroF1=0.5411 AUC=0.8947

SVM_RBF: Acc=0.9885 MacroF1=0.4971 AUC=0.4537

RandomForest: Acc=0.9894 MacroF1=0.4973 AUC=0.5692

GradientBoosting: Acc=0.9612 MacroF1=0.5118 AUC=0.6174

XGBoost: Acc=0.8279 MacroF1=0.4677 AUC=0.5269

Fold 4 — train=6803 | val=1133 | val_closed=60.19%

LogisticRegression: Acc=0.5428 MacroF1=0.4911 AUC=0.4971

SVM_RBF: Acc=0.5719 MacroF1=0.5343 AUC=0.5082

RandomForest: Acc=0.5569 MacroF1=0.5413 AUC=0.5281

GradientBoosting: Acc=0.5578 MacroF1=0.5245 AUC=0.5278

XGBoost: Acc=0.5287 MacroF1=0.5202 AUC=0.5244

Fold 5 — train=6803 | val=1133 | val_closed=8.03%

LogisticRegression: Acc=0.8923 MacroF1=0.6427 AUC=0.6111

SVM_RBF: Acc=0.8764 MacroF1=0.5596 AUC=0.5425

RandomForest: Acc=0.8782 MacroF1=0.5360 AUC=0.5039

GradientBoosting: Acc=0.8994 MacroF1=0.5718 AUC=0.4993

XGBoost: Acc=0.8711 MacroF1=0.5358 AUC=0.5199

Sliding-Window CV — Mean ± Std:

Feature Importance (RandomForest — 70/15/15 training partition):

11. Deep Learning Classification

All DL models use PyTorch with: (1) weighted CrossEntropyLoss (inverse class frequency) to handle imbalance, (2) AdamW + CosineAnnealingLR for stable training, (3) CV-optimised decision threshold to correct the accuracy paradox under concept drift, and (4) Macro-F1 as primary metric. Sequences are built per partition — no cross-boundary leakage.

11.0 Architecture Overview & Training Setup

Binary Cross-Entropy (weighted):

$$\mathcal{L} = -\frac{1}{N}\sum_{i=1}^{N} w_{y_i} \left[y_i \log(\hat{p}_i) + (1-y_i)\log(1-\hat{p}_i)\right]$$

where $w_c = \frac{N}{2 \cdot N_c}$ is the per-class weight. Sequence length: SEQ_LEN=64 samples (≈500ms at 128 Hz). Optimizer: AdamW, lr=1e-3, weight_decay=1e-4. Scheduler: CosineAnnealingLR over 25 epochs.

Split 70/15/15

Train=9524 (56.0% closed) | CV=2041 (35.9% closed) | Test=2041 (5.1% closed)

11.1 LSTM

Stacked bidirectional LSTM captures long-range temporal dependencies. Hidden state $h_t$ and cell state $c_t$ are updated via forget ($f_t$), input ($i_t$), and output ($o_t$) gates. Global average pooling over the sequence dimension produces the classification vector.

$$c_t = f_t \odot c_{t-1} + i_t \odot \tilde{c}_t, \quad h_t = o_t \odot \tanh(c_t)$$

Optimal threshold (CV-optimised): 0.92

Test Confusion Matrix:

TP=61 FP=816 FN=43 TN=1057

11.2 CNN_LSTM

Two 1D convolutional blocks extract local temporal features; a bidirectional LSTM then models the sequence dynamics of those features. The CNN acts as a learned front-end filter bank:

$$y_t^{(f)} = \text{ReLU}\left(\sum_{k,c} w_{k,c}^{(f)} \cdot x_{t+k,c} + b^{(f)}\right)$$

Optimal threshold (CV-optimised): 0.82

Test Confusion Matrix:

TP=48 FP=520 FN=56 TN=1353

11.3 EEGTransformer

CLS-token Transformer with sinusoidal positional encoding and pre-LN encoder layers. Multi-head self-attention captures global cross-electrode dependencies:

$$\text{Attn}(Q,K,V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$

The CLS token aggregates the full sequence into a single classification vector.

Optimal threshold (CV-optimised): 0.89

Test Confusion Matrix:

TP=0 FP=658 FN=104 TN=1215

11.4 EEGNet

EEGNet (Lawhern et al. 2018) uses depthwise-separable 2D convolutions that explicitly model temporal patterns (Block 1 temporal kernel ≈ 250ms) and cross-electrode spatial patterns (Block 1 depthwise spatial filter). Only ~400 parameters — highly resistant to overfitting on limited data.

Optimal threshold (CV-optimised): 0.65

Test Confusion Matrix:

TP=73 FP=590 FN=31 TN=1283

11.5 PatchTST_Lite

Patch-based Transformer (Nie et al. 2023) divides the 64-sample window into 15 overlapping patches (size=8, stride=4 ≈ 62ms each). Each patch is linearly embedded; a Transformer encoder with a CLS token captures both local (per-patch) and global (cross-patch) temporal context.

Optimal threshold (CV-optimised): 0.95

Test Confusion Matrix:

TP=17 FP=1153 FN=87 TN=720

11.6 Soft-Vote Ensemble — 70/15/15

Random-weight Dirichlet search (3000 trials) over the probability simplex to find the combination of DL models maximising CV Macro-F1. Weights are optimised on CV only — test set never touched during optimisation.

Optimal weights (CV Macro-F1 = 0.6035):

Ensemble Test (t=0.92): Acc=0.6874 | MacroF1=0.4852 | AUC=0.6511

TP=60 FP=574 FN=44 TN=1299

11.7 DL Model Comparison — 70/15/15

Split 60/20/20

Train=8163 (62.3% closed) | CV=2721 (34.6% closed) | Test=2722 (5.3% closed)

Optimal threshold (CV-optimised): 0.95

Test Confusion Matrix:

TP=85 FP=1144 FN=58 TN=1371

Optimal threshold (CV-optimised): 0.93

Test Confusion Matrix:

TP=87 FP=1257 FN=56 TN=1258

Optimal threshold (CV-optimised): 0.93

Test Confusion Matrix:

TP=0 FP=0 FN=143 TN=2515

Optimal threshold (CV-optimised): 0.84

Test Confusion Matrix:

TP=114 FP=824 FN=29 TN=1691

Optimal threshold (CV-optimised): 0.95

Test Confusion Matrix:

TP=43 FP=1600 FN=100 TN=915

11.6 Soft-Vote Ensemble — 60/20/20

Random-weight Dirichlet search (3000 trials) over the probability simplex to find the combination of DL models maximising CV Macro-F1. Weights are optimised on CV only — test set never touched during optimisation.

Optimal weights (CV Macro-F1 = 0.5446):

Ensemble Test (t=0.53): Acc=0.3657 | MacroF1=0.3099 | AUC=0.6090

TP=108 FP=1651 FN=35 TN=864

11.7 DL Model Comparison — 60/20/20

Split 80/10/10

Train=10884 (55.3% closed) | CV=1361 (6.7% closed) | Test=1361 (3.8% closed)

Optimal threshold (CV-optimised): 0.95

Test Confusion Matrix:

TP=48 FP=608 FN=4 TN=637

Optimal threshold (CV-optimised): 0.94

Test Confusion Matrix:

TP=39 FP=293 FN=13 TN=952

Optimal threshold (CV-optimised): 0.90

Test Confusion Matrix:

TP=0 FP=0 FN=52 TN=1245

Optimal threshold (CV-optimised): 0.84

Test Confusion Matrix:

TP=1 FP=59 FN=51 TN=1186

Optimal threshold (CV-optimised): 0.95

Test Confusion Matrix:

TP=10 FP=663 FN=42 TN=582

11.6 Soft-Vote Ensemble — 80/10/10

Random-weight Dirichlet search (3000 trials) over the probability simplex to find the combination of DL models maximising CV Macro-F1. Weights are optimised on CV only — test set never touched during optimisation.

Optimal weights (CV Macro-F1 = 0.4879):

Ensemble Test (t=0.95): Acc=0.8705 | MacroF1=0.4712 | AUC=0.7619

TP=1 FP=117 FN=51 TN=1128

11.7 DL Model Comparison — 80/10/10

12. Final Comparison and Inference

This section unifies all models across all evaluation protocols: classical ML (raw 14 channels, temporal splits, balanced weights, threshold-optimised) and deep learning (PyTorch, weighted loss, macro-F1 primary metric). Primary metric throughout: Macro-F1.

12.1 Unified Model Comparison

All test-partition results across all hold-out splits, sorted by Macro-F1.

Split 70/15/15

Split 60/20/20

Split 80/10/10

12.2 Inference and Recommendation

Best model per hold-out split (by Macro-F1):

Mean Macro-F1 across all three splits (stability ranking):

Best Overall Model: EEGNet

Based on mean Macro-F1 across all three temporal hold-out splits, EEGNet achieves the highest average score of 0.4964.

Key Observations:

• The last 15% of the recording is 8.1% closed-eye, creating a 44.9% distribution shift between training and test. This is the root cause of all metric paradoxes.
• Models with well-calibrated probabilities (LogReg, EEGNet) transfer thresholds across the distribution shift more reliably than uncalibrated models (CNN-LSTM).
• EEGNet achieves the best single-split Macro-F1 (0.6518 on 70/15/15) because its depthwise 2D convolutions match the neurophysiology of the alpha-band Berger effect, its threshold ≈ 0.58 is naturally calibrated, and ~400 parameters resist overfitting on limited data.
• GradientBoosting is the most robust ML model — lowest Walk-Forward CV variance and best ML performance on the hardest 60/20/20 split (51.8% distribution shift).
• PatchTST_Lite is the safety-critical choice: FN ≈ 0 across splits (near-perfect closed-eye recall) at the cost of high false positives — ideal for drowsiness detection in safety-critical BCI.

Recommended Model Per Use Case: