EmotionFlow

overview.md

EmotionFlow is a real-time affect-aware music system. A CNN ensemble (MiniXception + transfer-learning-based MobileNetV2/EfficientNet) classifies 7 facial emotions at 25–30 FPS; a state machine maps those emotions to Spotify playback; a rolling queue manager keeps 2–5 songs buffered per state. Validation accuracy improved from 74.7% to 85.4% through imbalance-aware sampling, label smoothing, Mixup augmentation, and confidence calibration. Inference latency was reduced 3.2× (3.08s → 0.96s) via multithreaded frame processing and optimised I/O pipelines.

architecture.drawio

1. Emotion Detection Pipeline — webcam feed → face detection → 48×48 grayscale preprocessing → MiniXception CNN inference → confidence filtering (threshold 0.5) → 15-frame stability buffer with weighted voting.
2. Music State Controller — maps the stable emotion to one of 7 playback states; enforces a 3-second hold before any state transition and a 5-second cooldown between switches to prevent thrashing.
3. Rolling Player Engine — monitors per-state queue depth; auto-refills below 2 songs, caps at 20; scores candidates 0.0–1.0 against target audio features (energy, valence, danceability, tempo, acousticness).
4. Spotify Integration — OAuth device detection, thread-safe concurrent queue updates, playlist creation on session close (format: "Emotion Mix - YYYY-MM-DD HH:MM").

stability.log

The core engineering challenge was making the system calm under noisy input. Three mechanisms work in sequence:

• Confidence thresholding — frames below 0.5 confidence are dropped before they reach state logic.
• 15-frame weighted-vote buffer — absorbs per-frame detection jitter so transient mis-detections never surface.
• State machine cooldown — a 3-second hold + 5-second cooldown prevents a momentary wrong prediction from triggering a full playlist switch.

model.json

per-class-results.md

lessons-learned.md

• Stability engineering (buffers, cooldowns, thresholds) matters more than raw model accuracy for real-time affect systems.
• Thread-safety in the queue manager prevented the hardest-to-reproduce bugs — race conditions during concurrent UI updates and background fetches.
• Framing the output as a live product (a playlist, not a label) forced every component to be robust rather than just correct in isolation.

future-work.md

• On-device inference to eliminate CPU latency variance.
• Personalized emotion → music mapping that adapts from skip signals.
• Export session analytics (emotion distribution, state transitions) as a visual summary.