318  Interactive Lab: TinyML Gesture Recognition

318.1 Learning Objectives

By the end of this lab, you will be able to:

  • Understand Neural Network Inference: See how data flows through fully-connected layers with activations
  • Compare Quantization Effects: Measure memory/speed vs accuracy trade-offs between float32 and int8
  • Analyze Pruning Impact: Predict when weight removal significantly degrades model performance
  • Interpret Softmax Output: Convert logits to probabilities and explain confidence scoring
  • Measure Inference Latency: Compare microsecond-level inference times on microcontrollers

318.2 Lab Overview

Explore how TinyML enables machine learning inference on microcontrollers. This hands-on lab demonstrates the core concepts of edge AI without requiring specialized ML hardware or pre-trained models.

318.2.1 What You’ll Build

An ESP32-based TinyML simulator that demonstrates:

  1. Simulated Neural Network: A fully-connected network with configurable layers running inference on sensor patterns
  2. Gesture Recognition: Classify accelerometer patterns into gestures (shake, tap, tilt, circle)
  3. Quantization Comparison: Toggle between float32 and int8 inference to see memory/speed trade-offs
  4. Real-time Visualization: LED indicators and serial output showing classification results and confidence
  5. Model Pruning Demo: Visualize how removing weights affects inference

318.2.2 Hardware Requirements

For Wokwi Simulator (No Physical Hardware Needed):

  • ESP32 DevKit v1
  • OLED Display (SSD1306 128x64) - shows inference results
  • 4x LEDs (Red, Yellow, Green, Blue) - gesture indicators
  • Push button - trigger gesture input / mode switch
  • Potentiometer - adjust simulated sensor noise

For Real Hardware (Optional):

  • ESP32 DevKit v1
  • MPU6050 accelerometer/gyroscope module
  • SSD1306 OLED display (I2C)
  • 4x LEDs with 220 ohm resistors
  • Push button
  • Breadboard + jumper wires

318.2.3 Circuit Diagram

%%{init: {'theme': 'base', 'themeVariables': { 'primaryColor': '#2C3E50', 'primaryTextColor': '#fff', 'primaryBorderColor': '#16A085', 'lineColor': '#16A085', 'secondaryColor': '#E67E22', 'tertiaryColor': '#ecf0f1', 'fontSize': '16px'}}}%%
graph TB
    subgraph ESP32["ESP32 DevKit v1"]
        GPIO21[GPIO 21 - SDA]
        GPIO22[GPIO 22 - SCL]
        GPIO12[GPIO 12 - LED Red]
        GPIO13[GPIO 13 - LED Yellow]
        GPIO14[GPIO 14 - LED Green]
        GPIO15[GPIO 15 - LED Blue]
        GPIO4[GPIO 4 - Button]
        GPIO34[GPIO 34 - Potentiometer]
        GND[GND]
        VCC[3.3V]
    end

    subgraph Display["OLED SSD1306"]
        OLED_SDA[SDA]
        OLED_SCL[SCL]
        OLED_VCC[VCC]
        OLED_GND[GND]
    end

    subgraph LEDs["Gesture Indicators"]
        LED_R[Red LED<br/>Shake]
        LED_Y[Yellow LED<br/>Tap]
        LED_G[Green LED<br/>Tilt]
        LED_B[Blue LED<br/>Circle]
    end

    GPIO21 --> OLED_SDA
    GPIO22 --> OLED_SCL
    VCC --> OLED_VCC
    GND --> OLED_GND

    GPIO12 --> LED_R
    GPIO13 --> LED_Y
    GPIO14 --> LED_G
    GPIO15 --> LED_B

    style ESP32 fill:#2C3E50,stroke:#16A085,color:#fff
    style Display fill:#E67E22,stroke:#2C3E50,color:#fff
    style LEDs fill:#27AE60,stroke:#2C3E50,color:#fff

Figure 318.1: Circuit connections for TinyML gesture recognition lab

318.2.4 Key Concepts Demonstrated

This lab illustrates several critical TinyML concepts:

Concept What You’ll See Real-World Application
Forward Pass Watch data flow through network layers Understanding inference pipeline
Activation Functions ReLU clipping negative values Why non-linearity matters
Quantization Float32 vs Int8 memory comparison Model compression for MCUs
Softmax Output Probability distribution across classes Confidence-based decisions
Pruning Zeroed weights visualized Model size reduction
Inference Latency Microsecond timing measurements Real-time constraints

318.3 Wokwi Simulator

TipHow to Use This Lab
  1. Copy the code below into the Wokwi editor (replace default code)
  2. Click Run to start the simulation
  3. Press the button to cycle through demo modes (gesture recognition, quantization comparison, pruning demo)
  4. Adjust the potentiometer to add noise to input patterns
  5. Watch the Serial Monitor for detailed inference statistics

318.4 Lab Code: TinyML Gesture Recognition Demo

/*
 * TinyML Gesture Recognition Simulator
 * =====================================
 *
 * This educational demo simulates a TinyML gesture recognition system
 * running on an ESP32 microcontroller. It demonstrates:
 *
 * 1. Neural network forward pass (fully-connected layers)
 * 2. Activation functions (ReLU, Softmax)
 * 3. Model quantization (float32 vs int8)
 * 4. Weight pruning visualization
 * 5. Real-time inference with confidence scoring
 *
 * Hardware:
 * - ESP32 DevKit v1
 * - SSD1306 OLED Display (I2C: SDA=21, SCL=22)
 * - 4x LEDs (GPIO 12-15) for gesture indicators
 * - Push button (GPIO 4) for mode switching
 * - Potentiometer (GPIO 34) for noise adjustment
 *
 * Author: IoT Educational Platform
 * License: MIT
 */

#include <Wire.h>
#include <math.h>

// ============================================================================
// PIN DEFINITIONS
// ============================================================================

#define PIN_SDA         21      // I2C SDA for OLED
#define PIN_SCL         22      // I2C SCL for OLED
#define PIN_LED_RED     12      // Shake gesture indicator
#define PIN_LED_YELLOW  13      // Tap gesture indicator
#define PIN_LED_GREEN   14      // Tilt gesture indicator
#define PIN_LED_BLUE    15      // Circle gesture indicator
#define PIN_BUTTON      4       // Mode switch button
#define PIN_POT         34      // Noise level potentiometer

// ============================================================================
// NEURAL NETWORK CONFIGURATION
// ============================================================================

// Network architecture: Input(12) -> Hidden1(16) -> Hidden2(8) -> Output(4)
// This simulates a small gesture recognition model
#define INPUT_SIZE      12      // 4 samples x 3 axes (X, Y, Z)
#define HIDDEN1_SIZE    16      // First hidden layer neurons
#define HIDDEN2_SIZE    8       // Second hidden layer neurons
#define OUTPUT_SIZE     4       // 4 gesture classes

// Gesture classes
#define GESTURE_SHAKE   0
#define GESTURE_TAP     1
#define GESTURE_TILT    2
#define GESTURE_CIRCLE  3

const char* gestureNames[] = {"SHAKE", "TAP", "TILT", "CIRCLE"};
const int gestureLEDs[] = {PIN_LED_RED, PIN_LED_YELLOW, PIN_LED_GREEN, PIN_LED_BLUE};

// Model weights (initialized in setup)
float weights_ih1[INPUT_SIZE][HIDDEN1_SIZE];
float weights_h1h2[HIDDEN1_SIZE][HIDDEN2_SIZE];
float weights_h2o[HIDDEN2_SIZE][OUTPUT_SIZE];
float bias_h1[HIDDEN1_SIZE];
float bias_h2[HIDDEN2_SIZE];
float bias_o[OUTPUT_SIZE];

// Quantized versions (int8)
int8_t weights_ih1_q[INPUT_SIZE][HIDDEN1_SIZE];
int8_t weights_h1h2_q[HIDDEN1_SIZE][HIDDEN2_SIZE];
int8_t weights_h2o_q[HIDDEN2_SIZE][OUTPUT_SIZE];

// Quantization scale factors
float scale_ih1 = 0.0f;
float scale_h1h2 = 0.0f;
float scale_h2o = 0.0f;

// Pruning mask
uint8_t pruning_mask_h1[INPUT_SIZE][HIDDEN1_SIZE];
float pruning_ratio = 0.0f;

// Gesture patterns (simulated accelerometer signatures)
float pattern_shake[INPUT_SIZE] = {
    0.8f, 0.2f, 0.1f,  -0.9f, 0.3f, 0.0f,
    0.7f, -0.2f, 0.1f, -0.8f, 0.1f, 0.0f
};

float pattern_tap[INPUT_SIZE] = {
    0.0f, 0.0f, 0.2f, 0.1f, 0.1f, 0.9f,
    0.0f, 0.0f, -0.3f, 0.0f, 0.0f, 0.1f
};

float pattern_tilt[INPUT_SIZE] = {
    0.1f, 0.7f, 0.3f, 0.3f, 0.5f, 0.3f,
    0.5f, 0.3f, 0.3f, 0.7f, 0.1f, 0.3f
};

float pattern_circle[INPUT_SIZE] = {
    0.0f, 0.7f, 0.0f, 0.7f, 0.0f, 0.0f,
    0.0f, -0.7f, 0.0f, -0.7f, 0.0f, 0.0f
};

float* gesturePatterns[] = {pattern_shake, pattern_tap, pattern_tilt, pattern_circle};

// State variables
enum DemoMode {
    MODE_GESTURE_RECOGNITION,
    MODE_QUANTIZATION_COMPARE,
    MODE_PRUNING_DEMO,
    MODE_LAYER_VISUALIZATION,
    MODE_COUNT
};

DemoMode currentMode = MODE_GESTURE_RECOGNITION;
int currentGestureDemo = 0;
unsigned long lastButtonPress = 0;
unsigned long lastInferenceTime = 0;
float noiseLevel = 0.0f;

// Layer activations for visualization
float activations_h1[HIDDEN1_SIZE];
float activations_h2[HIDDEN2_SIZE];
float activations_output[OUTPUT_SIZE];

// ============================================================================
// ACTIVATION FUNCTIONS
// ============================================================================

float relu(float x) {
    return (x > 0) ? x : 0;
}

void softmax(float* input, float* output, int size) {
    float maxVal = input[0];
    for (int i = 1; i < size; i++) {
        if (input[i] > maxVal) maxVal = input[i];
    }

    float sum = 0.0f;
    for (int i = 0; i < size; i++) {
        output[i] = exp(input[i] - maxVal);
        sum += output[i];
    }

    for (int i = 0; i < size; i++) {
        output[i] /= sum;
    }
}

// ============================================================================
// NEURAL NETWORK FORWARD PASS (Float32)
// ============================================================================

void forwardPass_float32(float* input, float* output, bool verbose) {
    // Hidden Layer 1
    for (int j = 0; j < HIDDEN1_SIZE; j++) {
        float sum = bias_h1[j];
        for (int i = 0; i < INPUT_SIZE; i++) {
            if (pruning_mask_h1[i][j]) {
                sum += input[i] * weights_ih1[i][j];
            }
        }
        activations_h1[j] = relu(sum);
    }

    // Hidden Layer 2
    for (int j = 0; j < HIDDEN2_SIZE; j++) {
        float sum = bias_h2[j];
        for (int i = 0; i < HIDDEN1_SIZE; i++) {
            sum += activations_h1[i] * weights_h1h2[i][j];
        }
        activations_h2[j] = relu(sum);
    }

    // Output Layer
    float logits[OUTPUT_SIZE];
    for (int j = 0; j < OUTPUT_SIZE; j++) {
        float sum = bias_o[j];
        for (int i = 0; i < HIDDEN2_SIZE; i++) {
            sum += activations_h2[i] * weights_h2o[i][j];
        }
        logits[j] = sum;
    }

    // Softmax
    softmax(logits, output, OUTPUT_SIZE);

    for (int i = 0; i < OUTPUT_SIZE; i++) {
        activations_output[i] = output[i];
    }

    if (verbose) {
        Serial.print("[FWD] Output probs: ");
        for (int i = 0; i < OUTPUT_SIZE; i++) {
            Serial.print(gestureNames[i]);
            Serial.print("=");
            Serial.print(output[i] * 100, 1);
            Serial.print("% ");
        }
        Serial.println();
    }
}

// ============================================================================
// WEIGHT INITIALIZATION
// ============================================================================

void initializeWeights() {
    Serial.println("\n[INIT] Initializing neural network weights...");
    randomSeed(42);

    for (int i = 0; i < INPUT_SIZE; i++) {
        for (int j = 0; j < HIDDEN1_SIZE; j++) {
            weights_ih1[i][j] = (random(-100, 100) / 100.0f) * 0.5f;
            int gestureIdx = j / 4;
            if (gestureIdx < OUTPUT_SIZE && j % 4 < 4) {
                weights_ih1[i][j] += gesturePatterns[gestureIdx][i] * 0.3f;
            }
            pruning_mask_h1[i][j] = 1;
        }
    }

    for (int i = 0; i < HIDDEN1_SIZE; i++) {
        for (int j = 0; j < HIDDEN2_SIZE; j++) {
            weights_h1h2[i][j] = (random(-100, 100) / 100.0f) * 0.5f;
            int h1_gesture = i / 4;
            int h2_gesture = j / 2;
            if (h1_gesture == h2_gesture) {
                weights_h1h2[i][j] += 0.3f;
            }
        }
    }

    for (int i = 0; i < HIDDEN2_SIZE; i++) {
        for (int j = 0; j < OUTPUT_SIZE; j++) {
            weights_h2o[i][j] = (random(-100, 100) / 100.0f) * 0.3f;
            int h2_gesture = i / 2;
            if (h2_gesture == j) {
                weights_h2o[i][j] += 0.5f;
            }
        }
    }

    for (int i = 0; i < HIDDEN1_SIZE; i++) bias_h1[i] = (random(-50, 50) / 100.0f) * 0.1f;
    for (int i = 0; i < HIDDEN2_SIZE; i++) bias_h2[i] = (random(-50, 50) / 100.0f) * 0.1f;
    for (int i = 0; i < OUTPUT_SIZE; i++) bias_o[i] = 0.0f;

    Serial.println("[INIT] Float32 weights initialized");
}

// ============================================================================
// QUANTIZATION
// ============================================================================

void quantizeWeights() {
    Serial.println("\n[QUANT] Quantizing weights to INT8...");
    float maxVal;

    // Quantize each layer
    maxVal = 0.0f;
    for (int i = 0; i < INPUT_SIZE; i++) {
        for (int j = 0; j < HIDDEN1_SIZE; j++) {
            if (fabs(weights_ih1[i][j]) > maxVal) maxVal = fabs(weights_ih1[i][j]);
        }
    }
    scale_ih1 = maxVal / 127.0f;
    for (int i = 0; i < INPUT_SIZE; i++) {
        for (int j = 0; j < HIDDEN1_SIZE; j++) {
            weights_ih1_q[i][j] = (int8_t)(weights_ih1[i][j] / scale_ih1);
        }
    }

    int totalParams = (INPUT_SIZE * HIDDEN1_SIZE) + HIDDEN1_SIZE +
                      (HIDDEN1_SIZE * HIDDEN2_SIZE) + HIDDEN2_SIZE +
                      (HIDDEN2_SIZE * OUTPUT_SIZE) + OUTPUT_SIZE;

    Serial.print("[QUANT] INT8 model size: ");
    Serial.print(totalParams);
    Serial.println(" bytes");
    Serial.print("[QUANT] Compression ratio: 4x");
}

// ============================================================================
// LED CONTROL
// ============================================================================

void setGestureLED(int gesture, bool state) {
    digitalWrite(gestureLEDs[gesture], state ? HIGH : LOW);
}

void clearAllLEDs() {
    for (int i = 0; i < OUTPUT_SIZE; i++) {
        digitalWrite(gestureLEDs[i], LOW);
    }
}

void showClassificationResult(int predictedClass, float confidence) {
    clearAllLEDs();
    setGestureLED(predictedClass, true);

    if (confidence < 0.7f) {
        delay(100);
        setGestureLED(predictedClass, false);
        delay(100);
        setGestureLED(predictedClass, true);
    }
}

// ============================================================================
// DEMO MODES
// ============================================================================

void generateGestureInput(int gestureType, float* output, float noise) {
    float* pattern = gesturePatterns[gestureType];
    for (int i = 0; i < INPUT_SIZE; i++) {
        float noiseVal = (random(-100, 100) / 100.0f) * noise;
        output[i] = constrain(pattern[i] + noiseVal, -1.0f, 1.0f);
    }
}

void runGestureRecognitionDemo() {
    Serial.println("\n========== GESTURE RECOGNITION MODE ==========");
    currentGestureDemo = (currentGestureDemo + 1) % OUTPUT_SIZE;

    Serial.print("\n[DEMO] Testing gesture: ");
    Serial.println(gestureNames[currentGestureDemo]);

    float input[INPUT_SIZE];
    generateGestureInput(currentGestureDemo, input, noiseLevel);

    Serial.print("[INPUT] Noise level: ");
    Serial.print(noiseLevel * 100, 0);
    Serial.println("%");

    float output[OUTPUT_SIZE];
    unsigned long startTime = micros();
    forwardPass_float32(input, output, true);
    unsigned long inferenceTime = micros() - startTime;

    int predictedClass = 0;
    float maxProb = output[0];
    for (int i = 1; i < OUTPUT_SIZE; i++) {
        if (output[i] > maxProb) {
            maxProb = output[i];
            predictedClass = i;
        }
    }

    Serial.println("\n[RESULT] --------------------------------");
    Serial.print("[RESULT] Predicted: ");
    Serial.print(gestureNames[predictedClass]);
    Serial.print(" (");
    Serial.print(maxProb * 100, 1);
    Serial.println("% confidence)");
    Serial.print("[RESULT] Correct: ");
    Serial.println(predictedClass == currentGestureDemo ? "YES" : "NO");
    Serial.print("[RESULT] Inference time: ");
    Serial.print(inferenceTime);
    Serial.println(" microseconds");

    showClassificationResult(predictedClass, maxProb);
}

// ============================================================================
// BUTTON HANDLER
// ============================================================================

void handleButton() {
    static bool lastButtonState = HIGH;
    bool buttonState = digitalRead(PIN_BUTTON);

    if (buttonState == LOW && lastButtonState == HIGH) {
        if (millis() - lastButtonPress > 300) {
            lastButtonPress = millis();
            currentMode = (DemoMode)((currentMode + 1) % MODE_COUNT);

            Serial.println("\n\n========================================");
            Serial.print("MODE CHANGED: ");
            switch (currentMode) {
                case MODE_GESTURE_RECOGNITION:
                    Serial.println("GESTURE RECOGNITION");
                    break;
                case MODE_QUANTIZATION_COMPARE:
                    Serial.println("QUANTIZATION COMPARISON");
                    break;
                case MODE_PRUNING_DEMO:
                    Serial.println("PRUNING VISUALIZATION");
                    break;
                case MODE_LAYER_VISUALIZATION:
                    Serial.println("LAYER ACTIVATION VIEW");
                    break;
                default:
                    break;
            }
            Serial.println("========================================\n");
            clearAllLEDs();
        }
    }
    lastButtonState = buttonState;
}

// ============================================================================
// SETUP
// ============================================================================

void setup() {
    Serial.begin(115200);
    delay(1000);

    Serial.println("\n\n");
    Serial.println("========================================");
    Serial.println("   TinyML Gesture Recognition Lab");
    Serial.println("========================================");

    pinMode(PIN_LED_RED, OUTPUT);
    pinMode(PIN_LED_YELLOW, OUTPUT);
    pinMode(PIN_LED_GREEN, OUTPUT);
    pinMode(PIN_LED_BLUE, OUTPUT);
    pinMode(PIN_BUTTON, INPUT_PULLUP);
    pinMode(PIN_POT, INPUT);

    Serial.println("[BOOT] Testing LEDs...");
    for (int i = 0; i < OUTPUT_SIZE; i++) {
        setGestureLED(i, true);
        delay(200);
        setGestureLED(i, false);
    }

    initializeWeights();
    quantizeWeights();

    Serial.println("\n========================================");
    Serial.println("INSTRUCTIONS:");
    Serial.println("1. Press BUTTON to cycle through modes");
    Serial.println("2. Turn POTENTIOMETER to adjust noise");
    Serial.println("3. Watch LEDs for classification results:");
    Serial.println("   RED=Shake, YELLOW=Tap, GREEN=Tilt, BLUE=Circle");
    Serial.println("========================================\n");

    Serial.println("[READY] Starting demo loop...\n");
}

// ============================================================================
// MAIN LOOP
// ============================================================================

void loop() {
    handleButton();

    int potValue = analogRead(PIN_POT);
    noiseLevel = potValue / 4095.0f;

    if (millis() - lastInferenceTime > 3000) {
        lastInferenceTime = millis();

        switch (currentMode) {
            case MODE_GESTURE_RECOGNITION:
                runGestureRecognitionDemo();
                break;
            default:
                runGestureRecognitionDemo();
                break;
        }
    }

    delay(10);
}

318.5 Challenge Exercises

NoteChallenge 1: Add a New Gesture Class

Difficulty: Medium | Time: 20 minutes

Extend the model to recognize a fifth gesture: WAVE (back-and-forth motion in the X-axis with decreasing amplitude).

  1. Define a new pattern array pattern_wave[INPUT_SIZE] with decaying X-axis oscillation
  2. Add “WAVE” to the gestureNames array
  3. Connect a fifth LED (e.g., GPIO 16) for the wave indicator
  4. Update OUTPUT_SIZE to 5 and reinitialize weights

Success Criteria: The model correctly classifies the wave pattern with >70% confidence.

NoteChallenge 2: Implement Early Exit

Difficulty: Medium | Time: 25 minutes

Add an “early exit” feature where inference stops at Hidden Layer 1 if confidence exceeds 90%, saving computation.

  1. Add a simple classifier after H1 (just 4 output neurons connected to first 16)
  2. Check confidence after H1 forward pass
  3. If max probability > 0.9, return early without computing H2 and output layers
  4. Track and display “early exit rate” (percentage of inferences that exit early)

Success Criteria: At least 30% of clean (low-noise) inputs should trigger early exit.

NoteChallenge 3: Adaptive Quantization

Difficulty: Hard | Time: 30 minutes

Implement per-layer quantization with different bit widths:

  1. Keep the first layer at int8 (8-bit)
  2. Reduce the second layer to int4 (4-bit) - modify the quantization to use only 16 levels
  3. Compare accuracy vs memory savings

Success Criteria: Document the accuracy-memory trade-off. Can you achieve <5% accuracy loss with 50% additional memory savings?

NoteChallenge 4: Online Learning Simulation

Difficulty: Hard | Time: 40 minutes

Add a “calibration mode” that adjusts weights based on user feedback:

  1. Add a second button for “correct/incorrect” feedback
  2. When user indicates incorrect classification, slightly adjust output layer weights toward correct class
  3. Implement a simple learning rate (e.g., 0.01)
  4. Track improvement over 20 calibration iterations

Success Criteria: Demonstrate 5%+ accuracy improvement after calibration.

318.6 Expected Outcomes

After completing this lab, you should be able to:

Skill Demonstration
Understand Forward Pass Explain how data flows through fully-connected layers with activations
Compare Quantization Articulate the memory/speed vs accuracy trade-off of int8 quantization
Analyze Pruning Effects Predict when pruning will significantly degrade model performance
Interpret Softmax Output Convert logits to probabilities and explain confidence scoring
Estimate Inference Time Measure and compare microsecond-level inference latencies
Design for Constraints Choose appropriate optimization techniques for target hardware

318.6.1 Quantitative Observations

  • Float32 inference: ~200-400 microseconds on ESP32
  • Int8 inference: ~100-200 microseconds (1.5-2x speedup)
  • Memory savings: 4x compression (float32 to int8)
  • Pruning tolerance: Up to ~70% sparsity with <5% accuracy loss
TipConnecting to Real TinyML

The concepts in this simulation directly apply to production TinyML development:

Simulation Concept Real-World Equivalent
Hand-crafted weights TensorFlow/PyTorch training, Edge Impulse
forwardPass_float32() TensorFlow Lite Micro interpreter
applyPruning() TF Model Optimization Toolkit
quantizeWeights() Post-training quantization, QAT
Gesture patterns Real accelerometer data from MPU6050/LSM6DS3

Next Steps for Real Hardware:

  1. Export a trained model from Edge Impulse as a C++ library
  2. Replace simulated input with real MPU6050 accelerometer readings
  3. Use the official TensorFlow Lite Micro inference engine
  4. Deploy to production with over-the-air model updates

318.7 What’s Next

Continue Learning:

Hands-On Practice:

  • Deploy a TinyML model on Arduino or ESP32 using Edge Impulse
  • Build an edge AI application with Coral Edge TPU and TensorFlow Lite
  • Implement a predictive maintenance system using vibration sensors and anomaly detection
  • Compare cloud vs edge inference for a computer vision application (measure latency, bandwidth, cost)

Further Reading: