37 Mobile Sensing: Features and Deployment

37.6 Windowed HAR Feature Record

A practical human activity recognition pipeline starts with a small label set, a repeatable collection protocol, and a feature record that can be reviewed. The phone samples acceleration and angular velocity, windows the stream, extracts features, runs a classifier, and applies a conservative output rule. If confidence is low or the setup differs from training, the output should be unknown rather than a confident but unsupported label.

Worked example: 50 Hz inertial stream
sample rate: 50 samples/second
window length: 2 seconds
window samples: 50 x 2 = 100 samples per sensor axis
overlap: 50 percent
step size: 1 second
new feature vector: every 1 second
channels: accel x/y/z + gyro x/y/z = 6 streams

example per-window features:
- mean and standard deviation per channel
- min and max per channel
- signal magnitude area for acceleration
- dominant frequency or spectral energy for periodic motion
- axis correlation for gait or phone orientation effects

Sequence and classifier choices should match the evidence boundary. Dynamic time warping (DTW) can compare motion-window shapes when timing varies. It aligns two time series with dynamic programming, nonlinearly stretching or compressing one motion against a reference motion, then accumulates the point-to-point Euclidean distances along the selected path. The useful DTW path should stay monotonic and continuous; its ordinary one-dimensional running time is O(n^2), while naive higher-dimensional versions become impractical because the search grows with the number of dimensions. In multi-sensor motion work, teams often calculate DTW per corresponding dimension, average the distances, and weight dimensions when some streams, such as gyroscope axes, carry more classification evidence than magnetometer axes.

PCA or another dimensionality-reduction step can expose dominant feature directions before a compact model is trained. PCA projects a high-dimensional dataset onto new axes so that the first component explains the largest possible variance in the original data, and each subsequent component explains the next largest remaining variance while staying orthogonal to the earlier components. That ordering is why PCA can highlight pertinent motion structure, reduce noise, and lower the cost of downstream comparisons such as DTW. The projected components still need scaling, provenance, and deployment-time reproducibility. Common first-pass classifiers for this kind of sensor stream include Naive Bayes, k-nearest neighbours, decision trees, logistic regression, multilayer perceptrons, and SVMs; hidden Markov models or other sequence models become useful when the transition pattern between activities is part of the claim. The review record should name the feature set, scaling step, model family, sequence assumption, and validation split together so the result is not just a list of algorithms.

Clinical gait monitoring is a stricter version of the same pipeline. A body-worn IMU system may label heel strike, foot flat, heel off, toe off, and midswing for the left and right legs across a normalized gait cycle. That output is not just a walking label; it is a timed phase sequence that must be compared with ground truth and reported with per-leg sensitivity, specificity, tolerance, sensor placement, and any missed or ambiguous transitions.

A stricter gait-study validation can pair wearable Orient IMUs with optical Vicon markers. The wearable units capture body-adjacent inertial streams, while the marker-based system supplies ground truth for normal walking. The record should name unit placement, reflector pairing, trigger source, sampling rates, interpolation rule, calibration period, walkway protocol, subject count, and trial count. If one stream is sampled at 42.67 Hz and the reference at 100 Hz, the comparison is not valid until the timestamp alignment and resampling rule are preserved with the labels.

One common sequence-classification pattern combines a feedforward neural network with a hidden Markov model. The FNN sees a fixed-size sliding window of sensor frames and assigns a gait-phase label to each window, which is useful for high-dimensional inputs but weak at enforcing temporal order. The HMM then treats those window labels as emissions from hidden gait-phase states, uses transition probabilities to encode legal phase order, and applies a decoder such as Viterbi to infer the most likely phase sequence. That pairing is reviewable only when the window size, central-frame label rule, emission confusion rates, transition model, and final phase-tolerance metric are stored with the validation record.

For a concrete gait-phase classifier, the states may be heel strike, foot flat, heel off, toe off, and midswing, with each phase named by its initiating event. A compact FNN record might use three hidden layers of ten units, five output units, 15-frame windows, and a one-frame step. With seven Orient units, six accelerometer/gyroscope channels per unit, and 15 frames per window, that becomes 7 x 6 x 15 = 630 input values before classification. The same record should preserve the training objective, such as cross-entropy, the optimizer, such as scaled conjugate-gradient backpropagation, the leave-one-trial-out validation split, the HMM transition matrix, and the emission matrix that captures how often each hidden gait phase is observed as each FNN label. Event scoring also needs its own tolerance statement: exact-frame sensitivity and specificity can differ from scores that allow a +/- N-frame window around the ground-truth event. Larger windows may improve phase accuracy, but smaller windows reduce compute and can make local wearable inference practical; if the sensor evidence drops to a two-channel foot-accelerometer setup, a longer window may be needed to approach the performance of a richer 42-channel gait frame.

The validation report should expose the result shape, not only one average accuracy. In the Orient gait study, sensitivity rises from the high 80s toward the low 90s as the neural-network sliding window grows, while specificity stays close to 97 to 98 percent. A five-subject table is more useful than a single headline: averaged over four trials each, subject sensitivity ranges from 87.72 to 96.07 percent and specificity from 96.70 to 98.47 percent. The gait-summary record should also keep cycle time, steps per minute, and left/right stance-to-swing ratios. In one example, cycle time spans 0.90 to 1.13 seconds, cadence spans 52.89 to 66.71 steps per minute, and stance-to-swing ratios cluster around 0.62 to 0.65, which gives a quick sanity check that the detected phases resemble expected walking mechanics.

Those metrics become more informative when plotted against symmetry expectations. Left-foot versus right-foot phase-duration plots let reviewers see whether points sit near the symmetry line or whether one subject is consistently biased. A light-blue subject with right-biased phase durations and a black subject with a left-biased swing phase but right-biased stance phase are not just visualization trivia; they show why automated gait analytics should assist practitioners by surfacing candidate asymmetries, not by hiding the raw comparison behind a pass/fail label. Sensor-placement ablations make the same point. Reducing the seven Orient units lowers FNN+HMM complexity but can change detection performance by phase and body location. Accel-plus-gyro evidence from any lower-body location can remain strong, all sensors together perform best, foot sensors are especially strong for heel strike, and thigh-only evidence can be weaker; in a gyro-only placement study, thigh data was comparatively poor for foot-centric events except midswing. When accelerometer-only or gyroscope-only streams are used in isolation, performance becomes more placement sensitive: specificity often stays high, but sensitivity varies because the evidence has been reduced to a single pair of triaxial sensors and split again by gait phase.

Instrumented skill-training simulators use the same evidence pattern outside walking. A take-home laparoscopic trainer can attach Orient-style inertial tags to tool handles and compare a trainee's instrument trace with cohort or expert records. The record should distinguish device setup evidence, such as tag placement, calibration pose, sampling rate, wireless link, participant id, and task id, from derived skill features. Roll and pitch from the accelerometer are gravity-referenced orientation cues, not a complete competence score: for a normalized acceleration vector G, common estimates use roll = atan2(G_y, G_z) and pitch = atan2(-G_x, sqrt(G_y^2 + G_z^2)), with signs tied to the mounted instrument frame. A simple economy-of-movement feature can compute the mean variance of x, y, and z acceleration over the previous one-second window; high variance suggests jerky novice motion while lower variance can indicate smoother expert control, but the claim is only reviewable when window length, axis units, filtering, and task segment are recorded with the raw trace.

The broader laparoscopic metric set should also be recorded, because it separates raw movement summaries from derived skill claims. Task duration is the elapsed time T; angular distance is the sum of frame-to-frame angle changes theta; average speed is theta / T; and average angular acceleration is omega / T. Motion smoothness is a normalized integral of instantaneous jerk, computed from the second derivative of frame-to-frame angle differences and averaged across left and right hands, with values closer to zero indicating smoother motion. Handedness subtracts left-hand frame distance from right-hand frame distance and averages the difference after multiplying by frame rate. Speed variance measures variance in consecutive-frame angular distances, and ambidexterity measures the correlation between left and right frame speeds, so positive correlation means concurrent instrument motion while negative correlation means one instrument dominates at a time.

Validation of those derived metrics should preserve the statistical question. In the Orient simulator study, one-way ANOVA tested whether novice, intermediate, and expert groups differed; Pearson correlation compared equivalent Orient and camera-based metrics; and a least-squares linear regression could combine the useful metrics into an overall scoring function only after an equal train/test split. The scoring equation should be documented as an equation, not summarized as "the model": a constant term plus weighted average acceleration, total duration, motion smoothness, angular distance, and ambidexterity. In the example record, acceleration raises the score, while duration, angular distance, and ambidexterity carry negative weights and motion smoothness is scaled by a very small coefficient because of its units. A leave-one-out validation view of the combined score still needs group evidence; the reported box plot separated novices from intermediates at p = 0.012, intermediates from experts at p = 0.001, and novices from experts at p < 0.001. The result pattern is not "all motion is faster for experts." Experts had shorter task duration, lower total angular distance, lower motion-smoothness values, and higher average acceleration, while average speed was not a strong separator and ambidexterity separated experts from less experienced groups more weakly. That shape is a useful teaching point: skill evidence comes from a bundle of bounded movement metrics, p-values, and validation splits, not one attractive box plot.

A cello tutor is another example of the same instrumented-skill-training pattern. A take-home setup can attach one Orient IMU to the player's wrist and another to the frog of the bow, then recognize whether the bowing technique matches the exercise and return feedback. The label set should be written in musical terms before it becomes a classifier target: legato is smoothly connected, staccato is a short detached stopped stroke, martele is a percussive accented stroke, spiccato is a detached or bouncing stroke, tremolo alternates rapid up and down strokes, col legno strikes with the bow stick, and ricochet groups multiple notes in one bow stroke. The protocol should also say whether the technique is performed on a single string or while crossing strings, because that changes the motion evidence without changing the technique name.

The cello record also shows why segmentation choices belong next to model accuracy. Orient data and video labels were aligned by timestamp to about 50 milliseconds, then split into candidate windows from 1 to 2 seconds. A 1-second segment can capture short bowing events while still encoding macroscopic motion; for example, a 25-second legato single-string capture yields 25 training examples. The per-window features were deliberately simple: average acceleration, acceleration variance, average smoothness of acceleration as the first derivative of the accelerometer signal, average angular velocity, angular-velocity variance, and average angular acceleration. A variance-in-acceleration plot for legato and martele illustrates the review question: the technique boundary can be much stronger than the single-string versus crossing-string condition, so the validation report should preserve both labels.

Model comparisons for this cello tutor are useful only inside their data-capture boundary. In the reported experiment, evaluation used 10-fold cross-validation on an artificially shuffled dataset made by randomly concatenating 1-second segments of individual bowing techniques. Within that setting, NaiveBayes and k-NN each reached 93.75 percent accuracy, J48 decision trees reached 90.83 percent, logistic regression reached 97.5 percent, and both a multilayer perceptron and an SVM with a normalised PolyKernel reached 98.33 percent. Those numbers are a compact model-selection example, not proof that a continuous live tutor is solved unless the same timing, segmentation, labels, users, and feedback latency are preserved.

Object-attached sensing extends the same pattern beyond body-worn coaching. A fast-moving consumer goods study can attach a coin-scale mini Speck or NanoSpeck accelerometer to a product, let a phone app collect the local stream, and infer daily-ritual events such as pouring liquid detergent, squeezing moisturiser or soap, and shaking ketchup. That setup replaces or complements interviews, diaries, and questionnaires: the manual methods are low tech, but they are intrusive, time-consuming, error-prone, and often produce noisy self-reported usage data. The sensor record should therefore name the product attachment point, the phone relay path, the programmed action labels, and the timestamp/server upload rule, because the business question is not "did acceleration change?" but "which product-use event happened, when, and with what confidence?"

A reviewable FMCG feature record can be deliberately small. In the slide example, MA is a 10-minute moving average of z-axis acceleration, JC is jerk count, and length is the number of z-axis accelerometer values in the event window. Scatter views of MA against JC x length separate pour, squeeze, and shake into visible clusters, while MA against JC exposes more overlap and outliers. A phone result screen that counts 13 daily-ritual events, for example four other events, four shakes, two squeezes, and three pours, is only useful if those feature definitions, window lengths, class labels, unknown/other handling, and app-device connection identifiers are stored with the model evidence.

Multi-IMU motion records become high-dimensional quickly. If a study uses N devices, S sensor types per device, and A axes per sensor, the raw dimensionality is D = N x S x A before any window features are added. A gait setup with seven body-worn units, three sensor streams, and three axes already produces 7 x 3 x 3 = 63 channels. The review should say whether the pipeline analyzes that full channel set, projects it into a lower-dimensional representation d < D, or converts it into joint-angle and joint-position features before segmentation, classification, or reference-motion matching.

Model-based gait segmentation adds a kinematic assumption on top of the inertial stream. Accelerometer and gyroscope fusion estimates joint angles for each frame; limb lengths and forward kinematics turn those angles into joint positions such as p_i = [x, y, z] relative to a fixed body reference, often the hip. The segmentation algorithm then scans sagittal, coronal, and transverse motion-plane traces for local minima and maxima, using successive extrema to define phase intervals. Its weakness is that the designer must explicitly choose the salient joints and planes: transverse knee motion, sagittal ankle motion, and left-versus-right joint traces may produce conflicting extrema sets, which is hard for a non-expert to resolve. That method is reviewable only when the sensor coordinate frames, body-plane transform, filtering, joint or plane selection, and extrema thresholds are recorded, because small discontinuities and sensor noise can create false positive segmentation points.

A model-free latent-space segmentation path reduces that selection burden. For J tracked joints, the pipeline can aggregate each frame into a single feature vector such as q_t = [x_1, y_1, z_1, x_2, y_2, z_2, ..., x_J, y_J, z_J]^T, learn a lower-dimensional representation r_t of the high-dimensional motion sequence q_t, and then run the same min-max segmentation logic in latent space. A manifold method such as Isomap is one way to learn that representation. The practical claim is not that Isomap is automatically better; it is that the review now has one segmentation set for the whole motion rather than separate, potentially conflicting sets for each hand-picked joint or plane.

Raw inertial traces are still useful evidence before classification. An arm-spin dataset, for example, may look like several noisy sensor channels over time; a PCA projection can turn those channels into a lower-dimensional trajectory that reveals repeated motion structure without pretending the projection is already an activity label. For a horizontal arm-spin or vertical-arm exercise record, the audit trail should keep the raw MAG, GYRO, and ACCEL x/y/z streams, the first and second principal-component trajectory, and the segmentation rule together. If the first principal component is used to place cycle boundaries, those boundaries are part of the method, not a visual afterthought.

Similarity matrices make the same point from another angle. A high-dimensional comparison can run DTW on each matching sensor channel and average the distances into one cell for each pair of motions. A PCA-reduced comparison can instead compute distances in the lower-dimensional component space, often making repeated arm-angle clusters easier to read. The learner should be able to see which distance rule, channel set, PCA fit scope, and dimensionality were used before accepting a coloured matrix as evidence that two exercises match. Keep that distinction when mobile sensing moves from personal feedback to social comparison or public-health summaries: the same sense-learn-act loop must preserve the raw-window source, feature transform, aggregation scope, and action boundary before it supports a decision.

Practitioner Knowledge Check

The model-quality discussion needs a resource ledger beside it. Figure 37.1 makes battery, memory, privacy, and missed-transition costs comparable before the chapter turns to deployment limits.

Use the BATTERY and PRIVACY rows in Figure 37.1 together: continuous GPS and CPU coverage spends about 180 mW plus 420 mW and observes location continuously, while ADAPTIVE gates expensive sensors and keeps shorter windows. The corresponding ACCURACY row exposes the trade: sleep boundaries can miss transitions.

37.7 Context Changes Break Deployment

Mobile sensing models often fail when training conditions and deployment conditions drift apart. The raw physics may still be valid, but the inference claim changes when the phone moves from pocket to hand, when sampling is throttled, when the user population changes, when labels were collected inconsistently, or when battery policy changes the available windows. The model should expose quality states instead of hiding these boundaries.

A common failure is training on clean, prompted activities but deploying into messy daily life. A participant may start walking before the label button is pressed, pause at a crossing, carry a bag in the same hand as the phone, or move through a transition that is neither sitting nor walking for the full window. If these transition windows were removed during training, the deployed model needs an unknown or transition state instead of forcing every window into a tidy class.

The deployment contract should preserve sample rate, timestamp tolerance, window length, overlap, placement assumption, feature code, classifier version, threshold, and fallback behavior together. If the operating system later batches sensors to save power, the same feature extractor may receive irregular intervals and stale windows. If a quantized model replaces a larger classifier, class boundaries and confidence calibration need a replay check before release. The review should treat those changes as evidence changes, not just implementation changes. A useful release record stores replay results by class and placement, so a later regression can be traced to sensor timing, feature drift, threshold choice, or model conversion instead of being described as a vague accuracy drop. Include failed windows too, because their pattern often reveals the actual deployment boundary.

Continuous sensing also has an energy contract. Always-on GPS, gyroscope, audio, and network upload can turn a useful prototype into a short-lived app, so many releases combine cheap motion triggers, lower-rate background checks, and duty cycling. A duty-cycled classifier should say which low-power signal keeps watch, how long the device sleeps when no event is detected, what shortens the sleep interval, and what accuracy or freshness is lost while sleeping. Audio keyword spotting follows the same rule: record the vocabulary, silence and unknown classes, sample rate, feature pipeline, class balance, and privacy boundary before treating a keyword label as an action trigger.

Some systems push the energy budget further by not sensing every attribute directly at all. If a context history shows that two cheap-to-sense values reliably co-occur with an expensive-to-sense one, a rule miner can learn that correlation and let the cheap values stand in for the expensive one. For a context history of 1,000 co-occurring readings where 200 include both A and B and 80 of those also include C, the rule {A, B} -> C has support 80/1,000 = 8 percent and confidence 80/200 = 40 percent: support says whether the pattern is common enough to bother with, and confidence says how often it holds once A and B are both true. A team tunes the support threshold to drop rare, noisy associations and tunes the confidence threshold to the accuracy the application needs, then can combine the surviving rules with ordinary AND/OR logic so, for example, "not walking and not driving" combined with "at home, in a meeting, or in the office" can stand in for a directly sensed indoor flag. That still leaves a design question: which attributes should be sensed directly and which should be inferred from cheaper ones. Searching every combination for the cheapest sensing plan that still meets an accuracy target is NP-hard, so production systems build the plan from measured per-attribute sensing costs and fall back to a heuristic once the attribute count is too large to search exhaustively. Either way, the rule set, its support and confidence thresholds, and the resulting sensing plan are model evidence and need the same retest discipline as any other classifier.

Under-the-Hood Knowledge Check

A concrete always-on case closes the chapter: Figure 37.2 separates the cheap listening gate from the expensive recognition burst.

Trace LISTEN and GATE in Figure 37.2 before paying the 42 mJ RECOGNISE burst. ADAPT lengthens sleep in quiet periods and shortens it after events, so the final design question is whether the saved charge still meets the missed-keyword and 180 ms latency contracts.