See also: 01-research-basics · 02-pose-estimation · 04-python-and-shell
Preprocessing turns raw (frames, 33, 3) pixel skeletons into windows the classifier can use.
The same steps run for every dataset, estimator and classifier. Otherwise a difference in
results could come from preprocessing rather than from the classifier.
Steps
- Pick the person and the joints. Keep one person (highest mean confidence) and about 13
body joints. Drop the face and finger points: they’re noisy and don’t help detect a fall. - Fill short gaps. If the person is missing for up to 5 frames, fill the gap by
interpolation (draw a straight line between the last known and next known position).
Keep a mask saying which frames were filled, so later you can tell made-up data from real.
Longer gaps stay missing, because inventing 2 seconds of motion is fiction. - Normalise.
p' = (p − hip) / L, where hip is the mid-hip point and L is torso length
(mid-shoulder to mid-hip). Subtracting the hip removes where in the image the person is.
Dividing by L removes how big they look (near or far from the camera). Clamp L to a minimum,
because when someone lies flat facing the camera, L can shrink towards 0 and dividing by it blows up. - Cut windows. Slide a window of W frames forward S frames at a time.
Count = ⌊(N − W) / S⌋ + 1. Example: N = 180 frames, W = 18 (≈1 s at 18 fps), S = 3 → (162 / 3) + 1 = 55 windows. - Label windows. Majority rule: if most frames in the window are inside the marked fall
period, it’s a fall window.
Two traps
- Compute hip speed before centring. After centring, the hip is always at (0, 0), so its
speed is always zero. The fastest signal of a fall (the body dropping) would be gone. - Don’t min-max normalise over a whole video. Scaling by the video’s own min and max uses
frames from the future, which a live system doesn’t have. It also makes a video’s numbers
depend on what happens in it, which quietly hands the classifier a clue.
In your own words
- Why do we normalise by torso length instead of by image size?
- N = 300, W = 18, S = 3. How many windows? (Show the formula.)
Jump to a subject
Software engineering
Infrastructure & security
Computer science
Hardware & engineering