5 min read
The Camera Already Knows Which Way Is Up
Stabilising 360 video is a rotation problem, and the camera recorded the answer six different ways. The work is deciding which sensor to believe, and my cleverest ideas lost to the baseline.
A 360 camera on a bicycle records everything, including every bump, lean and wobble. Flat video stabilisation crops and warps to hide that. A sphere has nothing to crop. But it has a better property: shake is only rotation. If you know how the camera was turned at each frame, you turn the sphere back and the world holds still.
The camera knows. That is the premise of rr360, a pipeline I have been building on and off since spring to turn my ride footage into something watchable. It does not try to recover the motion from the pixels. It reads what the camera wrote down.
Six witnesses
A GoPro MAX file carries, next to its two video streams, a stack of sensor tracks. On an eight-minute clip from last year:
| Track | Rate | What it claims |
|---|---|---|
| Camera orientation | 30 Hz | a fused quaternion per frame |
| Image orientation | 30 Hz | the in-camera stabiliser’s correction |
| Gyroscope | ~794 Hz | 382,868 raw rotation-rate samples |
| Accelerometer | ~198 Hz | 95,717 samples |
| Gravity vector | 30 Hz | which way is down |
| Magnetometer | ~25 Hz | which way is north, roughly |
Six witnesses to the same event, at five different rates, on at least two different clocks. They do not agree, and on any given clip one of them may be lying.
So the first thing rr360 does with a file is not render it. It writes a diagnostic: for every track, the sample count, the effective rate, the span it covers and anything anomalous. Then a fusion plan with explicit weights (for that clip: gravity 0.5, camera orientation 0.425, gyro 0.075), a recommended mode and a confidence.
That clip’s diagnostic contains this line:
IORI stream is all-identity — GoPro’s in-camera stabilization did not write orientations for this recording.
14,460 samples, every one of them “no rotation.” A pipeline that trusted that track would have produced beautifully encoded, completely unstabilised video and reported success.
Cross-examining them
Two checks do most of the work.
Clocks. The sensor tracks and the video frames are stamped by different parts of the camera. This is the one place rr360 looks at pixels: it measures apparent motion in the picture over a sixteen-second window, slides that against the telemetry and takes the best match: on that clip, 27 milliseconds, with a correlation of 0.67. The raw gyro sits a further 10.2 ms off. Twenty-seven milliseconds is most of a frame. Apply a rotation one frame late to a camera on handlebars and you have built a wobble generator.
Invariants. The orientation track and the gravity track are independent descriptions of one fact: where down is. Combine them and you should get the same answer for the whole clip. In a healthy file the spread is under two degrees. On that clip it was 9.1, so the log says the horizon reference may be unreliable instead of silently levelling to the wrong horizon.
I like that check a great deal. It needs no ground truth. Two sensors that should agree are each other’s test.
What “stable” means
With trustworthy rotations there are still several right answers, and they are different films:
- World-lock. The sphere stays fixed to the world. You turn a corner; north stays north.
- Horizon-lock. Level is level, but the view follows your heading.
- Direction-lock and smooth-follow. The view tracks where you are going, with the jitter taken out.
Then the unglamorous part. The camera stores its sphere as a cube with warped faces. rr360 builds its own lookup tables to unfold that into a 5376×2688 equirectangular frame and does the unfold and the rotation as a single resample on the GPU, with decode and encode on the same card. The eight-minute clip, 14,460 frames, stitched and stabilised in thirteen and a half minutes.
The clever ideas lost
In April I ran the experiment I was sure would pay off. Three upgrades from the literature: a better orientation filter (VQF), an L1-optimal smoother on the rotation group, and using the 800 Hz gyro instead of the camera’s own 30 Hz fusion. Two metrics: inter-frame fidelity weighted for the sphere, where higher means steadier, and a seam cost where the two lenses meet, where lower is better.
| Variant | Steadiness | Seam cost |
|---|---|---|
| Better filter | no change | no change |
| L1 smoother | +0.07 dB | 1.3% worse |
| High-rate gyro | −0.5 dB | 10.8% worse |
| High-rate gyro + better filter | +0.04 dB | 22.0% worse |
The baseline won. The high-rate path, the one with twenty-six times more data, was the worst thing I tried.
I do not think that means high-rate fusion is a bad idea. I think it means the camera’s own fused track already encodes calibration I do not have, and my version of “use the raw sensor” re-derives it worse. But that is a hypothesis, and this was one clip, so read it as a result about my pipeline and not about those methods. What I have is a table, and the table says keep the simple thing. So the simple thing is what runs.
It is uncomfortable to build sophistication and have a benchmark hand it back. It is much more uncomfortable to ship it unmeasured and wonder for a year why the seams shimmer.
The last step ate the first twelve
One more, because it is the most transferable.
Last week an overnight render of that clip was marked failed. Thirteen minutes of GPU work, 14,460 frames, a complete 2.9 GB file on disk with every frame accounted for. The failure was the final step: writing the small metadata tag that tells a video player “this is a sphere.” The tool doing it rewrites the whole file, had a budget of about ten minutes, and a 2.9 GB file took longer.
The pipeline treated a labelling step as equal in standing to the render it was labelling. The file was checked, the run re-marked as done, and the labelling step given a budget that grows properly with file size.
Order your steps by what they cost to redo, and never let a cheap one hold an expensive one hostage.
Status
It works on my footage, and I use it. I have not finished the comparison I most want, against the camera maker’s own desktop stabiliser on the same clip: the unstabilised reference render was interrupted and I have not rerun it. Until that table exists I will not claim it beats the manufacturer’s stabiliser. A preference is not a measurement.
The reels it now assembles from rides, matched against photos taken along the way, are a different story for a different post. That one starts with the discovery that a phone’s GPS timestamp is the time of its last location fix, not the time you pressed the shutter, and that on one ride those differed by up to forty seconds.