blog

5 min read

The Camera Already Knows Which Way Is Up

Stabilising 360 video is a rotation problem, and the camera recorded the answer six different ways. The work is deciding which sensor to believe, and my cleverest ideas lost to the baseline.

A 360 camera on a bicycle records everything, including every bump, lean and wobble. Flat video stabilisation crops and warps to hide that. A sphere has nothing to crop. But it has a better property: shake is only rotation. If you know how the camera was turned at each frame, you turn the sphere back and the world holds still.

The camera knows. That is the premise of rr360, a pipeline I have been building on and off since spring to turn my ride footage into something watchable. It does not try to recover the motion from the pixels. It reads what the camera wrote down.

Six witnesses

A GoPro MAX file carries, next to its two video streams, a stack of sensor tracks. On an eight-minute clip from last year:

TrackRateWhat it claims
Camera orientation30 Hza fused quaternion per frame
Image orientation30 Hzthe in-camera stabiliser’s correction
Gyroscope~794 Hz382,868 raw rotation-rate samples
Accelerometer~198 Hz95,717 samples
Gravity vector30 Hzwhich way is down
Magnetometer~25 Hzwhich way is north, roughly
Samples per second, by sensor track
Gyroscope 794 Hz
Accelerometer 198 Hz
Camera orientation 30 Hz
Image orientation 30 Hz
Gravity vector 30 Hz
Magnetometer 25 Hz
Measured rates on the clip, not the nominal ones. The video itself is 29.97 frames a second, so three of these tracks have exactly one sample per frame and the gyroscope has about twenty-six. rr360 diagnostic for an 8-minute clip

Six witnesses to the same event, at five different rates, on at least two different clocks. They do not agree, and on any given clip one of them may be lying.

So the first thing rr360 does with a file is not render it. It writes a diagnostic: for every track, the sample count, the effective rate, the span it covers and anything anomalous. Then a fusion plan with explicit weights (for that clip: gravity 0.5, camera orientation 0.425, gyro 0.075), a recommended mode and a confidence.

That clip’s diagnostic contains this line:

IORI stream is all-identity — GoPro’s in-camera stabilization did not write orientations for this recording.

14,460 samples, every one of them “no rotation.” A pipeline that trusted that track would have produced beautifully encoded, completely unstabilised video and reported success.

Cross-examining them

Two checks do most of the work.

Clocks. The sensor tracks and the video frames are stamped by different parts of the camera. This is the one place rr360 looks at pixels: it measures apparent motion in the picture over a sixteen-second window, slides that against the telemetry and takes the best match: on that clip, 27 milliseconds, with a correlation of 0.67. The raw gyro sits a further 10.2 ms off. Twenty-seven milliseconds is most of a frame. Apply a rotation one frame late to a camera on handlebars and you have built a wobble generator.

Invariants. The orientation track and the gravity track are independent descriptions of one fact: where down is. Combine them and you should get the same answer for the whole clip. In a healthy file the spread is under two degrees. On that clip it was 9.1, so the log says the horizon reference may be unreliable instead of silently levelling to the wrong horizon.

I like that check a great deal. It needs no ground truth. Two sensors that should agree are each other’s test.

What “stable” means

With trustworthy rotations there are still several right answers, and they are different films:

  • World-lock. The sphere stays fixed to the world. You turn a corner; north stays north.
  • Horizon-lock. Level is level, but the view follows your heading.
  • Direction-lock and smooth-follow. The view tracks where you are going, with the jitter taken out.

Then the unglamorous part. The camera stores its sphere as a cube with warped faces. rr360 builds its own lookup tables to unfold that into a 5376×2688 equirectangular frame and does the unfold and the rotation as a single resample on the GPU, with decode and encode on the same card. The eight-minute clip, 14,460 frames, stitched and stabilised in thirteen and a half minutes.

The clever ideas lost

In April I ran the experiment I was sure would pay off. Three upgrades from the literature: a better orientation filter (VQF), an L1-optimal smoother on the rotation group, and using the 800 Hz gyro instead of the camera’s own 30 Hz fusion. Two metrics: inter-frame fidelity weighted for the sphere, where higher means steadier, and a seam cost where the two lenses meet, where lower is better.

VariantSteadinessSeam cost
Better filterno changeno change
L1 smoother+0.07 dB1.3% worse
High-rate gyro−0.5 dB10.8% worse
High-rate gyro + better filter+0.04 dB22.0% worse
Steadiness against the baseline (higher is steadier)
Better filter 0.00 dB
L1 smoother +0.07 dB
High-rate gyro -0.52 dB
High-rate gyro + filter +0.04 dB
one clip, April benchmark; all four against the same baseline
Seam cost against the baseline (lower is better)
Better filter 0%
L1 smoother +1.3%
High-rate gyro +10.8%
High-rate gyro + filter +22.0%
Two measures, two charts. Nothing beat the baseline on both, and the options that used the fast gyroscope paid for it where the two lenses meet. same benchmark

The baseline won. The high-rate path, the one with twenty-six times more data, was the worst thing I tried.

I do not think that means high-rate fusion is a bad idea. I think it means the camera’s own fused track already encodes calibration I do not have, and my version of “use the raw sensor” re-derives it worse. But that is a hypothesis, and this was one clip, so read it as a result about my pipeline and not about those methods. What I have is a table, and the table says keep the simple thing. So the simple thing is what runs.

It is uncomfortable to build sophistication and have a benchmark hand it back. It is much more uncomfortable to ship it unmeasured and wonder for a year why the seams shimmer.

The last step ate the first twelve

One more, because it is the most transferable.

Last week an overnight render of that clip was marked failed. Thirteen minutes of GPU work, 14,460 frames, a complete 2.9 GB file on disk with every frame accounted for. The failure was the final step: writing the small metadata tag that tells a video player “this is a sphere.” The tool doing it rewrites the whole file, had a budget of about ten minutes, and a 2.9 GB file took longer.

Where the overnight job's 28 minutes went
Read the sensor tracks 3 min · 11%
Align clocks, compute corrections 1.2 min · 4.2%
Stitch and stabilise on the GPU14,460 frames 13.5 min · 48%
Write one metadata tagtimed out; the job was marked failed 10.7 min · 38%
render log, November 5

The pipeline treated a labelling step as equal in standing to the render it was labelling. The file was checked, the run re-marked as done, and the labelling step given a budget that grows properly with file size.

Order your steps by what they cost to redo, and never let a cheap one hold an expensive one hostage.

Status

It works on my footage, and I use it. I have not finished the comparison I most want, against the camera maker’s own desktop stabiliser on the same clip: the unstabilised reference render was interrupted and I have not rerun it. Until that table exists I will not claim it beats the manufacturer’s stabiliser. A preference is not a measurement.

The reels it now assembles from rides, matched against photos taken along the way, are a different story for a different post. That one starts with the discovery that a phone’s GPS timestamp is the time of its last location fix, not the time you pressed the shutter, and that on one ride those differed by up to forty seconds.