Skip to content

Status and limits

  • Not field validated. The benchmark is a synthetic fixture. Recorded IMU data (EuRoC, TUM VI) and one set of real camera frames (a real front end) are used with GNSS simulated from the ground truth. None of that data is vendored, and no number is a field measurement.
  • Visual aiding under GNSS denial is untrustworthy with the shipped single anchor, and disabled by default. See concepts. The opt-in stochastic clone (ADR-0017) is calibrated for independent visual errors. Against the measured errors of a real two-view front end it stays inside the gate only with the assumed noise inflated by four, in all six windows measured, which is a named opt-in setting (ADR-0023, ADR-0024).
  • FDIR detects implausible updates and can name only some causes. The chi-square gate in fdir/ isolates a channel and says how long it was out (ADR-0005). A multipath call exists as a function (ADR-0022). Spoofing and degradation are called in about three quarters of injected runs, weak faults are missed and the false-alarm rate is not zero (ADR-0021).
  • A slow-ramp spoof is followed. With an IMU and GNSS alone the filter follows a ramp of any rate tested (ADR-0019). A simulated visual source catches a fast ramp when its errors are independent (ADR-0020).
  • The calibrated-covariance failure is mitigated, not fixed. The ADR-0005 gate assumed a calibrated innovation covariance the filter does not have once position covariance has collapsed, and in outage_visual that cost 1.6 m of ATE. That was blocker B5, closed by ADR-0006: a per-channel NIS window plus adaptive GNSS covariance inflation, which re-gates a returning fix once under an inflated covariance. ATE 5.059 m to 2.541 m and rejections 51 to 5, with the false-alarm rate unchanged. B1 is still open, see below.

  • The filter is still overconfident under visual aiding. Mean NEES 286.2 against a nominal 3, 2σ coverage 20.5% where the SRS requires 95% (AC-04) and the roadmap gate for turning vision on is above 90%. It converges and is confidently wrong. This is blocker B1, and it is a pose-graph problem that no threshold in the FDIR subsystem will move. Robust across 10 noise seeds (worst case NEES 113.5, 6.8% coverage) and across all 8 scenes (286.4 [282.4, 290.2], 20.5%).

  • ADR-0008 frozen-anchor cross-check: implemented but unreachable under the current state-transition logic; the security gap remains documented as open.

  • The published numbers are single draws. A 10-seed sweep shows NEES varying by 4.4x to 45x between cases, and the two controls the tables call calibrated (gnss_only, outage_control) flip verdict across seeds, and outage_control is never clean in 10 draws. The shipped tables remain the seed-0 benchmark, which is the committed artefact; treat the magnitudes as order-of-magnitude and the verdicts as the claim.
  • Scene generalisation is demonstrated, external validity is not. The 8-scene sweep varies trajectory geometry over a 4.65x path-length range and all seven verdicts hold, so the failure above is not a property of one path. What is not shown: any real capture, any measured sensor characteristic, or any scene outside an analytic trajectory generator. Eight synthetic scenes bound the claim "the anchor model is structurally wrong under visual aiding"; they do not bound its behaviour on real imagery. Scenario duration remains fixed at 30 s.
  • The outage window matters for accuracy, not for calibration. The outage sweep (navkit sweep outages, 8 windows, 5 seeds each, one synthetic path) gives outage_visual a 2-sigma coverage of 10.4% to 39.0% and a mean NEES of 40.0 to 8 353 in all 40 runs, including 5 s outages; outage_control stays at 98.0% to 100.0% coverage in all 40. Accuracy is different. With the outage starting at 5 s, vision beats the control in all 5 seeds for 10 s, 15 s and 20 s outages (ATE ratio 0.14 to 0.89) and in 2 of 5 for 5 s. With the outage starting at 10 s, vision is worse than the control in all 20 runs (ATE ratio 1.54 to 7.30). The cause is not tested here. So "vision improves ATE" is a property of the benchmark's early outage, not a general result.

Outage sweep - Not flight-ready. No sensor driver, no live front end, no real-time loop, no failure-mode handling beyond a measurement gate. - Recorded-data checks skip without the data. The trajectory reader's orientation is checked against the recorded gyroscope on every fetched sequence, and always on a synthetic one. The claim of reproducing a published TUM VI ATE was retired (R5 in CONSTRAINTS.md).