Limitations¶

Purpose. A single list of what this repository cannot do, why, and what would be required to change it. Written so that a reader does not have to discover any of this by running the code.

Every entry is ordered by how much it would change the conclusions.

1. Blocking limitations¶

These prevent the repository from supporting any operational, product, or programme claim.

L1: No field data¶

The benchmark is a synthetic fixture: an analytic trajectory with the filter initialised exactly at the truth. Two public recorded datasets (EuRoC and TUM VI) can be fetched and evaluated, with GNSS simulated from their ground truth, and one study uses real camera frames. None of that data is vendored.

Consequence: no number here is a field measurement. GNSS is never real, so receiver latency, multipath, lever arm and real interference are not characterised. Bias stability, temperature behaviour and vibration response are only as good as the public recordings.

What it would take: a capture from the target platform with a real receiver and an independent reference. The recorded-data path exists; the field data does not.

VERIFIED BY REPOSITORY.

L2: The documented failure is not fixed¶

Visual aiding under GNSS denial is badly overconfident: mean NEES 286.2 against an expected 3, 2σ coverage 20.5%. The configuration ships disabled.

Consequence: the repository demonstrates a problem rather than solving it.

What it would take: implementing one of the established remedies: observability-constrained EKF, first-estimate Jacobians, an invariant or right-invariant error-state formulation, the Schmidt Kalman filter, or a pose-graph backend. None is present. This is the highest-value next step and it is not a research question, it is an implementation task.

VERIFIED BY REPOSITORY and VERIFIED BY EXTERNAL SOURCE, see S10–S16.

L3: No comparison against an established consistent estimator¶

The repository does not run OpenVINS, VINS-Mono, an invariant EKF, or any other published consistent filter on the same scenes.

Consequence: the repository cannot show that its formulation is worse than the state of the art, only that it is overconfident in isolation. Without a baseline, the numbers characterise a configuration, not the field.

What it would take: a harness interface and one reference implementation. OpenVINS is a reasonable first choice, and the synthetic scenes would need to be exported in a format it accepts.

VERIFIED BY REPOSITORY.

L4: No formal integrity target¶

FDIR performs chi-square innovation gating and per-channel fault detection, isolation and recovery.

Consequence: there is no protection level, no alert limit derived from a risk model, no availability requirement, and no argument that a fault is declared before it causes unacceptable navigation error. Receiver autonomous integrity monitoring is an established field with its own requirements machinery, and none of it is engaged.

What it would take: a specified integrity target, a derivation of alert limits from it, and validation against a fault model with defined detection and false-alarm probabilities.

VERIFIED BY REPOSITORY and VERIFIED BY EXTERNAL SOURCE: S1.

2. Material limitations¶

These do not block the repository, but they bound what its results mean.

L5: Synthetic scenes are not a distribution over reality¶

Eight scenes vary trajectory geometry over a 4.65× path-length range, holding duration, sample rate and the exact t=0 initial state fixed. All seven case verdicts are stable across them.

Consequence: the claim that survives is "the anchor model is structurally overconfident under visual aiding", not "this filter has X% error in conditions Y". Nothing here bounds behaviour on real imagery, real camera degradations, or real outage patterns.

Qualifier. The scene sweep holds the outage fixed (5 s to 20 s). A separate outage-window sweep (navkit sweep outages, MODEL.md) varies the start time and duration over 8 windows with 5 seeds each (40 runs). There the visual case is overconfident in every run, while its accuracy advantage over the control depends on when the outage starts. So "stable verdict" means the overconfidence is stable, not that the filter is more accurate in general.

Not varied: scene duration, sensor rate configurations, motion classes beyond smooth analytic paths, lighting, motion blur, rolling shutter, or calibration error.

VERIFIED BY REPOSITORY.

L6: No RF or adversary model¶

The benchmark models interference as a clean binary outage. There is no jamming model, no spoofing model, no multipath model and no interference characterisation.

Qualifier. The injection layer can apply more than outages: camera frame loss, IMU sample loss, timestamp offsets, a slow GNSS bias (a Gauss-Markov surrogate, not multipath) and measurement offsets. Apart from camera frame loss, none of these is a benchmark case. Most are covered by unit tests only, injectable vision outages by none, and the fault-mode table in MODEL.md lists each one with its gap.

Consequence: the FDIR layer is evaluated against a far simpler input than it would face. An adversary that produces plausible but wrong fixes is substantially harder than an outage, and is not tested at all.

Measured slow-ramp limit. A GNSS spoof that grows from 0.05 to 2 m/s was followed by the filter in every run on two recorded-IMU datasets with simulated GNSS, and a channel declared faulty did not stop it (see slow ramp, ADR-0019). With an IMU and GNSS alone this cannot be detected. A simulated visual second source with independent errors lowers the floor to about 2 m/s and with correlated errors does not help (vision ramp). A real front end is not tested.

ADR-0007 records the spoof-permanence problem (the detection signal that is still missing). It is tracked but unimplemented.

VERIFIED BY REPOSITORY.

L7: FDIR detects but does not classify¶

The gate isolates a channel and reports how long it was out. It does not distinguish multipath from spoofing from sensor degradation.

Consequence: an operator receives "this channel is faulty", not "this channel is being spoofed". No response is selected on the operator's behalf.

Measured. On injected faults and real recorded IMU data, the GNSS innovations separate multipath and large IMU faults, call spoofing in about three quarters of runs, and miss weak faults (fault classification). The false-alarm rate on the control is 18% in cross-validation and 35% on unseen runs; only the multipath call held up without a flaw.

VERIFIED BY REPOSITORY.

L8: The accelerometer-bias hypothesis was untested; it is now tested and was wrong¶

The research baseline said accelerometer bias is the dominant contributor to the error of a GNSS outage, and asked whether a bias fault would be detectable. Both are now run (Where the outage error comes from). On the synthetic fixture, removing every inertial error source leaves the peak outage error essentially unchanged, and the error grows with outage length at the same exponent with or without them. What dominates is the state error the filter has when GNSS is lost. A bias fault is not noticed during the denial, because nothing is measured; a small one is not noticed on return either, and a large one makes the filter reject the healthy GNSS and lock it out.

Consequence: the repository's mechanism library stated a physical error (constant bias gives linear growth) and a hypothesis the fixture falsifies; both are corrected. The result is about the synthetic fixture, whose inertial unit is good relative to its GNSS. It says nothing about the dominant source in a recording, where the bias random walk turned out to matter (EuRoC, TUM VI).

VERIFIED BY REPOSITORY.

L9: Recorded-data tests skip without external data¶

The orientation checks on recorded EuRoC and TUM VI sequences need data that is deliberately not vendored, and skip when it is absent. A synthetic sequence runs the same check in every checkout.

Consequence: in a clean checkout, the trajectory reader is checked on a synthetic sequence but not on recorded data.

VERIFIED BY REPOSITORY.

L10: Not independent¶

The estimator, the synthetic generator, the evaluation metrics and the benchmark configuration are all written by the same author.

Consequence: no adversarial review, and the risk of a shared conceptual error across generator and estimator is real. The strongest available mitigation is that the failure contradicts the code's intent: the filter is not overconfident because of a mistake that favours it, but despite every design choice the author made.

AUTHOR-STATED.

3. Minor limitations¶

No wall-clock budget. The code is synchronous and offline. Nothing is measured against a timing requirement, and "real-time" is not claimed.
No packaging for deployment. It is a library. A Dockerfile and a devcontainer exist to reproduce the results, not to deploy it; there is no hardware abstraction and no real-time operating system support.
Figures carry a matplotlib version difference. Regenerated figures show metadata drift. Deterministic run-to-run, but a reviewer diffing images sees noise.
Result JSON is gitignored. Numbers are reproducible from committed code and configuration, not from committed outputs. A reader must run the code.
similarity alignment is implemented but unused in headline numbers, so a scale-error result exists in the code and is not exercised by the benchmark.

VERIFIED BY REPOSITORY.

4. Claims that were removed rather than qualified¶

Recorded so the removal is auditable.

Claim	Why removed
"The real fix is a pose graph"	Implies novelty in identifying the remedy. Pose graphs are one established option among several; the literature is cited instead.
"Multi-scene validation is still open"	Became false when the scene sweep was added. Replaced with a scoped statement of what the 8 scenes do and do not establish.
"Demonstrates robustness"	Replaced with the actual scope: 10 noise seeds, 8 scenes, 7 stable verdicts.
"State of the art"	No algorithmic contribution exists to support it.
"The numbers characterise a configuration"	True but read as a disclaimer rather than a scoping statement; rewritten to state what the sweeps do establish.

VERIFIED BY REPOSITORY.

5. If you only read one section¶

Read L2 and L3 together. The repository shows a real, reproducible failure and does not compare itself against the field that already solved it. That single gap is what stands between this being a measurement artefact and being an engineering contribution.