Skip to content

Benchmark and command line

Reproducible benchmark for the 21-state ESKF.

Every number quoted in README.md and docs/ is produced by this module, from a YAML scenario in configs/, with a fixed seed. Nothing here is hand-entered. The point is that a reader can rerun it and get the same digits:

navkit run --out results/benchmark.json
navkit run --only gnss_only outage_visual

The same code runs from a source checkout without installing anything:

python scripts/run_benchmark.py --out results/benchmark.json

Design decisions worth stating, because they change the numbers:

  • The reference trajectory is the analytic one the synthetic generator is built from, so the ground truth is exact rather than another estimate. Motion is parameterised as 1 - cos(w t) so position, velocity, rotation and angular rate are all zero at t = 0; an estimator initialised at the identity pose is then exactly correct at t=0 and every later error belongs to the filter.

  • ATE is reported four ways, never one. none is the honest number for a filter that is already in the reference frame; rigid fits a transform to all poses; rigid_start fits on the first 20% only, so it cannot hide accumulated drift; similarity adds a scale degree of freedom. An ATE quoted without its alignment is not a result.

  • Calibration is scored with the filter's own covariance. A filter that is wrong by 31 m and claims 0.3 m of uncertainty is the failure this project exists to surface, so NEES and coverage are reported next to the position error, not instead of it.

  • All results are labelled as synthetic. These are numbers from a known-answer fixture, not from any real sensor capture, and no claim in this repository treats them as field performance.

Locate the default scenario file.

A source checkout (including an editable install) has configs/benchmark.yaml next to src/. An installed wheel carries a copy at navkit/configs/, put there at build time, so the command works from any directory.

Run one scenario end to end and return its result record.

seed overrides every stochastic stream in the scenario (GNSS, vision and camera drop) so a benchmark can be repeated over independent draws. Because the scenario is mutated before it is hashed, each override produces its own config_hash and the provenance stays truthful.

scene_seed is a separate axis: it varies the trajectory rather than the sensor noise, so the question becomes "is this a property of the filter" instead of "was this a property of one noise draw". The two are independent on purpose, and passing both sweeps the full cross product.

trajectories adds trajectory_est and trajectory_ref (time and positions, the reference resampled onto the estimate's timestamps) for a caller that draws the path. It is off by default so the benchmark JSON stays small and unchanged.

gnss_hook receives the GNSS stream after the scenario has been applied and returns the stream the filter sees. It is for a fault the scenario schema does not describe, such as a position offset (a spoof); the fault matrix uses it. The hook is not part of the scenario, so it is not in config_hash: a caller that uses it must record the fault itself.

imu_hook is the same for the inertial stream: it receives the IMU after the scenario's noise and bias are applied and returns what the filter sees. It is for a fault the scenario schema does not describe, such as a bias that appears part-way through. Like gnss_hook it is not in config_hash.

fdir_events adds the FDIR event log (declarations, lockouts, inflation grants) to the record as fdir_events. It is off by default so the benchmark JSON is unchanged.

Every scenario goes through the injection layer, including one with no outage or camera drop. The synthetic IMU has no noise of its own; the IMU noise and bias come from that layer, so skipping it ran some cases with a noiseless IMU (ADR-0012).

Render the results as the table quoted in README.md.

Generated rather than hand-written on purpose: a table that someone typed in by hand is a table that will silently disagree with the code the next time a default changes. docs/ and the README are both expected to be regenerated from this output.

Repeat a benchmark across independent seeds and report the spread.

Every number in README.md and docs/ comes from a single draw. configs/benchmark.yaml pins each case to seed: 0, so a result like mean NEES 286.2 is one sample of a stochastic process, not a measured property of the filter. This script re-runs the selected cases over N seeds and reports the distribution, so the difference between "this configuration overconfidently collapses coverage" and "this one fixture happened to" is settled by measurement rather than assumed either way.

navkit sweep seeds --seeds 10
navkit sweep seeds --seeds 10 --only outage_visual --markdown

It reuses run_case from :mod:navkit.benchmark rather than reimplementing the pipeline, so the sweep cannot drift away from what the committed benchmark measures. The baseline seed is still included by default: the committed numbers must remain reproducible, and a sweep that silently dropped the published draw would make the comparison harder, not easier.

The output is still synthetic. More seeds do not make a known-answer fixture into a field measurement, so the payload carries the same disclaimer and claim_type as results/benchmark.json and must never be cited as real-sensor performance.

Sweep the trajectory rather than the sensor noise, and report a distribution.

navkit sweep seeds varies GNSS, vision, and camera-drop noise over N seeds. That answers "is this filter overconfident under this noise", which is a narrower question than it looks: the trajectory is held fixed at the one in configs/benchmark.yaml, so every draw replays identical motion. A result that was really an artefact of that particular path -- a long straight excursion, a slow turn -- would be indistinguishable from a property of the estimator.

This script varies the scene instead, using synthetic.seeded_scene, and holds the noise at the configured seed. The two axes are independent, so navkit sweep seeds answers a robustness question and this one answers a generalisation question; they are complements, not replacements.

navkit sweep scenes --seeds 8
navkit sweep scenes --seeds 8 --markdown
navkit sweep scenes --seeds 8 --both-seeds   # full cross product

The strongest single number in the output is verdict_stable: whether the calibration verdict holds in every scene. A mean NEES that is stable while the verdict flips between scenes is a much weaker result than either alone, and the per-verdict counts are reported so that case cannot be hidden in an average.

Confidence intervals are bootstrapped rather than taken from a standard deviation. With 8-10 draws the normal approximation is not defensible, and the min/max are reported alongside so a reader can check the interval by eye rather than trusting it.

Percentile bootstrap CI for statistic over values.

Returns the point estimate, the interval, and how many of the resamples were degenerate. With very small samples a bootstrap can put a large share of its mass on a single observed value; n_distinct reports how many unique values went in, so a reader can tell a tight interval over a genuinely repeated quantity from a tight interval over one repeated draw.

Sweep the GNSS outage window: when it starts and how long it lasts.

The benchmark has one outage (5 s to 20 s). Everything it says about visual aiding through a denial is therefore a statement about a 15 s outage that begins 5 s in. This sweep asks whether the headline finding, that vision improves position error but the filter's reported uncertainty collapses, holds when the outage is shorter, longer or later, and whether the no-vision control stays calibrated across the same grid.

navkit sweep outages
navkit sweep outages --starts 5 10 --durations 5 10 15 20 --seeds 5 --markdown
navkit sweep outages --csv results/outage_sweep.csv --figure docs/figures/outage-sweep.png

Each cell of the grid is one (case, outage start, outage duration) and is repeated over N noise seeds; the table reports the mean with a percentile-bootstrap 95% interval. Only the outage window changes, so a difference between cells is a difference in the outage. Combinations whose window runs past the end of the scenario are skipped and listed.

The output is synthetic. A sweep over a known-answer fixture does not make it a field measurement, so the payload carries the same disclaimer and claim_type as results/benchmark.json. With few seeds the bootstrap interval is a rough guide, not a guarantee: n and n_distinct are reported beside it so that is visible.

Collapse the per-seed rows of one grid cell into mean, interval and verdict counts.

Return the cells to run and the (start, duration) pairs skipped for overrunning the scenario.

Copy of case whose only GNSS outage is the given window.

Render the benchmark figures committed under docs/figures/.

Figures are committed rather than generated on demand because they are the evidence behind the numbers in README.md and docs/. A reader should not have to run a 30 s filter to see what was measured, and a reviewer should be able to diff the figures between commits.

navkit figures --results results/benchmark.json

Every figure is regenerated from the JSON the benchmark wrote, never from in-memory state, so a figure can always be traced back to a recorded result.

The headline figure: how wrong the filter is against what it claims.

A dot on the diagonal is a calibrated filter. Dots far above it are the failure this project exists to document: the filter is more confident than the error justifies. The line is 1:1, not a fit, so the eye is not invited to look for a trend that is not being claimed.

Coverage at each per-axis sigma against the 3-dof expectation.

The expectation is not 68/95/99.7: those are the one-dimensional numbers, and quoting them for a 3-d position error is one of the easier ways to misreport a filter. The bars are the exact ellipsoid coverage.

Error against time through the GNSS outage, for the two outage cases.

Mean NEES against the expected value, which is the number a reviewer will check first when judging whether a filter is honest.

Error, NEES and coverage against outage length, one column per outage start.

Each line is a case and the band is its bootstrap 95% interval over noise seeds. The reference lines are what a calibrated filter would show: NEES 3 and 99.3% coverage of the 2-sigma ellipsoid. Written to out, not FIGDIR, because the sweep is a command with its own output path.

Error, NEES and coverage against the factor on the filter's assumed sensor noise.

One line per case; the band is the bootstrap 95% interval over noise seeds. The dashed lines are what a calibrated filter would show (NEES 3, 99.3% coverage), and the dotted vertical line marks a filter that is told the true noise. Written to out because the sweep is a command with its own output path.

The hero figure and its animation: the error the filter makes against the error it claims.

Two panels, one per case, with the same GNSS outage in both. The control (no vision) reports an uncertainty that grows through the outage and contains its error. The visual case reports a small, nearly flat uncertainty while its error leaves the claimed bound. That gap is the project's result, in one picture.

The claimed bound is the filter's own 2-sigma-per-axis ellipsoid, expressed as a radius. For an isotropic position covariance with per-axis sigma s the squared Mahalanobis radius is 2**2 * 3, so the bound on the error norm is 2 * sqrt(3) * s; the benchmark expects 99.3% of epochs inside it. The covariance is not exactly isotropic, so the bound is an approximation of the ellipsoid in each direction, which is why the coverage figures in the README come from the ellipsoid itself and not from this picture.

Both functions read only the benchmark result JSON, so a figure can always be traced to a recorded run.

The two cases the figure needs, in panel order. Empty if the result lacks them.

Write the static hero figure. Returns None when the result lacks the two cases.

Write the hero figure as a GIF that reveals the run in time order.

The navkit command line.

navkit run runs the reproducible benchmark, navkit sweep repeats it over seeds or scenes, navkit figures renders the committed plots from a result file, navkit euroc fetches and runs recorded sequences, and navkit --version prints the package version. The implementation of each subcommand lives in its own module, so this file only routes arguments.

navkit run --markdown
navkit run --only gnss_only outage_visual --out results/two_cases.json
navkit sweep seeds --seeds 10 --markdown
navkit sweep scenes --seeds 8
navkit sweep outages --markdown
navkit sweep mismatch --markdown
navkit sweep faults --markdown
navkit sweep attribution --markdown
navkit figures --results results/benchmark.json
navkit euroc fetch --sequence MH_01_easy
navkit tumvi fetch --sequence room1
navkit euroc run --sequence MH_01_easy --outage 60:20 --markdown
python -m navkit run --list

Run the command line and return a process exit code.