VehicleMapping HD AV Spatial Data

Measuring HD Map Accuracy Against Survey Control

An accuracy figure is only worth what its ground truth is worth, and the commonest way to produce a worthless one is to measure against control the map was built from. This task produces a defensible number for HD map quality assurance and certification: a deterministic holdout taken before the build, a resolution step that matches each withheld point to what it was surveyed on, a distribution rather than a mean, and a bias/scatter split that tells you whether you have a datum problem or a survey problem.

Where the withheld set has to sit relative to the pipeline — before the datum fit, not after it:

Where the Withheld Control Set Enters the Pipeline Control splits at the front into a fit path through the build stages and a withheld path that bypasses them, with a crossed-out leak edge from the withheld set into the datum fit. surveyed control split, seed pinned 70% 30% datum alignment fit set only geometry + topology the build proper candidate map accuracy measurement the first and only place the two paths meet the leak: withheld control reaching the datum fit the number it produces measures nothing

Prerequisites #

  • Python 3.10+, NumPy 1.24+, SciPy 1.11+ (spatial.cKDTree), pyproj 3.6+.
  • Input: a surveyed control set in a known datum and epoch, each point carrying the identifier of the feature it was observed on.
  • Upstream stage: the split must happen before any pipeline stage consumes control, including the datum alignment described in coordinate reference systems for AVs.
  • Output: an accuracy report — per-stratum counts, lateral and vertical statistics, and the mean error vector.

Step-by-Step #

1. Split the control set deterministically and by stratum #

An unstratified split concentrates the holdout wherever control is dense, which is normally motorways.

python
import numpy as np

def stratified_holdout(ids, points, road_class, frac=0.3, seed=20260809):
    """Withhold `frac` of control within each road class, reproducibly."""
    rng = np.random.default_rng(seed)
    fit, held = {}, {}
    for cls in sorted(set(road_class)):
        idx = [i for i, c in enumerate(road_class) if c == cls]
        rng.shuffle(idx)
        cut = int(len(idx) * (1.0 - frac))
        for i in idx[:cut]:
            fit[ids[i]] = points[i]
        for i in idx[cut:]:
            held[ids[i]] = points[i]
    return fit, held

Key parameters: seed is recorded in the validation record so the split can be reproduced by an auditor; frac at 0.3 leaves enough control to fit the datum while giving the holdout statistical weight. Expected output: two dicts whose key sets are disjoint and whose union is the input.

2. Resolve every withheld point against the map #

Match by feature identifier where the survey recorded one, and fall back to nearest-feature only when it did not — a nearest-feature match on a multi-lane carriageway can silently pick the wrong lane.

python
from scipy.spatial import cKDTree

def resolve(held: dict, map_features: dict, max_snap_m: float = 0.5):
    """Map each withheld point to the mapped position it should coincide with."""
    ids = sorted(map_features)
    tree = cKDTree(np.array([map_features[i] for i in ids]))

    resolved, unmatched = {}, []
    for pid, p in held.items():
        if pid in map_features:                    # surveyed feature still exists
            resolved[pid] = map_features[pid]
            continue
        d, j = tree.query(p, distance_upper_bound=max_snap_m)
        if np.isfinite(d):
            resolved[pid] = map_features[ids[j]]
        else:
            unmatched.append(pid)
    return resolved, unmatched

unmatched is not an error to suppress: a withheld point with no feature within half a metre is either a coverage gap or a gross error, and both belong in the report.

3. Report the distribution, split lateral from vertical #

python
def error_stats(resolved: dict, held: dict) -> dict:
    common = sorted(set(resolved) & set(held))
    d = np.array([resolved[k] for k in common]) - np.array([held[k] for k in common])
    lat = np.linalg.norm(d[:, :2], axis=1)
    vert = np.abs(d[:, 2])
    return {
        "n": len(common),
        "lateral": {"rmse": float(np.sqrt((lat ** 2).mean())),
                     "p95": float(np.percentile(lat, 95)),
                     "max": float(lat.max())},
        "vertical": {"rmse": float(np.sqrt((vert ** 2).mean())),
                      "p95": float(np.percentile(vert, 95)),
                      "max": float(vert.max())},
        "mean_vector_m": d.mean(axis=0).tolist(),
    }

The gate is written against lateral.p95 and lateral.max; vertical is reported and gated separately because its acceptable magnitude is different, not because it matters less.

Reading the mean error vector: the same p95 can mean two completely different defects.

Scatter Without Bias against the Same Scatter With a Datum Shift Two error-vector scatter plots with identical spread, one centred on the origin and one offset, with the mean vector drawn on each. mean vector ≈ 0 — genuine scatter nothing to correct; improve the survey or the extraction mean vector 0.08 m — a datum shift fix the mismatch upstream; never subtract it in the report identical p95 magnitude · completely different defect

Verification & Acceptance Criteria #

python
def assert_accuracy_report(report, held, fit, criteria) -> None:
    assert not (set(held) & set(fit)), "holdout leaked into the fit set"
    for cls, stats in report["by_class"].items():
        assert stats["n"] >= 100, f"{cls}: only {stats['n']} withheld points"
        assert stats["lateral"]["p95"] <= criteria.p95_m, f"{cls} p95 over budget"
        assert stats["lateral"]["max"] <= criteria.max_m, f"{cls} max over budget"
    bias = np.linalg.norm(report["mean_vector_m"][:2])
    assert bias <= 0.02, f"systematic shift of {bias:.3f} m — fix upstream"

Acceptance gate: holdout and fit sets disjoint; ≥100 withheld points per road class in the claim; lateral p95 ≤0.10 m and max ≤0.25 m; and a horizontal mean error vector ≤0.02 m, because anything larger is a bias rather than noise and has a cause worth finding.

How an accuracy claim decays when the holdout stops being held out:

Holdout Discipline and What the Number Means Three rows pairing a holdout regime with what the resulting accuracy figure actually measures. what happens to the accuracy number as the holdout is re-used holdout used once fresh split each release the number means what it says trustworthy holdout reused same points every release the pipeline is slowly tuned to them drifts optimistic holdout leaked into the fit datum solved on all control measures the fit, not the map meaningless a number that improves every release while the map does not is the signature of the middle row

Common Errors & Fixes #

Accuracy is suspiciously good on one road class. That class's holdout is tiny, so the percentile is estimated from a handful of points. Stratify the split and gate on per-class counts.

Every point resolves but the errors are large and directional. A datum or epoch mismatch, visible as the mean vector. Check that the survey epoch and the map epoch agree before touching the extraction.

Many points come back unmatched. Either the map has a coverage gap there, or the survey identifiers were not carried through the build. Both are findings; suppressing them by widening max_snap_m converts a coverage gap into a fictitious accuracy number.

The number changes when the pipeline is re-run on identical inputs. The split seed is not pinned, so a different holdout is measured each time. Pin the seed and record it.

FAQ #

How many control points does an accuracy claim need? #

Enough that the 95th percentile is stable, which in practice means at least a few hundred withheld points spread across the road classes in the claim. The count that matters is per stratum, not overall: two thousand motorway points and eleven residential ones support a motorway claim and nothing else. Report the count alongside the statistic so a reader can judge how much weight the tail estimate carries.

Should the error be measured in 3D or in the horizontal plane? #

Both, reported separately. Lateral error is what a lane-keeping controller consumes and it is the number a lane-level claim rests on; vertical error matters for grade, drainage and for sensors that project onto the road surface, and it is usually several times larger because GNSS height is weaker than GNSS position. Collapsing them into one 3D distance produces a figure that is dominated by the vertical component and answers neither question.

What does a non-zero mean error vector tell you? #

That the release has a systematic shift rather than random scatter — a datum, epoch or projection mismatch somewhere upstream. It is good news in the sense that a bias is correctable and scatter is not, but it must be fixed at its source rather than subtracted out: a pipeline that applies a measured offset is a pipeline that will apply the wrong offset the moment the underlying mismatch changes.

Up one level: HD Map Quality Assurance & Certification — the assurance stage this measurement is the core of.