VehicleMapping HD AV Spatial Data

Building a Map Release QA Gate in Python

A gate is a small piece of code with an unusual requirement: it has to be trustworthy when nobody is watching it, on the day the release is late. That rules out most of what makes ordinary code convenient — exceptions that stop at the first problem, thresholds embedded where they are used, override flags for the awkward case.

This task builds the gate for HD map quality assurance and certification: frozen criteria in version control, checks that return findings, a fixture suite that proves the gate can reject, and a record emitted on every run whether it passes or not.

The four properties that separate a gate from a test suite:

The Four Properties of a Release Gate Four panels naming a gate property, what it looks like in code, and the specific failure mode its absence produces. frozen criteria a frozen dataclass, version controlled without it a threshold drifts and nobody can say when relaxing it leaves a diff complete findings checks return lists, they do not raise without it eleven defects take eleven build cycles one run, whole work list proven rejection known-bad fixtures the gate must refuse without it an inverted comparison passes forever it has rejected something recorded outcome a record on pass and on fail alike without it the evidence is a log that rotates away hashed into the release the first two are about the gate being usable; the second two are about the gate being believable a gate with only the first two is a linter, which is a different and less demanding thing

Prerequisites #

  • Python 3.10+, standard library only for the gate core; the checks themselves pull in whatever the pipeline already uses.
  • Input: an accuracy report as produced by measuring HD map accuracy against survey control, a coverage report, and the defect tracker's open count.
  • Upstream stage: all validation has run; this gate consumes results, it does not compute them.
  • Output: an exit status, a human-readable summary and a JSON validation record.

Step-by-Step #

1. Freeze the criteria #

Every threshold lives in one immutable object, and that object is imported rather than constructed at the call site.

python
from dataclasses import dataclass, asdict

@dataclass(frozen=True, slots=True)
class ReleaseCriteria:
    lateral_p95_m: float = 0.10
    lateral_max_m: float = 0.25
    vertical_p95_m: float = 0.30
    bias_m: float = 0.02
    coverage_overall: float = 0.98
    coverage_min_class: float = 0.90
    min_holdout_per_class: int = 100
    open_defects: int = 0

CRITERIA = ReleaseCriteria()   # the single instance the pipeline uses

frozen=True means a check cannot mutate a threshold at run time, and defaults in the class body mean the whole criteria set appears in one diff when it changes. Expected output: an object that serializes straight into the validation record with asdict.

2. Write checks that return findings #

Each check takes the reports and the criteria and returns a list of findings. It never raises for a quality problem — only for a structural one, such as a missing report.

python
from dataclasses import dataclass

@dataclass(frozen=True)
class Finding:
    rule: str
    detail: str
    observed: float
    limit: float

def check_accuracy(report, crit) -> list[Finding]:
    out = []
    for cls, s in report["by_class"].items():
        if s["n"] < crit.min_holdout_per_class:
            out.append(Finding("holdout_size", f"{cls}", s["n"],
                               crit.min_holdout_per_class))
        if s["lateral"]["p95"] > crit.lateral_p95_m:
            out.append(Finding("lateral_p95", f"{cls}", s["lateral"]["p95"],
                               crit.lateral_p95_m))
        if s["lateral"]["max"] > crit.lateral_max_m:
            out.append(Finding("lateral_max", f"{cls}", s["lateral"]["max"],
                               crit.lateral_max_m))
    return out

Findings carry the observed value and the limit, not a formatted string, so the record stays machine-readable and the human summary is a rendering of it rather than the other way round.

3. Run every check, then decide once #

python
CHECKS = (check_accuracy, check_coverage, check_bias, check_defects)

def run_gate(reports, crit=CRITERIA) -> tuple[bool, list[Finding]]:
    findings: list[Finding] = []
    for check in CHECKS:
        findings.extend(check(reports, crit))
    return (not findings), findings

One decision point, taken after every check has run. Nothing short-circuits, so the work list is complete on the first attempt.

4. Prove the gate rejects #

python
def assert_gate_rejects(fixtures, crit=CRITERIA) -> None:
    for name, reports in fixtures.items():
        ok, findings = run_gate(reports, crit)
        assert not ok, f"gate accepted known-bad fixture {name!r}"
        print(f"{name}: refused with {len(findings)} finding(s)")

Fixtures are small hand-built report dicts, one per defect class the gate claims to catch. Run this in the same CI job as the gate itself: if the gate changes and a fixture starts passing, the change is a regression regardless of how it looked in review.

The fixture suite as a coverage matrix — each row is a defect the gate asserts it can see:

Known-Bad Fixtures Against the Checks That Catch Them A six-by-four matrix with marks where a fixture is refused by a check, showing full coverage in both directions. accuracybias coveragedefects datum shift 0.08 m NE one control point at 0.9 m residential coverage 0.84 holdout of 40 points two open defects vertical p95 0.6 m every column has a fixture · every row is refused

5. Emit the record on every run #

python
import json, hashlib
from dataclasses import asdict

def write_record(path, ok, findings, crit, build) -> str:
    record = {
        "release": build.release_id,
        "toolchain": build.toolchain_digest,
        "criteria": asdict(crit),
        "passed": ok,
        "findings": [asdict(f) for f in findings],
    }
    blob = json.dumps(record, sort_keys=True, separators=(",", ":")).encode()
    path.write_bytes(blob)
    return hashlib.sha256(blob).hexdigest()

Writing on failure as well as success is what makes the record useful later: the interesting question during an incident is usually "what did the gate say about the release before this one", and a record that only exists for passing releases cannot answer it. The returned digest is hashed into the release root alongside the tiles, as HD map version control does for every other artefact.

Verification & Acceptance Criteria #

python
def assert_gate_contract(fixtures, good_reports) -> None:
    ok, findings = run_gate(good_reports)
    assert ok and not findings, f"gate rejected a known-good release: {findings}"
    assert_gate_rejects(fixtures)
    digest = write_record(tmp, ok, findings, CRITERIA, build)
    assert len(digest) == 64, "no record was written"

Acceptance gate: a known-good release passes with zero findings; every fixture is refused; a record is written on both paths; and the criteria in the record match the criteria in version control byte for byte.

Three ways to report a failing release, on the same eleven defects:

Reporting Style and How Many Build Cycles It Costs Three rows pairing a reporting style with the number of build cycles a release with eleven defects would take. three failure-reporting styles on a release with eleven defects exception on first finding raise and stop one build cycle per defect 11 cycles warning, run continues print and carry on the decision moves to the reader 0 cycles, 0 fixes findings returned, one verdict collect then decide the whole work list in one run 1 cycle the middle row is the dangerous one: it looks like the bottom row and behaves like no gate at all

Common Errors & Fixes #

The gate has never failed a real release. Either the map is perfect or a comparison is inverted. The fixture suite distinguishes the two in seconds.

A threshold was relaxed and nobody noticed. Criteria were constructed at the call site instead of imported from the frozen instance. Import CRITERIA; make the constructor private if the team keeps reaching for it.

Findings are strings and the dashboard parses them with regexes. Return structured findings and render for humans at the edge, never the reverse.

Runs are slow enough that people skip them. Checks are independent, so run them concurrently — the gate is I/O bound on report loading, not CPU bound.

A release passed with a stale accuracy report. The gate consumed a report from a previous build. Bind reports to the build digest and refuse a report whose build does not match.

FAQ #

Why should checks return findings instead of raising? #

An exception stops at the first problem, so a release with eleven defects takes eleven build cycles to clean up and each cycle hides whatever comes after it. Returning a list lets one run report everything, which turns the gate from a tripwire into a work list. It also makes the validation record complete — the record should say what the run found, not what it found first.

How do you know the gate actually works? #

By feeding it releases it must refuse. A gate that has only ever seen good input has never demonstrated it can reject anything, and a check with an inverted comparison or a threshold in the wrong units passes silently forever. Keep a fixture set — a shifted datum, a coverage regression on one road class, a withdrawn lane still present — and assert on every pipeline change that each fixture is still refused.

Should a gate ever be overridden? #

Only by changing the criteria, in version control, with review. An ad-hoc override flag is a threshold with no history: nobody can later tell whether a release shipped because it met the bar or because somebody passed a flag. If a threshold is genuinely wrong, the fix is to change it and re-run, which leaves exactly the record an audit needs.

Up one level: HD Map Quality Assurance & Certification — the assurance stage this gate enforces.