VehicleMapping HD AV Spatial Data

Scoring and Triaging Map Change Candidates

A fleet produces far more change candidates than a survey team can look at, and the ordering of that queue determines whether map maintenance works. Get it wrong and the team spends its week on repainted parking bays while a lane closure sits at position four hundred.

This task turns the raw candidates from detecting HD map changes from fleet LiDAR into a ranked, workable queue for change detection and map maintenance — merged by identity, weighted by the quality of the observation, gated on independence, and ordered by what ignoring each one would cost.

Why confidence and consequence order the queue differently:

Confidence against Consequence for Six Candidates A scatter of candidates by confidence and consequence, with the two resulting queue orderings listed beneath. confidence → high low consequence lane closure, hourly route 0.72 conf repainted parking bay · 0.98 new kerb, weekly route by confidence 1 parking bay2 … 3 …4 new kerb 5 lane closure by consequence 1 lane closure2 new kerb 3 …4 … 5 parking bay confidence still gates promotion — it just does not order what a surveyor sees first ranking by confidence sorts by how easy each change was to detect, which nobody is trying to optimise

Prerequisites #

  • Python 3.10+, NumPy 1.24+.
  • Input: candidate observation records with stable identifiers, pose uncertainty, vehicle id and timestamp; plus route exposure statistics.
  • Upstream stage: onboard detection.
  • Output: a ranked queue of merged candidates with confidence and consequence.

Step-by-Step #

1. Merge observations by identity #

python
from collections import defaultdict

def merge(observations) -> dict[str, list]:
    """Group observations of the same change, whoever saw it."""
    out = defaultdict(list)
    for o in observations:
        out[o["id"]].append(o)
    return dict(out)

This works only because the detector derives identifiers from the tile, kind and affected feature rather than from the observer. An identifier that includes the vehicle turns agreement into a set of singletons, and nothing ever promotes.

2. Weight each observation by localization quality #

python
import numpy as np

def obs_weight(o, sigma_ref: float = 0.10) -> float:
    """Down-weight observations made while the pose was uncertain."""
    return float(sigma_ref / max(o["pose_sigma_m"], sigma_ref))

Weighting rather than filtering matters: the places hardest to localize in are often the places most in need of maintenance, and a hard filter removes the only evidence available there. A 0.9 m observation still counts, at about a ninth of a good one.

3. Require independence across vehicles and passes #

python
def confidence(obs) -> float:
    w = np.array([obs_weight(o) for o in obs])
    vehicles = len({o["vehicle"] for o in obs})
    agree = np.average([o["agrees_with_consensus"] for o in obs], weights=w)
    return float(agree
                 * (1 - np.exp(-vehicles / 2.0))
                 * (1 - np.exp(-w.sum() / 4.0)))

The pass term is driven by the weighted count, so four poor observations do not carry the weight of four good ones. The vehicle term is deliberately unweighted — independence is a structural property, and a poor observation from a second rig is still a second rig.

4. Rank by consequence #

python
SEVERITY = {"removed_lane": 1.0, "moved_boundary": 0.7, "added_boundary": 0.6,
            "changed_marking": 0.4, "repainted_bay": 0.1}

def consequence(cand, exposure_per_day: float) -> float:
    """What ignoring this change would cost: severity times how often it is met."""
    kind = cand["kind"]
    return SEVERITY.get(kind, 0.3) * np.log1p(exposure_per_day)


def queue(candidates, exposure, threshold: float = 0.6) -> list:
    promoted = [c for c in candidates if c["confidence"] >= threshold]
    return sorted(promoted, key=lambda c: -consequence(c, exposure[c["tile"]]))

log1p on exposure keeps a route driven a thousand times a day from dominating one driven fifty times by a factor of twenty — the difference matters, but not that much, and a linear term makes the queue nothing but the busiest corridor.

What the two-term confidence model does to a week of candidates:

One Week of Candidates Through the Two-Term Model Candidate counts split by which term they failed, with the promoted group's precision and the real changes lost to the independence requirement. 1 290 single-vehicle — never promote contains 9 real changes 402 too few passes will promote if they persist 148 promoted · 121 confirmed real · precision 0.82 the 9 real changes in the single-vehicle group are the price of the independence requirement they are found later, by a second vehicle or a scheduled survey — which is cheaper than an unworkable queue

Verification & Acceptance Criteria #

python
def assert_triage(queue, labelled, exposure) -> None:
    conf = [c for c in queue]
    tp = sum(1 for c in conf if c["id"] in labelled.real)
    precision = tp / max(len(conf), 1)
    assert precision >= 0.80, f"precision {precision:.2f} — the queue will be abandoned"

    for c in conf:
        assert len({o["vehicle"] for o in c["observations"]}) >= 2, \
            f"{c['id']}: promoted on a single vehicle"

    cons = [consequence(c, exposure[c["tile"]]) for c in queue]
    assert cons == sorted(cons, reverse=True), "queue is not ranked by consequence"

Acceptance gate: precision ≥0.80 on the promoted set; no candidate promoted on a single vehicle, which is structural rather than statistical; and a queue ordered by consequence rather than by confidence or arrival.

What a candidate that never promotes is actually telling you:

What a Never-Promoting Candidate Means Four rows pairing a never-promoting candidate pattern with what it indicates and where it should be routed. four ways a candidate can fail to promote seen once, never again no persistence a dynamic object discard many passes, one vehicle no independence a sensor fault on that vehicle route to fleet maintenance slow accumulation, weeks never clears the bar a marginal geometric disagreement route to the accuracy queue single-vehicle route independence unreachable a coverage problem, not evidence send a survey vehicle only the first should be deleted — the other three are findings about the fleet or the map rather than about the road

Common Errors & Fixes #

Nothing ever promotes. Identifiers include the vehicle or the timestamp, so every observation is its own candidate. Fix the identifier derivation in the detector.

A faulty rig promoted a dozen candidates. The vehicle term is missing, or agreement is being counted over observations rather than over vehicles. Count distinct vehicles.

The queue is all one corridor. Exposure is entering linearly. Use log1p.

Precision is high and surveyors still complain. The queue is precise and badly ordered — everything in it is real and the important ones are at the bottom. Check the ordering, not the threshold.

Candidates in an urban canyon never promote. Poor pose uncertainty is being filtered rather than weighted. Weight it; those are the places most likely to need maintenance.

FAQ #

Why rank by consequence rather than by confidence? #

Because confidence says how sure you are, not how much it matters. A candidate at 0.98 confidence describing a repainted parking bay is worth less surveyor time than one at 0.72 describing a lane closure on a route the fleet uses hourly. Ranking by confidence sorts the queue by how easy each item was to detect, which correlates with nothing anybody cares about. Consequence — exposure times severity — is the ordering that gets the important thing looked at first.

How do you weight an observation by localization quality? #

By the pose uncertainty the localizer reported when the observation was made. A candidate detected while the lateral sigma was 0.05 metres is evidence; the same candidate detected at 0.9 metres in an urban canyon might just be the pose. Weighting by the inverse of that uncertainty, rather than filtering on it, keeps the poor observations contributing a little instead of throwing away the only evidence available in the places that are hardest to localize in.

What happens to candidates that never reach the threshold? #

They age out, and the ageing is itself a signal. A candidate that accumulates observations slowly over weeks and never promotes is usually a marginal geometric disagreement rather than a change, and its persistence tells you a lane's mapped geometry is slightly off. Sweeping those into a low-priority accuracy queue, rather than deleting them, turns a stream of near-misses into a list of places the map is imprecise.

Up one level: Change Detection & Map Maintenance — the stage this triage sits in the middle of.