VehicleMapping HD AV Spatial Data

Streaming Map Tiles to Vehicles over gRPC

A tile transport has one job that a file download does not: it must fail usefully. A vehicle whose link degrades halfway through a tile needs to keep driving on the map it already has, and it needs to resume rather than restart when the link returns. This task builds that transport for map tile serving and distribution — a server-streaming RPC with feature-aligned chunks, a hard deadline, staged admission and offset-based resume.

The 200 ms budget in this guide is the whole path: request, transfer, decode and admission. Transfer-only measurements routinely mislead, because on lane-dense tiles the decode is the larger half.

Where streaming wins, drawn against the same tile delivered as one response:

Unary Transfer-Then-Decode against Overlapped Streaming Two horizontal timelines against a 200 millisecond budget line, the first showing serial transfer and decode overrunning it, the second showing chunked transfer with overlapping decode finishing just inside it. 200 ms budget unary transfer 200 ms decode 60 usable at 260 ms — 60 ms past the budget streaming four chunks · 200 ms decode, overlapped usable at 210 ms — inside the budget only the last chunk’s decode sits outside the transfer window, so the overlap saves almost the whole decode the saving grows with tile size, which is exactly where the unary path fails first

Prerequisites #

  • Python 3.10+, grpcio 1.60+ and grpcio-tools, protobuf 4.x.
  • Input: tiles cut and addressed as in tiling HD maps with quadkeys and H3, each with a content digest.
  • Upstream stage: the delta selection that decided which quadkeys this vehicle is missing.
  • Output: tiles staged, verified against their digest and promoted into the onboard cache.

Step-by-Step #

1. Define a server-streaming service #

The response message carries the quadkey it belongs to, an opaque payload chunk, the byte offset of that chunk within the tile, and a final flag. Nothing about map semantics belongs in the transport.

protobuf
service TileService {
  rpc StreamTiles(TileRequest) returns (stream TileChunk);
}

message TileRequest {
  repeated string quadkeys = 1;
  map<string, uint64> resume_offsets = 2;   // quadkey -> bytes already held
  string encoding = 3;
}

message TileChunk {
  string quadkey = 1;
  uint64 offset  = 2;
  bytes  payload = 3;
  bool   is_final = 4;
  string digest  = 5;    // set on the final chunk only
}

resume_offsets is what turns a failed transfer into a cheap retry: the client tells the server how far it got, and the server seeks rather than re-sending.

2. Chunk on feature boundaries #

A chunk that ends mid-feature forces the client to buffer across messages, which removes the incremental decode that justified streaming.

python
CHUNK_TARGET = 128 * 1024

def feature_aligned_chunks(features: list[bytes], target: int = CHUNK_TARGET):
    """Yield byte chunks that always end on a complete feature."""
    buf, size = [], 0
    for blob in features:
        buf.append(blob)
        size += len(blob)
        if size >= target:
            yield b"".join(buf)
            buf, size = [], 0
    if buf:
        yield b"".join(buf)

A single feature larger than the target is emitted as its own oversized chunk rather than being split — correctness beats the size hint. Expected output: chunks of roughly 128 KiB, each independently decodable.

3. Stage the transfer and promote only on the final chunk #

The client writes into a staging buffer that the planner cannot see, verifies the digest, and only then admits the tile.

python
import hashlib
import grpc

def fetch_tiles(stub, quadkeys, held_offsets, deadline_s=0.2):
    """Stream tiles, returning only those that completed and verified."""
    staged: dict[str, bytearray] = {}
    done: dict[str, bytes] = {}
    request = TileRequest(quadkeys=quadkeys, resume_offsets=held_offsets,
                          encoding="protobuf")
    try:
        for chunk in stub.StreamTiles(request, timeout=deadline_s):
            buf = staged.setdefault(chunk.quadkey, bytearray())
            if chunk.offset != len(buf):
                staged.pop(chunk.quadkey)          # gap: abandon this tile
                continue
            buf.extend(chunk.payload)
            if chunk.is_final:
                blob = bytes(buf)
                if hashlib.sha256(blob).hexdigest() == chunk.digest:
                    done[chunk.quadkey] = blob
                staged.pop(chunk.quadkey)
    except grpc.RpcError as exc:
        if exc.code() is not grpc.StatusCode.DEADLINE_EXCEEDED:
            raise
    return done, {k: len(v) for k, v in staged.items()}

The second return value is the resume state — how far each incomplete tile got — and it is fed straight back as resume_offsets on the next poll. Key parameter: timeout is on the call, so it bounds the whole batch rather than a single message; size the batch so that its worst-case transfer fits.

4. Apply flow control rather than buffering #

gRPC's HTTP/2 flow control is per stream, and it works only if the client actually pauses. A client that drains the iterator into a list as fast as it arrives has disabled it.

python
for chunk in stub.StreamTiles(request, timeout=deadline_s):
    admit_chunk(chunk)      # do the decode work here, inline

Consuming the iterator synchronously and doing the decode inside the loop is what applies backpressure: the window does not reopen until the client reads, so a slow decoder slows the server. Handing chunks to a thread pool and returning immediately reintroduces the unbounded queue that backpressure and queue tuning for sensor streams exists to prevent.

The three outcomes a transfer can have, and what the vehicle holds after each:

What the Vehicle Holds After Each Transfer Outcome Three rows pairing a transfer outcome with the resident map state and the resume state it leaves behind. outcomeresident mapresume state complete + digest matches is_final seen, sha256 equal new tile promoted cleared — nothing outstanding deadline expired mid-tile bytes arrived, no is_final unchanged — previous tile still serves offset kept → resume next poll digest mismatch complete but wrong bytes unchanged — nothing is promoted discarded → clean re-fetch

Verification & Acceptance Criteria #

Exercise the transport against a link simulator, not against localhost, because every interesting behaviour here is a behaviour under loss.

python
def assert_transport_contract(client, manifest, link) -> None:
    link.profile("urban-4g")
    done, resume = client.poll()
    assert client.p95_latency_ms() <= 200, "p95 over the 200 ms budget"

    link.drop_after_bytes(400_000)               # cut mid-tile
    done, resume = client.poll()
    assert client.resident_digests() == client.previous_digests(), \
        "a partial tile was promoted"
    assert resume, "no resume offset recorded"

    link.profile("urban-4g")
    done, _ = client.poll()
    assert set(done) and client.bytes_transferred() < 400_000, \
        "resume re-sent the whole tile"

Acceptance gate: p95 end-to-end ≤200 ms on the reference profile; zero partial promotions under an induced mid-tile cut; a resume that transfers strictly fewer bytes than a fresh fetch; and a digest mismatch that leaves the resident map untouched.

What a resume actually saves, measured against the naive retry:

Bytes Re-Sent After a Mid-Tile Cut Two bars for the recovery transfer after an interruption, one re-sending the whole tile and one re-sending only the remainder. 8.4 MB delivered before the cut link drops naive retry 12 MB re-sent offset resume 3.6 MB the saving grows the later the cut lands, which is when a weak link is most likely to drop and it costs one integer per incomplete tile on each side

Common Errors & Fixes #

Latency is fine on the bench and misses the budget in the field. The bench measured transfer only. Instrument from request start to admission, including decode, and profile the densest tile rather than an average one.

A truncated tile reaches the planner. Promotion is happening per chunk instead of on is_final. Stage outside the resident map and promote once, after the digest check.

Resume re-sends the whole tile. The client is not returning its staged lengths, or the server ignores resume_offsets. Both halves are needed; a resume field the server does not honour is worse than none because it hides the cost.

Throughput collapses on a slow decoder. Chunks are being queued to a worker pool, so flow control never engages and memory grows until the process is killed. Decode inline in the iterator loop.

Every tile fails the digest check after an encoder change. The digest is over encoded bytes, so re-encoding changes it legitimately. Recompute the manifest as part of the release rather than patching the client to skip verification.

FAQ #

Why server streaming rather than a plain unary RPC? #

A unary RPC hands the client the whole tile at once, so decode cannot begin until transfer ends and the two costs add. A server-streaming RPC overlaps them: the client decodes the first features while the last are still on the wire, which on a 12 megabyte tile turns a 200 millisecond transfer plus a 60 millisecond decode into roughly 210 milliseconds total. Streaming also gives HTTP/2 flow control per message, so a slow client pushes back on the server instead of silently buffering.

What happens when the deadline expires mid-tile? #

The partial payload is discarded and the vehicle keeps whatever version of that tile it already held. A half-delivered tile must never be promoted into the resident map, because a truncated tile reads to the planner as a region with no lanes rather than as a region it lacks data for — which is the difference between degraded freshness and a fabricated obstacle. The next poll requests the remainder from the last completed feature offset.

How large should a chunk be? #

Large enough that per-message overhead is negligible and small enough that the client can do useful decode work between messages — in practice 64 to 256 kibibytes, ending on a complete feature. Chunks that split a feature force the client to buffer across messages, which removes the incremental-decode benefit that justified streaming in the first place. Measure with your own encoder rather than adopting a number, since the feature-boundary constraint dominates the choice.

Up one level: Map Tile Serving & Distribution — the distribution stage this transport implements.