Benchmarking Routing Engine Latency and Accuracy

Choosing between engines on paper only gets you so far — the comparing routing engines for production guide covers feature coverage and architectural trade-offs, but SLA sign-off requires numbers measured against your own graph, your own query distribution, and your own concurrency profile. This page is the benchmarking harness that produces those numbers, and it sits inside the broader Routing API Automation & Fleet Integration toolchain alongside geocoding, matrix, and re-routing automation that all depend on knowing which engine can sustain load.

The harness below answers two separate questions that get conflated far too often: how fast is each engine under realistic concurrency, and how close is each engine’s answer to a trusted ground truth? A route that returns in 8ms but is 40% longer than the correct path is not a win. The implementation captures both dimensions from the same request set so latency and accuracy numbers are directly comparable per engine, per concurrency level.

When to Use This Approach

Run this harness, not a single-shot curl timing loop, whenever:

• You are choosing between osrm-routed, Valhalla, and GraphHopper for a production deployment and need defensible numbers, following on from OSRM vs Valhalla vs GraphHopper for freight routing.
• You changed a cost profile, re-extracted an OSM region, or upgraded an engine version and need a regression gate before promoting the build.
• You need to size a concurrency ceiling — how many simultaneous route requests a single engine instance can serve before p95 latency breaks your SLA.
• Product or ops is asking “is engine X actually faster than engine Y in our region” and a napkin comparison from vendor benchmarks won’t satisfy the question.

Skip it for one-off debugging of a single slow request — use engine-native profiling (osrm-routed --verbosity) or request tracing instead. This harness is for comparative, repeatable measurement across many requests, not root-causing a single outlier.

Implementation

The harness sends the same query set at each concurrency level to every engine in turn, timing each request individually and comparing the returned distance and duration against a pre-computed ground-truth file. It assumes engines are already deployed — see deploying OSRM with Docker for local routing for that step — and focuses only on the measurement layer.

# requires: aiohttp, numpy (pip install aiohttp numpy)
# Python 3.9+

import asyncio
import time
import json
from dataclasses import dataclass, field
from typing import Optional

import aiohttp
import numpy as np

ENGINES = {
    "osrm": "http://osrm-host:5000/route/v1/driving/{lon1},{lat1};{lon2},{lat2}?overview=false",
    "valhalla": "http://valhalla-host:8002/route",
    "graphhopper": "http://gh-host:8989/route",
}

CONCURRENCY_LEVELS = (1, 4, 16, 64)
WARMUP_REQUESTS = 50
REQUEST_TIMEOUT_S = 5.0


@dataclass
class QueryResult:
    engine: str
    concurrency: int
    latency_ms: float
    distance_km: Optional[float]
    duration_s: Optional[float]
    ground_truth_distance_km: float
    ground_truth_duration_s: float
    error: Optional[str] = None


async def fetch_osrm(session: aiohttp.ClientSession, o: tuple, d: tuple) -> dict:
    url = ENGINES["osrm"].format(lon1=o[1], lat1=o[0], lon2=d[1], lat2=d[0])
    async with session.get(url, timeout=REQUEST_TIMEOUT_S) as resp:
        payload = await resp.json()
        route = payload["routes"][0]
        return {"distance_km": route["distance"] / 1000, "duration_s": route["duration"]}


async def timed_request(
    session: aiohttp.ClientSession, engine: str, concurrency: int,
    sem: asyncio.Semaphore, query: dict,
) -> QueryResult:
    async with sem:
        t0 = time.perf_counter()
        try:
            result = await fetch_osrm(session, query["origin"], query["dest"])
            latency_ms = (time.perf_counter() - t0) * 1000
            return QueryResult(
                engine=engine, concurrency=concurrency, latency_ms=latency_ms,
                distance_km=result["distance_km"], duration_s=result["duration_s"],
                ground_truth_distance_km=query["gt_distance_km"],
                ground_truth_duration_s=query["gt_duration_s"],
            )
        except Exception as exc:
            latency_ms = (time.perf_counter() - t0) * 1000
            return QueryResult(
                engine=engine, concurrency=concurrency, latency_ms=latency_ms,
                distance_km=None, duration_s=None,
                ground_truth_distance_km=query["gt_distance_km"],
                ground_truth_duration_s=query["gt_duration_s"],
                error=str(exc),
            )


async def run_sweep(engine: str, queries: list[dict]) -> list[QueryResult]:
    results: list[QueryResult] = []
    async with aiohttp.ClientSession() as session:
        # Warmup — discarded, not timed into results
        warm_sem = asyncio.Semaphore(8)
        await asyncio.gather(*(
            timed_request(session, engine, 0, warm_sem, q)
            for q in queries[:WARMUP_REQUESTS]
        ))

        for c in CONCURRENCY_LEVELS:
            sem = asyncio.Semaphore(c)
            batch = await asyncio.gather(*(
                timed_request(session, engine, c, sem, q) for q in queries
            ))
            results.extend(batch)
    return results


def summarize(results: list[QueryResult]) -> dict:
    ok = [r for r in results if r.error is None]
    latencies = np.array([r.latency_ms for r in ok])
    dist_delta_pct = np.abs(
        np.array([r.distance_km for r in ok])
        - np.array([r.ground_truth_distance_km for r in ok])
    ) / np.array([r.ground_truth_distance_km for r in ok]) * 100
    dur_delta_pct = np.abs(
        np.array([r.duration_s for r in ok])
        - np.array([r.ground_truth_duration_s for r in ok])
    ) / np.array([r.ground_truth_duration_s for r in ok]) * 100

    return {
        "n": len(results),
        "error_rate_pct": (len(results) - len(ok)) / len(results) * 100,
        "p50_ms": round(float(np.percentile(latencies, 50)), 2),
        "p95_ms": round(float(np.percentile(latencies, 95)), 2),
        "p99_ms": round(float(np.percentile(latencies, 99)), 2),
        "distance_delta_p50_pct": round(float(np.percentile(dist_delta_pct, 50)), 2),
        "duration_delta_p50_pct": round(float(np.percentile(dur_delta_pct, 50)), 2),
    }


async def main():
    with open("queries_with_ground_truth.json") as fh:
        queries = json.load(fh)

    for engine in ENGINES:
        results = await run_sweep(engine, queries)
        for c in CONCURRENCY_LEVELS:
            subset = [r for r in results if r.concurrency == c]
            print(engine, c, summarize(subset))


if __name__ == "__main__":
    asyncio.run(main())

Swap fetch_osrm for a Valhalla or GraphHopper adapter that parses each engine’s native response shape — the timing, semaphore gating, and summarization logic stays identical across engines, which is what makes the resulting numbers comparable.

Key Parameters and Tuning

Semaphore vs connector limit. asyncio.Semaphore(c) bounds in-flight requests logically; also set aiohttp.TCPConnector(limit=c) on the session so the underlying connection pool doesn’t silently cap concurrency below what you intend to measure — a mismatch here produces latency numbers that look better than what the engine can actually sustain.

Percentile choice. p50 tells you typical experience; p95 and p99 tell you what your slowest dispatch calls look like under load — the numbers that actually blow SLAs during peak hours. Report all three; a single average masks tail behavior that matters most for fleet dispatch.

Integration Points

The harness output is a table of (engine, concurrency, p50_ms, p95_ms, p99_ms, distance_delta_pct, duration_delta_pct, error_rate_pct) rows — plain enough to feed several downstream systems:

• CI regression gate. Run the sweep against a staging engine after every OSM extract or costing-profile change and fail the pipeline if p95 latency regresses beyond a threshold or accuracy delta exceeds tolerance. This is the natural complement to the manual comparison in OSRM vs Valhalla vs GraphHopper for freight routing — that page tells you what to compare, this harness produces the numbers.
• Capacity planning. The concurrency level where p95 latency inflects sharply upward is your practical per-instance ceiling; use it to size horizontal scaling for the async matrix and geocoding workers described across the routing automation stack.
• Dashboarding. Emit each summary row as a JSON line and ship it to a time-series store (Prometheus pushgateway, InfluxDB) so latency and accuracy trends are visible across engine versions and OSM extract dates, not just single point-in-time runs.
• Ground-truth generation. If you don’t have a trusted reference, generate one by running the same query set once against a well-audited engine build (or a commercial routing API) immediately after a known-good extract, then freeze that file and re-derive it only when the graph itself changes.

Validation Checklist

1. Warmup is excluded from every reported percentile. Confirm the summarization step slices warmup requests out before calling np.percentile — mixing them in silently inflates p50.
2. Concurrency levels match the connector limit. Verify aiohttp.TCPConnector(limit=c) is set per sweep level; otherwise the semaphore is bottlenecked below the intended concurrency and your numbers understate real throughput.
3. Error rate is tracked separately from latency. A request that times out should not appear in the latency array as a huge outlier — it should increment error_rate_pct and be excluded from percentile math.
4. Ground truth is fresh relative to the extract under test. Re-run ground-truth generation any time the OSM extract, costing profile, or engine version changes; a stale baseline produces phantom accuracy regressions.
5. Percentiles are stable across repeated runs. Run the full sweep twice back-to-back; p50 and p95 should agree within a few percent. Large swings indicate an undersized query set or host-level noise (co-located processes, thermal throttling).
6. Client host is not the bottleneck. Confirm the benchmarking client’s own CPU and network utilization stay well under saturation during the highest concurrency level — otherwise you are measuring the client, not the engine.

Related

• Comparing routing engines for production — the decision framework this harness supplies measured evidence for
• OSRM vs Valhalla vs GraphHopper for freight routing — the feature-by-feature comparison this benchmark data validates
• Deploying OSRM with Docker for local routing — standing up the engine instances this harness measures
• Routing API Automation & Fleet Integration — the broader automation stack that depends on knowing which engine sustains production load