Sizing a routing deployment by guessing is common and expensive in both directions: too small and dispatch stalls at peak, too large and you pay for idle memory all year. A load test answers the question directly, and the work is in making the load resemble production rather than in running the tool. This page covers that, as part of comparing routing engines for production within Routing API Automation & Fleet Integration.

It complements the accuracy-focused harness in benchmarking routing engine latency and accuracy: that measures how good and how fast, this measures how much.

When to use this approach

Run a load test before committing to an instance size, after any change to the graph or engine version, and before a known demand step such as onboarding a large customer. The result — a capacity figure per instance — is what turns autoscaling policy from a guess into arithmetic.

The mix matters more than the volume. A workload that is ninety percent single routes and ten percent matrix requests behaves nothing like one that is fifty-fifty, because a matrix request costs one to two orders of magnitude more. Sampling the mix from production access logs is the difference between a useful number and a reassuring one.

Finding the knee p50 stays flat well past the point where p99 begins to rise. The knee, where p99 first exceeds about one and a half times its baseline, sits at around 210 requests per second; beyond it throughput plateaus while latency climbs steeply. Latency against offered load, single 8-core instance 50 180 280 400 req/s latency p99 p50 — still flat well past the knee knee ≈ 210 req/s provision here — 70 % of the knee Watching p50 alone would suggest capacity around 350 req/s, at which point one request in a hundred is taking several seconds. Dispatch experiences the tail, not the median, which is why the knee is read from p99.

Implementation

The locustfile encodes the mix, and the query set is sampled from production rather than invented.

# requires: locust (pip install locust)
import json
import random
from locust import HttpUser, task, between, events

# Sampled from production access logs, not synthesised
with open("od_pairs.json") as fh:
    PAIRS = json.load(fh)          # [{"from": [lon, lat], "to": [lon, lat]}, ...]


class RoutingUser(HttpUser):
    # Real clients think between requests; zero wait measures a different system
    wait_time = between(0.2, 1.5)

    @task(9)
    def single_route(self) -> None:
        p = random.choice(PAIRS)
        coords = f"{p['from'][0]},{p['from'][1]};{p['to'][0]},{p['to'][1]}"
        with self.client.get(
            f"/route/v1/driving/{coords}",
            params={"overview": "false"},
            name="/route",
            catch_response=True,
        ) as r:
            if r.status_code != 200 or r.json().get("code") != "Ok":
                r.failure(f"route failed: {r.status_code}")

    @task(1)
    def small_matrix(self) -> None:
        pts = random.sample(PAIRS, 12)
        coords = ";".join(f"{p['from'][0]},{p['from'][1]}" for p in pts)
        with self.client.get(
            f"/table/v1/driving/{coords}",
            name="/table",
            catch_response=True,
        ) as r:
            if r.status_code != 200:
                r.failure(f"table failed: {r.status_code}")

The @task weights are the mix, and they should come from counting endpoints in production logs rather than from intuition. A nine-to-one route-to-matrix ratio is common for dispatch platforms and produces a very different capacity figure from ninety-nine-to-one.

catch_response=True with an explicit failure check matters because a routing engine under load frequently returns 200 with a NoRoute body rather than an error status. Counting those as successes inflates the apparent capacity precisely where the engine is starting to struggle.

wait_time is the parameter most often set to zero, and doing so measures a system nobody operates. Real clients pause; a zero-wait test saturates with far fewer simulated users and reads a capacity figure that does not transfer.

Ramping in steps and holding each is what makes the knee visible:

# Step ramp: hold each level long enough for percentiles to settle
for users in 20 40 80 120 180 260 360; do
  locust -f locustfile.py --headless \
    --host http://osrm:5000 \
    --users "$users" --spawn-rate 20 \
    --run-time 2m \
    --csv "run_${users}" --only-summary
done

Two minutes per step is a floor rather than a target. Percentiles computed over a shorter window read the transient, and the transient is optimistic because internal queues have not yet reached their steady depth.

Two properties of the harness itself deserve as much attention as the engine under test. The first is that the load generator must not become the bottleneck: a single Locust process saturates a core well before a well-tuned engine saturates a machine, and a flattening throughput curve caused by the client looks identical to one caused by the server. Running the generator distributed, on separate hardware from the engine, removes the ambiguity. The second is that the coordinates driving the requests should be drawn from real traffic rather than sampled uniformly from the bounding box. Uniform sampling puts most origins in fields, where the graph is sparse and queries are cheap, and it will report a capacity figure the production mix never reaches.

Finally, hold the graph, the hardware, and the request mix fixed across any two runs you intend to compare. A capacity number is only ever a number for one configuration, and changing two things between runs produces a difference that cannot be attributed to either.

Key parameters and tuning

Parameter Recommended value Notes
Query set sampled from production Synthetic pairs miss the real length distribution
Task weights counted from access logs A matrix request costs 10–100× a single route
wait_time 0.2–1.5 s Zero measures a system nobody operates
Step duration 2 min minimum Shorter windows read the transient, which is optimistic
Ramp shape discrete steps A continuous ramp blurs the knee
Failure detection explicit body check A 200 with NoRoute is a failure, not a success
Provisioning target 70 % of the knee Leaves headroom for the burst the mean does not show

Integration points

With engine selection. Capacity per instance is one of the decision criteria in comparing routing engines for production, and it is the one that turns a latency difference into a cost difference. An engine three times slower needs three times the instances for the same throughput.

With the concurrency budget. The knee is the number that should set the client-side semaphore described in async route matrix automation. Sizing the client from the engine’s measured capacity rather than from its core count is what stops the automation layer driving the engine past its own knee.

With graph promotions. Re-run the test after any graph rebuild that changes size materially. Capacity is a function of the graph as well as the hardware, and a national extract replacing a regional one moves the knee — which is worth knowing before rather than after the promotion described in blue-green OSRM graph swaps without downtime.

The mix decides the capacity figure A route-only test measures 640 requests per second. A nine-to-one route-to-matrix mix measures 210. A five-to-one mix measures 96. All three are the same engine on the same hardware. Same engine, same hardware, three request mixes routes only 640 req/s 9:1 route:matrix 210 req/s — the production mix 5:1 route:matrix 96 req/s A route-only test would size the fleet at a third of what it needs, and the shortfall appears only at peak. Count the endpoints in a week of access logs before writing the locustfile — it is ten minutes that decides the whole result. Capacity translated into instance count Serving 600 requests per second at the production mix needs three OSRM instances, nine GraphHopper instances, or twenty-one Valhalla instances, once each engine’s measured knee is derated to seventy percent. Instances needed for 600 req/s at the production mix OSRM — knee 210/s 3 instances at 70 % headroom GraphHopper — knee 96/s 9 instances Valhalla — knee 41/s 21 instances This is the arithmetic that turns a latency comparison into a cost one, and it is usually more decisive than any feature difference. Re-measure after a graph change — capacity is a property of the graph and the hardware together, not of the engine alone.

Validation checklist

  • The query set matches production length distribution. Compare the sampled pairs’ straight-line distances against production. A test dominated by short routes overstates capacity.
  • Non-200 and NoRoute both count as failures. Confirm the failure rate is non-zero past the knee. A flat zero means the body check is missing.
  • Percentiles are stable within a step. Compare the first and second minute of a step. A large difference means the step is too short.
  • The client is not the bottleneck. Watch load-generator CPU. Above about seventy percent, the numbers describe Locust rather than the engine, and the test needs distributing.
  • The knee is reproducible. Run the ramp twice. A knee that moves by more than ten percent between runs means something else on the host is interfering.