Computing a travel-time matrix for 500 origins against 500 destinations means either one request that no engine will accept, or thousands of tiny requests that saturate a connection pool and starve each other. This page covers the specific mechanics of parallelizing origin-destination matrix requests with aiohttp inside the broader async route matrix automation workflow, which also covers semaphore-gated concurrency and chunk tiling at a conceptual level. It assumes the fleet automation context described in routing API automation and fleet integration — geocoded stops that need an NxN or NxM cost matrix before a solver can run.

The problem is not raw request volume; aiohttp can issue thousands of requests per second. The problem is that OSRM’s /table endpoint and Valhalla’s /sources_to_targets endpoint both enforce hard coordinate-count ceilings per request, and both degrade badly under uncontrolled concurrency — OSRM’s single-threaded Lua-free table handler queues behind itself, and Valhalla’s thread pool thrashes when the number of in-flight requests exceeds its worker count by a wide margin. Getting this right means matching three independent limits: the engine’s per-request coordinate cap, the engine’s server-side worker concurrency, and the client’s connection pool size.

When to Use This Approach

aiohttp parallel matrix request and NumPy assembly pipeline Four origin-destination chunk requests converge through a semaphore and TCPConnector gate into a single routing engine endpoint. Engine responses flow into a NumPy matrix, each chunk written into its own row band at the correct offset. O-D chunks chunk 1 (rows 0-49) chunk 2 (rows 50-99) chunk 3 (rows 100-149) chunk 4 (rows 150-199) TCPConnector(limit) + Semaphore(N) ≤ N in flight Routing engine OSRM /table Valhalla /sources_to_targets NumPy matrix chunk 1 chunk 2 chunk 3 chunk 4 Each chunk writes into its own row band at a known offset — no cross-chunk locking needed

Use parallel aiohttp fan-out when the origin-destination set exceeds a single engine request’s coordinate ceiling, or when a matrix that fits in one request still takes long enough (multiple seconds) that splitting it into concurrent sub-requests reduces wall-clock time. Typical triggers:

  • The combined origin and destination coordinate count exceeds OSRM’s --max-table-size (default 8000) or Valhalla’s configured max_matrix_distance / max_locations for the costing model in use.
  • A single-request matrix returns in acceptable time but you are computing matrices for many depots or many time-of-day snapshots back to back, and total batch throughput — not single-matrix latency — is the bottleneck.
  • The origin set changes faster than the destination set (or vice versa), so caching destination-side sub-matrices and only re-requesting the changed rows saves real request volume.

Skip this pattern when the full matrix fits comfortably under the engine’s request cap and the query is one-off — a single synchronous /table call is simpler and easier to debug. It also does not replace chunking large origin-destination matrices, which handles how chunk boundaries are chosen; this page assumes chunks already exist and focuses on the transport layer that fetches and reassembles them.

Implementation

The snippet below issues chunk requests concurrently against an OSRM /table endpoint (the same shape applies to Valhalla with a POST body swap, noted inline), gates concurrency independently from the connection pool, and writes each response directly into a pre-allocated NumPy matrix at its known offset — no post-hoc merge step required.

# requires: aiohttp, numpy (pip install aiohttp numpy)
# Python 3.9+

import asyncio
import numpy as np
import aiohttp

OSRM_BASE = "http://localhost:5000"
POOL_LIMIT = 20          # total open connections in the TCPConnector
CONCURRENCY = 12         # in-flight requests; must be <= POOL_LIMIT
REQUEST_TIMEOUT = aiohttp.ClientTimeout(total=30, connect=5, sock_read=20)


async def fetch_chunk(
    session: aiohttp.ClientSession,
    sem: asyncio.Semaphore,
    origins: list[tuple[float, float]],
    dests: list[tuple[float, float]],
    row_offset: int,
    col_offset: int,
    out: np.ndarray,
    attempt: int = 0,
) -> None:
    """Fetch one OSRM /table chunk and write durations directly into `out`."""
    coords = origins + dests
    coord_str = ";".join(f"{lon},{lat}" for lat, lon in coords)
    sources = ";".join(str(i) for i in range(len(origins)))
    destinations = ";".join(str(len(origins) + j) for j in range(len(dests)))
    url = f"{OSRM_BASE}/table/v1/driving/{coord_str}"
    params = {"sources": sources, "destinations": destinations, "annotations": "duration"}

    async with sem:
        try:
            async with session.get(url, params=params, timeout=REQUEST_TIMEOUT) as resp:
                resp.raise_for_status()
                payload = await resp.json()
        except (aiohttp.ClientError, asyncio.TimeoutError) as exc:
            if attempt < 3:
                await asyncio.sleep(0.5 * (2 ** attempt))
                return await fetch_chunk(
                    session, sem, origins, dests, row_offset, col_offset, out, attempt + 1
                )
            raise RuntimeError(f"chunk [{row_offset}:{col_offset}] failed after retries") from exc

    block = np.array(payload["durations"], dtype=np.float64)
    out[row_offset:row_offset + block.shape[0], col_offset:col_offset + block.shape[1]] = block


async def build_matrix(
    origin_chunks: list[list[tuple[float, float]]],
    dest_chunks: list[list[tuple[float, float]]],
) -> np.ndarray:
    n_rows = sum(len(c) for c in origin_chunks)
    n_cols = sum(len(c) for c in dest_chunks)
    matrix = np.full((n_rows, n_cols), np.nan, dtype=np.float64)

    connector = aiohttp.TCPConnector(limit=POOL_LIMIT, limit_per_host=POOL_LIMIT)
    sem = asyncio.Semaphore(CONCURRENCY)

    async with aiohttp.ClientSession(connector=connector) as session:
        tasks = []
        row_offset = 0
        for origins in origin_chunks:
            col_offset = 0
            for dests in dest_chunks:
                tasks.append(
                    fetch_chunk(session, sem, origins, dests, row_offset, col_offset, matrix)
                )
                col_offset += len(dests)
            row_offset += len(origins)
        await asyncio.gather(*tasks)

    return matrix

Valhalla variant. Swap session.get for session.post(f"{VALHALLA_BASE}/sources_to_targets", json={...}) with a body of {"sources": [{"lat": ..., "lon": ...}, ...], "targets": [...], "costing": "auto"}, and parse payload["sources_to_targets"] — a list of lists of {"time": ..., "distance": ...} dicts — into the same block-write pattern. Valhalla’s response nests time and distance per cell rather than returning two parallel arrays like OSRM, so extract with np.array([[c["time"] for c in row] for row in payload["sources_to_targets"]]).

Key Parameters and Tuning

Parameter Where set Typical value Effect
TCPConnector(limit=...) Client, once per session 15-30 Hard cap on total open connections; exceeding the engine’s worker thread count wastes sockets without adding throughput
TCPConnector(limit_per_host=...) Client equal to limit for single-host engines Prevents per-host throttling when all chunks target the same OSRM/Valhalla instance
asyncio.Semaphore(N) Client, wraps each request ≤ connector limit Caps requests actually issued at once; keeping it below the connector limit avoids queuing inside the pool
ClientTimeout(sock_read=...) Per-request or session default 15-25s Large chunks (near the coordinate ceiling) take longer per request; too low a sock_read timeout causes false-positive retries
Chunk cell budget Chunking layer, feeds this page 60-80% of --max-table-size Leaving headroom below the hard cap avoids 400 errors when coordinate counts vary slightly between chunks
Retry backoff fetch_chunk exponential, base 0.5s, max 3 attempts Absorbs transient 503s from an engine under momentary load without amplifying pressure via immediate retries
Where tasks wait with and without a semaphore With only a connector limit, gather schedules every chunk task at once and the surplus waits invisibly inside the connection pool, where the read timeout still applies. With a semaphore in front, the surplus waits at the semaphore, before any timeout has started. 64 chunk tasks, connector limit 20 — where do the other 44 wait? connector only gather · 64 tasks 44 blocked inside the pool 20 in flight to the engine the read timeout is already ticking for tasks that have not yet sent a byte — false retries follow semaphore first gather · 64 tasks 44 wait at the semaphore 20 in flight, each with a socket no timeout runs until a request is genuinely issued, so a retry means a real failure Both setups send the same number of requests. Only one of them can tell a slow engine apart from a saturated client. Keep the semaphore at or below the connector limit so the pool is never the thing that blocks.

Connector vs. semaphore, not either/or. A common mistake is setting only the TCPConnector limit and assuming it caps concurrency. It caps the connection pool, not in-flight coroutines — with no semaphore, asyncio.gather will schedule every chunk task immediately, and excess tasks block silently inside the connector waiting for a socket rather than failing or logging anything. Always pair the two, with the semaphore value at or below the connector limit.

Engine-side worker count matters more than client concurrency. OSRM’s HTTP server is typically compiled with a fixed thread pool (-t at launch, or environment default). Setting client concurrency higher than the OSRM thread count does not parallelize computation — it just queues requests server-side, which is functionally equivalent to sending them sequentially but with added HTTP overhead per request. Match CONCURRENCY to the engine’s actual worker count, verified with osrm-routed --help or the container’s startup logs, as described in deploying OSRM with Docker for local routing.

Retry policy is the other half of this, and the failure it has to distinguish is not transient versus permanent — it is engine under momentary load versus engine wedged. Exponential backoff with a small attempt ceiling separates them, because a genuinely transient 503 clears within a second or two while a wedged instance never does.

Retry timing against an engine that recovers after two seconds Three retry policies are plotted against a two-second outage. Immediate retry burns all attempts inside the outage and fails. Fixed 500 millisecond intervals also exhaust before recovery. Exponential backoff from 500 milliseconds spreads its attempts and succeeds on the third. Engine returns 503 for 2 s, then recovers — three retry policies engine unavailable 0 s 2 s 4 s 6 s immediate all three attempts inside the outage — fails fixed 500 ms exhausted at 1.0 s — still fails exponential third attempt lands after recovery — succeeds Add jitter so parallel chunks do not retry in lockstep — synchronised retries arrive as one spike and can re-trigger the outage they are waiting out.

Integration Points

Upstream: chunk boundaries. This page assumes origin_chunks and dest_chunks already exist as lists of coordinate lists sized under the engine’s per-request cap. See chunking large origin-destination matrices for how to compute chunk sizes from --max-table-size, exploit origin/destination symmetry, and reuse previously computed blocks.

Downstream: solver input. The assembled matrix array is a direct drop-in for OR-Tools’ RoutingIndexManager distance callback or any VRP solver expecting a dense cost matrix — index it as matrix[from_index][to_index] after converting to int (most solvers require integer costs; multiply by 100 and round if sub-second precision matters).

Engine choice affects chunk shape. If comparing engines for the same fleet workload, note that OSRM’s coordinate ceiling is a flat count across sources and destinations combined, while Valhalla enforces max_matrix_distance as a geographic bound in addition to a location count — a chunk sized correctly for OSRM may still be rejected by Valhalla if origins and destinations span a wide area. The trade-offs are covered in more depth for freight-specific routing in OSRM vs Valhalla vs GraphHopper for freight routing.

Caching partial results. Because each chunk writes to a known (row_offset, col_offset) block, a chunk-level cache keyed on the origin/destination coordinate hash lets you skip re-fetching unchanged blocks when only a subset of stops changes between runs — useful for daily depot-to-stop matrices where the depot side rarely moves.

Validation Checklist

  1. No NaN cells remain. After build_matrix completes without raising, assert not np.isnan(matrix).any(). Any surviving NaN marks a chunk that silently failed all retries and was swallowed — check asyncio.gather(*tasks, return_exceptions=True) output during development to catch this before it reaches production.

  2. Shape matches expected dimensions. matrix.shape == (n_origins, n_destinations) — a mismatch usually means a chunk’s coordinate order in the OSRM response did not match the order sources/destinations indices were built in.

  3. Semaphore value never exceeds connector limit. Assert this at startup: CONCURRENCY <= POOL_LIMIT. Violating it does not raise an error — it degrades silently into queuing, which is the hardest version of this bug to diagnose in production.

  4. Symmetric-route sanity check. For a handful of origin-destination pairs, compare matrix[i][j] against a direct single-pair /route request. Differences beyond a few percent usually indicate a coordinate order bug (lon,lat vs lat,lon — OSRM expects lon,lat in the URL, the opposite of most GeoJSON tooling).

  5. Timeout coverage under load. Run the full batch against a rate-limited or intentionally slow local proxy and confirm no request hangs past REQUEST_TIMEOUT.total. A missing sock_read timeout is the most common cause of a batch that never completes.

  6. Retry count stays bounded. Log retry attempts per chunk; if a large fraction of chunks hit the 3-attempt ceiling, CONCURRENCY is likely too high for the engine’s worker pool rather than the network being unreliable.

Why do some chunks return HTTP 400 while identical-sized neighboring chunks succeed?

This is usually a coordinate-count-off-by-one from chunk boundary math, not the engine. OSRM’s --max-table-size counts total coordinates (origins + destinations) in the request, not rows or columns separately. If chunk sizing was computed from --max-table-size without accounting for both dimensions combined, the last chunk in an uneven partition can exceed the cap by a handful of coordinates while earlier uniform chunks stay under it. Recompute chunk cell budgets as origin_chunk_len + dest_chunk_len, not origin_chunk_len * dest_chunk_len.