Async Batch Geocoding with Nominatim

Sequential geocoding of a delivery address list — one HTTP call, wait, next call — turns a 20,000-row import into a multi-hour job even against a local server. This page covers the specific throttling and error-handling pattern for driving a self-hosted Nominatim instance at its real concurrency ceiling with asyncio and aiohttp, as one implementation inside batch geocoding pipelines for logistics. That guide covers the full pipeline — normalization, caching, and confidence filtering — while this page focuses narrowly on the concurrent request layer that pipeline depends on. The pattern fits into the broader automation approach described in routing API automation & fleet integration, where geocoded coordinates feed directly into route-matrix and dispatch workflows.

The core problem is not raw HTTP concurrency — aiohttp handles thousands of open sockets without effort — it’s that Nominatim’s backend has a hard concurrency ceiling set by its worker pool and Postgres connection limit. Send requests faster than that ceiling and you get connection resets or 429s instead of faster throughput. The fix is a semaphore-gated worker pool sized to match the server, combined with exponential backoff so transient overload degrades gracefully instead of cascading into a retry storm.

When to Use This Approach

Use an async semaphore-gated pipeline against Nominatim when:

• You control the Nominatim server (self-hosted or a private cluster) and can size the semaphore to its actual capacity, rather than being bound by a public instance’s 1 request/second usage policy.
• Address volumes are in the low thousands to low hundreds of thousands per run — large enough that sequential requests take minutes to hours, small enough that a compiled bulk-load path (e.g. importing directly into your own address index) is not worth building.
• Most addresses are structured (street, city, postcode, country) rather than free-text search strings — structured queries are cheaper for Nominatim to resolve and easier to validate downstream.
• You need per-address success/failure visibility rather than an all-or-nothing batch job, because failed geocodes feed a manual-review or fallback-geocoder step.

Prefer a different approach when the address list already has cached results — check deduplicating and caching geocoding results first and only send cache misses through this pipeline. If you are hitting the public nominatim.openstreetmap.org endpoint rather than a self-hosted instance, this pattern still applies but the semaphore must drop to 1 concurrent request with a mandatory 1-second spacing to comply with the usage policy — at that point a synchronous loop with time.sleep is simpler and just as fast, since the bottleneck is the policy, not your I/O model.

Implementation

The pipeline below issues concurrent structured queries against a local Nominatim /search endpoint, gates concurrency with asyncio.Semaphore, and retries transient failures with exponential backoff. It assumes addresses arrive as a list of dicts with street, city, postalcode, and country keys — normalization into that shape is covered in the parent guide, not repeated here.

# requires: aiohttp, asyncio (pip install aiohttp)
# Python 3.9+

import asyncio
import random
from dataclasses import dataclass, field
from typing import Any

import aiohttp

NOMINATIM_URL = "http://localhost:8080/search"
MAX_CONCURRENCY = 12
MAX_RETRIES = 5
BASE_BACKOFF = 1.0        # seconds
REQUEST_TIMEOUT = 10.0    # seconds


@dataclass
class GeocodeResult:
    address_id: str
    query: dict
    lat: float | None = None
    lon: float | None = None
    display_name: str | None = None
    importance: float | None = None
    error: str | None = None
    attempts: int = 0


async def geocode_one(
    session: aiohttp.ClientSession,
    sem: asyncio.Semaphore,
    address_id: str,
    query: dict,
) -> GeocodeResult:
    result = GeocodeResult(address_id=address_id, query=query)
    params = {**query, "format": "jsonv2", "limit": 1}

    async with sem:
        for attempt in range(1, MAX_RETRIES + 1):
            result.attempts = attempt
            try:
                async with session.get(
                    NOMINATIM_URL,
                    params=params,
                    timeout=aiohttp.ClientTimeout(total=REQUEST_TIMEOUT),
                ) as resp:
                    if resp.status == 429 or resp.status >= 500:
                        raise aiohttp.ClientResponseError(
                            resp.request_info, resp.history, status=resp.status
                        )
                    resp.raise_for_status()
                    payload = await resp.json()
                    if not payload:
                        result.error = "no_match"
                        return result
                    match = payload[0]
                    result.lat = float(match["lat"])
                    result.lon = float(match["lon"])
                    result.display_name = match.get("display_name")
                    result.importance = match.get("importance")
                    return result
            except (aiohttp.ClientError, asyncio.TimeoutError) as exc:
                if attempt == MAX_RETRIES:
                    result.error = f"failed_after_retries: {exc!r}"
                    return result
                # exponential backoff with jitter to avoid retry synchronization
                delay = BASE_BACKOFF * (2 ** (attempt - 1)) + random.uniform(0, 0.5)
                await asyncio.sleep(delay)

    return result


async def geocode_batch(
    addresses: list[tuple[str, dict]],
    concurrency: int = MAX_CONCURRENCY,
) -> list[GeocodeResult]:
    sem = asyncio.Semaphore(concurrency)
    connector = aiohttp.TCPConnector(limit=concurrency, limit_per_host=concurrency)

    async with aiohttp.ClientSession(connector=connector) as session:
        tasks = [
            geocode_one(session, sem, addr_id, query)
            for addr_id, query in addresses
        ]
        return await asyncio.gather(*tasks)


def split_results(results: list[GeocodeResult]) -> tuple[list[GeocodeResult], list[GeocodeResult]]:
    successes = [r for r in results if r.error is None]
    failures = [r for r in results if r.error is not None]
    return successes, failures


# Example run
addresses = [
    ("addr_001", {"street": "1600 Amphitheatre Pkwy", "city": "Mountain View",
                  "postalcode": "94043", "country": "US"}),
    ("addr_002", {"street": "221B Baker Street", "city": "London",
                  "postalcode": "NW1 6XE", "country": "GB"}),
]

results = asyncio.run(geocode_batch(addresses))
ok, failed = split_results(results)
print(f"geocoded {len(ok)} / {len(results)}, {len(failed)} failed")

The TCPConnector(limit=..., limit_per_host=...) argument matters as much as the semaphore — without it, aiohttp defaults to 100 total connections, which will happily open more sockets than your Nominatim worker pool can service even while the semaphore appears to be throttling application-level concurrency correctly.

Key Parameters and Tuning

Structured vs. free-text queries. Passing street, city, postalcode, and country as separate parameters (structured search) is both faster and more accurate than concatenating them into a single q= free-text string — Nominatim can prune its search space using the postcode and country before running the more expensive name-matching pass. Reserve free-text q= queries for addresses where field-level parsing failed upstream.

Semaphore sizing under load. Start conservatively (8) and increase in steps of 4 while watching Nominatim’s request queue depth and Postgres pg_stat_activity connection count. The right value is the point just before p95 latency starts climbing non-linearly — past that point you are queueing inside Nominatim rather than saving wall-clock time.

Integration Points

Feeding the cache layer. Every successful GeocodeResult should be written to the cache described in deduplicating and caching geocoding results before the coordinates are used anywhere else. This turns repeat imports of overlapping address lists (common with recurring delivery manifests) into cache hits instead of re-geocoding.

Feeding route-matrix construction. The (lat, lon) pairs produced here are the direct input to origin-destination matrix building. If the resulting point set exceeds a few hundred locations, batch it through parallel distance matrix requests with aiohttp — that page reuses the same semaphore/connector pattern shown here, applied to OSRM /table and Valhalla sources_to_targets calls instead of Nominatim /search.

Confidence filtering before routing. Do not pass every geocoded point straight into a routing engine. Filter on importance and check that display_name contains the expected city or postcode before snapping the point onto a road graph — a low-confidence match silently routed through OSRM produces a plausible-looking but wrong delivery stop.

Snapping to the graph. Once coordinates are validated, they still need to be matched to a routable node or way, not treated as exact positions. Nominatim’s returned point is a building centroid or address interpolation, not a graph node — expect a nearest-edge snap step downstream, similar in spirit to how bike-share stations are matched to OSM nodes in snapping bike-share stations to OSM nodes.

Validation Checklist

1. Concurrency matches server capacity. Load-test at your chosen MAX_CONCURRENCY and confirm p95 latency stays flat rather than climbing — a rising p95 under fixed concurrency means the server, not the network, is saturated.
2. Retry counts stay low in steady state. Log result.attempts across a production run; if the mean attempt count exceeds 1.5, the concurrency setting is too aggressive for the server’s real throughput.
3. No duplicate work from retries. Confirm the retry loop reuses the same address_id and does not spawn a second concurrent task for an address already in flight — the semaphore-per-task pattern above guarantees this, but custom retry wrappers can violate it.
4. Error results are structured, not swallowed. Every failed GeocodeResult should carry a non-null error string distinguishing no_match (Nominatim returned zero results) from failed_after_retries (transient failure) — these need different downstream handling.
5. Coordinates are WGS84 decimal degrees. Nominatim returns lat/lon as strings in WGS84; verify the cast to float and that no reprojection step is silently applied before storage.
6. Confidence threshold is enforced before routing. Spot-check a sample of accepted results against their display_name and importance score to confirm the threshold used in your pipeline actually filters out mismatched addresses.

Related

• Batch geocoding pipelines for logistics — the full pipeline this page’s request layer plugs into, including normalization and confidence filtering
• Deduplicating and caching geocoding results — cutting geocoding volume before it reaches this async layer
• Parallel distance matrix requests with aiohttp — the same semaphore/connector pattern applied to OSRM and Valhalla matrix endpoints
• Routing API automation & fleet integration — the overview covering how geocoded points feed matrix construction and dispatch