A city extract parses in a comfortable Python handler with no thought given to memory. A continent does not, and the failure is abrupt: the process grows steadily for an hour and then the kernel kills it. This page covers the design that makes large extracts tractable, as part of building directed graphs from OSM PBF files within OSM Graph Architecture & Network Modeling.

Three decisions account for nearly all of it: what you filter before Python sees the data, which node-location index you choose, and whether you accumulate output or stream it.

When to use this approach

Below roughly ten million nodes — a large metro or a small country — none of this is needed and the straightforward handler is easier to read. Above that, the techniques become necessary in a fairly predictable order: pre-filtering first, then a disk-backed index, then streaming output.

The signal that you have crossed the threshold is a build whose resident memory grows linearly and never plateaus. A handler with bounded memory shows a rise during the way pass and then a flat line; one without shows a straight climb to whatever the machine has.

Two memory profiles for the same extract An accumulating handler grows linearly for the whole run and is killed by the kernel at the machine's limit. A streaming handler with a disk-backed node index rises during the way pass and then holds flat, finishing comfortably inside the same budget. Resident memory across a continental build, 32 GB host 32 GB 0 start way pass done host limit accumulating — killed here streaming with a disk-backed index — flat at 11 GB The plateau is the diagnostic. A handler that plateaus will finish on any machine large enough for the plateau; one that climbs will not finish anywhere. Measure at the peak rather than at exit — a handler that frees its accumulator before returning still needed the memory.

Implementation

Pre-filtering happens in C++, before Python is involved at all.

# Drop everything that cannot participate in a routing graph
osmium tags-filter planet.osm.pbf \
  w/highway=motorway,trunk,primary,secondary,tertiary,unclassified,residential,service,living_street,motorway_link,trunk_link,primary_link,secondary_link,tertiary_link \
  r/type=restriction \
  --output planet-routable.osm.pbf --overwrite

osmium fileinfo -e planet-routable.osm.pbf

Including r/type=restriction keeps the relations a turn-restriction pass needs later, which is easy to forget and expensive to discover afterwards. The filter keeps the nodes those ways reference automatically, so the output remains self-consistent.

The location index is the single largest memory decision:

# requires: osmium (pip install pyosmium)
import osmium


def index_for(node_count_estimate: int) -> str:
    """Pick a node location index appropriate to the extract's size."""
    if node_count_estimate < 50_000_000:
        return "flex_mem"                    # fastest, fully in RAM
    # Disk-backed. Dense is faster where ids are contiguous (full planet);
    # sparse is right for clipped extracts, where ids are scattered.
    return "sparse_file_array,/scratch/node-locations.idx"


class WayHandler(osmium.SimpleHandler):
    """Stream ways straight to disk rather than accumulating them."""

    def __init__(self, sink) -> None:
        super().__init__()
        self.sink = sink
        self.count = 0

    def way(self, w) -> None:
        coords = [(n.lon, n.lat) for n in w.nodes if n.location.valid()]
        if len(coords) < 2:
            return
        # Write immediately; nothing accumulates in the handler
        self.sink.write(w.id, w.tags.get("highway"), coords)
        self.count += 1


handler = WayHandler(sink)
handler.apply_file(
    "planet-routable.osm.pbf",
    locations=True,
    idx=index_for(400_000_000),
)

The sink.write call is the whole point of the design. A handler that appends to self.edges holds every edge for the duration of the run; one that writes each edge as it is produced holds none. The sink can be a Parquet writer, a CSV file or a database cursor — what matters is that it does not keep what it has written.

Choosing between sparse_file_array and dense_file_array is worth a moment because the wrong choice can double the index size. Dense allocates a slot for every node id up to the maximum, which is efficient on a full planet where ids are essentially contiguous and wasteful on a clipped extract where they are scattered across the id space.

# requires: osmium (pip install pyosmium)
import osmium


def id_density(path: str, sample: int = 2_000_000) -> float:
    """Fraction of the id range actually occupied — decides sparse vs dense."""
    lo = hi = None
    seen = 0

    class Probe(osmium.SimpleHandler):
        def node(self, n):
            nonlocal lo, hi, seen
            lo = n.id if lo is None else min(lo, n.id)
            hi = n.id if hi is None else max(hi, n.id)
            seen += 1
            if seen >= sample:
                raise StopIteration

    try:
        Probe().apply_file(path)
    except StopIteration:
        pass
    span = (hi - lo) or 1
    return seen / span

A density above roughly 0.5 favours dense; below it, sparse. On a Geofabrik country extract the density is typically well under 0.01, which makes dense catastrophically wasteful — it is the classic reason a build that worked on the planet file fails on a smaller one.

Key parameters and tuning

Parameter Recommended value Notes
Pre-filter always Removes ~90 % of a planet file in C++ rather than Python
Index, under 50 M nodes flex_mem Fastest; RAM cost is acceptable at this scale
Index, clipped large extract sparse_file_array Scattered ids make dense wasteful
Index, full planet dense_file_array Contiguous ids make dense faster and compact
Index location fast local disk Random access; network storage makes it unusable
Output streamed Accumulation is the usual cause of unbounded growth
Batch flush every 100 k features Small enough to bound memory, large enough to amortise writes

Integration points

Into graph assembly. The streamed way file is the input to the directed-edge construction in building directed graphs from OSM PBF files. Keeping the two stages separate means the expensive parse runs once even when edge construction is re-run with different tag rules.

With relation parsing. Restriction relations need the way node lists cached during the way pass, as described in parsing OSM relations for advanced constraints. At planet scale that cache is itself large, which is another reason to write it to disk rather than hold it.

With partitioning. Once the parse is bounded, the natural next step is to partition the output geographically so downstream stages work on pieces that fit comfortably in memory — the same partitions that incremental graph rebuilds without full reprocessing reuses for rebuild scoping.

Sparse against dense, by id density Dense index memory is governed by the id range rather than the node count, so it is enormous on a clipped extract with scattered ids and efficient on a full planet file where ids are contiguous. Sparse index memory tracks node count regardless. Index memory against node id density 0.001 0.1 1.0 — full planet memory dense — governed by id range sparse — governed by node count crossover ≈ 0.5 A country extract sits far to the left, which is why dense — the choice that works on the planet file — fails on a smaller one. Probe the density once per extract rather than assuming; it takes seconds and removes the guess entirely. What the pre-filter removes before Python starts A planet file of 9.1 billion objects is reduced to 780 million by the highway filter, plus 1.4 million restriction relations, leaving under nine percent of the input for the Python handler to stream. Where a planet file’s objects go after pre-filtering raw planet 9.1 B objects after w/highway filter 780 M kept after r/type=restriction + 1.4 M relations reaching the handler 8.6 % of the input Doing this in C++ costs minutes. Doing the same selection in the Python handler costs hours and holds the discarded objects in memory on the way past. Keep the relation clause — dropping it removes the turn restrictions a later pass needs, and nothing downstream will report their absence.

Validation checklist

  • Memory plateaus. Sample resident size every ten seconds through a full run and confirm a flat section. A monotonic climb means something is still accumulating.
  • The plateau matches the index. Peak memory should be close to the index’s expected footprint. A large excess points at Python-side accumulation.
  • Pre-filtering did not drop relations. Count type=restriction relations before and after filtering. A zero afterwards means the r/ clause was omitted.
  • Way count survives the parse. Compare the handler’s count against osmium fileinfo on the filtered file. A shortfall means ways are being skipped for invalid locations, which points at the wrong index or a missing locations=True.
  • The index file is on local disk. Confirm the scratch path is not a network mount. Random access over the network turns a two-hour build into a two-day one.