OSM Data Freshness and Incremental Updates

A routing graph is a snapshot of a dataset that thousands of people are editing continuously, and the gap between the snapshot and reality widens every hour it is left alone. Keeping that gap small is a distinct engineering problem from building the graph in the first place — it is about replication diffs, change classification, and rebuild scheduling rather than parsing and topology. This page sits within OSM Graph Architecture & Network Modeling and picks up where building directed graphs from OSM PBF files leaves off: you have a working pipeline that turns a PBF into a routable graph, and now you need it to keep producing a current one without reprocessing a continent every night.

The naive approach — download a fresh regional extract and rebuild everything — works and is genuinely the right answer for small extracts. It stops being the right answer once the rebuild takes longer than the interval you want between rebuilds, or once the bandwidth bill for re-downloading a multi-gigabyte extract daily becomes noticeable. Replication diffs solve both: OpenStreetMap publishes minute, hour and day change files, and applying a day’s worth of changes to an existing extract costs a fraction of re-downloading it.

Prerequisites

Tooling. osmium-tool 1.14+ provides apply-changes, derive-changes and merge-changes; pyosmium ships pyosmium-get-changes, which handles the replication-server bookkeeping. Both are available from conda-forge, which is the least painful installation route on most systems.

# Install the change-file toolchain
conda install -c conda-forge osmium-tool pyosmium

Data inputs. A base extract with a known replication timestamp, and the URL of a replication server covering it. Geofabrik publishes per-region replication directories alongside its extracts; the planet replication server at planet.openstreetmap.org/replication/ covers everything but requires you to clip each diff to your region.

State. A durable place to record the replication sequence number your extract currently reflects. A file on the same volume as the extract is sufficient and has the useful property of moving with it.

# Read the timestamp baked into an existing extract
osmium fileinfo -e region.osm.pbf | grep -i timestamp

Working space. Applying changes writes a complete new extract rather than modifying in place, so budget at least twice the extract size in free space, plus room for the downloaded diffs.

Conceptual architecture

The pipeline has four stages, and the important property is that only the last one is expensive. Fetching and applying diffs is cheap and can run frequently; classification decides whether the expensive stage runs at all; the rebuild is the only step whose cost scales with graph size.

The classification stage is the one people skip, and it is what makes the whole arrangement worthwhile. In a typical European metro region, a day of edits contains tens of thousands of object changes, of which a few hundred touch anything that changes a route: highway geometry, oneway, access, maxspeed, maxweight, turn restrictions, barriers. Everything else is buildings, addresses, shop opening hours and landuse polygons. Rebuilding a routing graph because someone corrected a café’s phone number is pure waste, and without classification that is exactly what a time-based schedule does.

Step-by-step implementation

1. Pin the replication sequence to the base extract

Every replication server exposes a state.txt giving the current sequence number and timestamp. The first job is to find the sequence corresponding to your extract’s timestamp, because that is where diff application must begin.

# requires: pyosmium, requests (pip install pyosmium requests)
from osmium.replication.server import ReplicationServer
import osmium
import datetime as dt

def find_start_sequence(pbf_path: str, server_url: str) -> int:
    """Return the replication sequence covering the extract's own timestamp."""
    # osmium records the extract's data timestamp in the file header
    header = osmium.io.Reader(pbf_path).header()
    stamp = header.get("osmosis_replication_timestamp")
    if not stamp:
        raise ValueError(
            f"{pbf_path} carries no replication timestamp — re-download from a "
            "source that sets one, or the diff start point cannot be determined."
        )
    ts = dt.datetime.fromisoformat(stamp.replace("Z", "+00:00"))
    server = ReplicationServer(server_url)
    seq = server.timestamp_to_sequence(ts)
    if seq is None:
        raise ValueError(
            "Extract is older than the replication server's retention window; "
            "a fresh base extract is required."
        )
    return seq

The failure mode worth guarding here is an extract with no replication timestamp, which some custom-clipped files lack. Without it there is no defensible starting sequence, and guessing produces either duplicated or — much worse — silently skipped changes.

2. Fetch and merge the interval’s diffs

Applying a hundred separate hourly diffs in sequence works but is needlessly slow. Merge them into one change file first, which also collapses objects edited repeatedly during the interval into a single final state.

# Fetch every diff since the recorded sequence into one merged change file
pyosmium-get-changes \
  --server https://download.geofabrik.de/europe/netherlands-updates/ \
  --start-id "$(cat replication.state)" \
  --outfile changes.osc.gz \
  --size 500

# Confirm what arrived before applying anything
osmium fileinfo -e changes.osc.gz

The --size cap bounds how much a single run will pull, which matters when catching up after an outage. A run that would otherwise try to fetch three months of diffs in one pass is better expressed as several bounded runs, each of which leaves the pipeline in a consistent state.

3. Apply the changes to the extract

# Write to a temporary path, never over the live extract
osmium apply-changes \
  region.osm.pbf changes.osc.gz \
  --output region.new.osm.pbf \
  --overwrite

# Verify before promoting
osmium fileinfo -e region.new.osm.pbf

# Promote atomically, then advance the recorded sequence
mv region.new.osm.pbf region.osm.pbf

Writing to a temporary path and moving it into place is not ceremony. apply-changes reads the input and writes the output as separate files, so an interrupted run leaves a truncated output; if that output was the live extract, the next pipeline stage silently consumes a partial dataset. The move is atomic on the same filesystem, so a reader either sees the old extract or the new one.

Region-clipped replication directories, such as Geofabrik’s -updates paths, contain only changes inside the region and can be applied directly. Planet-wide diffs cannot: they reference objects outside your clip, so the merged result needs re-clipping with a complete-ways strategy before it is usable.

4. Classify which changes actually matter

This is the step that turns a change file into a rebuild decision. Stream the diff, bucket each modified object by whether it touches a routing-relevant tag, and produce counts.

# requires: osmium (pip install pyosmium)
import osmium
from collections import Counter

ROUTING_KEYS = frozenset({
    "highway", "oneway", "access", "maxspeed", "maxheight", "maxweight",
    "maxaxleload", "hgv", "motor_vehicle", "junction", "barrier",
    "surface", "tracktype", "bridge", "tunnel", "toll",
})


class ChangeClassifier(osmium.SimpleHandler):
    """Count changed objects by whether they can alter a route."""

    def __init__(self) -> None:
        super().__init__()
        self.counts: Counter[str] = Counter()

    def _classify(self, obj, kind: str) -> None:
        tags = {t.k for t in obj.tags}
        if tags & ROUTING_KEYS:
            self.counts[f"{kind}:routing"] += 1
        else:
            self.counts[f"{kind}:other"] += 1

    def node(self, n) -> None:
        self._classify(n, "node")

    def way(self, w) -> None:
        self._classify(w, "way")

    def relation(self, r) -> None:
        # Restriction relations always matter, whatever else they carry
        if r.tags.get("type", "").startswith("restriction"):
            self.counts["relation:routing"] += 1
        else:
            self._classify(r, "relation")


handler = ChangeClassifier()
handler.apply_file("changes.osc.gz")

relevant = sum(v for k, v in handler.counts.items() if k.endswith(":routing"))
total = sum(handler.counts.values())
print(f"{relevant:,} routing-relevant of {total:,} changed objects")

Note that geometry changes reach this classifier as node modifications carrying no tags at all, and those are genuinely relevant when the node is a member of a highway way. Resolving that membership requires the way index, so a stricter classifier does a second pass; the tag-based count above is a deliberately cheap approximation that errs toward under-counting. Calibrate the threshold against that bias rather than against an idealised number.

5. Rebuild only what changed

For engines that partition the graph, the changed-object set maps onto a set of partitions, and only those need reprocessing. The mapping needs the changed nodes’ coordinates, which the merged extract now has.

# requires: geopandas, shapely (pip install geopandas shapely)
import geopandas as gpd
from shapely.geometry import Point

def affected_partitions(changed_coords: list[tuple[float, float]],
                        partitions: gpd.GeoDataFrame) -> set[str]:
    """Return the ids of partitions containing at least one changed element."""
    if not changed_coords:
        return set()
    pts = gpd.GeoDataFrame(
        geometry=[Point(lon, lat) for lon, lat in changed_coords],
        crs="EPSG:4326",
    ).to_crs(partitions.crs)
    hit = gpd.sjoin(pts, partitions, predicate="within", how="inner")
    return set(hit["partition_id"].unique())

Buffer the partition polygons before this join. A change one metre outside a partition boundary still affects routes that cross it, because the boundary node set on both sides is shared. A buffer roughly equal to the longest edge in the graph is a defensible default.

Configuration reference

Production optimization and scaling

Separate the fetch loop from the rebuild loop. They have different failure modes and different cadences. A fetch that fails should retry within minutes; a rebuild that fails should page someone. Running them as one job couples the two and makes the rebuild’s runtime the floor on freshness.

Keep the classification counts as a time series. The count of routing-relevant changes per interval is the single most useful operational signal this pipeline produces. A sudden spike means either a mapping party, an import, or a mass retagging — all of which warrant a look before their effects reach production weights.

Parallelise per partition, not per stage. Once the affected-partition set is known, each partition’s rebuild is independent. On a machine with cores to spare this turns a serial hour into a parallel ten minutes, and the partitions are already sized to fit in memory individually.

Cache the unchanged artefacts explicitly. The saving from incremental rebuilds only materialises if unchanged partitions’ outputs are reused rather than recomputed. Key the cache on partition id plus the content hash of that partition’s input, so a partition whose input genuinely did not change is a guaranteed hit and one whose input changed subtly is a guaranteed miss.

Watch for retention windows. Replication servers keep a finite history. An extract that falls behind the window can no longer be brought forward with diffs and needs a fresh base download. Alert on the gap between your sequence and the server’s current sequence well before that point.

Validation and testing

Every rebuild should clear the same gates as an initial build — connectivity, weight sanity, restriction counts — and one more that only applies to incremental pipelines: continuity. A graph produced by applying diffs must be equivalent to one produced by rebuilding from a fresh extract of the same timestamp, and drift between the two is the signature of a diff pipeline that is losing or duplicating changes.

# requires: networkx (pip install networkx)
import networkx as nx

def compare_builds(incremental: nx.DiGraph, from_scratch: nx.DiGraph) -> dict:
    """Quantify divergence between an incrementally updated and a fresh build."""
    inc_edges, fresh_edges = set(incremental.edges()), set(from_scratch.edges())
    only_inc = inc_edges - fresh_edges
    only_fresh = fresh_edges - inc_edges
    return {
        "edges_incremental": len(inc_edges),
        "edges_from_scratch": len(fresh_edges),
        "stale_in_incremental": len(only_inc),
        "missing_from_incremental": len(only_fresh),
        "divergence_pct": round(
            100 * (len(only_inc) + len(only_fresh)) / max(len(fresh_edges), 1), 4
        ),
    }

Run this comparison on a schedule — weekly is usually enough — and treat any non-zero divergence as a defect rather than as noise. A correctly sequenced diff pipeline produces an identical graph; divergence means changes are being skipped or applied twice, and the magnitude only ever grows.

Troubleshooting

Related

• Building directed graphs from OSM PBF files — the pipeline whose output this page keeps current, including PBF streaming and CSR serialisation
• Applying OSM change files with osmium derive-changes — deriving a change file between two arbitrary snapshots when replication history is unavailable
• Detecting breaking tag changes between PBF snapshots — the tag-count diff that turns silent weight drift into a reviewable event
• Incremental graph rebuilds without full reprocessing — mapping a changed-object set onto partitions and reusing unchanged artefacts
• Graph fragmentation prevention in OSM data — the connectivity audit every rebuild, incremental or not, has to clear