Eight Layers Nationwide, and the 19-Hour-52-Minute Land-Use Mesh Build

Hello from the MAPRISE team. Last time, we wrote about taking registry cadastral maps and PLATEAU nationwide. Work on other layers continued in parallel, and eight more layers have now gone nationwide in production. This isn't a feature announcement — it's a record of that rollout itself. In particular, we want to be honest about why the land-use detailed mesh, the heaviest of the eight, took 19 hours and 52 minutes, and what actually happened along the way.

The eight layers that went nationwide this round

For every one of these, we confirmed existing data was untouched before and after ingestion, and verified after deployment that the actual tile-serving endpoints return real data for the newly added areas.

Putting Kyushu's lessons to work at national scale

The approach we used for Kyushu's 7 prefectures doesn't scale cleanly to 47 prefectures and hundreds of municipalities if applied as-is — verification alone would never finish. This round, we built on what we'd learned in Kyushu and pushed several techniques further.

Stepwise execution: For the 21-year real-estate transaction backfill, we raised the processing load step by step while measuring the load on both the data provider and our own service. All 6 batches completed with zero errors.

Staged verification, every time: Every one of the eight ingestions followed the same pattern: dry-run (full rollback, confirming zero writes) → count verification → real ingestion → a before/after diff across all prefecture counts confirming untouched prefectures stayed exactly untouched. It's tedious, but skipping this step is exactly how the performance bugs and spec edge cases described below would have gone undetected until after launch.

Working around an environment constraint: In some working environments we repeatedly could not check the map visually in a browser. As a workaround, we fetched the data directly through the production delivery path and byte-compared it against the server-side result. We couldn't eyeball pixels, but we could verify the exact delivery path a real end user's browser would take — which, as a check, is arguably more rigorous than a screenshot.

Why the land-use mesh alone took 19 hours 52 minutes

Of the eight layers, the land-use detailed mesh (all 9 fiscal years: 1976, 1987, 1991, 1997, 2006, 2009, 2014, 2016, 2021) stood out by a wide margin. The pipeline started at 20:28 JST on August 16, 2026, and all 9 years finished at 16:20 JST on August 17 — a measured 19 hours 52 minutes.

Measuring on a small scale first

Before the nationwide run, we processed part of Kyushu as a trial and measured the time and data volume, then used those figures for the nationwide estimate. Map tile generation accounted for most of the total processing time.

Actual measurement came in at 9.28GB nationwide, smaller than a simple extrapolation of the Kyushu results would suggest: mountain, island, and offshore-boundary meshes are low in density and compress well, so a simple mesh-count ratio overestimates.

We ran the work within limits that keep the service unaffected, and we did not temporarily add capacity.

The two incidents

Two runtime blockers occurred over the 19h52m. In both cases we diagnosed the actual root cause by measurement before fixing anything — no guessing.

Incident 1: upload failure for fiscal year 1976

The cause: the upload step's authentication setting did not match the environment it ran in. The tile generation itself had already succeeded, so nothing was lost. We recovered with the correct authentication and, as a permanent fix, removed that setting from the process entirely.

Incident 2: fiscal year 2006 failed for lack of working space

The cause: our working-space planning was based on the Kyushu-only baseline. At national scale, output per fiscal year came to several times the Kyushu figure, and holding every year's intermediate files at once simply didn't hold up across all 9 years. We recovered by releasing the temporary files of fiscal years that had already been verified and uploaded. As a permanent fix, we now verify the size after each upload before deleting the temporary files (the original downloaded sources are retained as an audit trail).

In the end, roughly 64.1GB shipped nationwide, and an independent audit script — checking the delivered data directly rather than trusting the build's own completion report — confirmed a full pass across all 9 fiscal years.

Designing for ongoing operation

Finishing a nationwide rollout once doesn't mean these datasets are done. New fiscal years get published, and municipality-level readiness keeps shifting. This round, we revisited how we detect newly published data for both PLATEAU and the land-use mesh.

PLATEAU: On top of the existing monthly monitor (querying the G-Spatial Information Center's public listings), we added a mechanism to keep tracking the "unknown structure-type code" problem found during the nationwide ingestion. We now monitor monthly for revisions to MLIT's own official code list, and only when a revision is detected do we re-examine the currently excluded municipalities via a dry-run (a re-run of the quality gate that makes zero production writes). It's a two-layer design — a cheap monitor watches for a signal, and the expensive re-check only runs when that signal fires. Unconditional high-frequency automation simply isn't worth the cost.

Land-use mesh: The existing monitor watched a fixed URL. Investigating further, we found that the government's own site keeps three different URLs alive simultaneously for the same dataset — an old version, the current version, and a link surfaced from the master listing page — and that a new spec version can be published under an entirely new URL rather than as an update to the existing one. A fixed-URL monitor would miss exactly that case: a new version shipping at a new URL while the old one sits frozen and unchanged. So we switched to dynamically resolving the current representative URL from the government's own master listing page every time, and monitoring that.

Why not fully automate?: In both designs, the decision of "should we actually ingest this" is left to a human — a staged detect, human approval, ingest flow. The reason is simple: this ingestion actually hit 7 municipalities with quality-gate failures nobody had anticipated, so "a new municipality or fiscal year will always ingest safely" isn't a premise we can rely on. A single land-use mesh build alone also produces over ten hours of CPU and disk load — the kind of change a human should be aware of before it happens. What gets automated is detecting and surfacing the change; the final call on whether to actually ingest stays human. This round made that split explicit again.

Quality and regression checks

Going nationwide meant designing separate safeguards for two different concerns: not breaking the existing Kyushu data, and keeping the quality of the newly ingested data itself.

Fail-closed quality gates: For PLATEAU building ingestion, 238 of 245 targeted municipalities succeeded, but 7 were held back because their building data included structure-type codes that don't appear in the official code list. Rather than guessing and extending the code mapping to cover the unknown values, the gate simply held the ingestion back, leaving the actual investigation — checking the source spec's revision history — to a human. That pause is exactly what kept an unverified mapping out of production.

Non-contact verification before/after ingestion: For the real-estate transaction backfill, we diffed per-prefecture counts before and after each of the 6 batches, confirming that only the batch's target prefectures changed. The ground-boring ingestion process includes a safety guard that aborts immediately if even a single Kyushu record shows up in the incoming data.

Independent auditing: For the land-use mesh, rather than trusting the build script's own completion report, we ran a separate audit script that checks the delivered data directly. For the inland flood layer, we deliberately sent tile requests against excluded municipalities — areas where no data should exist — and confirmed the response correctly came back empty, a negative control.

None of this is glamorous work, but refusing to simply trust "ingestion complete" and instead cross-checking against the real data every time is, we think, how you catch the bugs that would otherwise only surface after launch.

Closing

The nationwide rollout of these eight layers, and the 19-hour-52-minute tile build behind one of them, isn't flashy work — but it's unavoidable when you're dealing with map data, which never truly finishes. We think of map data as a genuine trove of big data: it lets you stack fundamentally different kinds of information — statistics, land prices, disaster history, buildings, ground conditions — on top of one another through the one thing they share, location. Quantifying an accumulated, continuous history like that, and making it analyzable from many angles with digital tools and AI, is how more people get to benefit from it. We'll keep building toward that, one layer at a time.

Questions or feedback? Reach us any time at info@maprise.jp.

The data referenced in this post is sourced from open data published by Japan's Ministry of Land, Infrastructure, Transport and Tourism, the Ministry of Justice, the Geospatial Information Authority of Japan, and the Statistics Bureau of Japan, among others. Record counts, data volumes, and processing times are measured values drawn from each task's primary completion report and reflect this post's publish date; they may change as operations continue.