Replication package

The full Python pipeline that reproduces every number — public release in preparation.

Every number FAND publishes can be reproduced from public source data using the bundled Python pipeline. The package contains 128 builder scripts, organized as a 12-stage DAG, with SHA-256 checksums on every intermediate artifact and a MANIFEST file pinning the exact source-data snapshot used for each release.

Source code

Publication in preparation. The replication pipeline is built and runs today as part of fand-etl, the ETL project that reproduces the FAND builds end-to-end (raw agency sources → validated Postgres schema). fand-etl is not yet public; a standalone replication release, or a public cut of fand-etl itself, is planned but not yet published. This page will link the public artifact once it is available.

What the package contains

• A 12-stage Python pipeline that runs end-to-end from agency downloads through validated FAND output
• 128 builder scripts — 30+ World Tables (TL1), per-side cross-border builders (CBA / CBL), and US state/county builders
• run_all.py — the DAG entry point that resolves stage dependencies and produces a PASS/FAIL summary
• SHA-256 audit trail — every intermediate artifact carries a checksum so a reviewer can confirm bit-exact reproduction
• MANIFEST — pins the agency-data snapshot (source, table, retrieval date, URL) used for the release
• parameters.json — externalized valuation parameters (discount rate, severance rate, splice protocol, etc.)
• SQLite caches — oconeco_tl1_global.db (TL1) and oconeco_sub_usa.db (US subnational) for fast re-runs

How to run it

Once published, running the pipeline will look like:

pip install -r requirements.txt
python run_all.py

Useful flags:

python run_all.py --national   # TL1 builders only (skip US subnational)
python run_all.py --list       # show stages without running
python run_all.py --stage 7    # run a single stage

Every script is idempotent. A full run on a modern laptop completes in tens of minutes; subsequent runs against the cached intermediate artifacts complete in seconds. The final stage prints a PASS/FAIL validation summary that an institutional reviewer can spot-check against the published numbers.

License

The replication package will be released under the MIT License. The license applies to the pipeline code; the upstream source data remains governed by each agency’s own terms of use.

Where it sits in the architecture

The replication package is what gets handed to institutional reviewers. The public-facing summary lives here; the pipeline itself runs today inside fand-etl pending public release (see “Source code” above). Repository-level context on how the pipeline maps to the FAND data API is maintained internally alongside the site source.