Assessing National Statistics
DQAF, ODIN and NUSAP: three objects of assessment, and FAND's place among them.
Three communities grade official statistics, and each grades a different thing.
- The IMF’s Data Quality Assessment Framework (DQAF) grades the producer: can this agency be trusted to compile this dataset?1
- Open Data Watch’s Open Data Inventory (ODIN) grades the release: can a user find the data, read it, and reuse it?2
- NUSAP grades the number in its use: how much weight can this value bear for this purpose?3
A compiler that synthesizes official statistics across countries and decades, as FAND does, needs all three. None of them grades the compiler. FAND inherits the first two from its sources and owes its readers the third. That is what NUSAP-lite is for.
Three objects of assessment
| DQAF | ODIN | NUSAP | |
|---|---|---|---|
| Who | IMF Statistics Department | Open Data Watch | Funtowicz and Ravetz (1990); now van der Sluijs and colleagues |
| Grades | The producing agency, dataset by dataset | The release on the national statistical office’s website | The number, and the assumptions behind it |
| Unit | Agency × dataset (national accounts, prices, government finance, money, balance of payments) | 22 data categories × 197 countries | A value, or an assumption in a model |
| Scale | Each element Observed, Largely Observed, Largely Not Observed or Not Observed | Ten elements, five on coverage and five on openness, each scored 0, ½ or 1; totals out of 100 | Pedigree criteria scored 0–4, with spread and assessment alongside |
| Latest | Generic framework July 2003; dataset frameworks May 2012; a 2025 pilot version now used in data ROSCs | 2024/25 round; the 2026/27 round began in May 2026 | Applied in EU food safety and product-footprint rules (below) |
The DQAF has siblings on the producer side: the European Statistics Code of Practice, with its Quality Assurance Framework; the UN’s National Quality Assurance Frameworks; and the World Bank’s Statistical Performance Indicators, which in 2021 replaced its Statistical Capacity Index with five pillars (data use, services, products, sources and infrastructure).4 All four grade systems and processes. None grades a number.
How the DQAF lines up with NUSAP
The DQAF runs from prerequisites through five dimensions, each broken into elements, indicators and focal issues. Read against NUSAP, each dimension answers one part of the question NUSAP asks of a number.
| DQAF dimension | NUSAP | What FAND does with it |
|---|---|---|
| 0. Prerequisites (legal and institutional environment, resources, relevance) and 1. Assurances of integrity | P, the acceptance side of pedigree | Inherits it. FAND cannot audit a ministry; it can record which agency produced a value and when. See the source agency directory. |
| 2. Methodological soundness (in the 2025 pilot, alignment with international statistical standards) | P, the method side of pedigree | Posts its own Statements of Expectations as Operational Guidelines, and records departures as Exceptions to Expectations. |
| 3. Accuracy and reliability (source data, statistical techniques, validation, revision studies) | S and A | Measures one kind of spread today: the gap between a source’s world total and FAND’s sum of countries. |
| 4. Serviceability (periodicity and timeliness, consistency, revision policy) | U, read as the time scale of the unit | Completes ragged recent years by a stated rule and marks every completed value: the currentness switch. |
| 5. Accessibility (data, metadata, assistance to users) | N and U as published | Publishes the replication package that rebuilds every figure from public sources. |
The mapping shows where the frameworks part. The DQAF asks whether a process can be trusted. NUSAP asks how much a number can carry once it has left the process. An office rated Observed on every element can still publish a number that is wrong for a 75-year comparison across 200 economies: the concepts changed in 1993, the base year moved, the series was never meant to travel that far. And an office never assessed may publish the only number there is.
Revision studies (DQAF element 3.5) are the one place an official framework measures spread from evidence: how far first estimates moved when later ones arrived. They are the closest thing in official statistics to NUSAP’s S.
What ODIN adds, and what it cannot
ODIN counts a dataset’s reference metadata as present when it carries three things: the definition of the indicator, the date the data were published, and the agency responsible. Those three are the minimum of NUSAP’s U and P. In the 2024/25 round, only a quarter of the 14,136 dataset records ODIN examined carried all three. The source was named in 88 percent, the date in 52 percent, the definition in 44 percent. Across the 171 countries assessed in every round from 2018 to 2024, metadata scores rose 4 points while machine readability rose 17 and open licensing 19.5
Open Data Watch’s September 2026 brief argues that AI raises the bar: a chatbot that answers with an official number needs metadata it can read, not just metadata that exists. It points to SDMX for structure, to the 20 percent of ODIN records reachable by an API, and to the five national statistical or data agencies that now run their own MCP servers, the connectors AI assistants use to query an API.5 Per-cell pedigree is that kind of metadata.
ODIN grades availability, not accuracy. A perfect ODIN score means a number can be found and reused. It does not say whether the number is right, or right for the use a reader has in mind.
NUSAP now, especially in Europe
NUSAP has moved from the academy into the annexes of rules where numbers carry legal or market weight.
- Food safety. The European Food Safety Authority’s 2018 guidance on uncertainty lists NUSAP among its qualitative methods: ordinal scales for each source of uncertainty, plotted together to show which matter most.6
- Science advice. SAPEA, the academies’ arm of the European Commission’s Scientific Advice Mechanism, reviewed NUSAP as a tool for evidence under complexity and uncertainty in 2019.7 The Dutch environmental assessment agencies built their uncertainty guidance on it.8
- Product footprints. The EU’s Environmental Footprint methods, recommended by the Commission in 2021, require a Data Quality Rating on four criteria: technological, geographical and time-related representativeness, and precision.9 That rating descends from the life-cycle “pedigree matrix,” which was carried over from NUSAP in 1996.10 The ecoinvent database goes one step further and converts pedigree scores into uncertainty factors, so each score widens a value’s spread by a stated amount.11
- Energy models. A 2018 study scored the assumptions in a UK energy system model and found that the most influential ones, on carbon capture and bioenergy, had the weakest pedigree.12
Two of those threads matter for FAND. The footprint rating’s geographical and time representativeness are FAND’s distinction between the unit of observation and the unit of analysis: a county figure used for a county page is one thing; the same figure inferred from a state total is another. And the ecoinvent route shows how categorical grades can become spread, which is FAND’s largest gap.
Where FAND’s NUSAP-lite stands
FAND is a compiler, not a producer. It has no ROSC and no ODIN row. What it has, by letter:
- N and U. Enforced. Zero and blank are different values. Every leaf value obeys V = Q × P, and units are checked against an independent source before any series is loaded.
- P. Provenance tags on every branch, per cell on cross-border positions. The tags have been mapped to 0–4 scores for cross-border positions and for minerals (873 cells, 31 commodities). A four-column FAND pedigree matrix (source authority, temporal coverage, spatial coverage, valuation method) has been piloted on minerals.
- A. Recorded as Exceptions to Expectations. A grid that colors every value by trust status is being built into the FAND app.
- S. Aggregate only: the difference between a source’s world total and the sum of FAND’s countries. No value carries its own spread yet.
The “lite” is deliberate. It names what has not been done: spread per value, and the arithmetic that carries grades through sums and ratios.
Directions for further work
Four steps would close most of the distance, each building on the frameworks above rather than replacing them.
- Let the source-authority column inherit other people’s grades. Score it from the IMF dissemination standard the reporting country subscribes to for that dataset (SDDS Plus, SDDS, e-GDDS or none), from the most recent data ROSC rating on accuracy and reliability where one exists, and from ODIN’s openness score for the category. Each carries its date. FAND then grades none of its sources on its own authority.
- Derive spread from pedigree, then test it. Follow the ecoinvent route: give each pedigree score an uncertainty factor and compute a band for every value. Calibrate the factors on FAND’s own overlaps rather than stipulating them: wherever two sources cover the same cell (IMF IIP and the External Wealth of Nations; a source’s world total and the sum of countries; US national figures and the sum of states or counties), the observed disagreement estimates the factor.
- Publish what ODIN asks for. Put the definition, the publication date and the responsible source on every FAND download, machine-readable, under open terms.
- Offer the mapping back. The DQAF-to-NUSAP table above, and FAND’s pedigree matrix, are contributions the NUSAP community at nusap.net can test on a domain it has not covered: national balance sheets.
Related reading
- NUSAP-lite: how we grade what we publish: the five letters and FAND’s status on each.
- FAND’s switches: where two defensible choices stay side by side.
- For data scientists: national statisticians: the pathway for SNA practitioners.
- Four Types of Capital or Three: accuracy against precision, including timeliness.
Footnotes
-
International Monetary Fund, Data Quality Assessment Framework (DQAF): Generic Framework (Washington, DC, July 2003); IMF Statistics Department, Data Quality Assessment Framework for National Accounts Statistics (May 2012). The 2025 pilot version is the basis of recent data ROSCs, e.g. IMF, Cambodia: Report on the Observance of Standards and Codes—Data Module, Country Report No. 26/62 (March 2026). ↩
-
Open Data Watch, Open Data Inventory 2022/23 Methodology Guide; Open Data Watch, “The 2026-27 ODIN Assessments Begin in May,” opendatawatch.com (2026). ↩
-
Silvio O. Funtowicz and Jerome R. Ravetz, Uncertainty and Quality in Science for Policy (Dordrecht: Kluwer, 1990); Jeroen P. van der Sluijs et al., “Combining Quantitative and Qualitative Measures of Uncertainty in Model-Based Environmental Assessment: The NUSAP System,” Risk Analysis 25, no. 2 (2005): 481–492. ↩
-
Eurostat, European Statistics Code of Practice (rev. 2017); United Nations, National Quality Assurance Frameworks Manual for Official Statistics (2019); Hai-Anh Dang et al., “Statistical Performance Indicators and Index: A New Tool to Measure Country Statistical Capacity,” Scientific Data 10 (2023), doi:10.1038/s41597-023-01971-0. ↩
-
Open Data Watch (Jamison Henninger), “Making Official Statistics Work in an AI-Driven World,” Open Data Insights, September 2026. ↩ ↩2
-
EFSA Scientific Committee, “The Principles and Methods behind EFSA’s Guidance on Uncertainty Analysis in Scientific Assessment,” EFSA Journal 16, no. 1 (2018): e05122. ↩
-
SAPEA, Making Sense of Science for Policy under Conditions of Complexity and Uncertainty, Evidence Review Report (Berlin, 2019), section 5.5.3. ↩
-
Jeroen P. van der Sluijs et al., RIVM/MNP Guidance for Uncertainty Assessment and Communication (Bilthoven: RIVM/MNP, 2003). ↩
-
Commission Recommendation (EU) 2021/2279 of 15 December 2021 on the use of the Environmental Footprint methods, Official Journal of the European Union L 471 (30 December 2021), Annex I. ↩
-
Bo P. Weidema and Marianne S. Wesnæs, “Data Quality Management for Life Cycle Inventories: An Example of Using Data Quality Indicators,” Journal of Cleaner Production 4, no. 3–4 (1996): 167–174. ↩
-
Andreas Ciroth et al., “Empirically Based Uncertainty Factors for the Pedigree Matrix in ecoinvent,” International Journal of Life Cycle Assessment 21, no. 9 (2016): 1338–1348. ↩
-
Steve Pye et al., “Assessing Qualitative and Quantitative Dimensions of Uncertainty in Energy Modelling for Policy Support in the United Kingdom,” Energy Research & Social Science 46 (2018): 332–344. ↩