Article 53 asks what your model was trained on. We open-sourced the tool that answers it.

What it is. Shuddhi is an Apache-2.0 open-source data factory that turns raw text into an LLM training corpus sealed with a cryptographic receipt anyone can recompute.

Why now. It emits the EU AI Act Article 53(1)(d) training-content summary as a build artefact — enforcement powers have been active since 2 August 2026.

Proven. Reference run: 33,047,370 documents on one two-core VM, no GPU — 97.71% kept, an 84,275-copy duplicate cluster removed, 182,781 PII spans redacted, zero contamination.

Shuddhi's output isn't just a cleaner corpus. It's a chain of hashes anyone can recompute:

The registry declares where every shard came from — source, licence, acquisition date, data class. From the accepted documents Shuddhi mints a corpus build hash. Applying filters produces a filtered build hash — a content hash of the documents that survived — recorded alongside the corpus hash and a filter-configuration hash that pins every threshold used. Your training run cites the filtered hash in its ledger.

Three properties make it useful rather than decorative. It's order-independent — the hash is over a set, so parallelism can't change it. It's recomputable by anyone holding the same documents, with no key, no server and no configuration of ours — which is exactly why the filter configuration is recorded *beside* the hash rather than folded into it. And the manifest is self-describing: it names the exact configuration that produced the corpus. "Which documents did this model see?" stops being a memory exercise.

We didn't quite believe it ourselves at first, so we reproduced our own corpus hash across four independent full passes of 176 GB. The continuous-integration job still runs the sample pipeline twice on every commit and fails the build if a single hash differs.

The public numbers are all scale: trillions of tokens, as though the count were the achievement. It's the easiest number to publish and the least informative one, because a token count says nothing about what the tokens *are*. A corpus can be enormous and still be mostly the same page.

We know because it happened to us — and we only caught it because we measured every document rather than sampling. The largest near-duplicate cluster in a 33-million-document corpus was 84,275 copies of one template. Not similar pages — the same thing, eighty-four thousand times, every copy costing GPU hours to memorise something the model already knew after the first.

That's the argument for a data factory in one number. Deduplication, quality scoring and contamination screening aren't hygiene chores that happen before the interesting work; they decide how much of your compute budget teaches the model something new. A smaller corpus you can describe beats a larger one you can't. The frontier labs have known this for years — it's why their papers spend more pages on data than on architecture — and it's doubly true when you're training on a budget rather than a hyperscaler's cluster.

So the honest headline number isn't how many tokens we have. It's what fraction of them survived being measured, what was thrown away, and why — and whether anyone can check the answer. That's what a build hash is for.

Those are good tools and several of them will out-throughput Shuddhi on a web-scale crawl. Out-curating NVIDIA's GPU deduplication is a fight that helps nobody. What none of them emits is a receipt.

So Shuddhi will attest a corpus it did not build: `shuddhi attest --corpus ./out-from-datatrove/ --corpus-id fineweb-slice`. The fingerprint uses the same hash definition a native build uses, so an attested corpus and a Shuddhi-built one are verifiable the same way and comparable to each other.

The honest boundary, stated in the docs and worth repeating: an attestation proves content, not acquisition. It binds a corpus to a hash and reports what's inside it. It can't establish where the data came from — that's what a registry adds.

Permanently free, under Apache-2.0: the engine, every built-in filter, receipt generation *and verification*, the customer-data refusal, the plugin API, Article 53 summary generation, and attesting corpora built elsewhere. No seat limit. This won't move behind a licence later — several open-core projects relicensed after adoption, and what they lost wasn't revenue but anyone's willingness to build on them again.

Commercial: the hosted receipts registry — a signed, organisation-wide ledger with model-to-corpus-to-licence lookup, retention and legal hold; review workflow for the human sign-off the open engine already demands but gives you no way to manage; cross-build analytics; compliance packs; air-gapped deployment and support.

The line is simple: verification is free, operation at scale is paid. Nothing is ever removed from the open engine to create a paid tier, and commercial filters ship through the same public plugin API anyone else uses — where a paid filter still can't change a corpus without changing that corpus's receipt.