Design rationale (Appendix B)

Why metadata in the parent entry. Content blocks stay pure content: identical bytes dedupe to one block across any number of metadata variations; a metadata change rewrites one directory node instead of the content; and a directory listing with full stat is a single block read. The cost — an entry changes when its child's root ref changes — is exactly the spine rewrite the snapshot model wants anyway. (The same trade git trees make.)

Why block identity is the multihash alone. The CID's codec describes how a referrer interprets the block, not what the block is. Keying storage by multihash means bytes that happen to be referenced both as a chunk and as a node are stored once. The price is the two-set discipline of §4 — localized and testable — rather than duplicated storage. This is also the split the IPLD mainstream converged on: block stores there are keyed by the raw multihash, codec-agnostic (go-ipfs moved to multihash-keyed block storage in 0.12). A unfs store and an ordinary IPLD blockstore therefore agree on what a block is — and on when two refs name the same one.

Why tag 42, not the pre-IPLD tag 117. A pre-IPLD draft of this format wrapped every ref in a private CBOR tag 117, whose payload was a bare [kind byte] || hash — no CID, no codec, no multibase prefix — and that draft never shipped. Tag 42 is registered in the IANA "CBOR Tags" registry as "IPLD content identifier" with exactly the payload grammar this format's refs already need (§4); adopting it cost nothing and bought generic IPLD tooling that extracts and traverses links with no adaptation. Tag 117 is rejected, like every other non-42 tag (§5.3), for the same reason CIDv0 and non-minimal varints are: left standing, it would be a second encoding for a value this profile already has one form for.

Why sorted arrays instead of a hash-based directory structure. Sorted entry arrays give ordered listing with no extra work, point lookups in at most two block reads at any directory size, trivially verifiable structure (sortedness is checked locally), and deterministic layout without a hash function in the layout path. Single-level paging with content-defined segment boundaries keeps the worst case bounded (millions of entries at defaults), while making the paged layout canonical and edits segment-local — the same reasoning as CDC for file content. Filesystem directories, unlike arbitrary maps, do not need more. (The store hash does enter the paged layout path via the boundary predicate, but only over names, and it is the one hash the store already has.)

Why content-defined chunking by default. CDC makes chunk boundaries a function of content, so local edits invalidate only neighboring chunks and unchanged data keeps its refs — deduplication across file versions falls out of content addressing with no delta machinery. Delta compression is deliberately absent: dedupe-by-chunking plus (out-of-band) compression is the chosen complexity point, and delta chains are the complexity class this format exists to avoid.

Why the cut test reads the high bits. The gear hash is h = ((h << 1) + GEAR[b]) mod 2^32. Bit 0 of h is exactly bit 0 of GEAR[b] for the single most recent byte; bit 1 involves two bytes; and so on upward — the low bits of a gear hash are barely a function of the window at all. Testing the low bits with a contiguous mask — what this format's first draft did — therefore tests almost no content. On uniformly random bytes the deficiency is invisible, because every byte value occurs and a few mixed bits suffice. On real data, whose byte alphabets are narrow and repetitive, whole regions become structurally incapable of satisfying the predicate, so the chunker falls through to max. A cut at max is a fixed-size cut, sitting at a fixed offset from the previous boundary — so one inserted byte shifts every boundary after it. That is precisely the failure CDC exists to prevent. Measured rate of max-forced cuts at 65536/262144/524288 bounds, against the 1.5% an ideal uniform hash predicts:

The high-bit test (§9.4) is the same width — the same kS/kL, the same expected chunk length — read from the end of the register the whole window reaches. It buys what CDC promises: inserting one byte into 9 MB of minified JS re-stores 343 KB, where the low-bit test re-stored 933 KB.

Why the default bounds are small. Boundaries that land on content convert chunk size directly into dedupe, so once the predicate was fixed the old defaults (65536/262144/524288) were simply leaving matches on the table. At 8192/32768/98304, dedupe across 23 consecutive extracted npm package versions goes from 35.7% to 57.6%; the same corpus stored as .tar archives goes from 7.6% to 48.9%; three real Vite builds of one application go from 0.2% to 23.3% (each figure: old defaults with the low-bit test → new defaults with the high-bit test). The cost is roughly 1.7× the block count. Source trees are unaffected either way — nearly every source file is shorter than min, so dedupe there is whole-file identity, not chunking. max is 3 × avg rather than 2 × avg for the same reason the predicate changed: with the high-bit test the extra headroom drives the no-boundary-found rate to ~0%, so the fixed-size fallback essentially stops occurring on real data.

Why CARv1, and why a profile. A pre-IPLD draft of this format carried a bespoke pack framing — magic bytes, a private header, length-prefixed frames of [kind || hash || block]. CARv1 is byte-for-byte the same shape — varint-framed sections of self-describing ref + block — so the private framing bought nothing that CARv1 does not provide, while costing all interoperability. As CARv1, every pack can be ingested, relayed, pinned, and unpacked by generic IPLD tooling that has never heard of unfs. What CARv1 deliberately does not provide is determinism: it leaves block order, duplicate blocks, root cardinality, and DAG completeness unspecified. Those are exactly the properties this format's guarantees need pinned. Pinning unspecified freedoms is legal specialization: a canonical pack (§12.2) is still plain CARv1 to every other reader. The ecosystem itself already names this profile: IPFS trustless gateways advertise CAR responses as order=dfs; dups=n|y, and a canonical pack is the order=dfs; dups=n point of that space, made normative.

Why packs still have no index or trailer. Sections are self-delimiting and blocks are self-verifying (hash addressing), so the only integrity property a trailer could add — "the stream is complete" — is better checked semantically by walking the roots. In exchange, a prefix of a pack cut at a section boundary is itself a valid pack, which is what makes resumable transfer and thin packs representable with zero wire additions. The same reasoning keeps CARv2's index out of the canonical bytes: an index is derived data with no canonical form. Admitting it into the pack would trade "one DAG, one byte stream" for a seek table any consumer can rebuild locally in one pass — and one that a hostile producer could skew (§16). Determinism is therefore pinned to CARv1; CARv2 is accepted on input, unwrapped and unindexed (§12.5).

Why not UnixFS. UnixFS is the IPLD ecosystem's incumbent filesystem format. unfs deliberately shares its substrate — CIDs, raw and dag-cbor blocks, multihash-keyed stores, CAR interchange — so generic tooling can move, pin, and verify unfs graphs. It does not share the data model, for three reasons.

First, metadata placement. UnixFS 1.5 stores optional mode/mtime inside the file's own node, so touching metadata re-hashes the file and forks its identity away from every metadata variant of the same bytes. unfs keeps all unix metadata — a strictly larger set: owners, xattrs, device numbers, hardlink identity, fifos and sockets, none of which UnixFS can represent — in the parent's directory entry, so content blocks stay pure content and identical bytes dedupe across any metadata.

Second, determinism. UnixFS guarantees none: the same file yields different CIDs across chunkers, DAG layouts, raw-leaves settings, and CID versions, and its metadata support is optional and unevenly implemented. unfs, in contrast, makes "same content + same parameters ⇒ same CID" a conformance requirement, with normative vectors, canonical chunking, and canonical directory segmentation.

Third, one codec and one shape: UnixFS carries dag-pb envelopes around an inner protobuf message plus a HAMT sharding scheme; unfs is a single canonical-CBOR profile with sorted, locally verifiable directory arrays. unfs is therefore IPLD-compatible but not UnixFS: ecosystem tools can carry unfs DAGs without interpreting them; interpreting them as a filesystem takes this specification.

Why a DAG-CBOR subset rather than a private codec. The compatibility on offer is deliberately one-directional: every conforming block is valid DAG-CBOR, so generic IPLD tooling can decode, traverse, and transport unfs graphs unchanged, while unfs readers accept only the profile of §5. The reverse direction — accepting every valid DAG-CBOR block — is not offered, because it would admit floats, 64-bit integers, foreign CIDs, and, in lenient decoders, unordered maps. Each one is either a second encoding for a value this format can express, or a value it cannot. The DAG-CBOR specification sanctions exactly this posture: "A strictness opt-in may be offered for systems where round-trip determinism is a desirable feature and backward compatibility with old, non-strict data is unnecessary." unfs is that system; its strict readers are conforming strict DAG-CBOR decoders, not a deviation from the codec.

Why floats stay banned. No field of this format is float-typed, so the ban costs nothing in expressiveness; what it buys is the absence of an entire class of cross-language round-trip and canonicalization hazards. IPLD's own data-model guidance agrees: "it is recommended that Float values be avoided when developing systems on IPLD (and content-addressable systems in general)", citing "the broad scope for introducing variability in byte representations" and the round-trip ambiguity some languages introduce. DAG-CBOR itself already rejects NaN and ±Infinity; this profile extends the rejection to all floats.

Why strict-everything on the read path. Every "lenient" decode is a second accepted encoding for the same value, and every second encoding breaks either determinism (two writers, two byte forms) or identity (two byte forms, one decoded value — the malformed-UTF-8 name case). Strictness is not defensive garnish; it is what makes "same content ⇒ same ref" a two-way implication.