unfs

Pack interchange format (CAR)

A pack is a byte stream carrying a header (with root refs) followed by block sections. Packs move DAGs between stores; they carry blocks and roots only — no name bindings.

On the wire a pack is a CARv1 archive (Content Addressable aRchive, §2.5). Every pack this section admits is a valid CARv1, so generic IPLD tooling can read, transport, and store unfs packs unchanged.

CARv1 deliberately leaves block order, block duplication, and DAG completeness unspecified. This format pins all three; the result is a canonical pack — the only kind a conforming exporter emits (§12.2). Pinning those freedoms makes pack bytes deterministic without leaving the CARv1 format.

#12.1 Framing

varint(len(header)) || header       header = canonical CBOR (§5, DAG-CBOR)
                                        {roots: [Link…], version: 1}
repeat:
    varint(len(section)) || section     section = cid || blockBytes
  • There is no magic number: a CARv1 stream begins directly with the header frame. Identification is structural — a strict varint followed by the exact canonical header map below. (The interchange media type is application/vnd.ipld.car.) A stream that parses as a CAR but whose roots or blocks fail this format's validation (§4, §6) is a valid CAR that is not a unfs pack; implementations SHOULD report that case distinctly from "not a CAR at all". Version detection — including the CARv2 pragma — is §12.5 and §15.1.
  • Varints per §3.1 (strict). This is a subset of the CAR specification's unsigned LEB128: every canonical pack is therefore parseable by generic CAR readers, while a CAR framed with non-minimal varints is rejected here.
  • The header is a canonical-CBOR map (§5 — hence strict DAG-CBOR) with exactly the fields roots (a non-empty array of Links, §5.2 rule 5) and version (the integer 1). Canonical key order puts roots before version (§5.2 rule 2: length-first). Unknown header fields, a non-canonical encoding, and version ≠ 1 (§12.5) MUST be rejected. Roots MUST NOT repeat. Every root MUST pass ref validation (§4) carrying the store's multihash. (CARv1 leaves the number of roots partly unresolved; this format requires at least one. Importers MAY accept a foreign CAR whose roots array is empty, treating it as importing nothing — exporters MUST NOT produce one.)
  • A section is the referenced block's binary CID (§4) followed by blockBytes, the block verbatim. The CID here is bare — no multibase prefix; the 0x00 identity prefix belongs to the CBOR link payload only (§5.2 rule 5). The CID MUST be validated eagerly per §4 (CIDv1 only, codec raw or dag-cbor, the store's multihash, no identity multihash, minimal varints). blockBytes MUST be non-empty. The empty block does not exist in this format — no conforming structure can reference it, since chunk sizes are ≥ 1 and a node is a non-empty CBOR map (§6). The minimum legal section is therefore 37 bytes under SHA2-256 (36-byte CID + 1 byte), 38 on the wire once its length varint is counted; the §4 grammar's absolute floor is a 25-byte section (24-byte CID + 1).
  • There is no total length, no terminator, and no index. Consequence (deliberate, normative): a pack truncated at a section boundary is byte-for-byte a valid shorter pack carrying fewer blocks. A pack cut mid-field MUST be rejected. Completeness is therefore the receiver's check: after import, walk the roots and verify every reachable block is present. Implementations MUST NOT "fix" this with a length prefix, a trailer, or a CARv2 index (§12.5) — the same property is what makes thin packs (§12.3) expressible. (CARv1 itself neither promises nor precludes the prefix property; it holds because no CARv1 construct spans sections, and this format makes it normative.)

#12.2 Export (canonical packs)

Export of roots R₁ … Rₙ MUST emit:

  • The header frame, with roots = the argument roots deduplicated by full CID in first-occurrence order — at least one (§12.1).
  • For each root in that order, a depth-first preorder traversal: visit a ref; if its codec is dag-cbor, decode the block and recurse into each embedded Link in the order the links appear in the block's canonical encoding (arrays in element order, maps in key order, recursively).
    • Traversal is deduplicated by full CID across the whole export: a ref already visited is skipped, but a multihash first visited via a raw ref MUST still be traversed when later reached via a dag-cbor ref (§4).
    • Section emission is deduplicated by multihash alone across the whole export: one stored block = one section, even when referenced by both codecs or from multiple roots.

Under these rules pack bytes are a pure function of the store contents and the root list — export is deterministic. (The tree vectors of Appendix A.5 pin refs, not section order; the pack vector of Appendix A.7 pins the export bytes themselves.)

Raw blocks have no children; an exporter MUST NOT attempt to decode them.

Non-normative: in the CAR content-type vocabulary of the IPFS trustless gateway specifications, a canonical pack is exactly an application/vnd.ipld.car; version=1; order=dfs; dups=n stream — the profile this ecosystem already names, here made normative.

#12.3 Thin packs

An export MAY take an exclude set of refs. The exporter first computes the closure of the exclude refs — traversing dag-cbor blocks only, since raw refs are recorded but never fetched, so excluding a large base costs reads proportional to its node count, not its total size. This closure seeds BOTH the traversal set (full CID — prunes the walk) and the emission set (multihash — suppresses sections). All of this happens before the header, so an invalid exclude ref costs zero emitted bytes.

Nothing about the exclusion appears on the wire: a thin pack is indistinguishable from a full pack whose store happened to lack those blocks. CARv1 explicitly does not require its blocks to form a complete DAG, so a thin pack is as valid a CAR as a full one. It imports successfully anywhere; reads that need a missing excluded block fail at read time ("block not found"). A thin pack is a replication artifact against a receiver known to hold the exclude closure — never an archive.

The two dedupe sets MUST stay separate here as everywhere (§4): a multihash in the exclude closure first reached via a raw ref and later referenced as dag-cbor still needs traversal but MUST NOT be emitted as a section.

#12.4 Import

Import is deliberately more liberal than export: any CARv1 whose refs satisfy §4 imports, in any section order, so packs produced by generic IPLD tooling are accepted — canonical form (§12.2) binds exporters only. Importers MUST:

  • Parse with strict varints; enforce the header rules of §12.1 and version == 1. Any other version is a pack-version error, distinct from corruption (§12.5, §15.1). A foreign CAR with an empty roots array MAY be accepted as importing nothing; every other §12.1 header violation is fatal.
  • Enforce a frame-size limit checked the instant a length varint decodes, before buffering any of the frame — for the header frame and every section alike. Importers MUST accept every frame whose length is ≤ 2^30 bytes and MUST reject every frame whose length exceeds 2^31 − 1 bytes; between those bounds the limit is implementation-defined and MUST be documented. An importer MUST NOT allocate based on a declared length alone. (Non-normative: common CAR readers default tighter — e.g. 32 MiB headers and 8 MiB sections — and every block of a conforming tree is far below all of these limits.)
  • Validate each section's CID strictly (§4). Reject the pack if any CID — root or section — carries a multihash other than the store's (§10.3): importing it would verify blocks against the wrong hash function and mis-key them. (This is the CID-level successor of the retired header hash field: refs now self-describe their hash, and the header roots give the same up-front whole-pack rejection.)
  • Accept sections in any order. Canonical order is an export obligation; importers MUST NOT require it.
  • Copy block bytes and digests out of transient input buffers before storing (never alias stored data to the pack buffer or to producer-owned pieces).
  • Hash-verify each section's blockBytes against the CID's multihash digest using the store's hash function, storing under blocks:<base32(multihash)> (§10.1). Verification MAY be explicitly disabled by the operator for trusted input; skipping it can bind a block key to wrong bytes, and any cache of decoded nodes MUST be invalidated after an unverified import.
  • Treat duplicate sections for the same multihash as idempotent — a canonical export never emits them (§12.2), but importers MUST accept them. Their bytes MUST be confirmed identical: hash verification does this automatically; under an unverified import, a duplicate MUST instead be byte-compared against the already-stored value, and a mismatch is an error.
  • Offer, and perform by default, root-closure verification: after the last section, walk the closure of the header roots and verify every reachable block is present in the store (§12.1). Callers MAY opt out explicitly — required for thin-pack replication (§12.3), where completeness holds only jointly with blocks the receiver already has.
  • Return/report the header roots. Importing MUST NOT bind any names; the caller decides what, if anything, to point at the roots.

An incremental (streaming) importer MUST behave byte-identically to a buffered one over the same input, for every possible piece slicing, including 1-byte pieces. "Truncated varint / mid-field end of input" at the end of the stream is an error; end of input exactly at a section boundary is normal termination (§12.1). Import is not required to be transactional: a failed import may leave a subset of the pack's blocks stored (this is harmless — blocks are immutable and content-addressed).

#12.5 CAR versions

The pack format's version signal is the header's version field; CARv1 has no bytes outside its frames. CARv2 wraps a complete CARv1 payload behind a fixed 11-byte pragma,

0x0a 0xa1 0x67 0x76 0x65 0x72 0x73 0x69 0x6f 0x6e 0x02

which is deliberately itself a well-formed header frame encoding {version: 2} — so a v1 parser reads exactly far enough to report the found version (§15.1).

  • Exporters MUST NOT emit CARv2. Canonical pack bytes are CARv1: a CARv2 index is derived data with no canonical form. Determinism — and the pack vector of Appendix A.7 — is pinned on the CARv1 bytes. (Anyone may wrap or index a canonical pack locally; the wrapper is not the pack.)
  • Importers MAY accept CARv2 input: parse the pragma and the 40-byte CARv2 header, seek to the data offset, and process exactly data-size bytes of embedded CARv1 payload under §12.4. An index, if present, MUST be ignored for import: it is unverified derived data and MUST NOT substitute for hash verification or closure verification (§16).
  • An importer that does not accept CARv2 MUST report it as an unsupported pack version, distinct from corruption.
Edit this page
Previous Path semantics Next Conformance

unfs  A filesystem you can put in any key–value store.