Packs & Sync

Moving a tree between stores means copying the blocks the other side is missing, and nothing else.

A tree is fully described by its set of hash-named blocks, so nothing has to be re-serialized or re-hashed on the way, and the receiver can verify every byte it accepts. There are three things you'll want to do:

• Move a tree as a file — a whole tree in one portable pack.
• Sync two stores — two stores converging over a live connection.
• Clean up old blocks — reclaiming what nothing points at any more.

#Move a tree as a file

Export any path's tree to a single CARv1 pack file, then import it somewhere else. exportPack streams the bytes and exportPackBytes buffers them:

const pack = await fs.exportPackBytes("/docs"); // or "/", or one chunked file's path
for await (const part of fs.exportPack("/docs")) send(part); // the streaming form

On the receiving end you have three choices:

await other.importPack(pack, { ref: "incoming" }); // bind the root to a name you pick
await other.importPack(pack); // bind it to the fs's own head, replacing what's there
await other.graftPack(pack, "/vendored/docs"); // splice it in as a directory entry

Both importers accept a PackSource, which is either one Uint8Array or a sync or async iterable of pieces, and resolve to the imported root ref.

A few things worth knowing:

• Exports follow symlinks (the way stat does) and carry no history. Pass { history: true } to export the head commit and its whole prev chain, and importPack binds it back with the history intact.
• Grafting needs the pack's single root to be a directory. Otherwise it behaves like any other change: inside a batch it joins that batch's single commit, and graftPack(pack, "/") swaps the whole tree while keeping the head's history.
• Imported blocks are verified before any name moves, so a bad pack fails without touching your tree.

These helpers are async only. The synchronous facade doesn't carry them.

Because a pack is a plain CARv1 file, it's also how a tree travels to and from IPFS. ipfs dag import takes one directly, covered in IPFS.

#Sync two stores

Two stores can converge over any reliable, ordered byte stream. A short negotiation works out what the other side already has, then only the missing blocks move, as thin CARv1 packs.

import { syncPull, syncServe, transportPair } from "unfs";

const [clientEnd, serverEnd] = transportPair(); // or any byte stream, e.g. a socket
const [pulled] = await Promise.all([
  syncPull(mirror, clientEnd), // this store is the sink; it adopts what origin advertises
  syncServe(origin, serverEnd), // serves the session from the other end
]);
pulled.bound; // { "heads:main": Ref } — names bind only after the closure is verified

syncServe answers a session from the server end. From the client end, syncPull adopts the server's refs into your store, and syncPush offers yours to the server.

A transport is anything satisfying SyncTransport: a send(bytes), a receive async iterable of byte pieces, and an optional close(). TCP, a WebSocket, and an HTTP/2 stream all fit. transportPair() gives you a connected in-process pair.

Safety is built in. Every transferred block is hash-verified by the receiver, and completeness is checked with an exact closure walk that re-requests holes until none remain. A malicious or flaky peer can waste your bytes but can't corrupt your store.

Ref names move last, and fast-forward only by default. An unrelated head is reported as a conflict and left untouched, a per-name force replaces it unconditionally, and mode: "mirror" additionally tombstones names the source no longer advertises. A session killed at any point leaves refs untouched, and running it again converges.

The protocol is still a draft. Three of its capabilities are deferred (the filter and ancestry negotiation refinements, and path-want subtree syncs) and unfs never advertises them, so a peer that has them falls back to the core protocol. Like packs, sync is async only.

#Clean up old blocks

Blocks are immutable and writes only ever add, so superseded snapshots pile up until you collect them.

gc(store) is a mark and sweep. It walks the closure of every live named ref, then deletes every block nothing reached. Take a look before committing to it:

import { gc } from "unfs";

const plan = await gc(store, { dryRun: true }); // preview; writes nothing
plan.sweptBlocks; // blocks a real run would delete
plan.sweptBytes; //  …and how many bytes that frees
await gc(store); // the real sweep, same report, now applied

meta:format and the refs themselves are never touched. Marking finishes before the first delete, so a missing block aborts the run with NOT_FOUND without sweeping anything.

Your driver needs the optional keys and delete capabilities. memoryStore() has both, and a driver without them fails with UNSUPPORTED up front. gcSync does the same job for synchronous drivers.

Collection is cheap relative to what it reclaims, and a tree written with batch produces almost no garbage in the first place.