Layered Stores

Put a fast store in front of a slow one and use the pair as if it were a single store.

A layered store stacks key–value backends in order. Reads are answered by the highest layer that has the key. Writes are acknowledged as soon as the upper layers have them, then drained down to the durable layer in the background. flush() is the barrier that says "durable up to here".

It's a deployment arrangement, not a format. The stored bytes are indistinguishable from a single-layer store, and you can always open the bottom layer on its own.

#A write-back cache over a slow backend

multiStore takes its layers top first and returns a plain KVStore with flush(), withoutFills() and pendingWrites added, so everything else (createFs, packs, sync, gc) works over it unchanged:

import { createFs, initStore, memoryStore, multiStore } from "unfs";

const kv = await multiStore([
  { kv: memoryStore() }, // top: serves reads, absorbs writes
  { kv: slowRemoteKv }, //  durable: receives writes on flush()
]);

const store = await initStore(kv);
const fs = createFs(store);

await fs.writeFile("/notes.md", "instant\n"); // acknowledged from memory
kv.pendingWrites; // queued; a background drain is already scheduled

await kv.flush(); // the barrier: settles once everything above is durable

The factory is async because it establishes residency first. Every refs: and meta: key is copied up into the top layer before the store serves anything, so the mutable pointers, and a garbage collector's root set, always come from one authoritative layer. Every layer below the top must therefore support key enumeration, or construction is refused.

#Auto-flush

By default the log drains itself. Whenever there's queued work and no drain in flight, one is scheduled on a microtask, never inside your write, and writes that land during a drain coalesce into the next one.

Your writes stay instant, the crash-loss window stays roughly one in-flight drain, and flush() becomes a barrier you await at the moments that matter (before gc, at shutdown, after a burst you need durable) rather than a chore to remember.

If a background drain fails, the queue is kept and retried under exponential backoff. Wire up onFlushError to see those failures; an explicit flush() rejects to its caller instead. One backstop bounds the queue against a durable layer that keeps failing: once threshold entries are pending, the next write forces an immediate retry rather than waiting out the backoff.

const kv = await multiStore(layers, {
  autoFlush: { threshold: 256 }, // tune the backstop (1024 by default)…
  onFlushError: (error) => console.warn("drain failed, retrying:", error),
});

const manual = await multiStore(layers, { autoFlush: false }); // …or opt out:
// nothing reaches the durable layer until you call manual.flush()

#What a crash can and can't do

Writes drain to the durable layer in the order they happened. If a flush is interrupted by a crash or a lost connection, the durable layer is left holding a prefix of the write history.

Every prefix is a valid store, because a root is only ever written after every block it references. Reopening the durable layer alone gives you an earlier consistent snapshot, never a root with missing blocks. You lose un-flushed writes, and you never get corruption. A failed flush() keeps the undrained remainder queued, so calling it again resumes where it stopped.

Deletes hold up too. The write-back queue is part of the read path, so a deleted ref or block can't resurrect out of the durable layer in the window between the delete and the flush.

#A scratch overlay on a read-only base

Mark a layer readonly and it's only ever read, with no writes and no cache fills. That turns a published or shared store into a copy-on-write sandbox:

const kv = await multiStore([
  { kv: memoryStore() }, // all changes land here
  { kv: publishedKv, readonly: true }, // never touched
]);

With no writable layer below the top there's no write-back queue at all. flush() becomes a no-op, and the top layer simply is the store, with the read-only layers as fallbacks.

#Sharing one backend between stores

prefixedStore namespaces keys, so several independent stores, or several layers, can live in one physical backend:

import { prefixedStore } from "unfs";

const backend = memoryStore();
const storeA = await initStore(prefixedStore(backend, "a/"));
const storeB = await initStore(prefixedStore(backend, "b/"));

#Declaring a layer authoritative

Marking a layer authoritative promises that it holds every key. A miss in it is final, and the layers below are never consulted. That's what lets the fully synchronous createFsSync run over an arrangement whose lower layers are asynchronous, because the descent never reaches them.

Two rules make the promise safe to rely on:

• You populate it. unfs never fills a layer to make the claim true. Declare an incompletely populated layer authoritative and every miss becomes a silent wrong absence, where data that exists reads as deleted.
• It's revocable. If a write to an authoritative layer fails, the claim is withdrawn and reads widen to the layers below again. Never declare a layer authoritative if it evicts entries.

#Things to know

• Keep the handle. flush(), pendingWrites, and the auto-flush machinery all live on the object multiStore returns, so keep it alongside the Store you build from it. await kv.flush() is your graceful-shutdown story: with auto-flush on there's usually little left to drain, but only the barrier makes it a guarantee.
• One writer. A layered store assumes exclusive ownership of its backing layers. A second process writing to them behind its back is out of scope, the same rule garbage collection already has.
• Capabilities are honest. delete exists only if every layer supports it and none is read-only, and keys only if every layer can enumerate. A capability the arrangement can't serve is absent rather than present-and-failing, which is the same presence probe the rest of unfs uses.
• Batches descend a layer at a time. The batched read and the batched existence check (getMany, hasMany) are always available, and each asks a layer once for exactly the keys the layers above it left unanswered — so a batch costs a round trip per layer, not one per key. A layer that has its own batched method gets it; one that doesn't sees the same run of single calls it would have seen anyway. Presence checks read nothing and fill nothing, which is what lets gc prove a block present without pulling it up into the cache. The batched write (setMany) is deliberately not exposed, and you lose nothing by it: unfs's own fallback issues the batch as a run of set calls in your order, which is exactly how they have to reach the write-back log anyway.
• Collect against a drained store. Call await flush() before gc, since auto-flush keeps the log near-empty rather than empty. You don't have to give up caching to do maintenance: gc, gcSync and exportPackBytes already suppress cache fills for the duration of their own run, so a walk of the whole block space doesn't fault it up into the top layers. For anything else that sweeps a lot of blocks once — a streaming exportPack, your own traversal — bracket it yourself with kv.withoutFills(() => …), which returns whatever the callback returns and holds the suppression until a returned promise settles. { fills: false } at construction is still there for a store that should never cache at all.