Remote Stores

Talk to a unfs store served over plain HTTP the same way you'd talk to a local one.

unfs/http gives you two things: httpStore, a read-only KVStore backed by a service, and cachedStore, a wrapper that turns it into a writable local filesystem over a remote origin.

#Reading from a remote store

httpStore maps a store key onto a path under a base URL and drives it with fetch. It's a plain KVStore, so it plugs into openStore exactly like any other backend:

import { createFs, openStore } from "unfs";
import { httpStore } from "unfs/http";

const store = await openStore(httpStore("https://example.com/v1"));
const fs = createFs(store);

await fs.readFile("/docs/hello.txt", "utf8");
await fs.readdir("/docs", { withFileTypes: true });

Use openStore rather than initStore. The remote already has its meta:format, so you're opening an existing store, not creating one.

A few things worth knowing:

• It's read-only. set throws UNSUPPORTED. To write, layer it under cachedStore or a multiStore.
• Block content is verified, not trusted. Leave Store.trusted off (the default) and every block read through openStore is re-hashed against its key, so a hostile origin, or a CDN in front of one, can withhold a block but never substitute its bytes. Names are a different matter: nothing in the format can check the origin's claim about which tree a ref points at, and trusting the service to name the right root is the whole point of asking it.
• Reads batch automatically. Multi-block reads, such as fs.readdir fanning out or exportPack walking a subtree, go through a single batched QUERY request instead of one round trip per block.

#Options

httpStore(base: string | URL, options?: HttpStoreOptions) takes:

• fetch — a fetch implementation to use instead of the global one.
• headers — sent on every request, for auth or tracing.
• signal — an AbortSignal that cancels every in-flight request.
• batch — whether to use the batched QUERY /blocks route for multi-block reads. Defaults to true; turn it off against a plain static origin, such as an S3 bucket of blocks/* objects, that only serves per-key GET.
• maxBatch — keys per batch request. Larger runs are split into several. Defaults to 256.
• maxBytes — bytes accepted from one response before the read is abandoned. Defaults to 64 MiB. The origin is untrusted, and bounding how many bytes arrive is a separate concern from re-hashing which bytes did.

const store = await openStore(
  httpStore("https://example.com/v1", {
    headers: { authorization: `Bearer ${token}` },
    maxBytes: 8 * 1024 * 1024,
  }),
);

base is treated as a directory, so a missing trailing slash is added and any query or fragment is dropped. Put credentials in headers, not in the URL.

#When a request fails

A transport failure throws an HttpStoreError: the origin answered, but not with a value (a non-2xx status), or the request never completed at all.

It's a plain Error rather than an UnfsError, because the UnfsError code space describes the format of what came back, such as malformed bytes or an unsupported codec, and "the CDN returned 503" isn't one of those.

import { HttpStoreError } from "unfs/http";

try {
  await fs.readFile("/docs/hello.txt", "utf8");
} catch (error) {
  if (error instanceof HttpStoreError && error.status >= 500) {
    // retry — the origin is having a bad day, the data isn't wrong
  }
}

It carries status, the HTTP status or 0 when the request never completed, and url, the request URL.

#Writing to a remote-backed store

cachedStore reads through a local front store to a remote origin, and fills front with every block the origin serves. Blocks are content-addressed and immutable, so caching them is sound with no invalidation at all: the value under a block key can never change.

import { createFs, openStore, memoryStore } from "unfs";
import { cachedStore, httpStore } from "unfs/http";

const origin = httpStore("https://example.com/v1");
const kv = cachedStore(memoryStore(), origin);

const store = await openStore(kv);
const fs = createFs(store);

await fs.readFile("/docs/hello.txt", "utf8"); // fetched once, cached after
await fs.writeFile("/docs/notes.md", "local edit\n"); // lands in the front store

That turns an otherwise read-only remote into the backing for a writable local filesystem. Every key is looked up in front first, so a client reads its own writes, and only the blocks you actually touch are pulled from the origin.

Once a ref has been written locally it stays local, and the origin's value for that name is never consulted again, so a fork doesn't silently snap back to the origin's root. Names you haven't written locally are re-read from the origin on every access, which is how a client notices that a moving ref, a latest tag say, has moved.

front can be any writable KVStore: memoryStore() for a process-lifetime cache, or a persistent backend to survive restarts.

#Options

cachedStore(front: KVStore, origin: KVStore, options?: CachedStoreOptions) takes one option. populate decides whether blocks read from origin are persisted into front. It defaults to true; set it to false to read through without caching.

#Implementing the server side

readCapped, encodeBlocksBody, decodeBlocksBody, and BLOCKS_MEDIA_TYPE are exported from unfs/http too, but they're implementation details of the two functions above rather than something a store consumer reaches for.

They matter if you're implementing the server side of the protocol, or a compatible one. readCapped drains a Response under a byte cap without trusting Content-Length, and encodeBlocksBody and decodeBlocksBody are the codec for the batched-read wire format (BLOCKS_MEDIA_TYPE) that httpStore's QUERY /blocks route speaks on the way out and expects on the way back.