LMDB Store

Put a unfs filesystem in an LMDB environment, through lmdb-js.

LMDB is nearly what unfs's store layer already assumes: string keys in one ordered key space, byte values, and a read that's a pointer into a memory map rather than a copy out of a page cache. unfs/lmdb is the thin mapping between the two — you open the database, unfs uses it.

npm install unfs lmdb

import { createFs, initStore } from "unfs";
import { open } from "lmdb";
import { lmdbStore } from "unfs/lmdb";

const db = open({ path: ".data/store.lmdb", encoding: "binary" });
const fs = createFs(await initStore(lmdbStore(db)));

await fs.mkdir("/notes", { recursive: true });
await fs.writeFile("/notes/hello.txt", "hello from lmdb\n");

await fs.head(); // bafyrei… — the root CID of what's now on disk

Reopen it later, from another process or after a restart, and it's all still there:

import { createFs, openStore } from "unfs";
import { open } from "lmdb";
import { lmdbStore } from "unfs/lmdb";

const fs = createFs(
  await openStore(lmdbStore(open({ path: ".data/store.lmdb", encoding: "binary" }))),
);
await fs.readFile("/notes/hello.txt", "utf8"); // "hello from lmdb\n"

#encoding: "binary" is required

It's the one thing the store insists on, and it throws an UnfsError with code INVALID rather than accepting anything else:

lmdbStore(open({ path: ".data/store.lmdb" }));
// UnfsError: unfs lmdb store: the database is opened with encoding "msgpack", which would frame every block …

lmdb-js defaults to msgpack. Every value unfs stores is already an encoded block, so a second encoding on top of it is pure damage: a 3-byte block goes in as c4 03 01 02 03, and that framing is what comes back out. binary stores the bytes verbatim, which is what a blockstore wants.

A dupSort database is refused for the same reason: it keeps every value written to a key instead of replacing it, so set would not overwrite and setIfMatch would report a write that never happened.

Everything else about the database is yours. compression, an encryptionKey, a pageSize tuned for your working set, maxDbs — set them how you like. So is closing it: the store has no close(), because you opened the database and db.close() is yours to call.

#Sharing an environment

Pass a sub-database and unfs stays in its own key space, with other data alongside it in the same file:

const root = open({ path: ".data/app.lmdb", encoding: "binary", maxDbs: 8 });
const fs = createFs(await initStore(lmdbStore(root.openDB("unfs", { encoding: "binary" }))));

String keys round-trip verbatim (with one refused shape), so prefixedStore works too — several unfs stores in one database, each under its own prefix — and a key belonging to something else, including a non-string one, is skipped by keys() rather than mistaken for a block or taken as the end of the range.

#Using it synchronously

lmdb-js is synchronous where it counts and asynchronous where it pays, so this integration ships as a twin, like fsStore/fsStoreSync:

import { createFsSync, initStore } from "unfs";
import { lmdbStoreSync } from "unfs/lmdb";

const fs = createFsSync(await initStore(lmdbStoreSync(db)));
fs.writeFileSync("/hello.txt", "no await anywhere\n");

Reads are the same in both — get, has and keys() return plain values even from the async store, because LMDB reads are synchronous. The difference is writes:

lmdbStore is the one to reach for. Batching an event turn's writes into a single commit is what LMDB is fast at, and putSync gives that up: a filesystem write is many blocks, and the synchronous store commits each one in its own transaction. Use it when you need createFsSync — the API you're serving is synchronous — and not otherwise.

#What it promises

• Every optional capability but one. has, delete and keys all work, so packs, listRefs and gc do too — plus setIfMatch and deleteIfMatch, the compare-and-set and the compare-and-delete (see Multiple writers), and getMany on the async store and setMany on the synchronous one. Only hasMany is absent, because lmdb-js has no batched existence check and unfs's own fallback already issues the probes — and the async store needs no setMany for the same kind of reason, since it already folds an event turn's writes into one transaction.
• Enumeration is indexed. keys("refs:") seeks to the first ref and stops after the last, the same range scan the SQLite store does and the thing a directory has no index for.
• Enumeration is also a snapshot. A range runs against the version of the database it started with, so a concurrent write never makes it hand you a key twice — the hazard sqliteStore's live cursor has. The cost is that a long walk pins the pages it started with until it's done.
• Multiple processes, properly. LMDB is single-writer, multi-reader across processes, with readers never blocked by the writer. That's the concurrency model §15.3 asks for, enforced by the database rather than by convention.
• Durability is LMDB's, and it's commit-then-flush. On non-Windows platforms lmdb-js defaults to overlappingSync: a transaction is committed first and fsynced afterwards, in parallel with the transactions behind it. So an awaited write survives a process crash, and await db.flushed is what to reach for if you need it to survive a power cut. (That's weaker than unfs/sqlite, which fsyncs at every commit, and stronger than unfs/fs, which never fsyncs at all.)

Four things to know before it holds something you care about:

• A read can lag another process by an event turn. lmdb-js reuses one read transaction and resets it on new event turns and after its own commits, so a value another process just committed may not be visible to a get in the current turn. db.resetReadTxn() forces the refresh. Compare-and-set is unaffected — it reads inside its write transaction.
• Keys are limited to 1978 bytes, LMDB's maximum. Nothing unfs generates comes close; a long prefixedStore prefix plus a long ref name is the only way to find that wall.
• One key shape is refused. A key of 64 characters or more that carries a control character in U+0000–U+0004 is rejected with INVALID, because LMDB's key encoding stops escaping those past that length: such a key reads back fine but comes out of an enumeration as a different string, which would hide a ref from listRefs and let gc sweep the tree it names. §10.2 permits those characters in a ref name, so the store refuses the write rather than losing the tree later.
• It's a native module. lmdb-js ships prebuilt binaries for the common platforms, which is a different proposition from unfs/fs or unfs/sqlite — both of which install nothing at all.

#Multiple writers

setIfMatch here is a real transaction: the read, the comparison and the write happen inside one LMDB write transaction, and LMDB allows exactly one writer at a time across every process on the environment. lmdbStore runs it through db.transaction(), which takes the write lock off-thread and calls back once it holds it — so waiting for another writer costs your event loop nothing, though the callback itself runs on it like any other. lmdbStoreSync uses db.transactionSync(), which waits on the spot. deleteIfMatch is the same transaction around a remove, so unbinding a ref is conditional too — that's what unfs/git deletes a branch through.

The loop is the same one the on-disk store documents:

import { getRef, setRefIfMatch } from "unfs";

for (;;) {
  const head = await getRef(store, "head");
  const next = await buildFrom(head); // read the tree, write your change
  if (await setRefIfMatch(store, "head", head, next)) break;
  // Someone else committed first. Their tree is the new base — start over.
}

• false means the ref moved, so the tree you built is based on a root nobody has. Re-read and rebuild — and the re-read is fresh, because committing a transaction resets this thread's read snapshot.
• undefined as the expected value means "only if nothing is bound yet" — a create-only write.
• There's no BUSY here, unlike SQLite. A second writer waits on LMDB's write lock instead of failing, so a compare-and-set that returns at all is one that compared.

As everywhere, the guarantee only covers writers who all use it — a plain setRef still overwrites unconditionally.

#Which one