Feature: OPFS rather than Indexeddb for Persistence

[nickchomey]

Is your feature request related to a problem?

Indexeddb is known to be slow and limited. And, when surreal is an abstraction on top of it, it has to go through many levels of abstractions - surreal.wasm > js > indexeddb > leveldb/sqlite.

(see here for info about how chromium currently uses leveldb - which rocksdb evolved out of - as the backing store for indexeddb, but are changing it to sqlite https://issues.chromium.org/issues/40253999)

Describe the solution

The Origin Private File System (OPFS) is a new-ish browser api that allows for high performance file system access. https://developer.mozilla.org/en-US/docs/Web/API/File_System_API/Origin_private_file_system

It has support of all major browsers (look at the opfs-specific footnotes) https://caniuse.com/native-filesystem-api

A high profile example of it being used is SQLite Wasm, which seemingly allows for full sqlite usage in the browser. https://developer.chrome.com/blog/sqlite-wasm-in-the-browser-backed-by-the-origin-private-file-system/

Presumably surreal's rocksdb and surrealkb backends could be used in a similar manner, which would surely simplify and improve performance for your wasm efforts. Likewise surrealkv when that is ready.

Alternative methods

SQLite wasm is getting a lot of attention and would probably be the easiest and most stable option. But it doesn't appear that Surreal uses SQLite as a storage backend, so there's probably no point in looking into it.

SurrealDB version

N/a

Contact Details

Reply to this issue

Is there an existing issue for this?

• I have searched the existing issues

Code of Conduct

• I agree to follow this project's Code of Conduct

[nickchomey]

@tobiemh thanks for your attention in the AMA. I forgot that I had already created this issue previously. I just updated it with some more info.

I'm sure this is not a top priority, but surely is worth at least taking into consideration. After all, you added support for the various backends - and are creating your own - for a reason!

[nickchomey]

Ive been looking into OPFS and other APIs lately and found a lot of good discussions and work going on in these two repos. OPFS + Leader Election Tabs, etc...

rhashimoto/wa-sqlite#81

DallasHoff/sqlocal#39

It seems like OPFS is definitely where all sqlite wasm projects are focusing, for good reason.

[bitnom]

Recent updates in the wa-sqlite issue point to these adapters (see table): https://github.com/rhashimoto/wa-sqlite/tree/master/src/examples#vfs-comparison

These are claimed to greatly reduce dev overhead in contrast to previous suggestions. Looks fun.

[itsezc]

Would love this, but should be optional for you to pick from as OPFS is not supported on all browsers.

[nickchomey]

As far as I can tell (sites like mdn and caniuse don't really show specific stats on browser compatibility for opfs), it is pretty much universally available now

https://sqlite.org/wasm/doc/trunk/persistence.md#opfs

https://webkit.org/blog/12257/the-file-system-access-api-with-origin-private-file-system/

https://rxdb.info/rx-storage-opfs.html#opfs-limitations