Document how to create a custom file

system #9372
Open

Description
curiousdannii
opened on Sep 4, 2019 · edited by curiousdannii

I couldn't find any guide about how to create custom file systems.

 What functions need to be defined at a minimum?

Activity

kripken commented on Sep 5, 2019

kripken
on Sep 5, 2019
Member

There aren't much docs for this, yeah, sorry. But I wouldn't know much aside from starting from
one of the existing ones and duplicating it - which makes sense in the emscripten tree, but I'm not
sure how to do it outside.

In general the complexity comes from FSes being built on the same underlying JS infrastructure.
That's good in some ways (handles boring stuff like permissions etc. for you), but has downsides
too.

I'd also add that there are various thoughts about the future here, but no clear plan:

 One route is ASMFS, which avoids all JS and writes a new impl in compiled code,
allowing multithreading etc., but the story about extending it with more FSes isn't clear.
 Another route is to define the API at a lower level, like in the ASYNCFS pr, which
makes it much easier to add custom filesystems, including out of tree, and including
 async ones. We'd need to add helpers there for permissions, basic stdout, etc., as I
imagine most FSes would want those. If that interests you then perhaps that API would
be a good thing to use and collaborate on!

pmp-p commented on Sep 22, 2019

pmp-p
on Sep 22, 2019
Contributor

@curiousdannii you may also want to have a look at BrowserFS emscripten connectors
https://github.com/jvilk/BrowserFS#using-with-emscripten it provides already some handy
custom fs.

camilstaps
mentioned this on Jun 23, 2020

 Separate file modification/access times (or, alternatively, dropping access times) #11454

curiousdannii commented on Aug 5, 2020

curiousdannii
on Aug 5, 2020 · edited by curiousdannii
ContributorAuthor

I've started developing my custom FS, and am at the stage where very basic things work. So
here's the answers to my questions:

What functions need to be defined at a minimum?

In order for the app I am porting to read a file I needed to define: close, createNode, llseek,
lookup, mount, open, read. Obviously more will be necessary for writing files, but this does
show that it's not a lot of work to get started with a custom FS, and you can wait to flesh out the
implementation until you need more of it.

How do you compile it in? Just a regular JS library inclusion instead of a -S setting?
Would a custom FS still need to be mounted like the others?
It doesn't need to be compiled at all, you can provide a custom FS purely at runtime. You do
need to mount it, but that's easy, it's two function calls (FS.mkdir and FS.mount).

PS, I do love the sound of ASYNCFS.

curiousdannii commented on Aug 6, 2020

curiousdannii
on Aug 6, 2020 · edited by curiousdannii
ContributorAuthor

Documentation on what each of the node_ops and stream_ops functions are for would help. I've
got reading and writing of files working, but now I don't know how to say that a missing file
can't be opened. From reading library_fs.js it looks like it has to occur before stream_ops.open
is called, but I'm still trying to work out which other function call needs to fail. Edit: answer is
that lookup needs to throw.

stale commented on Aug 10, 2021

stale
on Aug 10, 2021

This issue has been automatically marked as stale because there has been no activity in the past
year. It will be closed automatically if no further activity occurs in the next 30 days. Feel free to
re-open at any time if this issue is still relevant.

stale
added
wontfix
and removed
wontfix
on Aug 10, 2021
curiousdannii
changed the title How to create a custom file system? Document how to create a custom file
system on Aug 10, 2021

stefnotch commented on Aug 12, 2021

It would be lovely to have a proper documentation for this. Which functions are required, which
are optional? And maybe even have some helper functions to easily add in a simplistic
filesystem.

The current state of custom file systems is a bit sad. As I understand it, the options are

 Use the undocumented emscripten file system. This involves things like writing code
similar to this or that yourself
 Use the seemingly abandoned BrowserFS. Looking at the commits, BrowserFS hasn't
received much attention since 2018
 Copy-paste the library_nodefs.js and then use a library that provides a node-like
filesystem. The issue here is that of the two libraries that do this in the browser, lightning-
fs probably doesn't support enough functions and filer.js isn't really being maintained
anymore
 Hack a single node from the in-memory-filesystem to have a custom content:
node.content = new Proxy(/* proxy filesystem */)
 Write my own fs.createLazyFile

stefnotch
mentioned this on Aug 12, 2021

 Accessing files within Python using native open() nettyso/uvic-localstar#10

curiousdannii commented on Aug 12, 2021

curiousdannii
on Aug 12, 2021 · edited by curiousdannii
ContributorAuthor

I've just thought of another option, and this may be one of the few working options if you need
Async file operations: use the linker's --wrap option to intercept stdio.h function calls, which
you could then replace with ones that implement a custom FS. This would only be a good option
if all file function calls should go to your custom file system; if you want to only use a custom
FS for some paths and keep using MEMFS or NODEFS for the rest, then mounting a custom
Emscripten FS would be better.
kripken commented on Aug 13, 2021

kripken
on Aug 13, 2021
Member

I hope that before the end of this year we do a rewrite of much of the FS code, fixing several
current issues along the way. I agree that adding proper docs for how to write a custom backend
would be a good addition to there, I'll make sure to mention this when planning begins.

stefnotch
mentioned this on Aug 14, 2021

 Discuss desired features/syntax for persistence convenience js/py API

stefnotch commented on Aug 17, 2021

stefnotch
on Aug 17, 2021

I just went down the road of implementing my own filesystem that talks to the Jupyter Contents
REST API. It is a simplistic networked filesystem. I thought it might be helpful to share my
experiences here, so here goes nothing:

1. I had no idea how to write it, so I ended up using a fair number of references.
Interestingly enough, the references don't really agree on what to do in certain cases. For
example, the in-memory filesystem will seemingly always throw an error when calling
the lookup function while the node filesystem will actually look the files up.
I suppose if one were to dig deeper into the filesystem code, this would make perfect
sense. This is also the sort of stuff that I'd love to see documented

 https://github.com/jvilk/BrowserFS/blob/master/src/generic/emscripten_fs.ts
 https://github.com/emscripten-core/emscripten/blob/main/src/library_nodefs.js
 https://github.com/emscripten-core/emscripten/blob/main/src/library_memfs.js
 https://github.com/emscripten-core/emscripten/blob/main/src/library_workerfs.js
 https://github.com/curiousdannii/emglken/blob/master/src/emglkenfs.js

2. I wasn't really sure what to do with the setattr and getattr functions, since the REST API
didn't expose any of that. So I had no way of actually doing any of that. What I ended up
going with was a somewhat primitive default implementation
 3. The next interesting tidbit when hooking up your own filesystem: All the errors don't
match up! Emscripten quite explicitly wants FS.ErrnoErrors. Here it might be useful to
document how to translate between typical errors and those. Like "ENOENT should be
thrown when ...". In the end, I opted for a best-effort translation. If that failed, I threw a
EPERM error, even if it's probably the wrong one to throw.
4. The symlink and readlink functions are also functions that frequently can't be
implemented with a given underlying filesystem.
5. I wish I didn't have to search so hard for the DIR_MODE and FILE_MODE constants
https://github.com/gzuidhof/starboard-python/blob/88327bb56f7e2f673cf1ea4c5ff1e0a68
c50e288/src/worker/emscripten-fs.ts#L120
6. And finally, the stream_ops. Since my underlying filesystem was a REST API with no
streaming API, I had to do quite a bit of work. I ended up writing it so that
stream_ops.open will download the file and save it in stream.fileData = ... as a
Uint8Array. Then, stream_ops.read and stream_ops.write will modify the
stream.fileData. And finally, stream_ops.close uploads the stream.fileData
again.

do a rewrite of much of the FS code, fixing several current issues along the way

On whole, I think that the Emscripten filesystem code is surprisingly decent once you figure out
all those things. Implementing a custom filesystem doesn't take too much code. My favorite ones
were functions like mknod, since they're very straightforward to implement

I'm most certainly looking forward to the rewrite and improvements! Can't wait to see what the
excellent Emscripten team will come up with

bollwyvl
mentioned this on Jun 1, 2022

 Implement a custom Emscripten File System which communicates with the JupyterLab
Content Manager, giving file access to pyolite jupyterlite/jupyterlite#655

reinhrst commented on Feb 20, 2023

reinhrst
on Feb 20, 2023

I hope that before the end of this year we do a rewrite of much of the FS code, fixing several
current issues along the way. I agree that adding proper docs for how to write a custom backend
would be a good addition to there, I'll make sure to mention this when planning begins.
@kripken I fully understand if the answer is "no" (priorities change, etc); just wondering if you
ever got around to this?

kripken commented on Feb 21, 2023

kripken
on Feb 21, 2023
Member

@reinhrst We did start the WasmFS rewrite of the FS code, and made a lot of progress on it. For
a lot of things it can work well enough already. However, we don't have much in terms of docs
for creating new filesystems in it yet. But the existing ones can be useful to read, for example
here is the WasmFS fetch backend:

 https://github.com/emscripten-core/emscripten/blob/main/system/lib/wasmfs/backends/
fetch_backend.cpp
 https://github.com/emscripten-core/emscripten/blob/main/src/library_wasmfs_fetch.js

to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata
Assignees

No one assigned

Labels

No labels

Type

No type

Projects

No projects

Milestone

No milestone
Relationships

None yet

Development

No branches or pull requests

Participants

Issue actions
Footer
© 2025 GitHub, Inc.

Footer navigation

 Terms
 Privacy
 Security
 Status
 Docs
 Contact


  

Document how to create a custom file system · Issue #9372 · emscripten-core/emscripten