`EmscriptenFS` and preloading #325

pmp-p · 2022-04-02T23:58:02Z

Hi, I have the following layout for running Pygame games apk directly on web

   // VM is emscripten Module, modularized or not

    VM.APK = "org.pygame.asteroids"
    var BFS = new BrowserFS.EmscriptenFS()

    const Buffer = BrowserFS.BFSRequire('buffer').Buffer;

    function apk_cb(e, apkfs){
        console.log(__FILE__,"APK", VM.APK,"received")
        window.document.title = VM.APK
        BrowserFS.FileSystem.InMemory.Create(
            function(e, memfs) {
                BrowserFS.FileSystem.OverlayFS.Create({"writable" :  memfs, "readable" : apkfs },
                    function(e, ovfs) {
                                BrowserFS.FileSystem.MountableFileSystem.Create({
                                    '/' : ovfs
                                    }, async function(e, mfs) {
                                        await BrowserFS.initialize(mfs);
                                        // BFS is now ready to use!
                                        await VM.FS.mount(BFS, {root: "/"}, "/data/data/" + VM.APK );
                                        await VM.FS.mkdir("/data/data/" + VM.APK + "/need-preload");
                                        VM.vfs = BFS
                                    })
                    }
                );

            }
        );
    }

    fetch(VM.APK + ".apk").then(function(response) {
        return response.arrayBuffer();
    }).then(function(zipData) {
        BrowserFS.FileSystem.ZipFS.Create({"zipData" : Buffer.from(zipData),"name":"apkfs"}, apk_cb)
    })

but emscripten_run_preload_plugins() cannot operate on Wasm/Image/Audio files located under /data/data//.
error messages are all like
Image blob:http://localhost:8000/b5c7a41e-0824-4c8c-b1b2-672608468e73 could not be decoded
`

browserfs 1.4.3

ref: https://emscripten.org/docs/api_reference/emscripten.h.html#c.emscripten_run_preload_plugins

The text was updated successfully, but these errors were encountered:

james-pre · 2023-03-09T23:06:26Z

@pmp-p Have you tried with 2.0.0-beta?

pmp-p · 2023-03-11T22:37:33Z

nvm i only use preload plugin now for libraries and it was an emscripten bug emscripten-core/emscripten#17956, forgot to close sorry

pmp-p · 2023-09-14T19:48:04Z

re-opening and quoting an emscripten expert on that "I guess FS.analyzePath in browserfs isn't setting .object.contents on the return value" if it can help to sched light.

Binary files that are expected to be preloaded by the emscripten plugins are truncated.

james-pre · 2023-09-15T16:33:53Z

@pmp-p

I'm not sure which parts of the BFS code you're referring to. analyzePath is not defined or used anywhere in the code.

If possible, could you create a minimally reproducible example with the latest code? Please note that the latest commit supports async/await on Create methods and configure. I recommend you use configure as it is much easier to use. For example:

import { configure, BFSRequire, ZipFS } from 'browserfs';

const mfs = await configure({
	fs: 'MountableFileSystem',
	options: {
		'/': {
			fs: 'OverlayFS',
			options: { 
				readable:  READABLE_FS,
				writable: { fs: 'InMemory' }
			}
		}
	}
});
const zipfs = await ZipFS.Create({ zipData: ZIP_FILE_BUFFER });
mfs.mount('/mnt/zip', zipfs);

pmp-p · 2023-09-18T00:42:14Z

So i built from git, using npm from emsdk with "npm run build" and used the un-minified
browserfs.js + its .map found in ./dist/

[PyDK:wasm] /data/git/pygbag-TODO/BrowserFS $ npm run build && mv -v dist/browserfs.js* /data/git/pygbag/0.0/

> browserfs@2.0.0 build
> node scripts/build.mjs

Building for browser, unminified...
Built for browser, unminified.
Building for browser, minified...
Built for browser, minified.
Building for ESM, unminified...
Built for ESM, unminified.
Building for ESM, minified...
Built for ESM, minified.
Building for node...
Built for node.
renamed 'dist/browserfs.js' -> '/data/git/pygbag/0.0/browserfs.js'
renamed 'dist/browserfs.js.map' -> '/data/git/pygbag/0.0/browserfs.js.map'

let consider "test.apk" zip file which contains a text file "assets/main.py"
eg https://v6p9d9t4.ssl.hwcdn.net/html/6836147/test.apk ( from an old working sample using older browserfs )

The goal is to mount in emscripten FS at path "/data/data/test" an overlayfs that read the zip file content and still allow to read/write/remove files and create/list/remove folders under "/data/data/test"

i tried without success ( folder and file structure is ok, reading zipped file content is corrupt)

var track_media
if (!vm) {
        vm={}

        // how is passed the FS object ???
        vm.BFS = new BrowserFS.EmscriptenFS()  // {FS:vm.FS}

        vm.BFS.Buffer = BrowserFS.BFSRequire('buffer').Buffer
        const data = await (await fetch('test.apk')).arrayBuffer()
        track_media = await vm.BFS.Buffer.from(data)
} else {
        track_media = MM[trackid].media
}


// how is passed the FS object ???
vm.BFS = new BrowserFS.EmscriptenFS()  // {FS:vm.FS}


vm.BFS.Buffer = BrowserFS.BFSRequire('buffer').Buffer

const data = await (await fetch('test.apk')).arrayBuffer()
const track_media = await vm.BFS.Buffer.from(data)


const zipfs = await BrowserFS.ZipFS.Create({
    zipData: track_media,
    "name": hint
})

const memfs = await BrowserFS.InMemory.Create();

const ovfs = await BrowserFS.OverlayFS.Create({
    writable : memfs,
    readable: zipfs
})

// this alone does not work ? why ??
// const mfs = await BrowserFS.MountableFileSystem.Create({"/" : ovfs})
// console.log( mfs )

const mfs = await BrowserFS.initialize( await BrowserFS.MountableFileSystem.Create({"/" : ovfs}) )

// where is the link beetween (BFSEmscriptenFS)vm.BFS and MountableFileSystem(mfs) ?
// vm.BFS.mount( ???????? )

const emfs = await vm.FS.mount(vm.BFS, {root: "/"}, "/data/data/test" );

needless to say i'm not very fond of javascript so i don't get some things in init sequence hence the above confused comments.

The working code (apart from corner case in preloading ) for older BFS

var bfs2 = false
if (!BrowserFS.InMemory) {
    console.warn(" ==================== BFS1 ===============")
    BrowserFS.InMemory = BrowserFS.FileSystem.InMemory
    BrowserFS.OverlayFS = BrowserFS.FileSystem.OverlayFS
    BrowserFS.MountableFileSystem = BrowserFS.FileSystem.MountableFileSystem
    BrowserFS.ZipFS = BrowserFS.FileSystem.ZipFS
} else {
    console.warn(" ==================== BFS2 ===============")
    bfs2 = true
}

            function apk_cb(e, apkfs){
                BrowserFS.InMemory.Create(
                    function(e, memfs) {
                        BrowserFS.OverlayFS.Create({"writable" :  memfs, "readable" : apkfs },
                            function(e, ovfs) {
                                BrowserFS.MountableFileSystem.Create({
                                    '/' : ovfs
                                    }, async function(e, mfs) {
                                        await BrowserFS.initialize(mfs);
                                        await vm.FS.mount(vm.BFS, {root: "/"}, "/data/data/test");
                                        //setTimeout(()=>{track.ready=true}, 0)
                                    })
                            }
                        );
                    }
                );
            }

            //await BrowserFS.FileSystem.ZipFS.Create(
            await BrowserFS.ZipFS.Create(
                {"zipData" : track_media, "name": "test.apk"},
                apk_cb
            )

i'm starting to think that code worked by accident and that my construction is broken.

james-pre · 2023-09-18T12:22:36Z

@pmp-p The non-fs polyfills have been removed (BFS is not a buffer/path/process polyfill, it is an FS polyfill). I recommend you use configure. For example:

import { configure, BFSRequire, EmscriptenFS  } from 'browserfs';

import ( Buffer } from 'buffer'; // polyfill. If running on node this is not needed.


const data = await (await fetch('test.apk')).arrayBuffer();

const track_media = await Buffer.from(data);

// assuming FS is from Emscripten

       

await configure({

	fs: 'MountableFileSystem',

	options: {

		'/': {

			fs: 'OverlayFS',

			options: {

				readable: { fs: 'ZipFS', options: { zipData: track_media, name: 'hint'  } },

				writable: { fs: 'InMemory' }

			}

		}

	}		

});

const fs =  new EmscriptenFS();

// ... set up emscripten ...

FS.mount(fs, { root: '/', }, '/data');

pmp-p · 2023-09-18T13:14:14Z

so

import { configure, BFSRequire, EmscriptenFS  } from 'browserfs';
import ( Buffer } from 'buffer'; // polyfill. If running on node this is not needed.

did not work for me ( syntax error, also i think import is not supported on some old/obsoleted mobile targets i have like safari/ios ). i load BrowserFS this way <script src="browserfs.js"></script> from html or from a script tag created from the main js file which has type=module but only when window.BrowserFS is undefined ( not the case during the test)

so i tried instead

const data = await (await fetch('test.apk')).arrayBuffer();
const track_media = await vm.BFS.Buffer.from(data);

// assuming FS is from Emscripten
await BrowserFS.configure({
	fs: 'MountableFileSystem',
	options: {
		'/': {
			fs: 'OverlayFS',
			options: {
				readable: { fs: 'ZipFS', options: { zipData: track_media, name: 'hint'  } },
				writable: { fs: 'InMemory' }
			}
		}
	}
});

vm.FS.mount(vm.BFS, { root: '/', }, '/data/data/test');

but same result zip content is garbled, folder structure is ok, file/folder creation works but reading newly created files back is garbled too.

james-pre · 2023-09-18T14:55:36Z

My bad... I had accidentally put a ( on the Buffer import. Here is the fixed version:

import { configure, BFSRequire, EmscriptenFS  } from 'browserfs';

import { Buffer } from 'buffer'; // polyfill. If running on node this is not needed.


const data = await (await fetch('test.apk')).arrayBuffer();

const track_media = await Buffer.from(data);

// assuming FS is from Emscripten

       

await configure({

	fs: 'MountableFileSystem',

	options: {

		'/': {

			fs: 'OverlayFS',

			options: {

				readable: { fs: 'ZipFS', options: { zipData: track_media, name: 'hint'  } },

				writable: { fs: 'InMemory' }

			}

		}

	}		

});

const fs =  new EmscriptenFS();

// ... set up emscripten ...

FS.mount(fs, { root: '/', }, '/data');

If you aren't using ESM, you will need to use BrowserFS from the browserfs.js, which defines It globally.

pmp-p · 2023-09-18T15:27:23Z

import { Buffer } from 'buffer';
But there is no buffer.js in dist. and i have no idea what ESM means

Maybe you are assuming all emscripten users have javascript knowledge, but that's not my case i only do C and python not even C++ ( emscripten is a C/C++ sdk ). Please don't take too much javscript shortcuts i'd really want to help solve that issue.

james-pre · 2023-09-18T15:32:24Z

You will have to install a buffer polyfill separately:

npm i buffer

pmp-p · 2023-09-18T15:53:24Z

Also i think i've made a shortcut too : the env testcase is not Node at all but only browser environnement ( emscripten target=web) and i'm testing on v8 not firefox

james-pre · 2023-09-18T16:17:30Z

I recommend you use a bundler (esbuild is fantastic) to bundle your code before running tests. This bundles dependencies which means you no longer need to worry about modules and imports. If you use esbuild, make sure you set bundle to true and platform to 'browser' in your esbuild config.

james-pre · 2023-10-25T00:49:21Z

Please use zen-fs/core#22

james-pre added the bug label Mar 9, 2023

pmp-p closed this as completed Mar 11, 2023

pmp-p reopened this Sep 14, 2023

pmp-p mentioned this issue Sep 14, 2023

file corruption when using emscripten_run_preload_plugins with BrowserFS emscripten-core/emscripten#17956

Open

james-pre changed the title ~~EmscriptenFS and preloading~~ EmscriptenFS and preloading Oct 25, 2023

james-pre mentioned this issue Oct 25, 2023

EmscriptenFS and preloading zen-fs/core#22

Closed

james-pre closed this as not planned Won't fix, can't repro, duplicate, stale Oct 25, 2023

Repository owner locked and limited conversation to collaborators Oct 25, 2023

james-pre reopened this May 17, 2024

Repository owner unlocked this conversation May 17, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`EmscriptenFS` and preloading #325

`EmscriptenFS` and preloading #325

pmp-p commented Apr 2, 2022 •

edited

Loading

james-pre commented Mar 9, 2023

pmp-p commented Mar 11, 2023 •

edited

Loading

pmp-p commented Sep 14, 2023 •

edited

Loading

james-pre commented Sep 15, 2023

pmp-p commented Sep 18, 2023 •

edited

Loading

james-pre commented Sep 18, 2023 •

edited

Loading

pmp-p commented Sep 18, 2023 •

edited

Loading

james-pre commented Sep 18, 2023 •

edited

Loading

pmp-p commented Sep 18, 2023

james-pre commented Sep 18, 2023

pmp-p commented Sep 18, 2023

james-pre commented Sep 18, 2023 •

edited

Loading

james-pre commented Oct 25, 2023 •

edited

Loading

EmscriptenFS and preloading #325

EmscriptenFS and preloading #325

Comments

pmp-p commented Apr 2, 2022 • edited Loading

james-pre commented Mar 9, 2023

pmp-p commented Mar 11, 2023 • edited Loading

pmp-p commented Sep 14, 2023 • edited Loading

james-pre commented Sep 15, 2023

pmp-p commented Sep 18, 2023 • edited Loading

james-pre commented Sep 18, 2023 • edited Loading

pmp-p commented Sep 18, 2023 • edited Loading

james-pre commented Sep 18, 2023 • edited Loading

pmp-p commented Sep 18, 2023

james-pre commented Sep 18, 2023

pmp-p commented Sep 18, 2023

james-pre commented Sep 18, 2023 • edited Loading

james-pre commented Oct 25, 2023 • edited Loading

`EmscriptenFS` and preloading #325

`EmscriptenFS` and preloading #325

pmp-p commented Apr 2, 2022 •

edited

Loading

pmp-p commented Mar 11, 2023 •

edited

Loading

pmp-p commented Sep 14, 2023 •

edited

Loading

pmp-p commented Sep 18, 2023 •

edited

Loading

james-pre commented Sep 18, 2023 •

edited

Loading

pmp-p commented Sep 18, 2023 •

edited

Loading

james-pre commented Sep 18, 2023 •

edited

Loading

james-pre commented Sep 18, 2023 •

edited

Loading

james-pre commented Oct 25, 2023 •

edited

Loading