WASM Support

[John0x]

Hey, first of all, this looks extremely promising. I've been looking for a modern airflow alternative for quite some time. Keep up the great work 👍

Have you thought about supporting languages apart from python? Web Assembly would be perfect for something like this. The reliance on python has been one of my biggest pain points with most data pipelines.

[saul-data]

Hi there @John0x, glad you find it promising and thanks for the feedback. I haven't worked with WASM before but my understanding is that its more web browser side. Are you referring to the web browser code editor and running code on the browser? On the server side, essentially any language that can run in Linux can be bundled into the workers for example Rust and called using the Bash processor - this can call any command in bash/sh. If you send me a hello world / basic example of what you are trying to achieve and give a bit of explanation, I can see how it might be built into a worker docker image.

[gedw99]

WASM is for browsers, WASI is for servers. They are both WASM in reality, except the way the Host functions are done is slightly different.

The advantages with WASM are many:

• the code is sandboxed, with no need for heavy docker.
• Can run anywhere, since you don't need docker.
• You can write the pipelines in any language that compiles to wasm. see: https://github.com/appcypher/awesome-wasm-langs
• execution is fast
• The API between the wasm guest and the warm host is controlled using host functions. See: https://www.wasm.builders/k33g_org/extend-wasm-with-host-functions-thanks-to-wazero-3n0n

I noticed that you use NATS. There is excellent going project using WASM / NATS here: https://github.com/bots-garden/capsule

[saul-data]

This is useful thank you. I am increasingly becoming interested in WASM workers and how it can benefit Dataplane.

[gedw99]

that's great !!! I use capsule and other wasm with golang.

[gedw99]

Another thing i use which you might like is yomorun. https://github.com/yomorun/yomo

Its great for driving the GUI with pushed data.

It uses QUIC and falls back to Websockets. Its crazy fast.

It uses WASM as the underlying codec to serialise the golang types between the server and browser.

demo code: https://github.com/yomorun/react-cursor-chat
demo: https://commerce-cursor-chat.vercel.app

You can see the wasm in the browser using the inspector.

[gedw99]

If you want to have a chat about all this let me know.

[saul-data]

@gedw99 Yes I’d very much like to discuss this with you. Would you be free for a video chat or what’s the best method to go through this with you? I have a number of questions and screen sharing would be useful here.

[gedw99]

https://t.me/gedw99 is best :)

[gedw99]

yomo has data frames build into it btw !! so its designed for these types of loose schema style coding.

[John0x]

Ah, sorry, didn't see your response. I meant WASI on the server side with proper integration for the code editor. It would be pretty awesome to have the same experience that you currently have for Python, for Rust or other languages.

[saul-data]

So the key here is about other coding language support?

Let's take Rust for example which requires being compiled. Would that be edited in code editor then compiled to WASI/M then run?

I would need to understand that flow.

[gedw99]

There are a few ways to make it easy for a user to edit their code without the ceremony of downloading an IDE and golang toolset. I assume that’s the reason your asking this

include the tinygo Compiler in the Tools of the a tual server so that golang can be easily compiled to wasm when ever they change the golang code in the web idiot.

min terms of a web ide there are many options. Ace editor , code-server , hackpad. These are all ways of giving a web user an IsdE that runs in the browser. Code server is the easiest because it’s vs code but designed to run as a golang app.

[saul-data]

Dataplane uses Monaco which is essentially vs code so the writing of the code and saving the files down is not an issue.

The issue is how the code gets compiled and run.

We are using Kubernetes to run Dataplane so another question is how this fits into those kind of deployment types.

Just trying to get a sense of how this will work end to end.

If there is a quick prototyping using Monaco editor would be greatly helpful.

[gedw99]

I am currently playing around with code server btw. It’s very good and production quality . Easy to include in the server side stack

the thing I am most interested in is flows and pipelining and Workflows . One wasm function using another wasm function . Capsule using nats is one way . Yomo is another.
Both have their strengths

yomo is not async and does not have a nats message broker and so is non durable .

Capsule is async , has the NATS mesh and can run durable or in memory . The durability is quite useful for the events from one function feeding into another function. Nats can be embedded into a golang program and can do about a million ops / second which I guess is enough .

[gedw99]

Dataplane uses Monaco which is essentially vs code so the writing of the code and saving the files down is not an issue.

The issue is how the code gets compiled and run.

We are using Kubernetes to run Dataplane so another question is how this fits into those kind of deployment types.

Just trying to get a sense of how this will work end to end.

If there is a quick prototyping using Monaco editor would be greatly helpful.

Well code server is also monoco so that’s the same thing but with a proper backend to go with it and easy to embed into the backend golang server and serve the web gui of vscode.

you also get debugging of the golang using delve :)

wasm debugging with a gui for that is being worked on by the Wazero wasm community . There is a gio web gui also but it’s very early days

[gedw99]

Yeah that’s exactly what I am trying to do also :)

[gedw99]

I am Building the gui out of wasm too using golang gio project .

https://gioui.org/doc/install/wasm

[gedw99]

Links

Vscode server and web gui
https://github.com/coder/code-server

golang assembly to golang gui. Built with gioui golang based gui. A wasm to golang gui has been spoken about - see issues
https://github.com/loov/lensm

Wazero is the wasm runner that so many golang projects like capsule use to run their wasm

https://github.com/tetratelabs/wazero

https://github.com/bots-garden/capsule

[saul-data]

I had a look at code server and the integration can be done but it’s not simple. Particularly around saving files - the backend is golang and code server uses nodejs. It pushes the run to the server side. It would need a lot of investigation on how to make it work.

[saul-data]

@John0x @gedw99 Would the below work? I think the easiest way of using different coding languages with Dataplane is simply to copy the compiled binary into workers through your CI/CD cycle.

FROM dataplane/dataplane-worker-python:latest

RUN mkdir -p /customcode && && chown -R appuser:appuser /customcode
COPY RustBinary /customcode/RustBinary

Then in Dataplane use a bash processor, change the command to run the binary.

[gedw99]

I am aiming to not use CI. I want the users to be able to compile to wasm via the backend . My use case is data scientists and scientists being able to do wasm and also to share the wasm with others . But at runtime, not ci tine. Hope that makes sense ?

[gedw99]

The gui has a simple AST that allows them to describe a gui and flow. I have that partially working.

[John0x]

@John0x @gedw99 Would the below work? I think the easiest way of using different coding languages with Dataplane is simply to copy the compiled binary into workers through your CI/CD cycle.

FROM dataplane/dataplane-worker-python:latest

RUN mkdir -p /customcode && && chown -R appuser:appuser /customcode
COPY RustBinary /customcode/RustBinary

Then in Dataplane use a bash processor, change the command to run the binary.

Well, it's definitely better than not being able to use rust at all, but I think it would be a killer feature to have the full experience for other languages as well(at least a few popular ones).
It's not an easy task, that's for sure, but it would be a huge selling point, in my opinion.

https://github.com/infinyon/fluvio
Allows to use Rust with WASM as sandboxed workers.

For the whole build pipeline stuff, you could probably take some inspiration from CI platforms such as Railway. They use nixpacks to support a multitude of languages: https://docs.railway.app/deploy/builds

[saul-data]

@John0x thanks that's helpful. Yes I agree this could be very powerful, I now understand where you are both coming from, this feature is now on my radar and I can look at your examples. We have some priority items that we are busy working on right now so this will take some time to architect and get in. I am hoping that workaround works for now to bridge the time/need gap. 😃

[saul-data]

Here is a good article on what would be required to get Rust to work. You can see the code needs to be compiled first. This is a good example because it uses Golang to run shell commands to compile the Rust code to WASM. This can be applied for WASI too.

https://blog.boot.dev/rust/rust-in-the-browser/

[gedw99]

That’s nice . I have seen exactly the same project to do it all with golang . Can remember the link.

the two grey areas are getting the wasm Web workers to be controlled by the window wasm / js.

you do multi threading by spinning up 4 web workers of the same type also, just like what blazor does .

Do you want to contact me on telegram to discuss approaches. Might be more fruitful than making issues . We probably have slightly different goals but maybe there is common ground too .

[gedw99]

For running rust or golang wasm I use Wazero. The runner is golang

https://www.fermyon.com/blog/introducing-spin is rust and can also run golang or rust wasm

Wazero is integrated with DAPR and nats and k8.

fluvio I have used but I changed to using NATS. They both use the co diners / publishers concepts but NATS is more mature and faster. It also has security integrated into the mesh .

https://docs.nats.io/using-nats/nats-tools/nsc

Fluvio started using Kafka and then rather than use NATS baked then own broker .

Nats is way more mature and more reused and open than fluvio. Imho

[gedw99]

This is similar to dataplane and built on nats

https://github.com/memphisdev/memphis-broker