add experimenal wasm32-wasi-threads support #274

yamt · 2022-12-26T07:48:02Z

possible blockers / known issues:

~~tls issue Initialize TLS of the main thread wasi-libc#372~~
memory import vs export should memory be exported as well? wasi-threads#22
thread-spawn ABI Define types and functions in the proposal's template wasi-threads#26 Experiment the new proposed abi of thread-spawn wasi-libc#385

yamt · 2022-12-26T07:48:32Z

yamt · 2022-12-29T12:53:18Z

with this and #275,
i was able to build ffmpeg with pthread.
https://github.com/yamt/FFmpeg-WASI
https://github.com/yamt/FFmpeg-WASI/releases/tag/yamt-20221229

i was able to use more than 1 cpus with the above binary.
eg.

~/git/toywasm/b/toywasm --wasi --wasi-dir videos -- ffmpeg.wasm -i videos/video-15s.avi -c:v libx264 -threads 16 videos/out.mp4

yamt · 2023-01-11T04:34:55Z

depending on WebAssembly/wasi-threads#22 we might need to wait for a llvm release with https://reviews.llvm.org/D135898

after WebAssembly/wasi-libc#381

abrown · 2023-01-13T06:54:01Z

I wouldn't worry too much about using your own fork here; even if wasi-libc doesn't have the TLS PR merged yet this would still be a step in the right direction. We just would want to make sure that any missing pieces like that PR get updated in the submodule before @sunfishcode publishes a new wasi-sdk release, which I hear might be in the next few weeks (?).

yamt · 2023-01-13T07:49:39Z

I wouldn't worry too much about using your own fork here; even if wasi-libc doesn't have the TLS PR merged yet this would still be a step in the right direction. We just would want to make sure that any missing pieces like that PR get updated in the submodule before @sunfishcode publishes a new wasi-sdk release, which I hear might be in the next few weeks (?).

ok. i will drop the "use my fork" commit. instead, i mentioned known issues in the description of this PR.

sunfishcode

Looks good to me.

abrown · 2023-01-20T00:15:42Z

@sunfishcode, you'll have to merge this (I don't have permissions here but I'm a +1 for this PR). @yamt, do we still need to update the wasi-libc commit in the submodule? If so, that can probably be its own PR.

yamt · 2023-01-20T02:54:53Z

@sunfishcode, you'll have to merge this (I don't have permissions here but I'm a +1 for this PR). @yamt, do we still need to update the wasi-libc commit in the submodule? If so, that can probably be its own PR.

the latest wasi-libc should be ok.
i haven't checked if the submodule is latest enough.

my concern at this point is the coming ABI change from: WebAssembly/wasi-threads#26
cf. WebAssembly/wasi-libc#387

loganek · 2023-01-20T11:52:52Z

How stable we consider wasi-sdk? Are we ok at that stage to introduce backward incompatible changes (that can be detected at compile time)?

sbc100 · 2023-01-20T14:33:26Z

How stable we consider wasi-sdk? Are we ok at that stage to introduce backward incompatible changes (that can be detected at compile time)?

Are you asking in general, or specifically regarding this PR? This PR doesn't break any existing thing right, it add a completely new set of target libraries while leaving the rest of the SDK untouched?

In the general case, WASI is not yet stable or standardized, and neither is wasi-sdk. Of course, we can and should always do our best to avoid breaking existing code, but we are not at the stage in development of WASI where we rule out such things. Even the core components on which we build wasi-sdk (llvm and musl) make breaking changes from time to time so it not practical to rule out all breaking changes.

loganek · 2023-01-20T17:05:34Z

Are you asking in general, or specifically regarding this PR?

Mainly asked in the context of @yamt 's concerns about ABI changes. Looks like wam32-wasi-threads target is even more experimental than the rest of the wasi-sdk, so I don't think we have to wait for the ABI to be finalized?

abrown · 2023-01-20T18:53:14Z

Right, to echo @sbc100, we cannot make too strong guarantees about about the wasi-threads ABI. We have an ABI that is good enough for users to experiment with and we can accompany the prerelease of all of this with some warnings in the documentation.

loganek

LGTM

TerrorJack · 2023-02-08T13:09:46Z

There doesn't seem to be libcxx/libcxxabi builds for wasm32-wasi-threads yet, is it an oversight?

abrown · 2023-02-08T17:55:22Z

No, just not enough time to do everything. Are you thinking of contributing something like this in WebAssembly/wasi-libc#392?

whitequark · 2023-02-23T07:28:05Z

There doesn't seem to be libcxx/libcxxabi builds for wasm32-wasi-threads yet, is it an oversight?

#301

pkgsCross.wasi32.llvmPackages: 12 -> 16 * It appears BULK_MEMORY_SOURCES no longer needs to be set to be empty to compile firefox. Without it, the build of wasilibc would fail if enableThreads is true. * Include preliminary support for the experimental threads support in wasilibc which provides pthreads API. If wasilibc is built with support, is exposed via passthru and libcxx / libcxxabi are built with threads support accordingly. See also: - WebAssembly/wasi-sdk#274 - WebAssembly/wasi-sdk#301 - WebAssembly/wasi-sdk#314 wasi-sdk ships it by default, but as a separate variant of libc which would be a hassle for us. Let's just make it optional for now. You can try it out using the following overlay: self: super: { wasilibc = super.wasilibc.override { enableThreads = true; }; } Flags for libc++abi are copied from wasi-sdk.

denis-belov · 2023-09-18T06:36:02Z

Hello guys! Does it mean that malloc() will be thread-safe from now?

pkgsCross.wasi32.llvmPackages: 12 -> 16 * It appears BULK_MEMORY_SOURCES no longer needs to be set to be empty to compile firefox. Without it, the build of wasilibc would fail if enableThreads is true. * Include preliminary support for the experimental threads support in wasilibc which provides pthreads API. If wasilibc is built with support, is exposed via passthru and libcxx / libcxxabi are built with threads support accordingly. See also: - WebAssembly/wasi-sdk#274 - WebAssembly/wasi-sdk#301 - WebAssembly/wasi-sdk#314 wasi-sdk ships it by default, but as a separate variant of libc which would be a hassle for us. Let's just make it optional for now. You can try it out using the following overlay: self: super: { wasilibc = super.wasilibc.override { enableThreads = true; }; } Flags for libc++abi are copied from wasi-sdk.

yamt mentioned this pull request Dec 29, 2022

wasi-threads SebastiaanYN/FFmpeg-WASI#2

Open

yamt mentioned this pull request Jan 12, 2023

Add the wasm32-wasi-pthread output to the sysroot #287

Closed

yamt added 4 commits January 13, 2023 14:16

Add a variation of wasi-sdk.cmake for pthread-using apps

f4fe55a

Install wasi-sdk-pthread.cmake

a3136f7

Build wasi-libc with THREAD_MODEL=posix as well

2d008b8

wasi-sdk-pthread.cmake: target rename to wasm32-wasi-threads

bab491d

after WebAssembly/wasi-libc#381

yamt force-pushed the pthread branch from d0ecd30 to 566b4d6 Compare January 13, 2023 05:36

yamt changed the title ~~wasm32-wasi-pthread~~ wasm32-wasi-threads Jan 13, 2023

yamt force-pushed the pthread branch from 566b4d6 to bab491d Compare January 13, 2023 07:50

yamt marked this pull request as ready for review January 13, 2023 07:50

yamt changed the title ~~wasm32-wasi-threads~~ add experimenal wasm32-wasi-threads support Jan 13, 2023

This was referenced Jan 19, 2023

Tracking issue for end-to-end wasi-threads support WebAssembly/wasi-threads#10

Closed

How to distribute THREAD_MODEL=posix builds? WebAssembly/wasi-libc#326

Closed

sunfishcode approved these changes Jan 19, 2023

View reviewed changes

loganek approved these changes Jan 20, 2023

View reviewed changes

abrown approved these changes Jan 20, 2023

View reviewed changes

sunfishcode merged commit f420453 into WebAssembly:main Jan 20, 2023

abrown mentioned this pull request Mar 13, 2023

cmake toolchain file for wasm32-wasi-pthread #272

Closed

theoparis mentioned this pull request Apr 27, 2023

Add support for wasm32-wasi-threads ziglang/zig#15484

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add experimenal wasm32-wasi-threads support #274

add experimenal wasm32-wasi-threads support #274

yamt commented Dec 26, 2022 •

edited

Loading

yamt commented Dec 26, 2022

yamt commented Dec 29, 2022

yamt commented Jan 11, 2023

abrown commented Jan 13, 2023

yamt commented Jan 13, 2023 •

edited

Loading

sunfishcode left a comment

abrown commented Jan 20, 2023

yamt commented Jan 20, 2023

loganek commented Jan 20, 2023

sbc100 commented Jan 20, 2023

loganek commented Jan 20, 2023

abrown commented Jan 20, 2023

loganek left a comment

TerrorJack commented Feb 8, 2023

abrown commented Feb 8, 2023

whitequark commented Feb 23, 2023

denis-belov commented Sep 18, 2023

add experimenal wasm32-wasi-threads support #274

add experimenal wasm32-wasi-threads support #274

Conversation

yamt commented Dec 26, 2022 • edited Loading

yamt commented Dec 26, 2022

yamt commented Dec 29, 2022

yamt commented Jan 11, 2023

abrown commented Jan 13, 2023

yamt commented Jan 13, 2023 • edited Loading

sunfishcode left a comment

Choose a reason for hiding this comment

abrown commented Jan 20, 2023

yamt commented Jan 20, 2023

loganek commented Jan 20, 2023

sbc100 commented Jan 20, 2023

loganek commented Jan 20, 2023

abrown commented Jan 20, 2023

loganek left a comment

Choose a reason for hiding this comment

TerrorJack commented Feb 8, 2023

abrown commented Feb 8, 2023

whitequark commented Feb 23, 2023

denis-belov commented Sep 18, 2023

yamt commented Dec 26, 2022 •

edited

Loading

yamt commented Jan 13, 2023 •

edited

Loading