Threading issue

[olivierlabayle]

Hi there,

I think a threading issue was introduced in 0.15. The following works on 0.14 but raises @threads :static cannot be used concurrently or nested in 0.15:

using MLJ
using EvoTrees
using MLJLinearModels
X, y = make_regression()
model = Stack(
    metalearner = LinearRegressor(),
    resampling  = CV(nfolds=2),
    tree        = EvoTreeRegressor()
)
mach = machine(model, X, y)
fit!(mach)

Not sure if it is platform specific, I am on MAC M1 with MLJ v0.19.2.

[jeremiedb]

The move to the :static threads scheduling was made in the context of several discussions arising around some race conditions:

• https://discourse.julialang.org/t/psa-thread-local-state-is-no-longer-recommended-common-misconceptions-about-threadid-and-nthreads/101274/45
• https://julialang.org/blog/2023/07/PSA-dont-use-threadid/

But the caveat with :static is what you just experienced, where MLJ apparently uses threading for some of its routines.
I'll have a second look at the above mentionned caveats as I don't think the code was subject to those. If such is the case, then I should be ably to reinstate the dynamic mode which allows for nested parallelism.

[jeremiedb]

@olivierlabayle Could you give a try at https://github.com/Evovest/EvoTrees.jl/tree/threading?
I've removed the :static to use :dynamic threading. The above MLJ test ran fine on my end (Windows).
I should then be able to make a fix release by tomorrow.

[olivierlabayle]

@olivierlabayle Could you give a try at https://github.com/Evovest/EvoTrees.jl/tree/threading? I've removed the :static to use :dynamic threading. The above MLJ test ran fine on my end (Windows). I should then be able to make a fix release by tomorrow.

Since you've closed this, I've just tried the new 0.16 which works just fine. Thank you for solving it so fast! :)