Utilizing weakdeps/extensions

[aplavin]

I wonder what the Makie devs think about embracing julia package extensions for various deps? Aside from reducing compilation/loading times, these may in practice help cleaning up the overall design.

Those that immediately come to mind are Distributions, OffsetArrays, Unitful – these probably can be made weakdeps without any changes to the interface.
There are a few more that are relatively heavy, but less straightforward to remove: eg KernelDensity that is used in some recipes.

[asinghvi17]

OffsetArrays is already pulled in by the image ecosystem, so there's no point removing it. Distributions and Unitful could possibly go into extensions though, especially since the Unitful support seems to be more or less isolated to one file.

The only potential issue that I can see is supporting older versions of Julia. Introducing Requires would have a pretty bad effect on load time...

[aplavin]

The only potential issue that I can see is supporting older versions of Julia. Introducing Requires would have a pretty bad effect on load time...

Oh I definitely think it shouldn't use Requires in any case. Either only support Julia 1.9+, or keep strong deps before 1.9.

[aplavin]

I looked at Makie heavy deps, and all/most of them are currently used in a manner that doesn't mesh well with switching to weakdeps.
For example, KernelDensity is among the heaviest, and it's currently used in the density() recipe.

I wonder what do you guys think about making it and other deps weak, so that the user has to do using KernelDensity if they want density() plots? Same with Distributions + qqplot, DelaunayTriangulation + tricontourf and some other.

This change would greatly reduce Makie size and installation time and I believe is generally beneficial to users.
Potentially, MakieFull.jl can be created that just loads Makie and all these packages (as Makie does now) without doing anything else.

[DanielVandH]

I like the idea of having something like a MakieFull.jl for that. It might be easier on the user to have Makie itself be the package that brings everything in, and then some MakieReduced (with a better name :-)) that loads what is needed (so that Makie does using X and then using MakieReduced). I imagine a lot of scripts would be broken otherwise (obviously there is SemVer for this purpose, but it might still be a burden). The problem with this being that it would probably be very annoying to do cleanly.

DelaunayTriangulation + tricontourf and some other.

This would be nice. DelaunayTriangulation is also used for triplot and voronoiplot, but I don't know how much those are used compared to tricontourf. I think from some of my discussion with Simon, DelaunayTriangulation.jl especially was probably a bit of a mistake to include directly here rather than as a package extension just inside DelaunayTriangulation.jl itself. The coupling has turned out to be annoying at times.

[aplavin]

I'm fine with either decision, either

• Makie is lighter with extensions for fancy functionality, MakieFull is Makie + all deps
• MakieLight is lighter with extensions for fancy functionality, Makie is MakieLight + all deps

Personally, I think the first one is more natural. And for the majority of users who only use simple plots its result will just be "Makie compiles and loads faster" :)

I imagine a lot of scripts would be broken otherwise (obviously there is SemVer for this purpose, but it might still be a burden). The problem with this being that it would probably be very annoying to do cleanly.

Indeed, this kind of change is obviously breaking, and requires a new version. Although, I think it's much easier to communicate it cleanly compared to other breaking changes (such as the recent switch from convert_arguments to expand_dimensions) – users who try eg qqplot() will see an error saying "please execute using Distributions to use qqplot()". It clear and unambiguous what exactly to do.

[DanielVandH]

users who try eg qqplot() will see an error saying "please execute using Distributions to use qqplot()". It clear and unambiguous what exactly to do.

I forgot about this obvious option 😅 . I think with this it would be a good idea. Any breakage is easy to fix with this type of error message.

MakieFull is probably better with that in mind. I only ever do using (Backend)Makie rather than using Makie but I don't think using MakieFull, (Backend)Makie would be too bad if I want everything.

[aplavin]

Yeah, backends like CairoMakie or GLMakie definitely shouldn't depend on MakieFull – that would lose the whole point of reducing deps :)

Let me know if this idea is supported by Makie devs, if so I myself can start moving deps -> weakdeps.

[DanielVandH]

There is a related issue here #2505

[bilderbuchi]

I wonder what do you guys think about making it and other deps weak, so that the user has to do using KernelDensity if they want density() plots? Same with Distributions + qqplot, DelaunayTriangulation + tricontourf and some other.

Since #2505 was converted into a discussion for some reason, let me harvest another potential candidate from over there to this list: FFMPEG could be a weakdep to avoid dragging it in if you don't plan to use video playback.

[ffreyer]

We want issues to be things that we can directly address and eventually close, so I'm moving the ones that are more open ended, e.g. plans and discussions for changes

[bilderbuchi]

OK -- they are marked "Done" by your automation, though, which seems confusing to me at least, as this does not seem to reflect that status? https://github.com/orgs/MakieOrg/projects/5/views/1 😕

I hope you'll manage to keep an overview, I find the Discussions interface confusing and suboptimal to retain a grasp on things 😅 , things just tend to get lost in there -- in Issues you can at least tag and milestone appropriately..

[ffreyer]

I don't think I have control over that. Github just does that when I convert to discussion

[aplavin]

Any hope for extensions in Makie?.. I believe the main thing required here is the decision from Makie devs that it's the right direction to follow. That is, whether it's ok that stuff like qqplot() would throw an error like "please load Distributions" unless they (or MakieFull) are loaded.

Turning everything into weakdeps in Makie, and creating MakieFull that just loads Makie + all weakdeps would even reduce TTFX for those who load MakieFull. Now, Makie is one of the longest-precompiling packages, and splitting stuff into extensions would allow a noticeable part of it to precompile in parallel.

[aplavin]

Thinking about this every time I see Makie precompiling yet again :)
Are Makie devs open to using weakdeps in such a way (see above)?

[jkrumbiegel]

I don't think weakdeps are the right tool for this. They are supposed to be used when you want to add code for interfacing between two packages without the cost of a strong dependency on one of them. But let's say a contourf plot is not interfacing between Makie and Isobands. It's Makie using a dependency to build a plot. If we were to make this a weakdep then having to load isobands to enable it would become part of the public API, while now it's just an implementation detail how contourfs are made.

There have been discussions about exactly these problems within the Sciml ecosystem. And I think I recently saw a discussion thread on Slack or so where something like "sub packages" was considered, which would need to be a new feature of Julia. But that would mean having one versioned package which can still be broken up into multiple parts that are precompiled and loaded separately.

Maybe @ChrisRackauckas could say if that was more of a brainstorming thing or actually somewhere on the horizon.

[asinghvi17]

A basic sub-package approach was just implemented for OrdinaryDiffEq, where https://github.com/SciML/OrdinaryDiffEq.jl/tree/master/lib contains 30 packages that each define some family of solvers, and there is a core package that holds all the shared functionality, types, function stubs, etc.

Then, OrdinaryDiffEq.jl loads all of these packages to provide the user experience.

In some ways, this is what we moved away from - it used to be that Makie was the wrapper package, and there was a package AbstractPlotting.jl which contained most of the code that's currently in the Makie repo. The backends depended on AbstractPlotting, and Makie.jl was a 5-line package which reexported AbstractPlotting + GLMakie.

Implementing a subpackage-like layout for Makie might mean:

• MakieCore on the bottom level (this is what we currently have)
• MakieBase that holds layouting, scene and block definitions, and recipes that don't need heavy external dependencies
• MakieContourf, MakieDensity, MakieQqplot, etc. that depend on external packages and MakieBase, and define recipes
• Makie that loads MakieCore, MakieBase, MakieContourf, etc and provides a smooth user experience

The only issue I can see is that backends would then have to be built on top of MakieBase, meaning that our current recommendation to load Makie via using GLMakie would not load the extra recipes that using Makie would load.

• Backends (GLMakie, CairoMakie, etc) built on top of MakieBase (not Makie as they are now)?

[jkrumbiegel]

But that's not what I mean. Those Sciml packages are all separate packages. That carries a large coordination burden. I mean subpackages that are actually all just part of one. That is currently not possible.

[asinghvi17]

Ah, you meant specifically for the purpose of precompilation. Yeah there's no way around that at the moment...

[aplavin]

They are supposed to be used when you want to add code for interfacing between two packages without the cost of a strong dependency on one of them. But let's say a contourf plot is not interfacing between Makie and Isobands.

Some stuff, like contourf & isobands, are nontrivial to split cleanly, I agree. But we don't have to start with them :)
Others are quite natural to turn into weakdeps for regular usage, eg:

• lines(UnivariateKDE(...)) instead of density(...)
• qqplot(Normal(), x) instead of qqnorm(x)

In these examples, it's easy to move the implementation to extensions. The first variant of each pair would just work (calling it requires having the other package loaded), and the second variants would require the user to load the corresponding package.

[ChrisRackauckas]

You likely want JuliaLang/julia#55516