make DepGraphs its own package?

[anandijain]

I started looking at the dependency graph of the general registry, and I copied a lot of the code from DepGraphs (which is very lightweight).

Preferably I'd like to just depend on that, instead of copying the code.
https://github.com/anandijain/MyPkgGraph.jl if you're curious

[tfiers]

Awesome! Yes, it makes sense to split out the DepGraphs module
(See also

• Split package #32,

which is about splitting out the Graphviz dot-specific code, i.e. DotString, SVG, dotcommand.jl, webapps.jl)

[anandijain]

i dont want to spam you with issues, we can just chat here.
i did @profview and we are just spending the whole 90 seconds just parsing TOML.

I do this when I create the graph of all of General.

    Pkg.Registry.create_name_uuid_mapping!(reg)
    map(x -> Pkg.Registry.init_package_info!(last(x)), collect(reg.pkgs))

Also I think a better approach to making the depgraph is to first convert the entire general registry to a graph, then take the induced subgraph of a tree traversal from the root. (im almost sure that this is exactly what you do, but i havent tested)

https://github.com/anandijain/MyPkgGraph.jl/blob/f9804ab383e68f7788ad0c18c89099b5f3bdc288/src/MyPkgGraph.jl#L138-L163

i realize though that i have a crazy amount of deps that you probably dont want

[tfiers]

Ha no, issues are fine; most issue spam here is from myself

Yes, all time is spent in toml parsing, indeed.
Btw what profiler do you use? My @profview doesn't look so nice

I'm trying to understand what your goals are. Doing graph analysis on the whole registry's dependency graph?

This package is more about dependency DAGs of specific packages under study
(I wanted to rename it to 'PkgGraphs' instead of 'PkgGraph' to highlight that, but alas I had already registered the singular name, and the General registry stewardship is quite strict on renaming)

[anandijain]

its @profview in vscode

the purpose is to reduce precompilation and using times. so im trying to look at which packages would have the most impact on the ecosystem

but for me the end goal would be something like https://perf.rust-lang.org/ but that you can explore the precompilation times and using times on the graph for different julia versions

i have some really messy code to attempt it https://github.com/anandijain/JuliaBenchmarker.jl that gives the import and precompile breakdowns that i now need to integrate with MyPkgGraph

ill then start to run benchmarks across maybe the top n packages and serve them as a rest api with oxygen.

ideally we can make put it on perf.julialang.org

[anandijain]

i should note that doing the induced subgraph method is ~100X faster (tested on DifferentialEquations 17 seconds vs 0.15) and it seems to only take a second or two to generate the registry graph

[tfiers]

I'm not working on this this week; but if you want to go fast, feel free to factor out DepGraphs to a separate package on your own

[anandijain]

no worries not a rush at all, i just realized i forgot to mention the benchmarks in my prior comment

[tfiers]

i should note that doing the induced subgraph method is ~100X faster (tested on DifferentialEquations 17 seconds vs 0.15) and it seems to only take a second or two to generate the registry graph

this is great. maybe PkgGraph.jl should then do this, and do it only once, saving it in a cache somewhere on disk

[tfiers]

I'm also very happy about benchmarking and tracking package load speed and TTFX online!

[tfiers]

I'm thinking of naming the split-off package PkgGraphs.jl (and not DepGraphs) -- and then renaming this package to PkgGraphViz.jl

What do you think @anandijain ?

[anandijain]

sounds good to me

[anandijain]

One thing that I'll want to add is not hardcoding in the General registry.
You can see the methods I added to allow people to generate depgraphs with private registry packages (and unions of registries), basically just extending the name and uuid functions

https://github.com/anandijain/MyPkgGraph.jl/blob/main/src/MyPkgGraph.jl#L61-L105