Dependency Visualization: A Breakdown

[JerryHue]

#2649 sparked the idea of improving the dependency graph of Telescope for the purposes of serving out more information about Telescope dependencies to Telescope contributors. This information would include things like open issues that people can work on, such as issues labelled "good first issue", "help wanted", etc. This way we can create a healthier open source community, where we not only contribute code to our project, but also to the projects we depend on.

I would like to start a discussion to how this dependency visualization should be broken down, so that we have an easier time on filing issues, opening PRs, writing documentation, among other things.

The end product we would like to have is a tool, preferably a web page, that we can easily visualize all the dependencies that have an available GitHub project for querying things like issues, PR, etc. The end product does not have to be exactly as described, so that means that the feature set can be expanded or reduced as long as it serves us to achieve our main objective: make contributing to dependency projects much easier!

[humphd]

For me, an MVP of this feature would do the following:

• extend our status service, or create a new microservice to do the following
• automatically run a job once every so often (daily? weekly?) using our current Bull queue and defining a repeatable job that runs on a schedule.
• when it runs, it finds all of the current dependencies for the project. Initially this could be done by parsing the lock file, but later we could make it more intelligent (e.g., looking at docker images, etc)
• for each dependency that's identified, a job can be created in the queue to get details about that repo. The jobs would use the URL of the project or something, and we'd de-dupe automatically on this key (i.e,. the job name would be unique).
• a processor function would be added to get these jobs out of the queue and process them in the background. Each job would get the name/URL of the dependency and make a call to the GitHub API to get data about open issues with certain labels that we care about (good first issue, bug, hacktoberfest, ...). We can then store the URL of the issues we get back for that project.
• a front-end dashboard would need to get added that lets you display and page through open issues. It would need to be sortable/filterable by project, date, language (e.g., JavaScirpt, C#, ...)
• Clicking an issue would take you to the issue in GitHub

This is pretty similar to what we do with feeds right now, only it's using the GitHub API vs. blogs. We'd have to get an API key from GitHub so we don't get rate-limited doing this. We'd also have to be smart about how quickly we do these requests so that we don't flood GitHub's API and get banned.

[JerryHue]

This is a very nice course of action, and it does leverage several other things that Telescope is already using.

I would like to give a fun way to visualize the dependencies for the dashboard: a tree-like diagram, where the root node is Telescope, and the children node are the 1st-level dependencies of Telescope, and each one of the children node has a similar approach. The user can then move around, to check every dependency and how all of them are connected. We can also decide other visual components to add to the tree diagram to convey information efficiently.

[humphd]

I would have that style as a later enhancement. You could use something like d3 to do it.

But I wouldn't start with this. Start with an API that gives the data you'd need to show the results in various forms. Build clients on top of that.

[JerryHue]

One important bit that we have to leave clear is figuring out what a dependency really is. Depending on our definition, this will become very straightforward, or somewhat difficult to discuss.

If we go with the broadest type of dependency, then that would include things like:

• JavaScript libraries (includes both regular and developer dependencies detailed in package.json),
• Docker images (detailed in Dockerfiles scattered throughout the codebase),
• formatting tools (prettier),
• linter tools (eslint),
• Visual Studio extensions (recommended in .vscode/extensions.json),
• runtime engine (nodejs),
• package manager (npm/pnpm),
• versioning control system used in Telescope (git),
• testing framework (jest).

The list, however, is not exhaustive. Maybe, when you think of the word "dependency", you might think of the package.json file of a project. However, notice some dependencies on the list are not necessarily enlisted in the package.json, such as nodejs, git, or the Docker images we have mentioned.

For starters, we can consider the package.json file to be the single source of truth for dependencies used in the project, as it includes several projects. After that, we would have to think of ways to automatically extract dependencies used by the project. This will come naturally as we figure everything out.

[humphd]

Agreed. I think eventually we probably need to do some of this manually and some with automation (i.e., it will be hard to write code that can extract everything, and easy to have some be defined in a .yml file or similar).

To start, reading what's in the pnpm-lock.yaml file will give us our deps to as many levels as we want. The pnpm command can export this list for us.

[sirinoks]

Does anyone already have an idea of how it would look like (front end)? Perhaps a reference to use as an example. Because it could be a list, a map, tables perhaps.. You could just send a link to some other project with the same structure (but different data)

[humphd]

https://deps.dev/npm/%40senecacdot%2Fsatellite/1.17.0/dependencies/graph is a similar tool that Google built (these are the dependencies for Satellite). I'm not sure the graph view provides the best overall experience tbh. But it depends on your goals.

[JerryHue]

A table of the dependencies seems like the easiest approach, which I think we will probably use for a prototype of the client. The graph in deps.dev looks amazing, but functionally speaking is somewhat poor, as it is difficult to see all the name of all dependencies. A tree-like graph might be more easier to look at:

However, it is a matter of prototyping the client and discover how much information we can convey so that it is more discoverable. Maybe the tree-like diagram will not be that great of visualization.

As long as we have our dependency data in a neutral format, we can choose infinite ways to show it, and it will be a matter of iterations to discover which formats are the most efficient to convey the information we want to convey.

[humphd]

I agree, keep it simple to start. The back-end API can be used to support various front-end views later on, since you're just returning JSON.

Let's get a version of this shipped in 2.6!

[sirinoks]

I'm removing a project for this from the old system and creating one in the Projects beta