Update tools/essence-feature-usage-stats

[gskorokhod]

Done

• Update to work with the new directory structure in conjure v2.5.1
• Specify multiple repos for Essence files
• Make repos sortable / hideable
• Add environment variable to cap the number of Essence files that get processed
• Add a way to exclude paths that match a given regex (By default, exclude all autogen files)
• Switched to data tables plugin for the table sorting features
• Added exporting data as CSV

This tool should be done and ready for review now!

[ozgurakgun]

@gskorokhod merged main one more time to bring in the new github actions changes, will stop messing up your PR now :)

[gskorokhod]

Good afternoon! I have fixed the sorting issues by switching to the DataTables plugin instead of my own sorting code, and added a link to download data as a CSV file. There is no Python lint errors and I've documented the new env variables in the project's README so should be ready to merge!
(Checks fail because it says it doesn't have permissions to deploy - The page builds fine in my fork so i assume it will be fine once this is merged?)

[gskorokhod]

@ozgurakgun I'm marking this as ready for review!

[niklasdewally]

(Checks fail because it says it doesn't have permissions to deploy - The page builds fine in my fork so i assume it will be fine once this is merged?)

This is not related to your changes, dw.

---

Ping: @ozgurakgun

Workflows don't get write access on PRs for security.

on:
  push:
    branches:
      - main # run for pushes to the main branch
  workflow_dispatch:
  pull_request:
    paths:
      - tools/essence-feature-usage-stats/**

I would remove PR from the above in the CI file.

Alternatively, if you want this to run in PRs successfully, you can use a workflow_run rule to do this.

In short, workflow A runs in the context of the PR, generates all the pages, and stores them as artifacts.
Workflow B runs in the context of the head of main, and is triggered when workflow A finishes. It loads the artifact saved by A, and deploys it to GH pages. As workflow B runs on main, it has write permissions.

See code-coverage / code-coverage-deploy for existing examples of how to do this.

[niklasdewally]

The above basically ensures that a malicious contributor does not edit CI in a PR, and that edited CI nukes the repo.
Workflow B always runs the copy of itself on main, so is safe.

[ozgurakgun]

We still want to build for PRs, but not push/deploy. An if statement to the deploy step should solve it?

[gskorokhod]

We still want to build for PRs, but not push/deploy. An if statement to the deploy step should solve it?

I will have a look!

[gskorokhod]

Added the if statement
By the way, why do we want to build on pull requests if it is not deployed? Is it just to ensure that it still builds correctly?

[ozgurakgun]

Added the if statement By the way, why do we want to build on pull requests if it is not deployed? Is it just to ensure that it still builds correctly?

exactly!

[ozgurakgun]

does this still need to be in the ENV? did you do multiple repos yet in this PR?

[gskorokhod]

Yes! Repos are specified in another variable (ESSENCE_FILE_REPOS I think?)
This is just the root directory for essence files, all repos will be cloned to its sub directories

[ozgurakgun]

is it not a bit confusing to have it set to EssenceCatalog, which is one of the repo names?

[gskorokhod]

true, changed that

[gskorokhod]

I'm rearranging my branches (making main an exact copy of the main in conjure-cp/conjure-oxide, and creating dev branches for every feature I'm working on), so I will have to close and re-open this PR from a different branch