Languages activity indepth mode refactor

[lowlighter]

Following the merge of #1118, the indepth mode from the 🈷️ Languages activity plugin has been entirely refactored for better maintainability, performances and accuracy.

It is currently available on @master/@main branches and will be officially released with v3.29 later on.
No configuration changes are required, unless you want to try out some of the new options.

If you have any feedbacks, issues or questions about this feature, please feel free to share them !

✨ New options and features

plugin_languages_indepth_custom

You can now manually specify additional repositories to analyze, specify a custom branch or a commit range (using git rev-list syntax).
It is also now possible to specify a repository hosted outside of github.com.

  plugin_languages_indepth_custom: |
    user/repo, 
    user/repo@main, 
    user/repo@main:v1.0..v2.0,
    https://my.domain.localhost/my-self-hosted-repo,

See documentation for more informations.

plugin_languages_analysis_timeout_repositories

In addition to plugin_languages_analysis_timeout, you can now specify a custom limit for the timeout of a single repository analysis.

If a repository timeout, other repositories will still be analyzed while the global timeout hasn't expired.

- plugin_languages_analysis_timeout: 15             # Maximum 15min for all repositories
  plugin_languages_analysis_timeout_repositories: 2 # Maximum 2min per repository

See documentation for more informations.

Support of repositories for recently used languages

Repositories now support the recently used languages section.
Only the commits from the default branch will be processed.

📈 What to expect ?

Reduced amount of unknown bytes

When a file has been deleted or moved around, the algorithm will now checkout the last commit at the time where the file still existed and will attempt to classify the file instead of simply mark it as "unknown".

More bytes should be detected as "languages bytes" rather than "unknown bytes"

Indirect commits detection

As the algorithm now also checks within the commit body using git log --grep, it should be able to detect commits that are based on your work but not published with your handle, such as co-authoring footers (like Co-authored-by: NAME NAME@EXAMPLE.COM)

Better handle of errors

The algorithm has been refactored to better handle errors, which should let the analysis perform further even if errors are encountered (failed to clone a repository, bad gpg, failing syscall, etc.)

⚙️ How does it work?

Below is a summary of the process used to compute indepth statistics (✨ signals changes from previous algorithm version):

Most used mode

• 1. Fetch GPG keys linked to your GitHub account
  • automatically add attached emails to commits_authoring
  • web-flow (GitHub's public key for changes made through web-ui) is also fetched
• 2. Import GPG keys so they can be used to verify commits later
• 3. Iterate through repositories
  • early break if plugin_languages_analysis_timeout is reached
  • skip repository if it matches plugin_languages_skipped
  • ✨ include repositories from plugin_languages_indepth_custom
    • ✨ a specific branch and commit range can be used
    • ✨ a source other than github.com can be used
• 4. Clone repository
  • ✨ target branch is checkout
• 5. List of authored commits is computed
  • using git log --author and commits_authoring to search in commit headers
  • ✨ using git log --grep and commits_authoring to search in commit body
  • ✨ ensure these are within the range specified by plugin_languages_indepth_custom (if applicable)
• 6. Process authored commits
  • ✨ early break if plugin_languages_analysis_timeout_repositories is reached
  • using git verify-commit to check authenticity against imported GPG keys
  • using git log --patch to extract added/deleted lines/bytes from each file
  • using GitHub linguist (linguist-js) to detect language for each file
    • respect plugin_languages_categories option
    • ✨ if a file has since been deleted or moved, checkout on the last commit file was present and run linguist again
• 7. Aggregate results

Recently used mode

• 1. Fetch push events linked to your account (✨ or target repository)
  • matching plugin_languages_recent_load and plugin_languages_recent_days options
  • matching committer emails from commits_authoring
• 2. Process authored commits
  • using GitHub linguist (linguist-js) to detect language for each file
    • respect plugin_languages_recent_categories option
    • ✨ directly pass file content rather than performing I/O and simulating a git repository
• 3. Aggregate results