[Feat]: Configure hooks directly in workflow step (i.e. without `templatesync.yml`) #467

kdeldycke · 2024-02-26T17:37:48Z

Describe the feature

actions-template-sync supports hooks and that's great! I'm planning to use them to run a couple of sed commands in the precommit phase, so I can adapt to the target repository some hard-coded strings from the source template repository.

For this I need a templatesync.yml. What I'd love to have is a way to encode these rules directly on the workflow step.

I.e. I'd like to write this:

on:
  (...)

jobs:
  repo-sync:
    (...)

    steps:
      - uses: actions/checkout@v4
        
      - name: actions-template-sync
        uses: AndreasAugustin/actions-template-sync@v1.6.2
        with:
          github_token: ${{ secrets.GITHUB_TOKEN }}
          source_repo_path: <owner/repo>
          upstream_branch: <target_branch> 
          pr_labels: <label1>,<label2>[,...] 
          is_allow_hooks: true
          hooks:
            precommit:
              commands:
                # Replace "/kdeldycke/awesome-template/" in URLs by "/kdeldycke/awesome-iam/"
                - sed -i "s/\/kdeldycke\/awesome-template\//\/kdeldycke\/awesome-iam\//g" ./*.md

And bypass entirely the creation of a templatesync.yml file.

Use Case

I have a collection of kdeldycke/awesome-* repositories, in which I push content from my kdeldycke/awesome-template repository.

To do that I rely on a reused workflow, that is called in the target awesome-* repositories.

With the current version of actions-template-sync, I have to add an maintain a dedicated templatesync.yml file in each target repository.

But with the addition of hook rules directly in the workflow step, I can create generic sed commands in my reuseable workflows. That's extremely powerful to automate tedious work.

Proposed Solution

Add support for the hooks > precommit > commands keywords under the with parameter.

Notes

This is a follow up on:

Acknowledgements

I may be able to implement this feature request

The text was updated successfully, but these errors were encountered:

AndreasAugustin · 2024-02-26T18:08:10Z

Hi @kdeldycke and thanks for the feature request.

Just some first fast thoughts:

If I remember right the reason for having the file approach was the missing possibility to use objects within GitHub action input parameters.

https://docs.github.com/en/actions/creating-actions/metadata-syntax-for-github-actions#inputs

If there haven't been changes lately the parameters within the with need to be a string.

Maybe 🤔 it is possible to split the current action so that you are able to execute the hooks even in another action. Not super sure if this is working (need to check how the filesystem is acting within the dockerized action)

I definitely need to do some research first.

kdeldycke · 2024-02-26T18:31:34Z

Maybe 🤔 it is possible to split the current action so that you are able to execute the hooks even in another action. Not super sure if this is working (need to check how the filesystem is acting within the dockerized action)

Yeah, that might be a workaround I'm ready to explore. Like having a second step right after actions-template-sync to amend the PR it created. But for this to work I would need at least the actions-template-sync action to output the branch name and/or PR number.

Here is how it would look like:

    steps:
      - uses: actions/checkout@v4.1.1

      - name: Sync from template repo
        id: template_sync
        uses: AndreasAugustin/actions-template-sync@v1.6.2
        with:
          # We need custom PAT with workflows permissions so we can update all the boilerplate .github files from
          # awesome-template.
          github_token: ${{ secrets.WORKFLOW_UPDATE_GITHUB_PAT || secrets.GITHUB_TOKEN }}
          source_repo_path: kdeldycke/awesome-template
          pr_title: "[sync] Update from `awesome-template` repository"
          pr_commit_msg: "[sync] Update from `awesome-template` repository"
          pr_labels: "📚 documentation"

      - name: Checkout template sync PR
        uses: actions/checkout@v4.1.1
        with:
          ref: ${{ steps.template_sync.outputs.pr_branch }}

      - name: Update repo URLs
        # Replace "/kdeldycke/awesome-template/" in URLs by "/kdeldycke/awesome-<repo_id>/".
        run: >
          find ./.github/ -iname "*.md" -or -iname "*.yaml" -exec
          sed -i "s/\/kdeldycke\/awesome-template\//\/kdeldycke\/${{ github.event.repository.name }}\//g" "{}" \;

      - name: Push to PR
        uses: peter-evans/create-pull-request@v6.0.0
        with:
          assignees: ${{ github.actor }}
          commit-message: "[sync] Update template URLs"
          branch: ${{ steps.template_sync.outputs.pr_branch }}

So as you can see, I would need a steps.template_sync.outputs.pr_branch for this to work in theory.

AndreasAugustin · 2024-02-29T08:59:16Z

Hi @kdeldycke .
As a current workaround you can do the following starting with version v1.7.0 (#469)

# using GITHUB_TOKEN here, please adjust to your needs
- name: Sync from template repo
        id: template_sync
        uses: AndreasAugustin/actions-template-sync@v1.7.0
       env:
         MY_ENV_VARIABLE_NAME: awesome-iam  # is possible to be used within the templatesync.yml
        with:
          github_token: ${{ secrets.GITHUB_TOKEN }}
          source_repo_path: kdeldycke/awesome-template
          pr_title: "[sync] Update from `awesome-template` repository"
          pr_commit_msg: "[sync] Update from `awesome-template` repository"
          pr_labels: "📚 documentation"
          is_allow_hooks: true

(still need the file)
and the file templatesync.yml

 hooks:
      precommit:
        commands:
          # Replace "/kdeldycke/awesome-template/" in URLs by "/kdeldycke/awesome-iam/"
          - sed -i "s/\/kdeldycke\/awesome-template\//\/kdeldycke\/${MY_ENV_VARIABLE_NAME}\//g" ./*.md

This should solve your current use case. Nevertheless I will think about the other options.

Remark
The output pr_branch is now available with #472 (v1.8.0), but will further think about the options

AndreasAugustin · 2024-02-29T09:07:33Z

@all-contributors please add @kdeldycke for idea

allcontributors · 2024-02-29T09:07:43Z

@AndreasAugustin

I've put up a pull request to add @kdeldycke! 🎉

Refs: AndreasAugustin/actions-template-sync#467

kdeldycke · 2024-03-02T07:37:18Z

I experimented yesterday with the new pr_branch but stumbled upon an issue of it being empty: #476

AndreasAugustin · 2024-03-02T10:47:55Z

I experimented yesterday with the new pr_branch but stumbled upon an issue of it being empty: #476

yes, sorry. Did not consider those 'edge-cases'. #479 (v1.8.1) should fix that issue.
Please consider further edge cases (https://github.com/AndreasAugustin/actions-template-sync#action-outputs), but those should not be sufficient for your current issue (related to the action run log).

There is likely to be a further feature (soon) #478 which will likely give some value to your workaround solution (actually that was the feature I meant with the split I mentioned within my comment above).

Just out of curiosity.
Have you also considered using the env variable solution I mentioned above? The idea is to use something like
(Remark within https://github.com/AndreasAugustin/actions-template-sync#lifecycle-hooks) mentioned here: #467 (comment)

kdeldycke · 2024-03-02T11:56:05Z

Thanks for identifying and listing the edge-case regarding pr_branch! To be honest these edge-cases are surprising for newcomers and I predict we'll get some tickets from users about these. 😅 But I'm happy with the current state of things and I'll find a way to workaround current limitations.

And good ideas about #478. That's a powerful feature! 💪

As for the env way, I'm not too fond of that. The syntax is awkward as it's bash syntax (${MY_VAR}) embedded in a YAML, where I expect to find GitHub's own ${{ }} string interpolation grammar. It too much cognitive overload to my small brain ahah! 😁

This, and I don't want to copy-n-past, keep in sync and maintain another Nth CI/CD file at the root of my repositories. My policy is: move every repository-specific config in a pyproject.toml or hide the gory details in a reused workflow. So if I go with templatesync.yml[^1], I'll have to generate it dynamically in-place, the way I generate a ruff.toml file then remove it before committing.[^2]

[^1] BTW, speaking of edge-cases, do you recognize the .yaml extension in addition to .yml? 😁

[^2] Another alternative would be to support remote templatesync.yml, like so:

      - name: actions-template-sync
        uses: AndreasAugustin/actions-template-sync@v1
        with:
          (...)
          templatesync: https://raw.githubusercontent.com/kdeldycke/workflows/v3.0.0/templatesync.yml

kdeldycke · 2024-03-04T10:57:37Z

It took me a week and 45 commits, but I finally found a workaround to do what I was looking for.

Tl;Dr:

  awesome-template-sync:
    name: Sync awesome template
    if: >
      startsWith(github.event.repository.name, 'awesome-')
      && github.event.repository.name != 'awesome-template'
    runs-on: ubuntu-22.04
    # We need custom PAT through the whole job so we get workflow permissions to update all the boilerplate .github
    # files from awesome-template.
    steps:

      - name: Initial checkout
        uses: actions/checkout@v4.1.1
        with:
          token: ${{ secrets.WORKFLOW_UPDATE_GITHUB_PAT || secrets.GITHUB_TOKEN }}
          fetch-depth: 0

      - name: Sync from template repo
        id: template_sync
        uses: AndreasAugustin/actions-template-sync@v1.8.1
        with:
          github_token: ${{ secrets.WORKFLOW_UPDATE_GITHUB_PAT || secrets.GITHUB_TOKEN }}
          source_repo_path: kdeldycke/awesome-template
          pr_title: "[sync] Updates from `awesome-template`"
          pr_commit_msg: "[sync] Updates from awesome-template"
          pr_branch_name_prefix: "sync-awesome-template"
          pr_labels: "📚 documentation"

      - name: Checkout new template sync branch
        uses: actions/checkout@v4.1.1
        with:
          token: ${{ secrets.WORKFLOW_UPDATE_GITHUB_PAT || secrets.GITHUB_TOKEN }}
          ref: ${{ steps.template_sync.outputs.pr_branch }}
          fetch-depth: 0

      # XXX We cannot rely on peter-evans/create-pull-request to manage the push of the new commit to the PR, as it
      # ends up with a: "Unexpected error: fatal: could not open '.git/COMMIT_EDITMSG': Permission denied".
      # See: https://github.com/AndreasAugustin/actions-template-sync/issues/484
      # So we manage the next steps with bare git calls.

      - name: Update repo URLs
        # Replace "/kdeldycke/awesome-template/" in URLs by "/kdeldycke/awesome-<repo_id>/".
        run: >
          find ./.github/ -type f -iregex ".*\.\(md\|yaml\)$" -print -exec sed -i
          "s/\/kdeldycke\/awesome-template\//\/kdeldycke\/${{ github.event.repository.name }}\//g" "{}" \;

      - name: Deactivate LFS
        # XXX Was introduced in https://github.com/AndreasAugustin/actions-template-sync/commit/5335a55
        run: |
          git config lfs.https://github.com/kdeldycke/${{ github.event.repository.name }}.git/info/lfs.locksverify false

      - name: Setup Git user
        run: |
          git config --global user.name "${{ github.actor }}"
          git config --global user.email "${{ github.actor }}@users.noreply.github.com"

      - name: Merge change to previous commit
        run: |
          git commit --all --amend --no-edit

      - name: Push new commit to PR
        run: |
          git push --force

This job is available at: https://github.com/kdeldycke/workflows/blob/ac65c9b5a5878d9b7c301cb5166302c147be55e3/.github/workflows/docs.yaml#L367-L419

kdeldycke · 2024-03-04T11:14:25Z

Note that my workaround above is only working after a second pass. The fist pass always ends up on this error:

  git commit --all --amend --no-edit
  shell: /usr/bin/bash -e {0}
error: insufficient permission for adding an object to repository database .git/objects
error: .github/ISSUE_TEMPLATE/new-link.yaml: failed to insert into database
error: unable to index file '.github/ISSUE_TEMPLATE/new-link.yaml'
fatal: updating files failed
Error: Process completed with exit code 128.

This might be related to: #484

The full log of that workflow run is available at: https://github.com/kdeldycke/awesome-billing/actions/runs/8139058758/job/22241336465

hooks can be configured directly within the workflow efinition

AndreasAugustin · 2024-03-05T20:31:07Z

@kdeldycke thanks for the input and super sorry for the hassle you are having getting this to work.
#489 should be the part you kinda looking for (available with v1.10.0)
Sad part: Was not able to define the hooks as yaml object (seems not supported by github actions so far). You need to use a yaml string (have a prefix > after the hooks input parameter), e.g.

- name: Test action step
  uses: AndreasAugustin/actions-template-sync@v1
  with:
    source_repo_path: AndreasAugustin/template.git
    upstream_branch: main
    is_dry_run: true
    is_allow_hooks: true
    hooks: >
      prepull:
        commands:
          - echo 'hi, we are within the prepull phase'
          - echo 'maybe you want to do adjustments on the local code'

Hope this is also ok for your use case. (unfortunately I did not think about this solution at first thoughts... )

AndreasAugustin · 2024-03-05T20:43:10Z

Thanks for identifying and listing the edge-case regarding pr_branch! To be honest these edge-cases are surprising for newcomers and I predict we'll get some tickets from users about these. 😅 But I'm happy with the current state of things and I'll find a way to workaround current limitations.

Yes, kinda agree 👍
The thing is that we have the following possible flows and therefore those edge cases 😸 (open for discussions)

template changes already merged -> in this case it does not make sense to open a PR because those changes are already part of the history
template changes are already in a PR -> in this case the action finishes because the PR is already open. (Your use case [Feat]: Overwrite open PR #482 not considered yet).

And good ideas about #478. That's a powerful feature! 💪

Thanks 😊 will try to implement this soon. Thanks btw for your found bugs. Think I need to do some investigations before fully implement the feature.

As for the env way, I'm not too fond of that. The syntax is awkward as it's bash syntax (${MY_VAR}) embedded in a YAML, where I expect to find GitHub's own ${{ }} string interpolation grammar. It too much cognitive overload to my small brain ahah! 😁

understood, but as a workaround would have possibly solve the issue faster 🥷 (but am super happy about the work and the mentioned bugs) . Agree on the bash interpolation part (YAML has a small difference within variable interpolation syntax)

This, and I don't want to copy-n-past, keep in sync and maintain another Nth CI/CD file at the root of my repositories. My policy is: move every repository-specific config in a pyproject.toml or hide the gory details in a reused workflow. So if I go with templatesync.yml[^1], I'll have to generate it dynamically in-place, the way I generate a ruff.toml file then remove it before committing.[^2]

[^1] BTW, speaking of edge-cases, do you recognize the .yaml extension in addition to .yml? 😁

you got me 🥷

[^2] Another alternative would be to support remote templatesync.yml, like so:

Yes, nice idea. Maybe also (or in addition) git submodules is possible.

      - name: actions-template-sync
        uses: AndreasAugustin/actions-template-sync@v1
        with:
          (...)
          templatesync: https://raw.githubusercontent.com/kdeldycke/workflows/v3.0.0/templatesync.yml

* feat(#467): ✨ option to set hooks directly hooks can be configured directly within the workflow efinition * fix: 🐛 * fix: 🐛 * fix: 🐛 * fix: 🐛 * fix: 🐛 * fix: 🐛 * fix: 🐛 * fix: 🐛

Refs: AndreasAugustin/actions-template-sync#467

kdeldycke · 2024-03-18T16:24:23Z

While the hooks are not proper YAML objects, having a way to add them directly in the action is a great progress! Thanks to this new parameter, I simplified a lot my workflow and everything is now self-contained within the actions-template-sync step.

Thanks a lot @AndreasAugustin for listening and considering my use-cases. actions-template-sync is a fantastic action and my little investment in it will pay off in the future by reducing the maintenance burden on my project. 💪

AndreasAugustin mentioned this issue Feb 29, 2024

[Feat]: hooks - support github actions env variables #468

Closed

1 task

allcontributors bot mentioned this issue Feb 29, 2024

docs: add kdeldycke as a contributor for ideas #471

Merged

AndreasAugustin mentioned this issue Feb 29, 2024

[Feat]: define output variables #472

Closed

1 task

kdeldycke added a commit to kdeldycke/workflows that referenced this issue Mar 1, 2024

Update repository URLs before recommiting symc template PR

11898af

Refs: AndreasAugustin/actions-template-sync#467

kdeldycke mentioned this issue Mar 2, 2024

[Bug]: Empty pr_branch output variable #476

Closed

1 task

AndreasAugustin mentioned this issue Mar 2, 2024

[Feat]: enable partially execution of the action #478

Closed

1 task

kdeldycke mentioned this issue Mar 3, 2024

[Feat]: Overwrite open PR #482

Closed

1 task

kdeldycke mentioned this issue Mar 4, 2024

[Bug]: Remote "origin" does not support the Git LFS locking API #485

Closed

1 task

AndreasAugustin added a commit that referenced this issue Mar 5, 2024

feat(#467): ✨ option to set hooks directly

5c1ab6e

hooks can be configured directly within the workflow efinition

AndreasAugustin mentioned this issue Mar 5, 2024

feat(#467): ✨ hooks now within action inputs #489

Merged

AndreasAugustin closed this as completed in #489 Mar 5, 2024

github-actions bot mentioned this issue Mar 5, 2024

chore(main): release 1.10.0 #490

Merged

AndreasAugustin self-assigned this Mar 13, 2024

kdeldycke added a commit to kdeldycke/workflows that referenced this issue Mar 18, 2024

Replace template URLs within precommit hook

74f4616

Refs: AndreasAugustin/actions-template-sync#467

kdeldycke mentioned this issue Mar 18, 2024

[Bug]: Action variables in PR body #507

Closed

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feat]: Configure hooks directly in workflow step (i.e. without `templatesync.yml`) #467

[Feat]: Configure hooks directly in workflow step (i.e. without `templatesync.yml`) #467

kdeldycke commented Feb 26, 2024 •

edited

Loading

AndreasAugustin commented Feb 26, 2024 •

edited

Loading

kdeldycke commented Feb 26, 2024

AndreasAugustin commented Feb 29, 2024 •

edited

Loading

AndreasAugustin commented Feb 29, 2024

allcontributors bot commented Feb 29, 2024

kdeldycke commented Mar 2, 2024

AndreasAugustin commented Mar 2, 2024

kdeldycke commented Mar 2, 2024 •

edited

Loading

kdeldycke commented Mar 4, 2024

kdeldycke commented Mar 4, 2024

AndreasAugustin commented Mar 5, 2024 •

edited

Loading

AndreasAugustin commented Mar 5, 2024

kdeldycke commented Mar 18, 2024

[Feat]: Configure hooks directly in workflow step (i.e. without templatesync.yml) #467

[Feat]: Configure hooks directly in workflow step (i.e. without templatesync.yml) #467

Comments

kdeldycke commented Feb 26, 2024 • edited Loading

Describe the feature

Use Case

Proposed Solution

Notes

Acknowledgements

AndreasAugustin commented Feb 26, 2024 • edited Loading

kdeldycke commented Feb 26, 2024

AndreasAugustin commented Feb 29, 2024 • edited Loading

AndreasAugustin commented Feb 29, 2024

allcontributors bot commented Feb 29, 2024

kdeldycke commented Mar 2, 2024

AndreasAugustin commented Mar 2, 2024

kdeldycke commented Mar 2, 2024 • edited Loading

kdeldycke commented Mar 4, 2024

kdeldycke commented Mar 4, 2024

AndreasAugustin commented Mar 5, 2024 • edited Loading

AndreasAugustin commented Mar 5, 2024

kdeldycke commented Mar 18, 2024

[Feat]: Configure hooks directly in workflow step (i.e. without `templatesync.yml`) #467

[Feat]: Configure hooks directly in workflow step (i.e. without `templatesync.yml`) #467

kdeldycke commented Feb 26, 2024 •

edited

Loading

AndreasAugustin commented Feb 26, 2024 •

edited

Loading

AndreasAugustin commented Feb 29, 2024 •

edited

Loading

kdeldycke commented Mar 2, 2024 •

edited

Loading

AndreasAugustin commented Mar 5, 2024 •

edited

Loading