Skip to content

Publishing a new release

Jump to bottom Edit New page
Robert Smith edited this page Dec 14, 2022 · 19 revisions

Quick summary of steps:

In the lead-up to release

Generate changelog

Use the fetch_changelog script to automate the process. This produces a first cut of the changelog based on the merge commits from the last tag on the branch currently checked out on your repo.

To use this:

  • [on initial setup] clone the fetch_changelog repo
  • navigate the top-level folder of your main local MRtrix3 repo (from where you will be generating the changelog)
  • [on initial setup] create the fetch_changelog_settings file with the following content, edited as appropriate (you'll need to generate your own GitHub personal authentication token: 
    repo = 'MRtrix3/mrtrix3'
user = 'jdtournier'
password = 'gngkasgndfgknldsafnfknxgkjd'
  • invoke fetch_changelog and redirect the output to file 
    ../fetch_changelog/fetch_changelog > changelog.txt
  • copy/paste the contents of the file to the community forum (or any other markdown editor), and edit as appropriate.

Prepare announcement post for community forum

  • create a post in the staff category of the forum, to ensure the formatting is as it will be when published, and for other developers to comment on and edit as required.

  • if the changelog is sufficiently long to warrant a separate post rather than a reply to the announcement, insert link to changelog post in the announcement (the URL will need to be updated when released, since the changelog will likely be copy/pasted into a new post to clean up its history).

Determine the branch from which to merge to master

You will need to generate a new git branch, within which the version information will be updated (more on this below). The construction of this branch will depend on the nature of the release:

  • If publishing a macro update (e.g. 3.0.x), containing only bug fixes, then create a new branch based off of master;

  • If publishing a micro update (e.g. 3.x.0), containing feature additions and/or changes to behavior, then create a new branch based off of dev.

For the rest of this document, this new branch will be referred to as "next_release"; but any non-reserved branch name can be used.

Ensure there are no merge conflicts

For a micro update, make sure that next_release can be merged to master without having to resolve merge conflicts. Attempting to create a pull request on GitHub to merge from next_release to master should show this information both prior to, and after, creation of the request. If such conflicts exist, this is typically resolved by:

  1. Merge master into next_release;
  2. Manually resolve any conflicts that arise;
  3. Push the updated branch to GitHub.

Merge to master

This is very fiddly given the number of interactions that need to be handled properly. The main issues to deal with are:

  • the version number in core/version.h needs to be updated (this is the version number that the build script falls back to in case no git history is available).
  • the version of the code checked out during container building needs to correspond to the new tag, so that containers built based on the code at a particular tag will contain code compiled from that same tag.
  • the most recent tag name needs to be updated to match: the build script will fail if they do not match (if the git history is available).
  • the tag needs to correspond to the actual merge commit on master that will constitute the release.
  • the documentation needs to be updated since the help page of some commands will make reference to the online documentation, and the release version will be substituted in these help pages (to ensure no broken links, etc.).

Getting all this right is tricky. This is the approach I recommend - note that it requires admin access for the final push:

  1. Make sure everything is up to date:

    git fetch --all
git checkout next_release
git pull

  2. Edit files core/version.h, Dockerfile and Singularity with desired version information and commit - but do not push (indeed none of the steps below are pushed to GitHub until step 8):

    vim core/version.h

    The Dockerfile and Singularity files will normally point to master, but need to be hard-coded to the exact tag name for that commit to make sure the release recipes always build the release version. Make sure to change these from master to the release tag; these will be reverted back to master immediately after release (later in the instructions).

    vim Dockerfile # Update content of environment variable "MRTRIX3_GIT_COMMITISH"
vim Singularity # Update "git clone" call

    Then commit the changes:

    git add core/version.h Dockerfile Singularity
git commit -m "updated version information"

  3. Tag the resulting commit with the same version information using a signed or annotated commit (so that this is what shows up with git describe), e.g. (edit as appropriate):

    git tag -s 3.0.0 -m "tagged version 3.0.0"

  4. Build and update the docs, then commit the changes:

    ./build
./docs/generate_user_docs.sh
git add docs
git commit -m "updated documentation to match version change"

  5. Merge next_release into master:

    git checkout master
git pull
git merge next_release

    Hopefully this won't generate any merge conflicts - these should have been resolved well in advance of reaching this point...

  6. Delete the previous tag, and re-create a signed tag with the same label on the merge commit from the previous step, e.g. (edit as appropriate):

    git tag -d 3.0.0
git tag -s 3.0.0 -m "tagged version 3.0.0"

  7. Merge master back onto next_release, ready to push for a final round of CI:

    git checkout next_release
git merge master

    The git merge operation should result in a fast-forward merge that leave the commit SHA unmodified. You should find that HEAD, master, next_release, origin/next_release, and your tag all point to exactly the same commit - but not origin/master (check with git log --decorate).

  8. Push next_release to GitHub for a final round of CI. This is important since it can catch issues introduced by the previous steps, and highlight any omissions, etc.. Best to make sure everything builds correctly before pushing out a release that immediately fails...

    git push --follow-tags

    and wait until the CI completes. Note that you need to push the tag as well, otherwise the CI builds will fail.

  9. Once all checks have passed, push the same commit directly to master (this can only be done by admin users). You can't merge using the GitHub interface since it'll create a merge commit, which would mean the tag will not longer point to the tip of master:

    git checkout master
git push --follow-tags

Publish the release

This needs to be done on the GitHub releases page. The tag you pushed in the previous steps should already show up. Edit this tag, give it a name and description (you can draw some inspiration from previous releases), and make sure it is not marked as pre-release.

Once created, the relevant GitHub Actions should trigger automatically to generate the packages. You can check progress on this page.

After release is published

Check generation of binary packages

  • GitHub Actions should generate 4 assets (at the time of writing) that will be uploaded to the release:

    • macOS application package installer
    • conda package for macOS
    • conda package for Linux
    • MSYS2 package for MSYS2

    If any of these fail, this will need to be investigated and fixed.

Upload anaconda packages

The macOS and Linux conda packages will need to be pushed to the MRtrix3 channel. This is not trivial, and needs to be done using the anaconda tool, within a working conda install:

  • [on initial setup] conda install anaconda-client
  • [on initial setup, or as required] conda activate
  • [on initial setup, or as required] anaconda login
    This assumes that you have an account with anaconda, and that you are a member of the MRtrix3 organisation.
  • download the two conda package files (conda-linux-mrtrix3-XXX-XXX_0.tar.bz2 and conda-macos-XXX-XXX_0.tar.bz2)
  • strip the conda-linux- and conda-macos- prefixes from the files.
    This is important: the next step will not work if the filenames are not of the form 'package-version-checksum_num.tar.bz2', and the command provides no information at all about the failure (or indeed that there even was a failure...).
  • upload both packages one at a time: 
    anaconda upload -u MRtrix3 mrtrix3-XXX-XXX_0.tar.bz2

Hopefully that should work OK.

Verify binary installers

This requires as many users and developers as possible to try installing the packages according to our own instructions, both on existing installs and on pristine environments if possible (although realistically we're not going to be able to check all of these scenarios). The main thing is to check that the packages install when they should, and that they work on the different platforms.

Verify updated docs on the documentation site

If everything looks OK, activate the new version in the builds section of readthedocs, to make sure it appears in the list of available versions (not just as latest)

Upload new container to DockerHub

  1. Access the code corresponding to the new tag:

    git checkout 3.#.#

  2. Locally build the Docker container:

    docker build . -t mrtrix3

  3. Push the container to DockerHub with the corresponding tag:

    docker tag mrtrix3 mrtrix3/mrtrix3:3.#.#
docker login
docker push mrtrix3/mrtrix3:3.#.#

Once everything checks out

Post release announcement on community forum

  • If the changelog is sufficiently long to warrant a standalone post, copy/paste the contents of the changelog into a fresh topic in the staff section, and once satisfied that it's good to go, re-categorise it to the 'uncategorised' section. This makes it available to view, but won't send notifications to users.

  • Copy/paste the contents of the announcement into a fresh topic in the Announcements section, and make sure the URL to the changelog points to the updated changelog created in the previous step. If the changelog is short, it can be posted as a reply to this topic.

  • Close both the announcement and changelog topics (prevents further comments), and pin them globally.

  • Make sure that the community forum announcement appears on the website as most recent announcement post. The script that does this runs once a day, but can be run manually by logging onto the DigitalOcean instance that hosts the forum, and running the website update script:

    ssh root@community.mrtrix.org
/etc/cron.daily/auto_gen_blog_post

    You'll obviously need the right credentials to do this...

  • Verify that the announcement shows up correctly on website, and fix up any issues with images and/or emojis, etc.

  • Add links to announcement and changelog on forum to the GitHub release description.

Post on social media, etc.

Revert container recipes

While it is important for consistency of container builds over time that those recipes pull the exact tagged version of code corresponding to the version of the recipe, anyone who clones the master branch and builds a container from one of the recipe files would expect the code compiled within the container to correspond to the current contents of master. As such, following a tag, it is recommended to revert the container recipes back to pulling code from the master branch:

git checkout master
git checkout -b revert_recipes_to_master
vim Dockerfile # Set content of environment variable "MRTRIX3_GIT_COMMITISH" back to "master"
vim Singularity # Revert "git clone" call to clone "master" only
git add Dockerfile Singularity
git commit --author="MRtrixBot <gitbot@mrtrix.org>" -m "Revert container recipes to pull master"
git push --set-upstream origin revert_recipes_to_master

Then generate a Pull Request to merge this change into master.

🍻

Add a custom footer
Add a custom sidebar
Clone this wiki locally