Development Workflow

E3SM Development Workflow

E3SM development workflow uses Git with topic branches and pull requests. If you are coming from Subversion, the most important differences are:

• Git is distributed: All repositories are equivalent up to conventions. Committing is a local operation and exchanged via explicit push and pull commands.
• Complete DAG history: The history is a directed acyclic graph (DAG). No information is lost in merges and the tooling offers advanced query and manipulation. Git commands are basically commands for manipulating the DAG.

Setup

• Set your name: git config --global user.name "Bobby Tables"
• Set your email: git config --global user.email "me@example.com"
• Do not push local branches nonexistent on upstream by default: git config --global push.default simple (older versions of git require git config --global push.default nothing)
• Download and source git-prompt to add branch information to your bash prompt (optional)
• Download and source git-completion.bash to enable tab completion of git commands (optional)

Early Users

If you would like to try out a new feature developed by a colleague, they may ask you to try a branch named something like alice/cool-feature. After cloning the repository or running git fetch, you can try their branch by running git checkout alice/cool-feature. If Alice adds additional features to her branch, you can obtain them by running git pull.

Developers

Developers work in topic branches and issue pull requests to an appropriate integration branch (master or a maint branch). After cloning the repository, the typical workflow is to start a new branch from 'master' (or a component integration branch, such as 'atm/master'):

$ git checkout -b yourgitusername/area-of-E3SM/short-description-of-feature master

Develop in your branch, test locally, and commit in logical, reviewable units with good commit messages. When you believe your branch to be complete and tested, push your new branch

$ git push -u origin yourname/short-description-of-feature

and create a pull request to a suitable integration branch (e.g., 'master') github repository.

Integrators

Need to determine which integration branches to have

Every branch has a purpose. Merging into branch branch-A is a declaration that the purpose of branch-A is better served by including those commits that were in branch-B. This is achieved with the command

(branch-A) $ git merge branch-B

Topic branches do not normally contain merge commits, but it is acceptable to merge from master or from other topic branches if your topic depends on a feature or bug fix introduced there. When making such a merge, use the commit message to state the reason for the merge. Never merge next into your branch.

For further philosophy on merges, see

• Junio Hamano: Fun with merges and purposes of branches
• LWN: Rebasing and merging: some git best practices
• Linus Torvalds: Merges from upstream
• petsc-dev mailing list

Racy integration

Two people occasionally attempt to merge at about the same time, in which someone will lose the race. It usually goes like this: you checkout 'next', pull to make sure you have everything from upstream (you didn't forget this, right?), merge 'my/topic-branch', test, and attempt to push, getting an error like:

To git@github.com/E3SM-Project/E3SM
 ! [rejected]        next -> next (fetch first)
error: failed to push some refs to 'git@github.com/E3SM-Project/E3SM'
hint: Updates were rejected because the remote contains work that you do
hint: not have locally. This is usually caused by another repository pushing
hint: to the same ref. You may want to first merge the remote changes (e.g.,
hint: 'git pull') before pushing again.
hint: See the 'Note about fast-forwards' in 'git push --help' for details.

Do NOT perform a non-fast-forward pull on an integration branch. (Doing so creates messy history that does not summarize nicely with git log --first-parent.) Instead, you have two choices. The cleanest is to gracefully lose the race and merge again.

$ git reset --hard origin/next
$ git merge my/topic-branch
... build and test ...
$ git push

This produces clean history with no evidence that you encountered a race and had to try again. If your merge had significant conflicts or if the testing you just did was especially onerous, you can switch hats and merge the result:

$ git reset --hard origin/next
$ git merge ORIG_HEAD
  # edit commit message to state which branch was actually merged
  # due to losing a race to the integration branch.
... build and test ...
$ git push

This keeps both merge commits (which is a record that there was a race, or just clutter) but git log --first-parent still produces an accurate and concise summary.

Day to day

Inspecting history

• Show logs: git log
• Show logs for file or folder: git log file
• Show changes for each log: git log -p (add file or folder name if required)
• Show diff:
  • Current working tree: git diff path/to/file
  • To other commit: git diff <SHA1> path/to/file
  • Compare version of file in two commits: git diff <SHA1> <SHA1> path/to/file
• Show changes that are in master, but not yet in my current branch:
  • At any path: git log ..master
  • Only affecting a path: git log ..master src/dm/impls/plex/
  • In my branch, but not yet in next: git log next.. src/dm/
  • Tabulated by author: git shortlog v3.3..master src/dm/impls/plex
• Showing branches:
  • Not yet stable: git branch --all --no-merged master
  • Being tested by early users: git branch --all --merged next
  • Will be in the next release: git branch --all --merged master
  • Remove --all to the above to not include remote tracking branches (work you have not interacted with yet).
• Find where to fix a bug:
  • Find the bad line (e.g., using a debugger)
  • Find the commit that introduced it: git blame path/to/file
  • Find the branch containing that commit: git branch --contains COMMIT (usually one topic branch, plus next)
  • Fix bug: git checkout topic-branch-name, fix bug, git commit, and merge to next, etc.

Miscellaneous

• Discard changes to a file which are not yet committed: git checkout path/to/file
• Discard all changes to the current working tree: git checkout -f
• Forward-port local commits to the updated upstream head on master: git rebase master (on feature branch)
• Delete local branch: git branch -D <branchname>
• Delete remote branch: git push origin :<branchname> (only after successful integration into master)

Write good commit messages

ComponentName: one-line explanation of commit

After a blank line, write a more detailed explanation of the commit.
Many tools do not auto-wrap this part, so wrap paragraph text at a
reasonable length. Commit messages are meant for other people to read,
possibly months or years later, so describe the rationale for the change
in a manner that will make sense later.

The message should prominently state its impact on users, such as a
different interface or a change of bit-for-bit results.

If this affects any known issues, include "fix #ISSUENUMBER" or
"see #ISSUENUM" in the message (without quotes). GitHub will create
a link to the issue as well as a link from the issue to this commit,
notifying anyone that was watching the issue. Feel free to link to
mailing list discussions.

* Commit messages may contain lists.

* Following these guidelines improves the ability of tools to quickly
  summarize changes according to various criteria.

If other people contributed significantly to a commit, perhaps by
reporting bugs or by writing an initial version of the patch,
acknowledge them using tags at the end of the commit message.

Reported-by: Helpful User <helpful@example.com>
Based-on-patch-by: Original Idea <original@example.com>
Thanks-to: Incremental Improver <improver@example.com>

Further reading.

Merges

Merge commits should explain why the merge occurs. For topic branches being integrated, this is a short summary of the feature or bug fix. If upstream (like 'master') is merged into a topic branch, the commit message should explain what is needed. If you don't know what to write, perhaps the merge should not take place.

External Resources

• Manual and cheat sheets - official documentation
• Collection of tutorials
• Git for Computer Scientists - graphical intro to Git data model