Skip to content

Home

Jump to bottom
dcaseGH edited this page May 31, 2023 · 5 revisions

This seems as good a place as any for the documentation. The important points, and worst crimes, of each branch will be detailed here (and struck out if addressed).

Overall structure:

There are three branches: as this is hpcg_27ptStencil, the stencil version is the main/master; the original has branch original; lfric_stencil_data is an extra app. This code is forked from HPCG.

It's possible that it could all be merged, but one app is very different from the other two, and this hasn't been attempted at this stage.

Interaction with reframe/spack

The easiest way to integrate with Spack is to tag a release. Several tags have been made [when stop tweaking code, make ones with definitive names and stick to them]

Original

This preserves the status of HPCG at the forking point. It is used to compare performance with the stencil version (main branch).

Main

This is the same calculation as original, but with the stencil hard coded. The files with subroutines which use direct addressing are: ComputeSPMV_ref.cpp, ComputeSYMGS_ref.cpp, ComputeRestriction_ref.cpp and ComputeProlongation_ref.cpp.

To generate the SYMGS and SPMV subroutines jupyter-notebooks were used: https://github.com/NCAS-CMS/hpcg_27ptStencil/tree/master/python_scripts These are quite untidy:

  • SYMGS is self-contained, but SPMV (done first) requires cat-ing files together
  • They are similar things, at the root, and could probably share subroutines
  • Things are badly named/written etc
  • The code could easily be simplified (SymPy will do this if required - perhaps it would help the compiler later?)
  • The generated subroutines are >10k lines, so I've just left things in this state as a first draft. An outstanding issue is that we want other people to make variants of this problem, with their own stencil, but it's not known whether these scripts could be simplified to the extent that they could generate a wide range of stencil based subroutines.
  • This has only been thoroughly run with the grid in the default hpcg.dat. In smaller grids (which can happen with no hpcg.dat in run directory, for e.g.) it can arise that MG symmetry fails. I'm flagging this as a known issue, but have not investigated the cause (sorry) due to noticing at end.

LFRic based

https://github.com/NCAS-CMS/hpcg_27ptStencil/tree/lfric_stencil_data lfric_stencil_data branch.

  • This is a very different problem - the grid and stencil are LFRic based.

  • No Multigrid is attempted for this

  • Halo exchange is commented out

  • The program begins as usual, but then rank 0 will read in an extra datafile: https://github.com/NCAS-CMS/hpcg_27ptStencil/blob/lfric_stencil_data/dinodump.dat and broadcast this to other ranks. They all solve the same CG problem, which has been symmetrized so as to converge

  • An additional data file, hpcg.dat is required. The could could be rewritten without this.

  • Some of the testing at the start is commented out too

Clone this wiki locally