-
Notifications
You must be signed in to change notification settings - Fork 24
Home
SPH-EXA is a Smoothed Particle Hydrodynamics code in 3D. It implements a few physical scenarios, referred below as test cases.
Code is structured as follows:
SPH-EXA
| README.md
└───include/ - folder containing all sph functions,
| utilities and helpers that are shared between all test cases.
└───src/ - folder containing test cases
└───testcase/ - folder containing test case main function and
other test case specific functions,
like reading input data from the file
Each test case is compiled separately and therefore there is the main function for each test case.
The main test case file, i.e. src/sqpatch/sqpatch.cpp
can be divided into the following steps.
-
Reading command line arguments
-
Reading from file or generating input for a test case
As a result of this step, object of the ParticleData class is created. That object holds all particles data and it's served as an input and output in all sph and domain distribution functions
-
Running a loop for s iterations (time-steps)
An essential part of each sph test case. It can be further divided into the following steps:
- Domain distribution
- Map particles in 3D to an Octree (
buildTree
) - Find neighbors of each particle
- Call sph functions calculating physics i.e.
computeDensity
,computeMomentumAndEnergy
functions - Update positions of all particles
- (Optionally) Dump particle data to file
Following sequence diagram illustrates the flow for the sample test case that doesn't need gravity calculations i.e. Square Patch:
Each function takes two parameters: TaskList
and Dataset
objects.
TaskList
is a list of Task
objects, which is used to determine which particle indexes should be calculated in the particular function call.
Dataset
contains data for all particles.
3D space is mapped to an Octree
in SPH-EXA. The octree is used to perform some sph calculations (such as sph::findNeighbors
) as well as to do a domain distribution across MPI ranks.
Octree
is divided into two parts, the global and local ones. The global part of the Octree
is known to all MPI ranks in terms of structure, but not in terms of data. This tree is used in domain distribution to assign MPI ranks to the tree nodes. When MPI ranks are assigned, the data exchange happens. If rank was assigned to the particular tree node, it will receive all particle data belonging to that node. Then the given node can build the local tree from this data. If the node was not entirely assigned to the MPI rank, the assignee value will be equal to -1. It's an indication that this tree node is shared between at least two MPI ranks. This assumption is later used in calculating gravity in the distributed manner, see the description below.
It's important to calculate gravity contributions from the shared tree nodes only once. Hence remoteGravityTreeWalks
function will skip calculating contribution from the tree nodes that have assignee equal to -1, because those were calculated during the self-gravity calculations.
This assumption is also used in the build global gravity tree phase (sph::buildGlobalGravityTree
). It's needed to synchronize gravity parameters only in the tree nodes that are shared. Only this and local tree nodes will be used to calculate gravity in the self-gravity calculation phase. Hence the function sph::buildGlobalGravityTree
has to be called before sph::gravityTreeWalk
for MPI runs. If given particle needs contribution from the tree node that is assigned to other MPI rank, that particle will be sent to that MPI rank with a request to calculate contribution there.
Gravity is calculated using the octree data structure (GravityOctree
class). Each particle is traversing the tree recursively and calculates the gravity contribution of the tree nodes (sph::gravityTreeWalk
).
To perform gravity calculations, the Domain object must be created with the template parameter of GravityOctree
. Thanks to that, the Octree
will be built with additional parameters per node, which will be used later to calculate gravity contributions in the sph::gravityTreeWalk
function.
To calculate gravity in the test case, the following steps are needed:
0. ensure that you build the code with -DGRAVITY
compile flag
- pass a
GravityOctree
class as a template parameter to Domain class Then for every iteration (time-step) in the test case: - In the main iteration loop: call function
sph::gravityTreewalk
The steps above will only work for non-MPI versions of the code. To be able to calculate gravity in a distributed manner, additional steps are needed.
The Octree
in SPH-EXA is divided into two parts, the global and the local ones. The global part, referred as a Global Octree or Global Gravity Octree is known to all computing nodes (MPI ranks) in terms of structure. See section explaining these two trees in details
Every compute node has data of only its particles and its neighbors. Gravity is a global force, therefore one node will need to calculate the gravity contribution not only from data it contains, but from some other nodes, or even all other nodes in the worst case.
The first step is to build the Global Gravity tree (function sph::buildGlobalGravityTree
). This step is needed to synchronize the gravity parameters of the global part of the tree with all the nodes. Every node needs to have the same gravity parameters in the tree nodes, to be able to correctly distinguish if a particle needs gravity contribution from the other node.
If the particle during calculations in the sph::gravityTreeWalk
function needs information from the other node to perform calculations, this particle is marked as a particle to send to that node, to perform gravityTreewalk
there again and gather back the needed data. Hence the gravityTreeWalk
calculations are divided into two parts, gravityTreewalk
for particles that are in the node (self Gravity) and gravityTreewalk
for particles that other nodes request (remote or foreign particles gravityTreeWalk: sph::remoteGravityTreewalks
).
The sph::gravityTreeWalk
function returns a rank to particle indexes map. Each entry in the map contains particle indexes that are needed to send to the rank, to be able to get the gravity contribution from this particular rank. This map is then used as a parameter to sph::remoteGravityTreeWalks
function, where the data exchange happens and additional treewalks for foreign particles are performed.
To perform gravity calculations with MPI, two additional steps are needed. Building Global Gravity Tree and performing treewalk for remote particles.
- ensure that you build the code with
-DGRAVITY
compile flag - pass a
GravityOctree
class as a template parameter toDomain
class Then for every iteration (time-step) in the test case: - Build GravityOctree (
sph::buildGlobalGravityTree
) - Perform self gravity treewalk (
sph::gravityTreeWalk
) - Perform treewalk for remote particles needing data from a given node (
sph::remoteGravityTreeWalks
)
Following sequence diagram illustrates the flow for test case with gravity calculations i.e. Evrard Collapse:
-
Add a new folder in path
src/
. The folder name must be a test case name. This name will be used later to determine the test case in the build phase -
If the test uses gravity, compile with
-DGRAVITY
flag or add its name to Makefile to line stating :
ifeq ($(TESTCASE),evrard)
TESTCASE_FLAGS = -DGRAVITY
endif
- Implement generating or reading input data from the file as well as writing output following the
IFileReader
andIFileWriter
interfaces. - Create a main function in file in path
src/your_testcase/your_testcase.cpp
and implement test scenario