From 8203dd75ac7981d95759c23c3044ff22826a5420 Mon Sep 17 00:00:00 2001 From: Kate Fossell Date: Tue, 14 Jun 2022 12:03:28 -0600 Subject: [PATCH 1/3] Update 2D documentation --- docs/2D-decomp.md | 48 ++++++++++++++++++++++++++++++++++- sorc/ncep_post.fd/MPI_FIRST.f | 19 ++++++++------ 2 files changed, 58 insertions(+), 9 deletions(-) diff --git a/docs/2D-decomp.md b/docs/2D-decomp.md index 105a28cf0..517bc7a54 100644 --- a/docs/2D-decomp.md +++ b/docs/2D-decomp.md @@ -17,5 +17,51 @@ The structural changes needed for the 2D decomposition are implemented in MPI_FI Both MPI_FIRST.f and EXCH.f contain significant additional test code to exchange arrays containing grid coordinates and ensure EXACT matches for all exchanges before the domain exchanges are performed. This is intended to trap errors in the larger variety of 2D decomposition layouts that are possible and most of it can eventually be removed or made conditional at build and run time. -Indices and variables to facilitate the 2D decomposition are found in CTLBLK.f and shared in the rest of UPP through use of CTLBLK.mod. +The following is found in CTLBLK.f and shared in the rest of UPP through use of CTLBLK.mod: + +| Variable | Type | Description | +|---------|------|-------------| +| im | integer | full longitude domain| +| jm | integer | full latitude domain| +| | | | +| jsta | integer | start latitude on a task subdomain| +| jend | integer | end latitude on a task subdomain| +| ista | integer | start longitude on a task subdomain| +| iend | integer | end longitude on a task subdomain| +| | | | +| ista_2l | integer |start longitude -2 of the subdomain| +| iend_2u | integer |end longitude +2 of the subdomain| +| jsta_2l | integer |start latitude -2 of the subdomain| +| jend_2u | integer |end latitude +2 of the subdomain| + +The shape of the subdomain is ista_2l:iend_2u,jsta_2l:jend_2u so it includes the halos although the halos are not populated until exchange is done in EXCH.f. Because of halos we need more bounds defined: + +| Variable | Type | Description | +|---------|------|-------------| +| jsta_m | integer | Beginning latitude loop index in subdomain for halo depth 1 | +| jend_m | integer | ending latitude loop index in subdomain for halo depth 1 | +| jsta_m2 | integer | second latitude below begin latitude of subdomain for halo depth 2 (in NGMFLD.f) | +| jend_m2 | integer | second latitude above end latitude of subdomain for halo depth 2 ( in NGMFLD.f) | + +Note: + +| Variable | Type | Description | +|---------|------|-------------| +| ista_m | integer | begining longitude loop index in subdomain for halo depth 1| +| iend_m | integer | end longitude loop index in subdomain for halo depth 1 | +| ista_m2 | integer | second longitude before begin longitude for halo depth 2 (not used as of 6/22)| +| iend_m2 | integer | second longitude after end longitude for halo depth 2 (not used as of 6/22) | + +Note: + + +| Variable | Type | Description | +|---------|------|-------------| +| ileft | integer | MPI rank containing the last longitude before ista_m +| iright | integer | MPI rank containing the first longitude after iend_m +| iup | integer | MPI rank containing the first latitude after jend +| idn | integer | MPI rank containing the last latitude before jsta +| | | | +| ileftb | integer |MPI rank containing the last longitude before ista_m but for cyclic boundary conditions where "last" at the beginning is the other end of the domain (apparently unused and replaced with local calculation) | +| irightb | integer | MPI rank containing the first longitude after iend_m but for cyclic boundary conditions where "first" at the beginning is the other end of the domain (apparently unused and replaced with local calculation) | diff --git a/sorc/ncep_post.fd/MPI_FIRST.f b/sorc/ncep_post.fd/MPI_FIRST.f index 1b399877c..0536acf0d 100644 --- a/sorc/ncep_post.fd/MPI_FIRST.f +++ b/sorc/ncep_post.fd/MPI_FIRST.f @@ -130,14 +130,17 @@ SUBROUTINE MPI_FIRST() ! global loop ranges ! -! para_range2 supports a 2D decomposition. The rest of the post -! supports 1D still and the call here is the special case where each -! processor gets all of the longitudes in the latitude 1D subdomain -! jsta:jend. The X decomposition will be specified by the third -! argument (currently 1) and the Y decoposition will be specified by -! the fourth argument (currently all of the ranks) When X is -! subdivided the third and fourth arguments will have to be integral -! factors of num_procs +! para_range2 supports a 2D decomposition. +! The X decomposition is specified by the third +! argument and the Y decoposition is specified by +! the fourth argument. The product of the third and fourth arguments +! must be num_procs and the third and fourth arguments must be integral +! factors of num_procs. +! +! for the special case of 1D decomposition, numx is set to 1 and the +! fourth argument becumes the number of MPI ranks for the job. numx=1 +! makes the code fully compatible with the old 1D decomposition. + call para_range2(im,jm,numx,num_procs/numx,me,ista,iend,jsta,jend) From d63652cbb67edbb27650437b30bfffd2589c1f4c Mon Sep 17 00:00:00 2001 From: Kate Fossell Date: Tue, 14 Jun 2022 12:51:24 -0600 Subject: [PATCH 2/3] Minor updates to indices descriptions --- docs/2D-decomp.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/2D-decomp.md b/docs/2D-decomp.md index 517bc7a54..16fa468ac 100644 --- a/docs/2D-decomp.md +++ b/docs/2D-decomp.md @@ -57,11 +57,11 @@ Note: