Merge pull request #517 from hertneky/2d_doc_updates

Documentation/run script updates for 2D decomposition
NOAA-EMC · Jun 13, 2022 · 79f68d9 · 79f68d9
2 parents 6b4a79c + 85cc971
commit 79f68d9
Show file tree

Hide file tree

Showing 5 changed files with 58 additions and 25 deletions.
diff --git a/docs/AddNewVariable.rst b/docs/AddNewVariable.rst
@@ -79,7 +79,7 @@ with examples in the sections below.
             This flat file (instead of the xml file) is read in by UPP as it was much faster to read a text file
             than an xml file.
 
-2.  Allocate the field: ALLOCATE.f
+2.  Allocate and initialize the field: ALLOCATE.f
 
     This file is the instantiation or allocation of the variable. Note that the variables are defined
     based on the parallel processing capability of UPP - use an example from the file.
@@ -171,7 +171,8 @@ with examples in the sections below.
             e) Add the new variable to the UPP/parm/post_avblflds.xml, which lists all fields available
                for output in GRIB2 format. This file is generally not modified unless adding a new field or
                modifying an existing one.
-                - Post_avblfldidx: the unique array index number used to store this variable.
+                - Post_avblfldidx: the unique array index number used to store this variable. The number chosen here
+                  is just an example and it is important to pick one that is not yet in use.
                 - Shortname: name describing the variable and level type
                 - Pname: the abbreviation for your variable (should match what is used in params_grib2_tbl_new)
                 - Table info: table used if not standard WMO
@@ -185,7 +186,7 @@ with examples in the sections below.
                 ::
 
                  <param>
-                   <post_avblfldidx>999</post_avblfldidx>
+                   <post_avblfldidx>1003</post_avblfldidx>
                    <shortname>DEEP_TSOIL_ON_DEPTH_BEL_LAND_SFC</shortname>
                    <pname>TG3</pname>
                    <fixed_sfc1_type>depth_bel_land_sfc</fixed_sfc1_type>
@@ -220,16 +221,21 @@ with examples in the sections below.
           This flat file (instead of the xml file) is read in by UPP as it was much faster to read a text file
           than an xml file.
 
-2. Allocate the new variable in ALLOCATE_ALL.f
+2. Allocate and initialize the new variable in ALLOCATE_ALL.f
    This file is the instantiation or allocation of the variable. Note that the variables are defined
    based on the parallel processing capability of UPP - use an example from the file.
 
    User Procedure
-    - Add in VRBLS2D GFS section as:
+    - Allocate in VRBLS2D GFS section as:
+
+    ::
+
+      allocate(tg3(ista_2l:iend_2u,jsta_2l:jend_2u))
+    - Initialize in the initialization section that comes after the allocations section you added to.
 
     ::
 
-      allocate(tg3(im,jsta_2l:jend_2u))
+      tg3(i,j)=spval
 
 3. De-allocate the variable to give the resources back in DEALLOCATE.f
    All good programmers give back their resources when they are done. Please update this
@@ -270,7 +276,7 @@ with examples in the sections below.
 
      ! deep soil temperature
            VarName='tg3'
-           call read_netcdf_2d_para(ncid2d,im,jsta,jsta_2l,jend,jend_2u, &
+           call read_netcdf_2d_para(ncid2d,ista,ista_2l,iend,iend_2u,jsta,jsta_2l,jend,jend_2u, &
            spval,VarName,tg3)
 
 6. Determine the appropriate routine to add the new variable to (e.g. SURFCE.f, MDLFLD.f,
@@ -299,16 +305,17 @@ with examples in the sections below.
     ::
 
      ! DEEP SOIL TEMPERATURE
-     IF ( IGET(999).GT.0 ) THEN
+     IF ( IGET(1003).GT.0 ) THEN
        ID(1:25) = 0
        If(grib=='grib2') then
          cfld=cfld+1
-         fld_info(cfld)%ifld=IAVBLFLD(IGET(999))
+         fld_info(cfld)%ifld=IAVBLFLD(IGET(1003))
      !$omp parallel do private(i,j,jj)
          do j=1,jend-jsta+1
            jj = jsta+j-1
-           do i=1,im
-             datapd(i,j,cfld) = TG3(i,jj)
+           do i=1,iend-ista+1
+           ii = ista+i-1
+             datapd(i,j,cfld) = TG3(ii,jj)
            enddo
          enddo
        endiF

diff --git a/docs/CodeOverview.rst b/docs/CodeOverview.rst
@@ -30,4 +30,8 @@ Examples of UPP products include:
 
 A full list of fields that can be generated by the UPP is provided in :doc:`UPP_GRIB2_Table`.
 
+The UPP has 2D decomposition capabilities (V11.0.0) and is also backwards compatible for 1D decomposition.
+The functionality demonstrates run-time improvements, especially for larger domains. Support for this
+feature is available for standalone UPP applications.
+
 Support for the community UPP is provided through the `UFS Forum <https://forums.ufscommunity.org/>`_.
diff --git a/docs/InputsOutputs.rst b/docs/InputsOutputs.rst
@@ -36,26 +36,31 @@ UFS Unified Model Variables
 ITAG
 ----
 
-The file called :bolditalic:`itag` is a text file that contains the fortran namelist &model_inputs. It is
-read in by the :bolditalic:`upp.x` executable from stdin (unit 5) and is
-generated automatically within the UFS application workflow or stand-alone run script based on
-user-defined options. It should not be necessary to edit this. For description purposes, the namelist
-&model_inputs (:bolditalic:`itag` file) contains 7 lines for FV3:
+The file called :bolditalic:`itag` is a text file that contains the fortran namelist &model_inputs as
+well as the 2D decomposition specification in &nampgb. It is read in by the :bolditalic:`upp.x` executable
+from stdin (unit 5) and is generated automatically within the UFS application workflow or stand-alone run script based on user-defined options. It should not be necessary to edit this. For description purposes,
+the namelist &model_inputs and &nampgb (:bolditalic:`itag` file) contain the following lines for FV3:
 
-#. Name of the FV3 (pressure level) output file to be post-processed.
+&model_inputs
 
-#. Format of FV3 model output (netcdfpara).
+#. fileName: Name of the FV3 (pressure level) output file to be post-processed.
 
-#. Format of UPP output (GRIB2)
+#. IOFORM: Format of FV3 model output (netcdfpara).
 
-#. Forecast valid time (not model start time) in YYYY-MM-DD_HH:00:00 format (the forecast time desired
-   to be post-processed).
+#. grib: Format of UPP output (grib2)
 
-#. Model used (GFS, FV3R - regional FV3 also known as the LAM - Limited Area Model).
+#. DateStr: Forecast valid time (not model start time) in YYYY-MM-DD_HH:00:00 format (the forecast time
+   desired to be post-processed).
 
-#. Name of the FV3 (surface) output file to be post-processed.
+#. MODELNAME: Model used (GFS, FV3R - regional FV3 also known as the LAM - Limited Area Model).
 
-#. Name of configuration file (postxconfig-NT.txt)
+#. fileNameFlux: Name of the FV3 (surface) output file to be post-processed.
+
+#. fileNameFlat: Name of configuration file (postxconfig-NT.txt)
+
+&nampgb
+
+#. numx: Number of subdomains in the x-direction used for 2D decomposition. 
 
 ------------
 Control File

diff --git a/docs/Running.rst b/docs/Running.rst
@@ -107,7 +107,14 @@ Run Script Overview
        |       >> IBM: ``mpirun.lsf upp.x < itag > outpost``
        |       >> SGI MPT: ``mpiexec_mpt upp.x < itag > outpost``
 
-6. Set naming convention for prefix and extension of output file name
+6. Set the value for numx.
+
+       | **numx**: The number of subdomains in the x-direction used for decomposition.
+
+       |     - For 1D decomposition, set numx=1 (default)
+       |     - For 2D decomposition, set numx>1
+
+7. Set naming convention for prefix and extension of output file name
     - **comsp** is the initial string of the output file name. By default it is not set and the prefix
       of the output file will be the string set in the XML file DATSET parameter. If set it will
       concatenate the setting to the front of the string specified in the XML file DATSET parameter.

diff --git a/scripts/run_upp b/scripts/run_upp
@@ -26,6 +26,7 @@ set -x
 #           Added netcdfpara; Removed netcdf;
 #           Changed UPP directory name and path             
 #
+# June 2022: Add 2D decomposition capabilities, TH (DTC)
 #--------------------------------------------------------
 #
 # This script runs the stand-alone community version of UPP
@@ -59,6 +60,9 @@ set -x
 #                 * Do not set to 0 or the script will loop continuously *
 # RUN_COMMAND   : System run command for serial or parallel runs, examples below.
 #
+# numx          : The number of subdomains in the x-direction for 2D decomposition.
+#                 (default is 1)
+#
 #----------------------------------------------------------------------------------
 #--- BEGIN USER EDIT HERE ---------------------------------------------------------
 #----------------------------------------------------------------------------------
@@ -96,6 +100,9 @@ export RUN_COMMAND="mpirun -np 1 ${POSTEXEC}/upp.x "
 #export RUN_COMMAND="mpirun.lsf ${POSTEXEC}/upp.x "
 #export RUN_COMMAND="mpiexec_mpt ${POSTEXEC}/upp.x "
 
+# The number of subdomains in the x-direction for 2d decomposition
+export numx=2
+
 # Shouldn't need to edit these.
 # tmmark is an variable used as the file extention of the output
 # filename .GrbF is used if this variable is not set
@@ -330,6 +337,9 @@ MODELNAME='${tag}'
 fileNameFlux='${flxFileName}'
 fileNameFlat='postxconfig-NT.txt'
 /
+&nampgb
+numx=${numx}
+/
 EOF
    fi
 fi