Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

mksurfdata_map: replace source (SRC) files of various masks with SRC files with no mask #823

Merged
merged 45 commits into from
May 25, 2021
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
6fa062b
Replace source (SRC) files of various masks with SRC files with no mask
slevis-lmwg Oct 17, 2019
46ae6a9
Now apply SRC mask in mksurfdata_map, starting with mklaiMod
slevis-lmwg Oct 21, 2019
7798006
Revisions in response to code review by @billsacks and @ekluzek
slevis-lmwg Oct 21, 2019
12506f9
Apply SRC mask in mksurfdata_map, continued: mkpftMod
slevis-lmwg Oct 22, 2019
7e3a915
Return to original tolerance tol = 1.e-12_r8 with these changes
slevis-lmwg Oct 22, 2019
84c1630
Additional changes that were required for mkpft to work
slevis-lmwg Oct 23, 2019
5f20517
Revisions in response to code review
slevis-lmwg Oct 28, 2019
d93f1d6
Revisions in response to latest review
slevis-lmwg Oct 30, 2019
163a231
Apply SRC mask in mksurfdata_map (cont'd): mkglcmecMod
slevis-lmwg Nov 2, 2019
edb5c2e
Now setting scripgriddata_lrgfile_needed for 3x3min_nomask grid
slevis-lmwg Nov 4, 2019
faf2aac
mkglcmecMod.F90 part 2
slevis-lmwg Nov 5, 2019
25e3624
File change required to eliminate large diffs + an optimization change
slevis-lmwg Nov 5, 2019
57e4b24
Apply SRC mask in remaining mksurfdata_map modules
slevis-lmwg Nov 7, 2019
7c8f3e2
Minor cleanup
slevis-lmwg Nov 7, 2019
acf99eb
Documentation in ChangeLog and updates to namelist_defaults
slevis-lmwg Nov 11, 2019
a2ba7fe
Replaced 360x720 SRC and map files with 0.5x0.5
slevis-lmwg Nov 13, 2019
0e43e79
Added 0.125x0.125_nomask SRC resolution (previously mislabeled 3x3min)
slevis-lmwg Nov 14, 2019
0673a26
Making checkmapfiles.ncl consistent with previous commit
slevis-lmwg Nov 14, 2019
9b66478
Merge tag 'ctsm1.0.dev079' into mksurfdata_maps_wo_src_masks
slevis-lmwg Nov 14, 2019
a977c8a
Adding code to handle UNSTRUCT grid files
slevis-lmwg Nov 15, 2019
01f2a56
Updated ChangeLog and reverted mods in mkmapdata.sh
slevis-lmwg Nov 18, 2019
d3ebddf
Revisions in response to code review: PART 1
slevis-lmwg Nov 20, 2019
9c3ade8
Revisions: part 2
slevis-lmwg Nov 21, 2019
da20c87
Revisions: part 3
slevis-lmwg Nov 21, 2019
33dc03b
Revisions part 3b: update namelist_defaults_ctsm_tools.xml
slevis-lmwg Nov 21, 2019
9f60042
Revisions part 1b: Organizational update of part 1
slevis-lmwg Nov 21, 2019
66f14c9
Revisions Part 4: mostly responding to @billsacks question 8
slevis-lmwg Dec 2, 2019
d976063
Revisions Part 4b: update subr. gridmap_check
slevis-lmwg Dec 2, 2019
14f2ba6
Revisions Part 4c: A better option than (4b)
slevis-lmwg Dec 6, 2019
bba7202
Revisions Part 5: in reponse to @billsacks question 9
slevis-lmwg Dec 9, 2019
f39a043
Fix CMake-based unit tests
billsacks Dec 15, 2019
4856791
Revisions Part 6: in response to latest comments
slevis-lmwg Dec 18, 2019
0a4f996
Revisions Part 7: adding comment to mkmapdata.sh
slevis-lmwg Dec 18, 2019
3e5a099
Merge tag 'ctsm1.0.dev080' into mksurfdata_maps_wo_src_masks
slevis-lmwg Dec 18, 2019
50d3085
Revisions required for expected tests in test_driver.sh to PASS
slevis-lmwg Dec 22, 2019
5700ef2
Pointing to newly created files
slevis-lmwg Feb 25, 2020
9bee59b
Changes to scripts associated with latest changes to files
slevis-lmwg Feb 25, 2020
0043191
Committing changes that lead to all but one type of test to PASS
slevis-lmwg Mar 3, 2020
5dfe8dd
Committing mods from early March to make them available
slevis-lmwg May 7, 2020
573009a
Merge tag 'ctsm5.1.dev038' into mksurfdata_maps_wo_src_masks
slevis-lmwg May 4, 2021
4235c43
Bring back 0.125x0.125_nomask SRC resolution (dropped out w the merge)
slevis-lmwg May 6, 2021
79a9219
Updated map_3x3min_nomask_to_ne120 and ne16 files
slevis-lmwg May 7, 2021
d409574
mkmapdata.sh that points to Bob Oehmke's fix
slevis-lmwg May 20, 2021
f81262c
Merge tag 'ctsm5.1.dev039' into mksurfdata_maps_wo_src_masks
slevis-lmwg May 20, 2021
0a769de
Updated ChangeLog/ChangeSum
slevis-lmwg May 20, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion bld/namelist_files/checkmapfiles.ncl
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,7 @@ begin
print( "query string="+querynml )


mapgrids = (/"0.5x0.5_MODIS", "0.5x0.5_AVHRR", "0.5x0.5_MODIS", "5x5min_nomask", "5x5min_IGBP-GSDP", "5x5min_ISRIC-WISE", "10x10min_nomask", "3x3min_MODIS", "3x3min_LandScan2004", "3x3min_GLOBE-Gardner", "3x3min_GLOBE-Gardner-mergeGIS", "0.9x1.25_GRDC", "360x720cru_cruncep", "1km-merge-10min_HYDRO1K-merge-nomask"/);
mapgrids = (/"0.5x0.5_nomask", "0.25x0.25_nomask", "3x3min_nomask", "5x5min_nomask", "10x10min_nomask", "0.9x1.25_nomask", "1km-merge-10min_HYDRO1K-merge-nomask"/);
do i = 0, dimsizes(resolutions)-1
res = resolutions(i);
print( "Go through maps for Resolution: "+res );
Expand Down
1,181 changes: 244 additions & 937 deletions bld/namelist_files/namelist_defaults_ctsm.xml

Large diffs are not rendered by default.

42 changes: 6 additions & 36 deletions bld/namelist_files/namelist_defaults_ctsm_tools.xml
Original file line number Diff line number Diff line change
Expand Up @@ -55,48 +55,18 @@ attributes from the config_cache.xml file (with keys converted to upper-case).
>lnd/clm2/mappingdata/grids/SCRIPgrid_0.1x0.1_nomask_c110712.nc</scripgriddata>
<scripgriddata hgrid="0.5x0.5" lmask="nomask"
>lnd/clm2/mappingdata/grids/SCRIPgrid_0.5x0.5_nomask_c110308.nc</scripgriddata>
<scripgriddata hgrid="0.5x0.5" lmask="AVHRR"
>lnd/clm2/mappingdata/grids/SCRIPgrid_0.5x0.5_AVHRR_c110228.nc</scripgriddata>
<scripgriddata hgrid="0.5x0.5" lmask="MODIS"
>lnd/clm2/mappingdata/grids/SCRIPgrid_0.5x0.5_MODIS_c110228.nc</scripgriddata>
<scripgriddata hgrid="0.25x0.25" lmask="MODIS"
>lnd/clm2/mappingdata/grids/SCRIPgrid_0.25x0.25_MODIS_c170321.nc</scripgriddata>
<scripgriddata hgrid="0.25x0.25" lmask="nomask"
>lnd/clm2/mappingdata/grids/SCRIPgrid_0.25x0.25_nomask_c191014.nc</scripgriddata>
<scripgriddata hgrid="5x5min" lmask="nomask"
>lnd/clm2/mappingdata/grids/SCRIPgrid_5x5min_nomask_c110530.nc</scripgriddata>
<scripgriddata hgrid="5x5min" lmask="IGBP-GSDP"
>lnd/clm2/mappingdata/grids/SCRIPgrid_5x5min_IGBP-GSDP_c110228.nc</scripgriddata>
<scripgriddata hgrid="5x5min" lmask="ISRIC-WISE"
>lnd/clm2/mappingdata/grids/SCRIPgrid_5x5min_ISRIC-WISE_c111114.nc</scripgriddata>
<scripgriddata hgrid="5x5min" lmask="ORNL-Soil"
>lnd/clm2/mappingdata/grids/SCRIPgrid_5x5min_ORNL-Soil_c170630.nc</scripgriddata>
<scripgriddata hgrid="10x10min" lmask="nomask"
>lnd/clm2/mappingdata/grids/SCRIPgrid_10x10min_nomask_c110228.nc</scripgriddata>
<scripgriddata hgrid="10x10min" lmask="IGBPmergeICESatGIS"
>lnd/clm2/mappingdata/grids/SCRIPgrid_10x10min_IGBPmergeICESatGIS_c110818.nc</scripgriddata>
<scripgriddata hgrid="3x3min" lmask="MODIS"
>lnd/clm2/mappingdata/grids/SCRIPgrid_3minx3min_MODIS_c110915.nc</scripgriddata>
<scripgriddata hgrid="3x3min" lmask="MODIS-wCsp"
>lnd/clm2/mappingdata/grids/SCRIPgrid_3minx3min_MODISwcspsea_c151020.nc</scripgriddata>
<scripgriddata_lrgfile_needed hgrid="3x3min" lmask="MODIS" >64bit_offset</scripgriddata_lrgfile_needed>
<scripgriddata_lrgfile_needed hgrid="3x3min" lmask="MODIS-wCsp">64bit_offset</scripgriddata_lrgfile_needed>
<scripgriddata hgrid="3x3min" lmask="USGS"
>lnd/clm2/mappingdata/grids/SCRIPgrid_3x3_USGS_c120912.nc</scripgriddata>
<scripgriddata_lrgfile_needed hgrid="3x3min" lmask="USGS">64bit_offset</scripgriddata_lrgfile_needed>
<scripgriddata hgrid="3x3min" lmask="LandScan2004"
>lnd/clm2/mappingdata/grids/SCRIPgrid_3minx3min_LandScan2004_c120517.nc</scripgriddata>
<scripgriddata_lrgfile_needed hgrid="3x3min" lmask="LandScan2004">64bit_offset</scripgriddata_lrgfile_needed>
<scripgriddata hgrid="3x3min" lmask="GLOBE-Gardner"
>lnd/clm2/mappingdata/grids/SCRIPgrid_3minx3min_GLOBE-Gardner_c120922.nc</scripgriddata>
<scripgriddata_lrgfile_needed hgrid="3x3min" lmask="GLOBE-Gardner">64bit_offset</scripgriddata_lrgfile_needed>
<scripgriddata hgrid="3x3min" lmask="GLOBE-Gardner-mergeGIS"
>lnd/clm2/mappingdata/grids/SCRIPgrid_3minx3min_GLOBE-Gardner-mergeGIS_c120922.nc</scripgriddata>
<scripgriddata_lrgfile_needed hgrid="3x3min" lmask="GLOBE-Gardner-mergeGIS">64bit_offset</scripgriddata_lrgfile_needed>
<scripgriddata hgrid="360x720cru" lmask="cruncep"
>lnd/clm2/mappingdata/grids/SCRIPgrid_360x720_cruncep_c120830.nc</scripgriddata>
<scripgriddata hgrid="3x3min" lmask="nomask"
>lnd/clm2/mappingdata/grids/SCRIPgrid_3x3min_nomask_c191014.nc</scripgriddata>
<scripgriddata hgrid="360x720cru" lmask="nomask"
>lnd/clm2/mappingdata/grids/SCRIPgrid_360x720_nomask_c120830.nc</scripgriddata>
<scripgriddata hgrid="0.9x1.25" lmask="GRDC"
>lnd/clm2/mappingdata/grids/SCRIPgrid_0.9x1.25_GRDC_c130307.nc</scripgriddata>
<scripgriddata hgrid="0.9x1.25" lmask="nomask"
>lnd/clm2/mappingdata/grids/SCRIPgrid_0.9x1.25_nomask_c191014.nc</scripgriddata>
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regarding the diffs in VIC variables mentioned in 57e4b24,
I have confirmed that the two files above differ only in that the mask = 1 everywhere in the nomask file.


<!-- Other raw data grids -->
<!-- Note that the following is a UGRID file rather than a SCRIP grid file -->
Expand Down
2 changes: 1 addition & 1 deletion bld/namelist_files/namelist_definition_ctsm.xml
Original file line number Diff line number Diff line change
Expand Up @@ -1849,7 +1849,7 @@ CLM run type.
<entry id="res" type="char*30" category="default_settings"
group="default_settings"
valid_values=
"512x1024,360x720cru,128x256,64x128,48x96,32x64,8x16,94x192,0.23x0.31,0.47x0.63,0.9x1.25,1.9x2.5,2.5x3.33,4x5,10x15,0.125nldas2,5x5_amazon,1x1_camdenNJ,1x1_vancouverCAN,1x1_mexicocityMEX,1x1_asphaltjungleNJ,1x1_brazil,1x1_urbanc_alpha,1x1_numaIA,1x1_smallvilleIA,0.1x0.1,0.25x0.25,0.5x0.5,3x3min,5x5min,10x10min,0.33x0.33,0.125x0.125,ne4np4,ne16np4,ne30np4,ne60np4,ne120np4,ne240np4,1km-merge-10min">
"512x1024,360x720cru,128x256,64x128,48x96,94x192,0.23x0.31,0.47x0.63,0.9x1.25,1.9x2.5,2.5x3.33,4x5,10x15,0.125nldas2,5x5_amazon,1x1_camdenNJ,1x1_vancouverCAN,1x1_mexicocityMEX,1x1_asphaltjungleNJ,1x1_brazil,1x1_urbanc_alpha,1x1_numaIA,1x1_smallvilleIA,0.1x0.1,0.25x0.25,0.5x0.5,3x3min,5x5min,10x10min,0.33x0.33,0.125x0.125,ne4np4,ne16np4,ne30np4,ne60np4,ne120np4,ne240np4,1km-merge-10min">
Horizontal resolutions
ekluzek marked this conversation as resolved.
Show resolved Hide resolved
Note: 0.1x0.1, 0.25x0.25, 0.5x0.5, 5x5min, 10x10min, 3x3min and 0.33x0.33 are only used for CLM tools
</entry>
Expand Down
22 changes: 6 additions & 16 deletions tools/mkmapdata/mkmapdata.sh
Original file line number Diff line number Diff line change
Expand Up @@ -263,22 +263,12 @@ fi

if [ "$phys" = "clm4_5" ]; then
grids=( \
"0.5x0.5_AVHRR" \
"0.25x0.25_MODIS" \
"0.5x0.5_MODIS" \
"3x3min_LandScan2004" \
"3x3min_MODIS-wCsp" \
"3x3min_USGS" \
"0.5x0.5_nomask" \
"0.25x0.25_nomask" \
"5x5min_nomask" \
"5x5min_IGBP-GSDP" \
"5x5min_ISRIC-WISE" \
"5x5min_ORNL-Soil" \
"10x10min_nomask" \
"10x10min_IGBPmergeICESatGIS" \
"3x3min_GLOBE-Gardner" \
"3x3min_GLOBE-Gardner-mergeGIS" \
"0.9x1.25_GRDC" \
"360x720cru_cruncep" \
"0.9x1.25_nomask" \
"3x3min_nomask" \
"1km-merge-10min_HYDRO1K-merge-nomask" \
)

Expand All @@ -288,7 +278,7 @@ else
fi

# Set timestamp for names below
ekluzek marked this conversation as resolved.
Show resolved Hide resolved
CDATE="c"`date +%y%m%d`
CDATE="c"`date -d "-0 days" +%y%m%d`
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see that we need this change.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was convenient for me when rerunning mkmapdata.sh because I didn't have to recreate all the map_ files from scratch. For example a couple of days ago I set this line to -28 days and focused on the files that hadn't been created, yet. I'm happy to revert to its original form, but I found this days flag to be a huge time saver.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, if you want to keep it, add a comment that explains why it's there and how it can help, and how you would use it. Otherwise, it's extra syntax that doesn't actually do anything.


# Set name of each output mapping file
# First determine the name of the input scrip grid file
Expand Down Expand Up @@ -346,7 +336,7 @@ case $hostname in
intelvers=17.0.1
module load esmf_libs/$esmfvers
module load intel/$intelvers
module load ncl
module load ncl/6.6.2
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a need for a specific version of NCL? If not using the generic one seems better.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

module load ncl returns this error:
Lmod has detected the following error: These module(s) exist but cannot be loaded as requested: "ncl"
Try: "module spider ncl" to see how to load the module(s).

...while module load ncl/6.6.2 works.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had to load intel/18.0.5 recently to get the generic ncl to work with mksurfdata_map.

intelvers=18.0.5

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was just going to say the same thing, if you use intelvers=18.0.5 you can load generic ncl, but otherwise you have to name the version. And you should do the update to intel 18 anyway, as we have that on the release branch.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's a bug on CISL's part that you can't load ncl without naming a specific version. We could create a ticket for them about this.

module load nco

if [ "$interactive" = "NO" ]; then
Expand Down
9 changes: 0 additions & 9 deletions tools/mksurfdata_map/src/mkdomainMod.F90
Original file line number Diff line number Diff line change
Expand Up @@ -866,15 +866,6 @@ subroutine domain_checksame( srcdomain, dstdomain, tgridmap )
end if
do n = 1,ns
ni = src_indx(n)
if ( srcdomain%maskset )then
if (srcdomain%mask(ni) /= mask_src(ni)) then
write(6,*) trim(subname)// &
' ERROR: input domain mask and gridmap mask are not the same at ni = ',ni
write(6,*)' domain mask= ',srcdomain%mask(ni)
write(6,*)' gridmap mask= ',mask_src(ni)
call abort()
end if
end if
ekluzek marked this conversation as resolved.
Show resolved Hide resolved
if (abs(srcdomain%lonc(ni) - xc_src(ni)) > eps) then
write(6,*) trim(subname)// &
' ERROR: input domain lon and gridmap lon not the same at ni = ',ni
Expand Down
12 changes: 4 additions & 8 deletions tools/mksurfdata_map/src/mkgridmapMod.F90
Original file line number Diff line number Diff line change
Expand Up @@ -688,21 +688,19 @@ subroutine gridmap_areaave_srcmask (gridmap, src_array, dst_array, nodata, mask_
! !LOCAL VARIABLES:
integer :: n,ns,ni,no
real(r8):: wt
real(r8), allocatable :: wtnorm(:)
character(*),parameter :: subName = '(gridmap_areaave_srcmask) '
!EOP
!------------------------------------------------------------------------------
call gridmap_checkifset( gridmap, subname )
ns = size(dst_array)
allocate(wtnorm(ns))
wtnorm(:) = 0._r8
gridmap%frac_dst(:) = 0._r8
billsacks marked this conversation as resolved.
Show resolved Hide resolved

do n = 1,gridmap%ns
ni = gridmap%src_indx(n)
no = gridmap%dst_indx(n)
wt = gridmap%wovr(n)
if (mask_src(ni) > 0) then
wtnorm(no) = wtnorm(no) + wt*mask_src(ni)
gridmap%frac_dst(no) = gridmap%frac_dst(no) + wt*mask_src(ni)
end if
end do

Expand All @@ -712,16 +710,14 @@ subroutine gridmap_areaave_srcmask (gridmap, src_array, dst_array, nodata, mask_
no = gridmap%dst_indx(n)
wt = gridmap%wovr(n)
if (mask_src(ni) > 0) then
dst_array(no) = dst_array(no) + wt*mask_src(ni)*src_array(ni)/wtnorm(no)
dst_array(no) = dst_array(no) + wt*mask_src(ni)*src_array(ni)/gridmap%frac_dst(no)
end if
end do

where (wtnorm == 0._r8)
where (gridmap%frac_dst== 0._r8)
dst_array = nodata
end where

deallocate(wtnorm)

end subroutine gridmap_areaave_srcmask

!==========================================================================
Expand Down
10 changes: 5 additions & 5 deletions tools/mksurfdata_map/src/mklaiMod.F90
Original file line number Diff line number Diff line change
Expand Up @@ -247,7 +247,7 @@ subroutine mklai(ldomain, mapfname, datfname, ndiag, ncido)
! Loop over pft types to do mapping

do l = 0, numpft_i - 1
mask_src(:) = 1._r8
mask_src(:) = tdomain%frac(:)
call gridmap_areaave(tgridmap, mlai_i(:,l) , mlai_o(:,l) , nodata=0._r8, mask_src=mask_src)
call gridmap_areaave(tgridmap, msai_i(:,l) , msai_o(:,l) , nodata=0._r8, mask_src=mask_src)
call gridmap_areaave(tgridmap, mhgtt_i(:,l), mhgtt_o(:,l), nodata=0._r8, mask_src=mask_src)
Expand Down Expand Up @@ -311,13 +311,13 @@ subroutine mklai(ldomain, mapfname, datfname, ndiag, ncido)
do l = 0, numpft_i - 1
do ni = 1, ns_i
glai_i(l) = glai_i(l) + mlai_i(ni,l) *tgridmap%area_src(ni)*&
tgridmap%frac_src(ni)*re**2
mask_src(ni)*re**2
gsai_i(l) = gsai_i(l) + msai_i(ni,l) *tgridmap%area_src(ni)*&
tgridmap%frac_src(ni)*re**2
mask_src(ni)*re**2
ghgtt_i(l) = ghgtt_i(l)+ mhgtt_i(ni,l)*tgridmap%area_src(ni)*&
tgridmap%frac_src(ni)*re**2
mask_src(ni)*re**2
ghgtb_i(l) = ghgtb_i(l)+ mhgtb_i(ni,l)*tgridmap%area_src(ni)*&
tgridmap%frac_src(ni)*re**2
mask_src(ni)*re**2
end do
end do

Expand Down