mksurfdata_map: replace source (SRC) files of various masks with SRC files with no mask #823
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,4 +1,230 @@ | ||
| =============================================================== | ||
| Tag name: ctsm1.0.dev081 | ||
| Originator(s): slevis (Samuel Levis,SLevis Consulting LLC,303-665-1310) | ||
| Date: Tue Dec 17 18:26:42 MST 2019 | ||
| One-line Summary: Replace masked with "nomask" SRC files for use with mkmapdata and mksurfdata_map | ||
|
There was a problem hiding this comment. I'm interested in your comments regarding this documentation. There was a problem hiding this comment. @billsacks
I will wait for the go-ahead from you guys, since I think this will be a time consuming step. By the way, is there a shortcut for generating ALL the sfc datasets at once? There was a problem hiding this comment. @slevisconsulting there is a Makefile "Makefile.data" that we use to send simultaneous jobs to create the standard datasets needed. Using "all" makes most of the standard ones outside of some of the high resolution ones. You could also name each of those to send them to make as well. There was a problem hiding this comment. This old post reminded me:
|
||
|
|
||
| Purpose of changes | ||
| ------------------ | ||
|
|
||
| 1) Collapse multiple source (SRC) files of a given resolution to a | ||
| single "nomask" SRC file per SRC resolution. The mask variable is now | ||
| set to 1 everywhere in all SRC files and mkmapdata.sh generates | ||
| a single map_ file (aka weight file) per destination (DST) resolution. | ||
| 2) Now applying the SRC mask found in the raw datasets while | ||
| running mksurfdata_map. | ||
|
|
||
| These changes simplify the introduction of new surface variables with | ||
| raw data in one of our existing SRC resolutions because we need not | ||
| create new SRC files for such variables. To introduce new surface | ||
| variables with raw data in new resolutions, the corresponding new SRC | ||
| files will have mask equal to 1 everywhere. | ||
|
|
||
|
|
||
| Bugs fixed or introduced | ||
| ------------------------ | ||
|
|
||
| Issues fixed (include CTSM Issue #): #286 | ||
|
|
||
|
|
||
| Significant changes to scientifically-supported configurations | ||
| -------------------------------------------------------------- | ||
|
|
||
| Does this tag change answers significantly for any of the following physics configurations? | ||
| (Details of any changes will be given in the "Answer changes" section below.) | ||
|
|
||
| [Put an [X] in the box for any configuration with significant answer changes.] | ||
|
|
||
| [X] clm5_0 | ||
|
|
||
| [X] ctsm5_0-nwp | ||
|
|
||
| [X] clm4_5 | ||
|
|
||
| Notes of particular relevance for users | ||
| --------------------------------------- | ||
|
|
||
| Caveats for users (e.g., need to interpolate initial conditions): | ||
| These changes are transparent to users making surface datasets using | ||
| default SRC resolutions. Users using non-default SRC resolutions or | ||
| users introducing new surface variables with raw data in non-default | ||
| SRC resolutions, will create new SRC file(s) with mask always set to 1 | ||
| everywhere. | ||
| Changes to CTSM's user interface (e.g., new/renamed XML or namelist variables): | ||
| None. | ||
|
|
||
| Changes made to namelist defaults (e.g., changed parameter values): | ||
| Changed the names of most default SRC files and default map_ files to | ||
| the names of the "nomask" versions of these files. | ||
|
|
||
| Changes to the datasets (e.g., parameter, surface or initial files): | ||
| Replaced SRC files and map_ files with "nomask" versions. Most SRC | ||
| resolutions already had nomask files available, so I created such files | ||
| for 0.25x0.25, 0.9x1.25, 3x3min, 5x5min, and 0.125x0.125. I changed all | ||
| of the map_ files except | ||
| map_1km-merge-10min_HYDRO1K-merge-nomask SRC resolution. | ||
|
|
||
| Substantial timing or memory changes: | ||
| - We end up with fewer SRC and map_ files by replacing multiple custom | ||
| grids per SRC resolution with one nomask grid. I list here resolutions | ||
| that had multiple custom grids per resolution before the switch to | ||
| nomask: | ||
| 0.5x0.5: AVHRR, MODIS, and 360x720cru_cruncep | ||
| 3x3min: GLOBE-Gardner-mergeGIS, GLOBE-Gardner, LandScan2004, MODIS-wCsp | ||
| 5x5min: IGBP-GSDP, ISRIC-WISE, ORNL | ||
| 10x10min_IGBPmergeICESatGIS | ||
|
|
||
| UNSTRUCT info shown only for comparison here. | ||
| NOT switching to UNSTRUCT files at this time. | ||
|
|
||
| - The nomask map_ files are larger now. E.g.: | ||
| slevis ncar 2754246572 Nov 1 map_3x3min_GLOBE-Gardner_to_0.9x1.25_nomask_aave_da_c191101.nc MASKED SCRIP | ||
| slevis ncar 3051641404 Nov 9 map_3x3min_nomask_to_0.9x1.25_nomask_aave_da_c191109.nc NOMASK SCRIP | ||
| slevis ncar 2974797264 Nov 15 map_3x3min_nomask_to_0.9x1.25_nomask_aave_da_c191115.nc NOMASK UNSTRUCT | ||
|
|
||
| - Although the nomask SCRIP cases shown below are slower because each | ||
| individual mapping file takes longer to create, the total time to | ||
| generate mapping files is reduced, possibly significantly, due to the | ||
| need for fewer mapping files in total. Note the increase in time | ||
| savings from the 0.5x0.5 to the 10x10min resolution, which suggests | ||
| even greater time savings for the 3x3min and 5x5min that went from a | ||
| total of 7 to 2 map_ files. Sample timing data: | ||
| mkmapdata 0.5x0.5 to 0.9x1.25 MASKED (2 grids) 78.0u 1.30s 1:20.08 99.1% | ||
| mkmapdata 0.5x0.5 to 0.9x1.25 NOMASK SCRIP 51.199u 0.85s 0:52.61 98.8% | ||
| mkmapdata 0.5x0.5 to 0.9x1.25 NOMASK UNSTRUC 38.316u 0.69s 0:39.51 98.7% | ||
|
|
||
| mkmapdata 0.25x0.25 to 0.9x1.25 MASKED SCRI 129.288u 1.46s 2:11.38 99.5% | ||
| mkmapdata 0.25x0.25 to 0.9x1.25 NOMASK SCRI 168.443u 1.69s 2:50.77 99.6% | ||
| mkmapdata 0.25x0.25 to 0.9x1.25 NOMASK UNST 110.875u 1.46s 1:52.80 99.5% | ||
|
|
||
| mkmapdata 0.125x0.125 to 0.9x1.25 MASKED SC 489.651u 3.67s 8:14.58 99.7% | ||
| mkmapdata 0.125x0.125 to 0.9x1.25 NOMASK SC 646.372u 5.07s 10:54 99.5% | ||
| mkmapdata 0.125x0.125 to 0.9x1.25 NOMASK UN 388.842u 4.20s 6:33 99.7% | ||
|
|
||
| mkmapdata 10x10min to 0.9x1.25 MASKED (2 grids) 639.7u 4.8s 10:46 99.7% | ||
| mkmapdata 10x10min to 0.9x1.25 NOMASK SCRIP 380.3u 3.1s 6:24 99.7% | ||
| mkmapdata 10x10min to 0.9x1.25 NOMASK UNSTRUCT 264.2u 2.6s 4:27 99.6% | ||
|
|
||
| mksurfdata_map 0.9x1.25 MASKED SCRIP: 65.846u 18.204s 1:42.64 81.8% | ||
| mksurfdata_map 0.9x1.25 NOMASK SCRIP: 74.439u 18.502s 1:57.48 79.1% | ||
|
|
||
| Notes of particular relevance for developers: (including Code reviews and testing) | ||
| --------------------------------------------- | ||
| NOTE: Be sure to review the steps in README.CHECKLIST.master_tags as well as the coding style in the Developers Guide | ||
|
|
||
| Caveats for developers (e.g., code that is duplicated that requires double maintenance): | ||
| None | ||
|
|
||
| Changes to tests or testing: | ||
| None | ||
|
|
||
| Code reviewed by: | ||
| @billsacks @ekluzek | ||
|
|
||
| CTSM testing: | ||
|
|
||
| [... Remove before making master tag. Available test levels: | ||
|
|
||
| a) regular (must be run before handing off a tag to SEs and must be run | ||
| before committing a tag) | ||
| b) build_namelist (if namelists and/or build_system changed)) | ||
| c) tools (only if tools are modified and no CTSM source is modified) | ||
| d) short (for use during development and in rare cases where only a small | ||
| change with known behavior is added ... eg. a minor bug fix) | ||
| e) doc (no source testing required) | ||
|
|
||
| ... ] | ||
|
|
||
| [PASS means all tests PASS and OK means tests PASS other than expected fails.] | ||
|
|
||
| build-namelist tests: | ||
|
|
||
| cheyenne - | ||
|
|
||
| tools-tests (test/tools): | ||
|
|
||
| cheyenne - | ||
|
|
||
| PTCLM testing (tools/shared/PTCLM/test): | ||
|
|
||
| cheyenne - | ||
|
|
||
| python testing (see instructions in python/README.md; document testing done): | ||
|
|
||
| (any machine) - | ||
|
|
||
| regular tests (aux_clm): | ||
|
|
||
| cheyenne ---- | ||
| izumi ------- | ||
|
|
||
| If the tag used for baseline comparisons was NOT the previous tag, note that here: | ||
|
|
||
|
|
||
| Answer changes | ||
| -------------- | ||
|
|
||
| Changes answers relative to baseline: | ||
|
|
||
| Summarize any changes to answers, i.e., | ||
| - what code configurations: ALL | ||
| - what platforms/compilers: ALL | ||
| - nature of change: larger than roundoff/same climate | ||
|
|
||
| I performed testing at the 0.9x1.25 degree resolution. | ||
|
|
||
| Code changes were tested by comparing baseline files (a) | ||
| surfdata_0.9x1.25_hist_78pfts_CMIP6_simyr2000_c191101.nc | ||
| surfdata_0.9x1.25_hist_78pfts_CMIP6_simyr2000_c191101.log | ||
| generated by running baseline code pointing to masked map_ files | ||
| to (b) | ||
| surfdata_0.9x1.25_new_hist_78pfts_CMIP6_simyr2000_c191101.nc | ||
| surfdata_0.9x1.25_new_hist_78pfts_CMIP6_simyr2000_c191101.log | ||
| generated by running the new code still pointing to masked map_ files | ||
| and then comparing to (c) | ||
| surfdata_0.9x1.25_nomask_78pfts_CMIP6_simyr2000_c191101.nc | ||
| surfdata_0.9x1.25_nomask_78pfts_CMIP6_simyr2000_c191101.log | ||
| generated by running the new code pointing to nomask map_ files. | ||
|
|
||
| We set an expectation of max abs relative differences of less than | ||
| roughly 1e-13. I got an actual max abs relative difference of 7e-12 in | ||
| PCT_NAT_PFT for (a) vs (b) and (a) vs (c). I refer to these here as | ||
| small diffs. | ||
|
|
||
| All other variables gave smaller diffs except the four VIC variables | ||
| binfil, Ws, Ds, Dsmax between (b) and (c). These diffs appear in 5 grid | ||
| cells (along two meridians that got shifted in longitude by less than | ||
| 1e-13 degrees) that vic considers land in the baseline and | ||
| ocean in the nomask case. Vic provides valid generic data for ocean | ||
| grid cells and, therefore, also where the discrepancies occur. This | ||
| means that if the land model treats these grid cells as land, the | ||
| model will work. | ||
|
|
||
| I had to generate new masked map_ files for (a) and (b) because | ||
| changes in ESMF codes over the years were leading to large differences | ||
| between all three (a) vs (b) vs (c). Generating new masked map_ files | ||
| brought the diffs back to small. | ||
|
|
||
| For the 3x3min and 5x5min SRC resolutions, I made six temporary | ||
| nomask SRC files, one per surface variable pointing to 3x3min | ||
| map_ files and one per surface variable pointing to 5x5min map_ files. | ||
| I did this to get small differences relative to the baseline. I created | ||
| the temporary files by making copies of the correspoding masked SRC | ||
| files and changing mask to 1 everywhere. The temporary files had diffs | ||
| less than roughly 1e-5 from generic nomask files generated by running | ||
| mkscripgrid.ncl for 3x3min and 5x5min. Ultimately we are replacing the | ||
| temporary files with the generic nomask files, which leads to larger | ||
| diffs in the surface datasets. | ||
|
|
||
|
|
||
| Detailed list of changes | ||
| ------------------------ | ||
|
|
||
| Pull Requests that document the changes (include PR ids): | ||
| https://github.com/ESCOMP/CTSM/pull/823 | ||
|
|
||
| =============================================================== | ||
| =============================================================== | ||
| Tag name: ctsm1.0.dev080 | ||
| Originator(s): sacks (Bill Sacks) | ||
| Date: Mon Dec 2 12:23:22 MST 2019 | ||
|
|
||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The old file was an incorrectly labeled 0.125-degree file, as first documented in this comment.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This most recent file name corresponds to the "trusted" generic 3x3min nomask file generated by mkscripgrid.ncl.
Same is true of the 5x5min file a few lines up from here.