How to upgrade your fork to use the latest stable release of MDTF-diagnostics

[wrongkindofdoctor]

A list of MDTF-diagnostics releases and notes can be found here. Stable releases have the naming convention v##.##, where numbers before the decimal point indicate a major release, and numbers after indicate small updates to the major release such as bug fixes or brief documentation extensions (e.g., v2.0, v3.2). Note that checking out the main branch without specifying a tag, or specifying a tag with an alpha or beta suffix, means that the code that is being actively developed and tested; therefore, it is subject to change (i.e., potentially break).

To upgrade your fork to use the latest stable release of MDTF-diagnostics:

1. On your machine, go to your local copy of the MDTF-diagnostics repository: cd <...>/MDTF-diagnostics
2. Commit any uncommitted changes to the current branch. If you have changes but aren’t ready to commit them, temporarily save them with git stash save.
3. Check out the main branch: git checkout main
4. Pull in updates from the NOAA-GFDL repository: git pull https://github.com/NOAA-GFDL/MDTF-diagnostics.git main
5. Push the main branch updates to your remote repository: git push -u origin main
6. Check out a new temporary branch from release tag: git checkout -b <temporary_branch_name> [release tag; e.g., v3.0]
7. Update all POD development branches:
   Let <branch_name> be the name of the branch you want to update (git branch will list all local branches).

 git checkout <branch_name>`
 git merge --no-ff <_temporary_branch_name_>

8. Update all the MDTF conda environments:
   a. Verify that you have a working conda installation by running conda --version and verifying that the version number is newer than 4.7.6.
   b. Determine the root directory of the conda installation you’d like the MDTF package to use; e.g. for the current user’s installation, it’s the path returned by conda info --root. If you’re using an input configuration file, this is the value of conda_root. If you’ve opted to install the conda environment files in a location other than the root conda directory, determine this path. If you’re using an input configuration file, this is the value of conda_env_root.
   c. Run the environment installation script:
   ./src/conda/conda_env_setup.sh --all --conda_root <conda_root> --env_dir <conda_env_root>
9. Commit the merge when prompted. No merge conflicts should be reported; POD development should not have altered framework code in src/ (with the potential exception of adding new conda environment definitions.)
10. Verify that your updated branch runs correctly, then push the changes to the remote copy of the branch:
    git push -u origin <branch_name>
11. If you saved uncommitted changes with git stash save in step 2, git checkout the branch you were initially on and execute git stash pop to recover those changes.
12. Delete the temporary branch: git branch -d <temporary_branch_name>

[bitterbark]

I followed these instructions but ran into two different problems. Hopefully I'm just missing something easy but I post here in case anyone else has these issues in the future.

Commands below denoted by %, and I'm attaching a log of my commands and their output as well as the default_tests file (appended .txt otherwise it wouldn't attach here).

1. at step (8), the first development branch I checked out failed to run with the error:
   % ./mdtf -f src/default_tests.jsonc
   ./mdtf: line 6: /home/bundy/mdtf/MDTF-diagnostics/mdtf_framework.py: No such file or directory

2. When I go back to main
   % git checkout main
   it will run (mdtf_framework.py exists) but the default_tests.jsonc file is failing:

% ./mdtf -f src/default_tests.jsonc
=== Starting /home/bundy/mdtf/MDTF-diagnostics/mdtf_framework.py

ERROR: unrecognized arguments: verbose
CRITICAL: ERROR: no valid entries in case_list. Please specify model run information.
Received:

The case_list looks fine regarding quotes and commas, and I've checked it matches what I've used before. However this error seems familiar for something being wrong in the file. It is concerning to me that the file still has verbose if that is unrecognized!

Thanks for any thoughts!

mdtf_update_to_v3.0beta.txt
default_tests.jsonc.txt

[wrongkindofdoctor]

Thanks for trying this out, Dani.

Try re-running the environment installation script (step 6c) on the other branch you are trying to update to re-generate the executable. I'm not sure about the second failure on main, but try running ./mdtf -f src/default_tests.jsonc -v (-v for "verbose) on any branch with an updated executable, then open an issue with detailed log output.

I also noticed that you are using version 2 of anaconda. I don't think it's responsible for the errors you are having, but we recommend using version 3 of anaconda or miniconda since the framework uses Python 3.

[bitterbark]

Is default_tests.jsonc the right input file to use? Is the fact that the version in my git dir has the apparently deprecated verbose setting in it a sign of a problem? I also see it in the code on github, so is it expected to give a warning?

…

On Fri, Feb 26, 2021 at 9:36 AM Jess ***@***.***> wrote: Thanks for trying this out, Dani. Try re-running the environment installation script (step 6c) on the other branch you are trying to update to re-generate the executable. I'm not sure about the second failure on main, but try running ./mdtf -f src/default_tests.jsonc -v (-v for "verbose) on any branch with an updated executable, then open an issue with detailed log output. I also noticed that you are using version 2 of anaconda. I don't think it's responsible for the errors you are having, but we recommend using version 3 of anaconda or miniconda since the framework uses Python 3. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#152 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHLJVQFD554E3HGHIDEWWFLTA7EXLANCNFSM4YDFXTIA> .

[bitterbark]

Thanks for putting your time into this, @wrongkindofdoctor. Having the commands you ran were helpful. I found I had two issues:

1. complete user error in that I merged with $branch_name instead of $temporary_branch_name. No suprise it said there was nothing to merge!
2. Like you pointed out, I needed to run conda_env_setup.sh after the merge. I'm still not sure why there are any differences between gfdl-remote/main (step 6 in the above instructions) and after the merge (step 8), but since it appears to make a difference, I propose the instructions be changed in the original post.

BTW I timed the conva_env_setup and it took 32 minutes. This is typical and a major reason I wanted to do everything possible before trying that.

Thanks again for your help!

[wrongkindofdoctor]

Woohoo! Glad this worked out. I reorganized the instructions as requested. The conda_env_setup.sh script is a major sticking point in the installation process, and holds up the CI on github actions, too (though it takes more like 15 min than 30!). I'll open an issue and make sure to discuss with the rest of the crew.

[bitterbark]

Did we ever get a way to speed up the conda_env_setup.sh script? I'm trying to update to the latest code and getting the same error as before and once again, this slow script is making it quite difficult. Thanks!

[tsjackson-noaa]

@bitterbark: yes, performance of conda_env_setup.sh was addressed in PR #166 that was merged back in April.

[bitterbark]

I just re-installed the latest version on a new machine and it took 2 hours 58 minutes to create the conda envs. The stdout indicates it used mamba through the _MDTF_install_temp environment. Is there something else I could be doing to speed it up? This will be an issue when convincing model developers to use the framework. Also, it makes me hesitant to update my own environment with new versions.

Happy to move this discussion elsewhere if it's inappropriate to use this old thread.

[jhafner2]

Hello
I am just wondering if anybody completed the upgrade to framework v3.0-beta3 and ran their POD successfully? I am experiencing the problems with the upgrade. It seems to be updated and it starts the POD run, but when it comes to actually execute the POD code it is exiting with errors. One of the main is missing python modules in conda environment. Please, if anybody successfully completed the upgrade process, could you share your insight into the process? thanks Jan

[jkrasting]

Sorry you're still having trouble @jhafner2. I am curious if you rebuilt your conda environments as part of the upgrade.

[jhafner2]

For some reason after the rebuild the python modules (e.g. xarray) do not load in the script.
One possibility I can think of which could be the problem is in the settings.jsonc file. I have noticed it differs for the upgraded version v3.0-beta3. I am attaching the file here for your reference. Please, if you could kindly have a look and let me know if you will see any obvious errors. thank you

{
"settings": {
"driver": "ENSO_MSE.py",
"long_name": "ENSO moist static energy budget",
"description": "ENSO moist static energy budget",
"runtime_requirements": {
"python3": ["numpy", "xarray", "netCDF4"],
"ncl": ["contributed", "gsn_code", "gsn_csm", "shea_util", "calendar_decode2"]
}
},
"data": {
"frequency": "mon"
},
"dimensions": {
"lat": {"standard_name": "latitude"},
"lon": {"standard_name": "longitude"},
"lev": {
"standard_name": "air_pressure",
"units": "hPa",
"positive": "down",
"axis": "Z"
},
"time": {"standard_name": "time"}
},

"pod_env_vars": {
	"slon1": "160",
	"slon2": "200",
	"slat1": "-10",
	"slat2": "5",
	"ENSO_OBS": "1",
	"ENSO_COMPOSITE": "1",
	"ENSO_MSE": "1",
	"ENSO_MSE_VAR": "1",
	"ENSO_SCATTER": "1"
},
"varlist": {
	"zg_var": {
                "standard_name": "geopotential_height",
                "units": "m",
                 "dimensions": ["time", "lev", "lat", "lon"],
		"freq": "mon"
	},
	"ua_var": {
                 "standard_name": "eastward_wind",
                 "units": "m s-1",
                 "dimensions": ["time", "lev", "lat", "lon"],
		"freq": "mon"
	},
	"va_var": {
                 "standard_name": "northward_wind",
                 "units": "m s-1",
                 "dimensions": ["time", "lev", "lat", "lon"],
		"freq": "mon"
	},
	"ta_var": {
                  "standard_name": "air_temperature",
                 "units": "K",
                 "dimensions": ["time", "lev", "lat", "lon"],
		"freq": "mon"
	},
	"qa_var": {
                 "standard_name": "specific_humidity",
                 "units": "1",
                 "dimensions": ["time", "lev", "lat", "lon"],
		"freq": "mon"
	},
	"omega_var": {
                 "standard_name": "lagrangian_tendency_of_air_pressure",
                 "units": "Pa s-1",
                 "dimensions": ["time", "lev", "lat", "lon"],
		"freq": "mon"
	},
	"pr_var": {
                 "standard_name": "precipitation_flux",
                 "units": "kg m-2 s-1",
                 "dimensions": ["time", "lat", "lon"],
		"freq": "mon"
	},
	"ts_var": {
                 "standard_name": "surface_temperature",
                 "units": "K",
                 "dimensions": ["time", "lat", "lon"],
		"freq": "mon"
	},
	"hfss_var": {
                 "standard_name": "surface_upward_sensible_heat_flux",
                 "units": "W m-2",
                 "dimensions": ["time", "lat", "lon"],
		"freq": "mon"
	},
	"hfls_var": {
                 "standard_name": "surface_upward_latent_heat_flux",
                 "units": "W m-2",
                 "dimensions": ["time", "lat", "lon"],
		"freq": "mon"
	},
	"rsus_var": {
                 "standard_name": "surface_upwelling_shortwave_flux_in_air",
                 "units": "W m-2",
                 "dimensions": ["time", "lat", "lon"],
		"freq": "mon"
	},
	"rsds_var": {
                 "standard_name": "surface_downwelling_shortwave_flux_in_air",
                 "units": "W m-2",
                 "dimensions": ["time", "lat", "lon"],
		"freq": "mon"
	},
	"rsdt_var": {
                 "standard_name": "toa_incoming_shortwave_flux",
                 "units": "W m-2",
                 "dimensions": ["time", "lat", "lon"],
		"freq": "mon"
	},
	"rsut_var": {
                 "standard_name": "toa_outgoing_shortwave_flux",
                 "units": "W m-2",
                 "dimensions": ["time", "lat", "lon"],
		"freq": "mon"
	},
	"rlus_var": {
                 "standard_name": "surface_upwelling_longwave_flux_in_air",
                 "units": "W m-2",
                 "dimensions": ["time", "lat", "lon"],
		"freq": "mon"
	},
	"rlds_var": {
                 "standard_name": "surface_downwelling_longwave_flux_in_air",
                 "units": "W m-2",
                 "dimensions": ["time", "lat", "lon"],
		"freq": "mon"
	},
	"rlut_var": {
                 "standard_name": "toa_outgoing_longwave_flux",
                 "units": "W m-2",
                 "dimensions": ["time", "lat", "lon"],
		"freq": "mon"
	}
}

}

[wrongkindofdoctor]

@jhafner2 Seconding @jkrasting 's recommendation. If you are still having trouble, push whatever you have on your local machine to the remote branch(es) that you are trying to update and I can test them on my machine.

[bitterbark]

@thomas Jackson - NOAA Affiliate ***@***.***> So when I'm running the latest GFDL main or develop code, it should be using mamba automatically, in a way that is transparent to me, and should not be taking more than 30 minutes? Or do I need to do something to make it work? (Because it is still taking more than 30 minutes for this up-to-date code).

…

On Thu, Aug 26, 2021 at 6:21 PM Thomas Jackson ***@***.***> wrote: @bitterbark <https://github.com/bitterbark>: yes, performance of conda_env_setup.sh was addressed in PR #166 <#166> that was merged back in April. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#152 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHLJVQCTVOHWVXPQKZ23AU3T63LCLANCNFSM4YDFXTIA> .

[tsjackson-noaa]

Yes, installation and use of mamba is transparent, and the user instructions are up to date.

conda_env_setup.sh --all appears to typically take under 10 minutes in the CI builds, and locally on my laptop. Because it's installing lots of packages via a package manager, it will make appreciable use of CPU (for solving package dependencies), network and disk, so performance will differ from system to system.

Is the script finishing successfully? The final line of output should be "Created MDTF wrapper script at < repo directory > ." It's only necessary to run the script once to update all environments.

[bitterbark]

It finishes successfully every time. It took 32 minutes this morning which is typical. I'm attaching the output here; I don't see the connectivity errors you mentioned in the other thread that you told me not to use. It is really not my main problem, which is that I am not able to run the latest MDTF (will post as a separate issue), but having over 30 min delay to switching versions makes it difficult to debug.

[bitterbark]

I've followed the above instructions in order to just test the GFDL main branch before merging with my changes but it is failing to run. It quits with the error:
CRITICAL: No valid entries in case_list. Please specify model run information.

despite receiving entries in case_list in a validated input file that differs as little as possible from default_tests.jsonc.

Last time (on this thread in the spring) I was able to get past this by re-running conda_env_setup but this time it isn't helping.

I'm running with -v debug option which tells me
DEBUG: Console loggers configured.
DEBUG: Using selector: EpollSelector

I've spent quite some time putting in debug statements to try to figure it out how the input is parsed but I haven't cracked it yet.

The commands I've used:
git checkout main
git pull upstream main
git reflog shows the code to be at b16e49d which matches the latest GFDL main branch on GitHub.

Any advice on how else to debug? I'm assuming the code works for you all. I started off posting on the commit in question but was told to bring the discussion here.

[bitterbark]

For the record for anyone else who runs into this simple error: the -f flag is not optional (as I had mistakenly believed)
FAILS: ./mdtf src/input_dani.jsonc
WORKS: ./mdtf -f src/input_dani.jsonc

Thanks to Rich @swrneale for providing notes on getting the latest version to work for him.

[wrongkindofdoctor]

@bitterbark Glad you were able to resolve this. The documentation for the cli doesn't explicitly state whether options are required or optional, so I can add a some lines specifying this to the descriptions.

[bitterbark]

Thanks @wrongkindofdoctor . To clarify, I thought we could now give the input file as an argument without the -f flag. Not sure where I got this idea!

[jkrasting]

I also remember hearing that the `-f` flag was optional at some point. I usually run with the flag anyway.

[tsjackson-noaa]

Hi,
The problem encountered by @bitterbark is not whether the input_file setting is optional/required (it's always been optional), but whether it's parsed as a positional argument or requires the -f/--input-file flag beforehand. The input file used to be recognized as a positional setting, but this functionality was removed in December 2020 (87bdf88) to simplify the CLI parsing logic, since both input_file and CASE_ROOT_DIR were optional positional arguments.

The CLI options were correctly documented in both the sphinx documentation and CLI help (mdtf -h) before and after this change.

Omitting the -f flag currently results in an unhelpful error message about an empty caselist, because the input file path has been used to set CASE_ROOT_DIR, and this attribute isn't validated until relatively late in the setup procedure. PR #276 fixes this.

[bitterbark]

Thanks for the clarification; I meant that the -f flag was optional vs required, not the input file itself. It is always challenging for existing users to know when things have changed, even if they are correctly documented, so thank you for working on the error message. It seems obvious in hindsight I should have picked up on my error, but since I'd seen the same error message in the past when I had incompatible conda envs, I went down that rabbit hole. Or maybe that had been my problem previously too and I inadvertently fixed it with the -f flag at the same time as I rebuilt my conda env!

…

On Thu, Sep 16, 2021 at 12:42 AM Thomas Jackson ***@***.***> wrote: Hi, The problem encountered by @bitterbark <https://github.com/bitterbark> is not whether the input_file setting is optional/required (it's always been optional), but whether it's parsed as a positional argument or requires the -f/--input-file flag beforehand. The input file used to be recognized as a positional setting, but this functionality was removed in December 2020 ( 87bdf88 <87bdf88>) to simplify the CLI parsing logic, since both input_file and CASE_ROOT_DIR were optional positional arguments. The CLI options were correctly documented in both the sphinx documentation and CLI help (mdtf -h) before and after this change. Omitting the -f currently results in an unhelpful error message about an empty caselist, because the input file path has been used to set CASE_ROOT_DIR, and this attribute isn't validated until relatively late in the setup procedure. PR #276 <#276> fixes this. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#152 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHLJVQG6AGTAADOD7KMRFEDUCGGWZANCNFSM4YDFXTIA> .