Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New feature: non-coarray LUT #172

Merged
merged 3 commits into from
Nov 28, 2023

Conversation

scrasmussen
Copy link
Member

@scrasmussen scrasmussen commented Sep 28, 2023

TYPE: new feature

KEYWORDS: LUT, look-up table, coarrays

SOURCE: Soren Rasmussen, NCAR

DESCRIPTION OF CHANGES:

  • The large LUT coarray has been changed to a regular array and two new smaller coarrays are used to communicate the data. This change has been done so the Cray compiler on Derecho can handle large LUT files.
  • added check if to see if LUT file paths are writeable when reading namelist options
  • there were issues with Crays debugslow mode causing issues with the the namelist being read incorrectly. Seems to be from the -eZ preprocessor option creating .i files mixed with some other compile options. The new preprocessor option is now -eT. The following shows the different preprocessor options and their descriptions
  • Preprocessing flags:
    • -eT Control preprocessing of Fortran source files. When enabled, source preprocessing is performed. Macro expansion within Fortran source lines is enabled but can be controlled by the -e/d F command line option. When disabled (-dT), preprocessing of the Fortran source file is not performed, even for files with upper case suffixes such as file.F90.
    • -eZ Perform source preprocessing and compilation on Fortran source files. When specified, source code is included by both #include directives and Fortran INCLUDE lines. Generates file file.i, which contains the source code after the preprocessing has been performed and the effects applied to the source program.
    • -eP Perform source preprocessing on Fortran source files but do not compile. When specified, source code is included by #include directives but not by Fortran INCLUDE lines. Generates file.i, which contains the source code after the preprocessing has been performed and the effects applied to the source program. If the -o out_fileargument is also specified, the preprocessed source is written to out_file instead of file.i.
    • -eF Control preprocessor expansion of macros in Fortran source lines.

TESTS CONDUCTED:

  • A Cray 6km_conus_west.nc testcase that broke on Derecho from lack of memory now runs.
  • Ran the PR executable and the upstream executable with 1,2,4,8 cores. I then compared the resulting look-up table files and they were always identical between the PR and upstream output. Note, this set of comparisons was done with the idealized test case and the following lt_parameters
&lt_parameters
n_dir_values=2,
n_spd_values=2,
n_nsq_values=2,
buffer = 2,               
stability_window_size = 1,    
vert_smooth = 1,
variable_N = False,
read_LUT  = True,  
write_LUT = True,  
LUT_filename = "Lineary_Theory_LUT.nc"
/

Checklist

  • Closes issue #xxxx (An issue must exist or be created to be closed. The
    issue describes and documents the problem and general solution, the PR
    describes the technical details of the solution.)
  • Tests added (unit tests and/or regression/integration tests)
  • Backwards compatible
  • Requires new files? If so, how to generate them.
  • Fully documented

… smaller coarrays are used to communicate the data.

This change has been done so the Cray compiler on Derecho can handle large LUT files.
do img = 1, num_images()
associate(ims => grids(img)%ims, &
ime => grids(img)%ime, &
jms => grids(img)%jms, &
jme => grids(img)%jme &
)
!$omp critical
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

double check that !$omp critical isn't needed. Are there other OpenMP calls in the LUT routines

LUT(LUT_k, LUT_i, LUT_j, 1:ime-ims+1, LUT_z, 1:jme-jms+1) = LUT_co(1:ime-ims+1, 1:jme-jms+1, img)
end do
end associate
sync all
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can remove this sync all since one is called outside these subroutine calls

call copy_data_to_remote(temporary_v, v_grids, hi_v_LUT, i,j,k, z)
call copy_data_to_remote(temporary_u, u_grids, hi_u_LUT, hi_u_LUT_co, LUT_index_co, i, j, k, z)
call copy_data_to_remote(temporary_v, v_grids, hi_v_LUT, hi_v_LUT_co, LUT_index_co, i, j, k, z)
sync all
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove this sync all

@gutmann
Copy link
Member

gutmann commented Nov 28, 2023

may still be able to remove one of the sync alls, but need to get this code into the develop branch for further testing.

@gutmann gutmann merged commit c3609d5 into NCAR:develop Nov 28, 2023
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants