Update to Parthenon with Kokkos 4.4.1 #13
Conversation
…brryan/kokkos_441
brryan
changed the title
env/bash
Outdated
| PARTITION="venado-gh" | ||
| elif [[ "$HOSTNAME" =~ ^ve-rfe[4-7]$ || ( $SLURM_CLUSTER_NAME == "venado" && $SLURM_JOB_PARTITION == "cpu" ) ]]; then | ||
| elif [[ "$HOSTNAME" =~ ^ve-rfe[4-7]$ || "$HOSTNAME" =~ ^ve-fe[4-7]$ || ( $SLURM_CLUSTER_NAME == "venado" && $SLURM_JOB_PARTITION == "cpu" ) ]]; then |
There was a problem hiding this comment.
I don't think you can distinguish cpu/gpu from the hostname. You can be on a grace-grace frontend but submit to a grace-hopper backend. The hostname you check on the backend is still the grace-grace one. I think if you did hostname in [1-3] or $SLURM_GPUS_ON_NODE > 0, that would always work
There was a problem hiding this comment.
Yeah you're right, thanks for noticing this. I thought at some point in the past HOSTNAME wasn't defined on venado backends? But it definitely is now, maybe I'm just misremembering. Yes I can update this logic to fix this.
This also doesn't work for e.g. SLURM_JOB_PARTITION=gpu_debug (it's only been sneaking through because I always use gpu frontends for gpu backends etc.)
There was a problem hiding this comment.
OK this should be fixed now. I tested it on cpu frontend, and gpu backend via either cpu frontend or gpu frontend. I'm not sure I have access to the CPU backends actually to test those
Background
Chicoma and Venado are failing at runtime when running with multiple GPUs using CUDA-aware MPI. Forrest found that moving to Kokkos 4.4.1 fixes this issue at least on Venado.
Description of Changes
env/bashscript to support Venado (sort of)constexpr ifcapture)and warning (unused var) on recentnvccChecklist