-
Notifications
You must be signed in to change notification settings - Fork 44
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
grafana monitoring not working for static resources #35
Comments
I propose this line 1click-hpc/scripts/post.install.sh Line 79 in 7a833d4
the reason is the stack name is also unique even if it is recycled, and we can use the SLURM_CLUSTER_NAME Name of the cluster executing the job inside the prolog scripts, thus avoiding undesired calls to I will eventually submit PR, now at the stage of brainstorming to get the best idea to work (hm, the current value of SLURM_CLUSTER_NAME is always |
when we use a non-zero minimum in cluster config for resources, they get alive at cluster launch. then this job-related check will never have a value of True:
1click-hpc/modules/40.install.monitoring.compute.sh
Line 59 in 7a833d4
because this must be run in the root context, the only chance to do it is in the prolog script to attach it to a job, so basically the plan would be to
the problem is how to send a signal about the job to prolog and epilog since the custom user env variables are not sent, and the job comment is not sent. Because per slurm manuals, we should not perform
scontrol
from prolog; this will impair the scaling of the jobs similarly to the API calls (this is related to #34 )Looking at the variables available at prolog/epilog time I only have 2 ideas so far:
#SBATCH --nice 0
or some sensible value to uniquely identify the intention then use the TaskProlog and TaskEpilog to start/stop the monitoring container[GM] my job name
then pick and interpret this from SLURM_JOB_NAME Name of the job. Available in PrologSlurmctld, SrunProlog, TaskProlog, EpilogSlurmctld, SrunEpilog and TaskEpilog. Meaning also the use of TaskProlog and TaskEpilog to start/stop the monitoring containerThe text was updated successfully, but these errors were encountered: