-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
7 changed files
with
166 additions
and
14 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,25 @@ | ||
#!/bin/bash | ||
|
||
# Check if the correct number of arguments is provided | ||
if [ "$#" -ne 7 ]; then | ||
echo "Usage: $0 <GITHUB_ACTIONS_WKDIR> <REPO_URL> <REGISTRATION_TOKEN> <LABELS> <SLURMD_NODENAME> <SLURM_JOB_ID> <REMOVAL_TOKEN>" | ||
exit 1 | ||
fi | ||
|
||
# Assign input arguments to variables | ||
GITHUB_ACTIONS_WKDIR=$1 | ||
REPO_URL=$2 | ||
REGISTRATION_TOKEN=$3 | ||
LABELS=$4 | ||
SLURMD_NODENAME=$5 | ||
SLURM_JOB_ID=$6 | ||
REMOVAL_TOKEN=$7 | ||
|
||
# Run the commands inside the Apptainer shell | ||
export DOCKER_HOST=unix:///tmp/run/docker.sock | ||
export RUNNER_ALLOW_RUNASROOT=1 | ||
export PYTHONPATH=/home/runner/.local/lib/python3.10/site-packages | ||
|
||
/home/runner/config.sh --work "${GITHUB_ACTIONS_WKDIR}" --url "${REPO_URL}" --token "${REGISTRATION_TOKEN}" --labels "${LABELS}" --name "slurm-${SLURMD_NODENAME}-${SLURM_JOB_ID}" --unattended --ephemeral | ||
/home/runner/run.sh | ||
/home/runner/config.sh remove --token "${REMOVAL_TOKEN}" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,4 @@ | ||
[proxy_plugins] | ||
[proxy_plugins.stargz] | ||
type = "snapshot" | ||
address = "/tmp/run/containerd-stargz-grpc/containerd-stargz-grpc.sock" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,4 @@ | ||
#!/bin/bash | ||
ls -l /mnt/wato-drive/alexboden | ||
ls -l /mnt/wato-drive/alexboden/provisioner-cache | ||
mkdir /mnt/wato-drive/alexboden/provisioner-cache/111312526769 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,54 @@ | ||
# Issues encountered | ||
1. Docker in docker | ||
2. Cloudflare 502 errors | ||
3. Caching docker images | ||
|
||
|
||
# 3 | ||
Does caching occur in the the machine? | ||
1. Testing | ||
|
||
# Testing basic | ||
Average times for each task: | ||
Register Runner: 4.02 seconds | ||
Configure Container: 0.37 seconds | ||
Run Runner: 128.38 seconds | ||
Start Container: 69.29 seconds | ||
Start Docker: 4.65 seconds | ||
Remove Container: 12.67 seconds | ||
Create Parent Directory: 0.02 seconds | ||
Remove Runner: 0.44 seconds | ||
|
||
Average times for each task: | ||
Register Runner: 19.64 seconds | ||
Configure Container: 3.00 seconds | ||
Run Runner: 258.25 seconds | ||
Start Container: 33.52 seconds | ||
Start Docker: 10.39 seconds | ||
Remove Container: 0.02 seconds | ||
Create Parent Directory: 0.02 seconds | ||
Remove Runner: 0.72 seconds | ||
|
||
|
||
Received status code: 401 Unauthorized. Refreshing creds... | ||
|
||
Options: | ||
Apptainer: | ||
- sif file in apptainer | ||
- cvmfs with apptainer | ||
|
||
stargz-snapshotter: | ||
- stargz-snapshotter | ||
|
||
Docker: | ||
try docker load from wato-drive | ||
|
||
look at the preprovision to see how to provision the image | ||
debugging | ||
- ./kuberenetes.sh k9s -A | ||
|
||
make new uid for container and user. | ||
base on slurm-dist, hard code version | ||
|
||
|
||
- try caching on ram for client image |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,61 @@ | ||
# Description: This script is used to poll the status of SLURM jobs that have been submitted by the user. | ||
# It was used mainly for debugging purposes to ensure that the jobs were being submitted and running as expected. | ||
|
||
import time | ||
|
||
def check_slurm_status(): | ||
if not allocated_jobs: | ||
return | ||
|
||
to_remove = [] | ||
frozen_jobs = allocated_jobs.copy() | ||
for job_id, runningjob in frozen_jobs.items(): | ||
if not runningjob or not runningjob.slurm_job_id: | ||
continue | ||
|
||
# Use 'sacct' to get job status and start time for a single job | ||
sacct_cmd = ['sacct', '-n', '-P', '-o', 'JobID,State,Start,End', '--jobs', str(runningjob.slurm_job_id)] | ||
try: | ||
sacct_result = subprocess.run(sacct_cmd, capture_output=True, text=True) | ||
sacct_output = sacct_result.stdout.strip() | ||
|
||
if sacct_result.returncode != 0: | ||
logging.error(f"sacct command failed with return code {sacct_result.returncode}") | ||
logging.error(f"Error output: {sacct_result.stderr}") | ||
continue | ||
|
||
for line in sacct_output.split('\n'): | ||
parts = line.split('|') | ||
if line == '' or len(parts) < 4: | ||
continue # Sometimes it takes a while for the job to appear in the sacct output | ||
|
||
job_component = parts[0] # e.g., '3840.batch' | ||
status = parts[1] | ||
start_time_str = parts[2] | ||
end_time_str = parts[3] | ||
|
||
# Focus only on the main job ID and ignore '.batch' or '.extern' components | ||
if '.batch' in job_component or '.extern' in job_component: | ||
continue | ||
|
||
# Convert time strings to datetime objects | ||
start_time = datetime.strptime(start_time_str, '%Y-%m-%dT%H:%M:%S') if start_time_str != 'Unknown' else None | ||
end_time = datetime.strptime(end_time_str, '%Y-%m-%dT%H:%M:%S') if end_time_str != 'Unknown' else None | ||
|
||
if status in ['COMPLETED', 'FAILED', 'CANCELLED', 'TIMEOUT']: # otherwise job is not finished running | ||
duration = "[Unknown Duration]" | ||
if start_time and end_time: | ||
duration = end_time - start_time | ||
logging.info(f"Slurm job {job_component} {status} in {duration}. Running Job Info: {str(runningjob)}") | ||
to_remove.append(job_id) | ||
|
||
except Exception as e: | ||
logging.error(f"Error querying SLURM job details for job ID {runningjob.slurm_job_id}: {e}") | ||
|
||
for job_id in to_remove: | ||
del allocated_jobs[job_id] | ||
|
||
def poll_slurm_statuses(sleep_time=1): | ||
while True: | ||
check_slurm_status() | ||
time.sleep(sleep_time) |