Update preprocessing scripts

[yashkurkure]

Preprocessing scripts

This will contain all the code, tests, and data used in preprocessing. Create a new directory called /preprocessing at the root of the project. This directory will contain the following:

• folder data which contains the input data from the ANL website
• script preprocessing_theta_23_24.py
• script preprocessing_polaris_23_24.py
• a README.md on how to use the preprocessing.py scripts

data

The data folder will contain the following:

• ANL-ALCF-DJC-POLARIS_20230101_20231231.csv
• ANL-ALCF-DJC-POLARIS_20240101_20240930.csv
• ANL-ALCF-DJC-THETA_20230101_20231231.csv
• ANL-ALCF-DJC-THETA_20240101_20240930.csv

preprocessing_theta_23_24.py

This will read the files ANL-ALCF-DJC-THETA_20230101_20231231.csv and ANL-ALCF-DJC-THETA_20240101_20240930.csvthen output a file called theta_23_24.swf.

• first job id should begin with 1
• file should be sorted by submit column
• all entries should be integers (no decimal points in the swf)
• ignore jobs whose used_proc is 0
• ignore jobs whose req_proc is 0
• ignore jobs whose req_time is 0
• ignore jobs part of the debug queue
• theta_23_24.swf should contain jobs in the range Sep 1, 23 to Oct 7, 24
• Include the following header in theta_23_24.swf:

; UnixStartTime: 0
; MaxNodes: 4360
; MaxProcs: 4360

preprocessing_polaris_23_24.py

This will read the files ANL-ALCF-DJC-POLARIS_20230101_20231231.csv and ANL-ALCF-DJC-POLARIS_20240101_20240930.csv then output a file called polaris_23_24.swf

• first job id should begin with 1
• file should be sorted by submit column
• all entries should be integers (no decimal points in the swf)
• ignore jobs whose used_proc is 0
• ignore jobs whose req_proc is 0
• ignore jobs whose req_time is 0
  These two requirements are specific to Polaris trace:
• ignore jobs whose req_proc <= 8 and part of the debug queue
• for jobs whose req_proc > 8 and part of debug queue, keep the job but in the swf file the req_proc should be procs in alcf file - 8
• polaris_23_24.swf should contain jobs in the range Sep 1, 23 to Oct 7, 24
• Include the following header in theta_23_24.swf:

; UnixStartTime: 0
; MaxNodes: 552
; MaxProcs: 552

SWF file structure

https://www.cs.huji.ac.il/labs/parallel/workload/swf.html

swf_columns = [
    'id',             #1
    'submit',         #2
    'wait',           #3
    'run',            #4
    'used_proc',      #5
    'used_ave_cpu',   #6
    'used_mem',       #7
    'req_proc',       #8
    'req_time',       #9
    'req_mem',        #10 
    'status',         #11
    'user_id',     #12
    'group_id', #13
    'num_exe',        #14
    'num_queue',         #15
    'num_part',       #16
    'num_pre',        #17
    'think_time',     #18
    ]

[yashkurkure]

Also upload the final traces here: https://drive.google.com/drive/folders/1vKV5f65mNxKA14xT5yt-YWuG6G0FIF7Y?usp=share_link

[mochiiten9158]

#4 (comment)

Preprocessing for theta and polaris

[mochiiten9158]

Updated: New Pull Request with the correct preprocessing files
#5 (comment)