psij and Covalent job submission

[Andrew-S-Rosen]

ExaWorks/psij-python#398 (comment)

@RedStar-Iron:

@Andrew-S-Rosen It also makes me confused a lot when I check this setting in the quacc. I tried to submit the vasp/emt jobs from my local env (wsl) to the Expanse, but always failed, without error file left in ~/.psij . I was wondering whether it can read the setting of job script from a Json or YAML file directly which has been implemented by dpdispatcher.

[Andrew-S-Rosen]

@RedStar-Iron: Thanks for your discussion on the psij issue tracker. For context to those reading this, @RedStar-Iron is referring to the use of the psij-python dependency in the covalent-hpc-plugin used to dispatch Covalent calculations via quacc.

For a bit of backstory, Covalent already offers a covalent-slurm-plugin. So, if you are using Expanse which indeed uses Slurm, you might as well give that a go. The reason I made the separate covalent-hpc-plugin was because I wanted to have a plugin that interfaced to all the major job schedulers, including new ones (i.e. Flux). I also liked the idea of building around psij, which has a clear/unified standard for how to interface with the various job scheduler systems.

Unfortunately, in practice, this didn't end up being as useful as I had wanted it to be. The psij dependency simply adds on another layer of complexity, and I simply don't have the bandwidth to maintain the plugin myself to add features or make it more user-friendly.

Independent of Covalent, there are dozens and dozens of job script generators. dpdispatcher is one, but I could go on all day listing alternatives. The psij-python project has no interest in interfacing with external projects, so I don't see that being a viable route.

If you are interested in using quacc with Covalent on a machine with slurm, my suggestion is to use the covalent-slurm-plugin. If you have issues, chat with the Covalent folks and try to get things patched there. The job submission aspect is a bit beyond the scope of quacc itself. We offer interfaces to several workflow engines, and anything on the workflow dispatching side is handled elsewhere.

[Andrew-S-Rosen]

You may also be interested in trying Parsl as the workflow engine: https://parsl.readthedocs.io/en/stable/userguide/configuring.html#expanse-sdsc

[RedStar-Iron]

@Andrew-S-Rosen Thanks a lot! I am trying the Parsl, hopefully, it can work well with Expanse. I did try the covalent-slurm-plugin to bind with quacc, but still failed with No executor found by name: slurm when I run the following code. But I can smoothly ssh to the expanse through my local env.

n_nodes = 1
n_cores_per_node = 32

executor = ct.executor.SlurmExecutor(
    username=username,
    address="login.expanse.sdsc.edu",
    ssh_key_file="/home/xxx/.ssh/id_rsa",
    cert_file="/home/xxx/.ssh/id_rsa.pub",
    # launcher="srun",
    # launcher="mpirun",
    create_unique_workdir=True,
    options={
        "partition":"debug",
        "time": "00:30:00",
        "account": f"{account}",
        "nodes": f"{n_nodes}",
        "ntasks-per-node": f"{n_cores_per_node}",
        "cpus-per-task": f"{n_cores_per_node}",
        "export": "ALL",
    },
    prerun_commands=[
        "module purge", 
        "module load slurm",
        "module load cpu/0.17.3b",
        "module load ucx/1.10.1/wla3unl",
        "source /home/xxx/intel/oneapi/setvars.sh"],
    
    srun_options={
        "N" : f"{n_nodes}",
        "ntasks-per-node" : f"{n_cores_per_node}",
        "cpu_bind": "cores"},
    # environment={
    #     "QUACC_VASP_PARALLEL_CMD": f"srun -N {n_nodes} --ntasks-per-node={n_cores_per_node} --cpu_bind=cores"
    # },
    conda_env="quacc",
    remote_workdir="/home/xxx/test",
    cleanup=False,
)


@flow(executor=executor)
def workflow(atoms):
    relax_output = relax_job(atoms)
    return static_job(relax_output["atoms"])


atoms = bulk("C")
dispatch_id = ct.dispatch(workflow)(atoms)
result = ct.get_result(dispatch_id, wait=True)
print(result) 

[Andrew-S-Rosen]

but still failed with No executor found by name: slurm when I run the following code.

This sounds like something that can be solved with the slurm_path keyword argument to the executor.

[RedStar-Iron]

After adding the slurm_path="/cm/shared/apps/slurm/current/bin/ to the executor, it still shows the same error.

[Andrew-S-Rosen]

It is a very buggy interface, sadly... :(

[Andrew-S-Rosen]

@RedStar-Iron: In case it's of interest, I just overhauled the executor documentation to hopefully provide you and others with clearer information about your options.

[Andrew-S-Rosen]

Unfortunately, I can't explain why your unique setup is leading to an OpenSSH error. It is almost certainly due to a failure with the asyncssh.connect() call in the Covalent plugin, but this is not something I can debug for you.

Should I run the covalent server on Expanse as well?

No, if you are using the Slurm or HPC executors, they are meant to be used on a local machine that SSHs into the remote machine. The only plugins that run locally are the Dask and local executors.

[RedStar-Iron]

@Andrew-S-Rosen Thanks for your help! Now, I can submit the vasp job through the Covalent via the below code:

import covalent as ct
from ase.build import bulk
from quacc import flow
from quacc.recipes.vasp.core import relax_job, static_job

username = "username"
account = "xxx"
n_nodes = 1
n_cores_per_node = 32
vasp_parallel_cmd = (
    f"srun -N {n_nodes} --ntasks-per-node={n_cores_per_node} --cpu_bind=cores"
)

executor = ct.executor.SlurmExecutor(
    username=username,
    address="login.expanse.sdsc.edu",
    ssh_key_file="/your/home/path/.ssh/id_rsa",
    # cert_file="/your/home/path/.ssh/id_rsa.pub",
    slurm_path="/cm/shared/apps/slurm/current/bin/",
    create_unique_workdir=True,
    options={
        "job-name":"quacc",
        "partition":"debug",
        "time": "00:30:00",
        "account": f"{account}",
        "export": "ALL",
        "nodes":f"{n_nodes}",
        "ntasks-per-node":f"{n_cores_per_node}",
    },
    variables={"COVALENT_CONFIG_DIR": "/tmp"},
    use_srun=False,
    prerun_commands=[
        "module purge", 
        "module load slurm",
        "module load cpu/0.17.3b",
        "module load ucx/1.10.1/wla3unl",
        "source /home/xxx/intel/oneapi/setvars.sh",
        f"export QUACC_VASP_PARALLEL_CMD={vasp_parallel_cmd}",
        ],
    conda_env="quacc",
    remote_workdir="/xxx/test/vasp_test",
    cleanup=False,
)


@flow(executor=executor)
def workflow(atoms):
    relax_output = relax_job(atoms,kpts=[1,1,1])
    return static_job(relax_output["atoms"],kpts=[1,1,1])


atoms = bulk("C")
dispatch_id = ct.dispatch(workflow)(atoms)
result = ct.get_result(dispatch_id, wait=True)
print(result)

Noted that, it will cause the OpenSSH error if I didn't comment the line of cert_file. And I have to designate the "ntasks-per-node":f"{n_cores_per_node}" in the options, otherwise, the error of

sbatch: error: bank_limit plugin: Please set the ntasks or ntasks-per-node parameter
sbatch: error: Batch job submission failed: Requested operation not supported on this system

will come.
Another thing that I want to mention is the path of QUACC_VASP_PP_PATH in the Environment Variable should be the parent path of potpaw_PBE which include all the pseudopotential files used by vasp, rather the the path of potpaw_PBE directly which is in line with pymatgen.

Last thing is the tag in INCAR include the EFERMI = midgap by default, which would be problemic if the vasp verison below 6.4.

[RedStar-Iron]

By the way, could you please give me some hints about how to modify the input parameters of VASP used for relax_job, for example comment or remove the EFERMI = midgap from the INCAR.

[Andrew-S-Rosen]

@RedStar-Iron:

Apologies about the confusion. The documentation has a tooltip on that line that states: "This a certificate file used to validate your SSH credentials. This is often not needed but is required at NERSC facilities due to the use of sshproxy-based multi-factor authentication."

Another thing that I want to mention is the path of QUACC_VASP_PP_PATH in the Environment Variable should be the parent path of potpaw_PBE which include all the pseudopotential files used by vasp, rather the the path of potpaw_PBE directly which is in line with pymatgen.

The documentation states "The VASP_PP_PATH setting should point to the directory containing your VASP pseudopotentials. There should be two subdirectories name potpaw_PBE and potpaw for the PBE and LDA pseudopotentials, respectively. If your pseudopotential directories have a different name, create a symbolic link with the required naming scheme." This is based off the ASE documentation. The setting itself also has the following description: "Path to the VASP pseudopotential library. Must contain the directories potpaw_PBE and potpaw for PBE and LDA pseudopotentials, respectively." Is there anything that needs to be made clearer in this regard?

Last thing is the tag in INCAR include the EFERMI = midgap by default, which would be problemic if the vasp verison below 6.4.

Yes, this was intentional. The VASP manual states that EFERMI should be set to MIDGAP. Yes, this is problematic for older versions of VASP, but there is no way to check for the VASP version beforehand, and eventually most users will be running 6.4. I felt this was a worthwhile tradeoff.

By the way, could you please give me some hints about how to modify the input parameters of VASP used for relax_job, for example comment or remove the EFERMI = midgap from the INCAR.

As noted in the function singature, the **kwargs can be any value accepted by the ASE Vasp calculator. So, you can simply do efermi="legacy" to override the default.

[RedStar-Iron]

Thanks a lot for your help, I am more clear now!

[RedStar-Iron]

One modification for the VASP submission script on Expanse.

If the VASP used on Expanse is compiled by intel oneAPI toolkit by yourself, you'd better add export I_MPI_PMI_LIBRARY=/cm/shared/apps/slurm/current/lib64/libpmi.so in the prerun_commands, otherwise, there would be something wrong if you use the srun command. Like,

Abort(1090063) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Unknown error class, error stack:
MPIR_Init_thread(189): 
MPID_Init(1454)......: 
MPIR_pmi_init(113)...: 
(unknown)(): Unknown error class

or

MPI startup(): PMI server not found. Please set I_MPI_PMI_LIBRARY variable if it is not a singleton case.

These errors can not be handled by just adding --mpi=pmi2 to the srun command.

For clarification, the revised job script for vasp are shown below:

import covalent as ct
from ase.build import bulk
from quacc import flow
from quacc.recipes.vasp.core import relax_job, static_job

username = "xxx"
account = "user acount"
n_nodes = 1
n_cores_per_node = 32
vasp_parallel_cmd = (
    f"srun --mpi=pmi2 -N {n_nodes} --ntasks-per-node={n_cores_per_node} --cpu_bind=cores"   # without --mpi=pmi2  is fine.
)

executor = ct.executor.SlurmExecutor(
    username=username,
    address="login.expanse.sdsc.edu",
    ssh_key_file="/home/liug/.ssh/id_rsa",
    # cert_file="/home/liug/.ssh/id_rsa.pub",
    slurm_path="/cm/shared/apps/slurm/current/bin/",
    create_unique_workdir=True,
    options={
        "job-name":"quacc",
        "partition":"debug",
        "time": "00:30:00",
        "account": f"{account}",
        "export": "ALL",
        "nodes":f"{n_nodes}",
        "ntasks-per-node":f"{n_cores_per_node}",
    },
    variables={"COVALENT_CONFIG_DIR": "/tmp"},
    use_srun=False,
    prerun_commands=[
        "module purge", 
        "module load slurm",
        "module load cpu/0.17.3b",
        "module load ucx/1.10.1/wla3unl",
        # "module load cm-pmix3/3.1.7",   you don't need to module load this one.
        "export I_MPI_PMI_LIBRARY=/cm/shared/apps/slurm/current/lib64/libpmi.so",   # **This line should be added.**
        "source /home/wli7/intel/oneapi/setvars.sh",   
        f"export QUACC_VASP_PARALLEL_CMD={vasp_parallel_cmd}",
        ],
    conda_env="quacc",
    remote_workdir="/home/xxx/test/vasp_test",
    cleanup=False,
)


@flow(executor=executor)
def workflow(atoms):
    relax_output = relax_job(atoms,kpts=[5,5,5],efermi=None)
    return static_job(relax_output["atoms"],kpts=[5,5,5],efermi=None)


atoms = bulk("C")
dispatch_id = ct.dispatch(workflow)(atoms)
result = ct.get_result(dispatch_id, wait=True)
print(result)

[Andrew-S-Rosen]

Thanks for reporting. Yes, unfortunately there are unique subtleties for every machine out there so the docs are just a starting point. Glad you could sort it out.