working directory of Custodian does not match with that of VASP

[honghuikim]

Details about the quacc environment

• quacc version: 0.10.0
• Python version: 3.10.14
• VASP version: 5.4.4

What is the issue?

When running VASP with Custodian, Custodian does not save the corrected input files (e.g., INCAR, KPOINTS) in the working directory of VASP.

It seems Custodian saves them in the base working directory where the calculation script is submitted. I have checked that this bug still remains the same when I'm not using workflow engines. I guess directory of the temporary VASP working directory (e.g., tmp-quacc-xxx) is not transferred to Custodian in Vasp._run.

In the below example, all 5 corrections from custodian are Algo=Normal, but it's not applied on the failed-quacc-2024-07-02-06-06-37-930779-39481/INCAR while it's applied on the INCAR (./INCAR) in the base working directory.

How can we easily reproduce the issue?

from ase.build import bulk
from quacc.recipes.vasp.core import static_job

static_job(bulk("Cu"), efermi=None, nelm=1)

.
├── failed-quacc-2024-07-02-06-06-37-930779-39481
│   ├── AECCAR0
│   ├── AECCAR1
│   ├── AECCAR2
│   ├── ase-sort.dat
│   ├── CHG
│   ├── CHGCAR
│   ├── CONTCAR
│   ├── custodian.json
│   ├── DOSCAR
│   ├── EIGENVAL
│   ├── error.1.tar.gz
│   ├── error.2.tar.gz
│   ├── error.3.tar.gz
│   ├── error.4.tar.gz
│   ├── error.5.tar.gz
│   ├── IBZKPT
│   ├── INCAR
│   ├── INCAR.orig
│   ├── KPOINTS
│   ├── KPOINTS.orig
│   ├── OSZICAR
│   ├── OUTCAR
│   ├── PCDAT
│   ├── POSCAR
│   ├── POSCAR.orig
│   ├── POTCAR
│   ├── POTCAR.orig
│   ├── PROCAR
│   ├── REPORT
│   ├── std_err.txt
│   ├── vasp.out
│   ├── vasprun.xml
│   ├── WAVECAR
│   └── XDATCAR
├── INCAR
├── run.py
├── test.e430226
├── test.o430226
└── test.qsub

[Andrew-S-Rosen]

@honghuikim: thank you for this important bug report. This looks like a regression in behavior (ie it worked in a prior version but not now). I will ensure this is fixed in the afternoon and will add a test to make sure it does not happen again.

[honghuikim]

@Andrew-S-Rosen Thank you for the quick fix. Do you know what version of Quacc worked before? It could be a problem with my environment tbh. I can run the same test on that version.

[Andrew-S-Rosen]

I must admit, I'm not sure! It could also (potentially) be a bug with a recent version of custodian. I'll have to track down the ultimate cause. Doing pip install custodian 2024.4.18 would quickly tell us if it's a custodian bug or not. But I think it is reasonably likely it is a quacc bug.

[honghuikim]

I see. Actually, I had a similar problem before the recent update of Quacc (probably a month ago), but didn't take a look since it wasn't a big deal and I thought there was something to do with the old change_settings_wf and Parsl.
And I just checked that I'm using custodian 2024.4.18...

[Andrew-S-Rosen]

Looks like there is some digging for me to do when I'm back at my machine...!

[honghuikim]

Thank you! Take your time. I'll also dig some more meantime.

[Andrew-S-Rosen]

@honghuikim: It looks like this is a Custodian error. I determined this was likely the case because the Custodian files (e.g. error.1.tar.gz, custodian.json) are written in the correct directory, but only the updated INAR is not.

We can further confirm this with the following minimal example I have reported upstream: materialsproject/custodian#340.

I won't be able to solve this one immediately, but I think it's safe to say the problem is unfortunately further within the depths of the quacc ecosystem. In the meantime, unfortunately the only suggestion I have is to quacc set VASP_USE_CUSTODIAN False until it's patched.

[Andrew-S-Rosen]

Looks like it's been patched. Doing pip install git+https://github.com/materialsproject/custodian.git should fix it (until a new Custodian release is minted).

I'll keep this issue report open until Custodian has a new build on PyPI.

[honghuikim]

@Andrew-S-Rosen: Thank you very much for reporting the issue in Custodian. I just checked that it works now with the updated Custodian. Thank you!

[Andrew-S-Rosen]

Closing as patched upstream.