Splitting a NetCDF input file into multiple files for calibration

[Nooshdokht-Bayatafshary]

Dear developers,

I am currently working on calibrating a Lisflood model using 6-hourly data spanning 7 years at a relatively fine resolution. The process involves generating meteorological data as a unified NetCDF file, which incurs a significant computational cost (more than 370 GB of RAM, and about 1 TB of storage are required for each parameter).

I am wondering if it's possible to enhance the calibration mode to accept data in a more manageable format. Specifically, instead of providing a single 7-year precipitation file, would it be feasible to submit 7 separate one-year precipitation files for the calibration input?

Your guidance on this matter would be greatly appreciated, and I am open to discussing potential solutions or workarounds to optimize the calibration process.

Thank you for your time and consideration.

Best regards,
Nooshdokht

[StefaniaGrimaldi]

Dear @Nooshdokht-Bayatafshary,

thank you for reaching out and for your willingness to collaborate!
Your application sounds very interesting: the size of the files points to a quite fine resolution model set-up.

We are pleased to let you know that OS LISFLOOD is capable of reading meteorological forcings split into multiple files (e.g. yearly chuncks). To use this functionality, it is enough to add the symbol '' after the file name (e.g. ET0_).
This functionality can be used in calibration mode ().
We realized that this functionality was not explained in the documentation: we added a sentence, the revised documentation will be publicly available with the next minor update of OS LISFLOOD.

We welcome suggestions for improvement, and we would be pleased to work with you!

Kind regards,
on behalf of the developers team,
Stefania

[Nooshdokht-Bayatafshary]

Dear Stefania and the OS LISFLOOD Developers Team,

Thank you so much for your prompt and informative response! I'm thrilled to hear that OS LISFLOOD supports reading meteorological forcings split into multiple files. I'm also delighted to learn about the upcoming documentation update that will include this functionality. Your team's responsiveness is truly commendable.

The addition of the symbol after the file name for this functionality is a valuable insight, and I appreciate your clarification. To ensure I've grasped the naming convention correctly, for a seven-year setup, I should label the files as follows: ET0_1.nc, ET0_2.nc, ..., ET0_7.nc, correct? Additionally, could you confirm if any specific setting changes in the configuration files are necessary to implement this functionality, or will the default settings suffice?

Once again, thank you for your support and openness to collaboration. I look forward to exploring the capabilities of OS LISFLOOD further.

Kind regards,
Nooshdokht

[StefaniaGrimaldi]

Dear @Nooshdokht-Bayatafshary,

you can name the files using the relevant year, e.g. ET0_2023.nc.

We look forward to the next exchange,
Kind regards,
on behalf of the developers team,
Stefania

[Nooshdokht-Bayatafshary]

Dear @StefaniaGrimaldi,

I am trying to run the calibration code (CAL_7A_CALIBRATION.py), but I encountered an error when opening the forcing file. As previously mentioned, LISFLOOD can accept inputs as split files, and I am attempting to use this option. I have set the following options in my XML file:

**************************************************************
                PREFIXES OF METEO AND VEGETATION RELATED VARIABLES
**************************************************************
</comment>

<textvar name="PrefixPrecipitation" value="tp_">
    <comment>
        prefix precipitation maps
    </comment>
</textvar>

<textvar name="PrefixTavg" value="ta_">
    <comment>
        prefix average temperature maps
    </comment>
</textvar>

<textvar name="PrefixE0" value="e0_">
    <comment>
        prefix E0 maps
    </comment>
</textvar>

<textvar name="PrefixES0" value="es_">
    <comment>
        prefix ES0 maps
    </comment>
</textvar>

<textvar name="PrefixET0" value="et_">
    <comment>
        prefix ET0 maps
    </comment>
</textvar>

However, when I run the CAL_7A_CALIBRATION script, I get the following error:
OSError: Could not open file ../Karkheh/outcal/CATCHMENTS_DIR/21413/maps/tp_.nc

As you can see in the attached picture, I have provided the files with the appropriate names.

Could you please provide guidance on how to resolve this issue? I have attached my template XML file for your reference (settings_calibration.zip).

Thank you for your assistance and your support.
Best regards,
Nooshdokht

[StefaniaGrimaldi]

Dear @Nooshdokht-Bayatafshary,

could you please edit the forcing file names in your settings xml as follows?
tp_ --> tp_*
ta_ --> ta_*
e0_--> e0_*
es_--> es_*
et_--> et_*

This change is expected to solve your problem.

Please do not hesitate to let us know whether you encounter other issues,
Kind regards,
on behalf of the developers team,
Stefania