Problems with convert_and_aggregate for long timespans?

[Tasqu]

Description

It seems that there's some problem with the conversion and aggregation functionality when used for long timespans.
E.g. The building_stock_weather_aggregation.ipynb stops working when cutout time includes multiple months.

I'm hardly an expert, but as far as I can tell, the common problem seems to be the xr.apply_ufunc(...) call in aggregate_matrix(da, matrix, index). This problem can be circumvented by passing allow_rechunk=True in dask_gufunc_kwargs (as suggested by the error messages), but this significantly slows the performance down from what I remember it being.

My best guess is that this might be related to #252. Maybe with the data being downloaded in monthly increments, it's organized differently, causing issues with the aggregate_matrix function?

Expected Behavior

The cutout.temperature and cutout.irradiation used to work no problems even for year-long timespans.

Actual Behavior

Now, with cutouts spanning more than 1 month the cutout.temperature and cutout.irradiation become unreliable, throwing errors like the ones included below (from examples\building_stock_weather_aggregation.ipynb).

Error Message

When used on a cutout with 2 months of data, cutout.irradiation fails:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
[c:\atlite\examples\building_stock_weather_aggregation.ipynb](file:///C:/atlite/examples/building_stock_weather_aggregation.ipynb) Cell 18 line 1
     [13](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=12) dirs = {"north": 0.0, "east": 90.0, "south": 180.0, "west": 270.0}
     [14](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=13) for name, lout in layouts.items():
---> [15](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=14)     irr_total[name] = {
     [16](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=15)         d: cutout.irradiation(
     [17](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=16)             orientation={"slope": 90.0, "azimuth": az}, layout=lout.fillna(0.0)
     [18](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=17)         )
     [19](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=18)         .squeeze()
     [20](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=19)         .to_series()
     [21](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=20)         for d, az in dirs.items()
     [22](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=21)     }
     [23](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=22)     irr_direct[name] = {
     [24](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=23)         d: cutout.irradiation(
     [25](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=24)             orientation={"slope": 90.0, "azimuth": az},
   (...)
     [31](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=30)         for d, az in dirs.items()
     [32](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=31)     }
     [33](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=32)     irr_diffuse[name] = {
     [34](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=33)         d: cutout.irradiation(
     [35](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=34)             orientation={"slope": 90.0, "azimuth": az},
   (...)
     [41](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=40)         for d, az in dirs.items()
     [42](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=41)     }

[c:\atlite\examples\building_stock_weather_aggregation.ipynb](file:///C:/atlite/examples/building_stock_weather_aggregation.ipynb) Cell 18 line 1
     [13](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=12) dirs = {"north": 0.0, "east": 90.0, "south": 180.0, "west": 270.0}
     [14](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=13) for name, lout in layouts.items():
     [15](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=14)     irr_total[name] = {
---> [16](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=15)         d: cutout.irradiation(
     [17](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=16)             orientation={"slope": 90.0, "azimuth": az}, layout=lout.fillna(0.0)
     [18](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=17)         )
     [19](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=18)         .squeeze()
     [20](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=19)         .to_series()
     [21](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=20)         for d, az in dirs.items()
     [22](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=21)     }
     [23](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=22)     irr_direct[name] = {
     [24](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=23)         d: cutout.irradiation(
     [25](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=24)             orientation={"slope": 90.0, "azimuth": az},
   (...)
     [31](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=30)         for d, az in dirs.items()
     [32](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=31)     }
     [33](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=32)     irr_diffuse[name] = {
     [34](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=33)         d: cutout.irradiation(
     [35](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=34)             orientation={"slope": 90.0, "azimuth": az},
   (...)
     [41](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=40)         for d, az in dirs.items()
     [42](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=41)     }

File [C:\atlite\atlite\convert.py:600](file:///C:/atlite/atlite/convert.py:600), in irradiation(cutout, orientation, irradiation, clearsky_model, **params)
    597 if not callable(orientation):
    598     orientation = get_orientation(orientation)
--> 600 return cutout.convert_and_aggregate(
    601     convert_func=convert_irradiation,
    602     orientation=orientation,
    603     irradiation=irradiation,
    604     clearsky_model=clearsky_model,
    605     **params,
    606 )

File [C:\atlite\atlite\convert.py:174](file:///C:/atlite/atlite/convert.py:174), in convert_and_aggregate(cutout, convert_func, matrix, index, layout, shapes, shapes_crs, per_unit, return_capacity, capacity_factor, show_progress, dask_kwargs, **convert_kwds)
    171 if index is None:
    172     index = pd.RangeIndex(matrix.shape[0])
--> 174 results = aggregate_matrix(da, matrix=matrix, index=index)
    176 if per_unit or return_capacity:
    177     caps = matrix.sum(-1)

File [C:\atlite\atlite\aggregate.py:19](file:///C:/atlite/atlite/aggregate.py:19), in aggregate_matrix(da, matrix, index)
     17 if isinstance(da.data, dask.array.core.Array):
     18     da = da.stack(spatial=("y", "x"))
---> 19     return xr.apply_ufunc(
     20         lambda da: da * matrix.T,
     21         da,
     22         input_core_dims=[["spatial"]],
     23         output_core_dims=[[index.name]],
     24         dask="parallelized",
     25         output_dtypes=[da.dtype],
     26         dask_gufunc_kwargs=dict(output_sizes={index.name: index.size}),
     27     ).assign_coords(**{index.name: index})
     28 else:
     29     da = da.stack(spatial=("y", "x")).transpose("spatial", "time")

File [c:\Users\trtopi\AppData\Local\miniconda3\envs\atlitetoolbox\Lib\site-packages\xarray\core\computation.py:1249](file:///C:/Users/trtopi/AppData/Local/miniconda3/envs/atlitetoolbox/Lib/site-packages/xarray/core/computation.py:1249), in apply_ufunc(func, input_core_dims, output_core_dims, exclude_dims, vectorize, join, dataset_join, dataset_fill_value, keep_attrs, kwargs, dask, output_dtypes, output_sizes, meta, dask_gufunc_kwargs, on_missing_core_dim, *args)
   1247 # feed DataArray apply_variable_ufunc through apply_dataarray_vfunc
   1248 elif any(isinstance(a, DataArray) for a in args):
-> 1249     return apply_dataarray_vfunc(
   1250         variables_vfunc,
   1251         *args,
   1252         signature=signature,
   1253         join=join,
   1254         exclude_dims=exclude_dims,
   1255         keep_attrs=keep_attrs,
   1256     )
   1257 # feed Variables directly through apply_variable_ufunc
   1258 elif any(isinstance(a, Variable) for a in args):

File [c:\Users\trtopi\AppData\Local\miniconda3\envs\atlitetoolbox\Lib\site-packages\xarray\core\computation.py:308](file:///C:/Users/trtopi/AppData/Local/miniconda3/envs/atlitetoolbox/Lib/site-packages/xarray/core/computation.py:308), in apply_dataarray_vfunc(func, signature, join, exclude_dims, keep_attrs, *args)
    303 result_coords, result_indexes = build_output_coords_and_indexes(
    304     args, signature, exclude_dims, combine_attrs=keep_attrs
    305 )
    307 data_vars = [getattr(a, "variable", a) for a in args]
--> 308 result_var = func(*data_vars)
    310 out: tuple[DataArray, ...] | DataArray
    311 if signature.num_outputs > 1:

File [c:\Users\trtopi\AppData\Local\miniconda3\envs\atlitetoolbox\Lib\site-packages\xarray\core\computation.py:754](file:///C:/Users/trtopi/AppData/Local/miniconda3/envs/atlitetoolbox/Lib/site-packages/xarray/core/computation.py:754), in apply_variable_ufunc(func, signature, exclude_dims, dask, output_dtypes, vectorize, keep_attrs, dask_gufunc_kwargs, *args)
    752             for axis, dim in enumerate(core_dims, start=-len(core_dims)):
    753                 if len(data.chunks[axis]) != 1:
--> 754                     raise ValueError(
    755                         f"dimension {dim} on {n}th function argument to "
    756                         "apply_ufunc with dask='parallelized' consists of "
    757                         "multiple chunks, but is also a core dimension. To "
    758                         "fix, either rechunk into a single array chunk along "
    759                         f"this dimension, i.e., ``.chunk(dict({dim}=-1))``, or "
    760                         "pass ``allow_rechunk=True`` in ``dask_gufunc_kwargs`` "
    761                         "but beware that this may significantly increase memory usage."
    762                     )
    763     dask_gufunc_kwargs["allow_rechunk"] = True
    765 output_sizes = dask_gufunc_kwargs.pop("output_sizes", {})

ValueError: dimension spatial on 0th function argument to apply_ufunc with dask='parallelized' consists of multiple chunks, but is also a core dimension. To fix, either rechunk into a single array chunk along this dimension, i.e., ``.chunk(dict(spatial=-1))``, or pass ``allow_rechunk=True`` in ``dask_gufunc_kwargs`` but beware that this may significantly increase memory usage.

When used for a cutout with 3+ months of data, cutout.temperature fails as well:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
[c:\atlite\examples\building_stock_weather_aggregation.ipynb](file:///C:/atlite/examples/building_stock_weather_aggregation.ipynb) Cell 18 line 6
      [3](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=2) layouts = {"Weighted": layout, "Uniform": uniform, "Jyväskylä": jyvaskyla}
      [4](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=3) for name, lout in layouts.items():
      [5](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=4)     temperature_dict[name] = (
----> [6](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=5)         cutout.temperature(layout=lout.fillna(0.0)).squeeze().to_series()
      [7](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=6)     )
      [9](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=8) # Calculate radiation data into dictionaries
     [10](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=9) irr_total = {}

File [C:\atlite\atlite\convert.py:216](file:///C:/atlite/atlite/convert.py:216), in temperature(cutout, **params)
    215 def temperature(cutout, **params):
--> 216     return cutout.convert_and_aggregate(convert_func=convert_temperature, **params)

File [C:\atlite\atlite\convert.py:174](file:///C:/atlite/atlite/convert.py:174), in convert_and_aggregate(cutout, convert_func, matrix, index, layout, shapes, shapes_crs, per_unit, return_capacity, capacity_factor, show_progress, dask_kwargs, **convert_kwds)
    171 if index is None:
    172     index = pd.RangeIndex(matrix.shape[0])
--> 174 results = aggregate_matrix(da, matrix=matrix, index=index)
    176 if per_unit or return_capacity:
    177     caps = matrix.sum(-1)

File [C:\atlite\atlite\aggregate.py:19](file:///C:/atlite/atlite/aggregate.py:19), in aggregate_matrix(da, matrix, index)
     17 if isinstance(da.data, dask.array.core.Array):
     18     da = da.stack(spatial=("y", "x"))
---> 19     return xr.apply_ufunc(
     20         lambda da: da * matrix.T,
     21         da,
     22         input_core_dims=[["spatial"]],
     23         output_core_dims=[[index.name]],
     24         dask="parallelized",
     25         output_dtypes=[da.dtype],
     26         dask_gufunc_kwargs=dict(output_sizes={index.name: index.size}),
     27     ).assign_coords(**{index.name: index})
     28 else:
     29     da = da.stack(spatial=("y", "x")).transpose("spatial", "time")

File [c:\Users\trtopi\AppData\Local\miniconda3\envs\atlitetoolbox\Lib\site-packages\xarray\core\computation.py:1249](file:///C:/Users/trtopi/AppData/Local/miniconda3/envs/atlitetoolbox/Lib/site-packages/xarray/core/computation.py:1249), in apply_ufunc(func, input_core_dims, output_core_dims, exclude_dims, vectorize, join, dataset_join, dataset_fill_value, keep_attrs, kwargs, dask, output_dtypes, output_sizes, meta, dask_gufunc_kwargs, on_missing_core_dim, *args)
   1247 # feed DataArray apply_variable_ufunc through apply_dataarray_vfunc
   1248 elif any(isinstance(a, DataArray) for a in args):
-> 1249     return apply_dataarray_vfunc(
   1250         variables_vfunc,
   1251         *args,
   1252         signature=signature,
   1253         join=join,
   1254         exclude_dims=exclude_dims,
   1255         keep_attrs=keep_attrs,
   1256     )
   1257 # feed Variables directly through apply_variable_ufunc
   1258 elif any(isinstance(a, Variable) for a in args):

File [c:\Users\trtopi\AppData\Local\miniconda3\envs\atlitetoolbox\Lib\site-packages\xarray\core\computation.py:308](file:///C:/Users/trtopi/AppData/Local/miniconda3/envs/atlitetoolbox/Lib/site-packages/xarray/core/computation.py:308), in apply_dataarray_vfunc(func, signature, join, exclude_dims, keep_attrs, *args)
    303 result_coords, result_indexes = build_output_coords_and_indexes(
    304     args, signature, exclude_dims, combine_attrs=keep_attrs
    305 )
    307 data_vars = [getattr(a, "variable", a) for a in args]
--> 308 result_var = func(*data_vars)
    310 out: tuple[DataArray, ...] | DataArray
    311 if signature.num_outputs > 1:

File [c:\Users\trtopi\AppData\Local\miniconda3\envs\atlitetoolbox\Lib\site-packages\xarray\core\computation.py:754](file:///C:/Users/trtopi/AppData/Local/miniconda3/envs/atlitetoolbox/Lib/site-packages/xarray/core/computation.py:754), in apply_variable_ufunc(func, signature, exclude_dims, dask, output_dtypes, vectorize, keep_attrs, dask_gufunc_kwargs, *args)
    752             for axis, dim in enumerate(core_dims, start=-len(core_dims)):
    753                 if len(data.chunks[axis]) != 1:
--> 754                     raise ValueError(
    755                         f"dimension {dim} on {n}th function argument to "
    756                         "apply_ufunc with dask='parallelized' consists of "
    757                         "multiple chunks, but is also a core dimension. To "
    758                         "fix, either rechunk into a single array chunk along "
    759                         f"this dimension, i.e., ``.chunk(dict({dim}=-1))``, or "
    760                         "pass ``allow_rechunk=True`` in ``dask_gufunc_kwargs`` "
    761                         "but beware that this may significantly increase memory usage."
    762                     )
    763     dask_gufunc_kwargs["allow_rechunk"] = True
    765 output_sizes = dask_gufunc_kwargs.pop("output_sizes", {})

ValueError: dimension spatial on 0th function argument to apply_ufunc with dask='parallelized' consists of multiple chunks, but is also a core dimension. To fix, either rechunk into a single array chunk along this dimension, i.e., ``.chunk(dict(spatial=-1))``, or pass ``allow_rechunk=True`` in ``dask_gufunc_kwargs`` but beware that this may significantly increase memory usage.

Your Environment

• The atlite version used: Latest master 764d206
• How you installed atlite (conda, pip or github): pip install -e <locally_cloned_repository>
• Operating System: Windows 11
• My environment: Python 3.11.6 Conda environment with atlite, rioxarray, ipykernel, and matplotlib for running the examples\building_stock_weather_data_aggregation.ipynb

# packages in environment at C:\Users\trtopi\AppData\Local\miniconda3\envs\atlite:
#
# Name                    Version                   Build  Channel
affine                    2.4.0                    pypi_0    pypi
asttokens                 2.4.0              pyhd8ed1ab_0    conda-forge
atlite                    0.2.12.dev44+g764d206.d20231017          pypi_0    pypi
attrs                     23.1.0                   pypi_0    pypi
backcall                  0.2.0              pyh9f0ad1d_0    conda-forge
backports                 1.0                pyhd8ed1ab_3    conda-forge
backports.functools_lru_cache 1.6.5              pyhd8ed1ab_0    conda-forge
bottleneck                1.3.7                    pypi_0    pypi
bzip2                     1.0.8                h8ffe710_4    conda-forge
ca-certificates           2023.7.22            h56e8100_0    conda-forge
cdsapi                    0.6.1                    pypi_0    pypi
certifi                   2023.7.22                pypi_0    pypi
cftime                    1.6.2                    pypi_0    pypi
charset-normalizer        3.3.0                    pypi_0    pypi
click                     8.1.7                    pypi_0    pypi
click-plugins             1.1.1                    pypi_0    pypi
cligj                     0.7.2                    pypi_0    pypi
cloudpickle               3.0.0                    pypi_0    pypi
colorama                  0.4.6              pyhd8ed1ab_0    conda-forge
comm                      0.1.4              pyhd8ed1ab_0    conda-forge
contourpy                 1.1.1                    pypi_0    pypi
cycler                    0.12.1                   pypi_0    pypi
dask                      2023.10.0                pypi_0    pypi
debugpy                   1.8.0           py311h12c1d0e_1    conda-forge
decorator                 5.1.1              pyhd8ed1ab_0    conda-forge
exceptiongroup            1.1.3              pyhd8ed1ab_0    conda-forge
executing                 1.2.0              pyhd8ed1ab_0    conda-forge
fiona                     1.9.5                    pypi_0    pypi
fonttools                 4.43.1                   pypi_0    pypi
fsspec                    2023.9.2                 pypi_0    pypi
geopandas                 0.14.0                   pypi_0    pypi
idna                      3.4                      pypi_0    pypi
importlib-metadata        6.8.0              pyha770c72_0    conda-forge
importlib_metadata        6.8.0                hd8ed1ab_0    conda-forge
ipykernel                 6.25.2             pyh60829e3_0    conda-forge
ipython                   8.16.1             pyh5737063_0    conda-forge
jedi                      0.19.1             pyhd8ed1ab_0    conda-forge
jupyter_client            8.4.0              pyhd8ed1ab_0    conda-forge
jupyter_core              5.4.0           py311h1ea47a8_0    conda-forge
kiwisolver                1.4.5                    pypi_0    pypi
libexpat                  2.5.0                h63175ca_1    conda-forge
libffi                    3.4.2                h8ffe710_5    conda-forge
libsodium                 1.0.18               h8d14728_1    conda-forge
libsqlite                 3.43.2               hcfcfb64_0    conda-forge
libzlib                   1.2.13               hcfcfb64_5    conda-forge
locket                    1.0.0                    pypi_0    pypi
matplotlib                3.8.0                    pypi_0    pypi
matplotlib-inline         0.1.6              pyhd8ed1ab_0    conda-forge
nest-asyncio              1.5.8              pyhd8ed1ab_0    conda-forge
netcdf4                   1.6.4                    pypi_0    pypi
numexpr                   2.8.7                    pypi_0    pypi
numpy                     1.26.1                   pypi_0    pypi
openssl                   3.1.3                hcfcfb64_0    conda-forge
packaging                 23.2               pyhd8ed1ab_0    conda-forge
pandas                    2.1.1                    pypi_0    pypi
parso                     0.8.3              pyhd8ed1ab_0    conda-forge
partd                     1.4.1                    pypi_0    pypi
pickleshare               0.7.5                   py_1003    conda-forge
pillow                    10.1.0                   pypi_0    pypi
pip                       23.3               pyhd8ed1ab_0    conda-forge
platformdirs              3.11.0             pyhd8ed1ab_0    conda-forge
progressbar2              4.2.0                    pypi_0    pypi
prompt-toolkit            3.0.39             pyha770c72_0    conda-forge
prompt_toolkit            3.0.39               hd8ed1ab_0    conda-forge
psutil                    5.9.5           py311ha68e1ae_1    conda-forge
pure_eval                 0.2.2              pyhd8ed1ab_0    conda-forge
pygments                  2.16.1             pyhd8ed1ab_0    conda-forge
pyparsing                 3.1.1                    pypi_0    pypi
pyproj                    3.6.1                    pypi_0    pypi
python                    3.11.6          h2628c8c_0_cpython    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python-utils              3.8.1                    pypi_0    pypi
python_abi                3.11                    4_cp311    conda-forge
pytz                      2023.3.post1             pypi_0    pypi
pywin32                   306             py311h12c1d0e_2    conda-forge
pyyaml                    6.0.1                    pypi_0    pypi
pyzmq                     25.1.1          py311h7b3f143_1    conda-forge
rasterio                  1.3.8.post2              pypi_0    pypi
requests                  2.31.0                   pypi_0    pypi
rioxarray                 0.15.0                   pypi_0    pypi
scipy                     1.11.3                   pypi_0    pypi
setuptools                68.2.2             pyhd8ed1ab_0    conda-forge
shapely                   2.0.2                    pypi_0    pypi
six                       1.16.0             pyh6c4a22f_0    conda-forge
snuggs                    1.4.7                    pypi_0    pypi
stack_data                0.6.2              pyhd8ed1ab_0    conda-forge
tk                        8.6.13               hcfcfb64_0    conda-forge
toolz                     0.12.0                   pypi_0    pypi
tornado                   6.3.3           py311ha68e1ae_1    conda-forge
tqdm                      4.66.1                   pypi_0    pypi
traitlets                 5.11.2             pyhd8ed1ab_0    conda-forge
typing-extensions         4.8.0                hd8ed1ab_0    conda-forge
typing_extensions         4.8.0              pyha770c72_0    conda-forge
tzdata                    2023.3                   pypi_0    pypi
ucrt                      10.0.22621.0         h57928b3_0    conda-forge
urllib3                   2.0.6                    pypi_0    pypi
vc                        14.3                h64f974e_17    conda-forge
vc14_runtime              14.36.32532         hdcecf7f_17    conda-forge
vs2015_runtime            14.36.32532         h05e6639_17    conda-forge
wcwidth                   0.2.8              pyhd8ed1ab_0    conda-forge
wheel                     0.41.2             pyhd8ed1ab_0    conda-forge
xarray                    2023.9.0                 pypi_0    pypi
xz                        5.2.6                h8d14728_0    conda-forge
zeromq                    4.3.4                h0e60522_1    conda-forge
zipp                      3.17.0             pyhd8ed1ab_0    conda-forge

[Tasqu]

I briefly tested the historic-comparison-germany.ipynb example as well just to be safe, and it seems to suffer from the same problem. At least on my machine.

[FabianHofmann]

Thanks for reporting @Tasqu, I will look into it.

[FabianHofmann]

@Tasqu I cannot reproduce the bug. All works fine for me, having xarray 2023.09.0 and dask 2023.10.0 installed. I am seeing that you have mixed installations from pypi and conda-forge. I would recommend to create a new environment with conda/mamba and retest.

[Tasqu]

@FabianHofmann Ok, good to know. I'll have to try different environment setups, hopefully that solves this. I'll try to get back to you later today.

[Tasqu]

@FabianHofmann meanwhile, I can't seem to get rid of this error 😅 I tried creating new conda environments using solely conda to install the packages for both Python 3.11 and 3.9, but I still keep running in (almost) the same errors. Here's the latest conda environment I used for testing this out:

# packages in environment at [c:\Users\trtopi\AppData\Local\miniconda3\envs\atlite](file:///C:/Users/trtopi/AppData/Local/miniconda3/envs/atlite):
#
# Name                    Version                   Build  Channel
affine                    2.4.0              pyhd8ed1ab_0    conda-forge
asttokens                 2.4.0              pyhd8ed1ab_0    conda-forge
atlite                    0.2.11             pyhd8ed1ab_0    conda-forge
attrs                     23.1.0             pyh71513ae_1    conda-forge
aws-c-auth                0.7.4                hd571b1d_2    conda-forge
aws-c-cal                 0.6.2                hd5965a7_2    conda-forge
aws-c-common              0.9.3                hcfcfb64_0    conda-forge
aws-c-compression         0.2.17               hd5965a7_3    conda-forge
aws-c-event-stream        0.3.2                hef061cf_2    conda-forge
aws-c-http                0.7.13               h064cb6d_2    conda-forge
aws-c-io                  0.13.33              ha16e049_0    conda-forge
aws-c-mqtt                0.9.7                h37bb463_0    conda-forge
aws-c-s3                  0.3.18               h263813f_0    conda-forge
aws-c-sdkutils            0.1.12               hd5965a7_2    conda-forge
aws-checksums             0.1.17               hd5965a7_2    conda-forge
aws-crt-cpp               0.24.3               hd745c5a_1    conda-forge
aws-sdk-cpp               1.11.156             hc1d519e_5    conda-forge
backcall                  0.2.0              pyh9f0ad1d_0    conda-forge
backports                 1.0                pyhd8ed1ab_3    conda-forge
backports.functools_lru_cache 1.6.5              pyhd8ed1ab_0    conda-forge
blas                      1.0                         mkl  
blosc                     1.21.5               hdccc3a2_0    conda-forge
bokeh                     3.3.0              pyhd8ed1ab_0    conda-forge
bottleneck                1.3.7            py39hd88c2e4_1    conda-forge
branca                    0.6.0              pyhd8ed1ab_0    conda-forge
brotli                    1.1.0                hcfcfb64_1    conda-forge
brotli-bin                1.1.0                hcfcfb64_1    conda-forge
brotli-python             1.1.0            py39h99910a6_1    conda-forge
bzip2                     1.0.8                h8ffe710_4    conda-forge
c-ares                    1.20.1               hcfcfb64_0    conda-forge
ca-certificates           2023.7.22            h56e8100_0    conda-forge
cairo                     1.18.0               h1fef639_0    conda-forge
cdsapi                    0.6.1              pyhd8ed1ab_0    conda-forge
certifi                   2023.7.22          pyhd8ed1ab_0    conda-forge
cfitsio                   4.3.0                h9b0cee5_0    conda-forge
cftime                    1.6.2            py39hd88c2e4_2    conda-forge
charset-normalizer        3.3.0              pyhd8ed1ab_0    conda-forge
click                     8.1.7           win_pyh7428d3b_0    conda-forge
click-plugins             1.1.1                      py_0    conda-forge
cligj                     0.7.2              pyhd8ed1ab_1    conda-forge
cloudpickle               3.0.0              pyhd8ed1ab_0    conda-forge
colorama                  0.4.6              pyhd8ed1ab_0    conda-forge
comm                      0.1.4              pyhd8ed1ab_0    conda-forge
contourpy                 1.1.1            py39h1f6ef14_1    conda-forge
cycler                    0.12.1             pyhd8ed1ab_0    conda-forge
cytoolz                   0.12.2           py39ha55989b_1    conda-forge
dask                      2023.10.0          pyhd8ed1ab_0    conda-forge
dask-core                 2023.10.0          pyhd8ed1ab_0    conda-forge
debugpy                   1.8.0            py39h99910a6_1    conda-forge
decorator                 5.1.1              pyhd8ed1ab_0    conda-forge
distributed               2023.10.0          pyhd8ed1ab_0    conda-forge
exceptiongroup            1.1.3              pyhd8ed1ab_0    conda-forge
executing                 1.2.0              pyhd8ed1ab_0    conda-forge
expat                     2.5.0                h63175ca_1    conda-forge
fiona                     1.9.5            py39h48fce2b_0    conda-forge
folium                    0.14.0             pyhd8ed1ab_0    conda-forge
font-ttf-dejavu-sans-mono 2.37                 hab24e00_0    conda-forge
font-ttf-inconsolata      3.000                h77eed37_0    conda-forge
font-ttf-source-code-pro  2.038                h77eed37_0    conda-forge
font-ttf-ubuntu           0.83                 hab24e00_0    conda-forge
fontconfig                2.14.2               hbde0cde_0    conda-forge
fonts-conda-ecosystem     1                             0    conda-forge
fonts-conda-forge         1                             0    conda-forge
fonttools                 4.43.1           py39ha55989b_0    conda-forge
freetype                  2.12.1               hdaf720e_2    conda-forge
freexl                    2.0.0                h8276f4a_0    conda-forge
fsspec                    2023.9.2           pyh1a96a4e_0    conda-forge
gdal                      3.7.2            py39hbe60bc6_7    conda-forge
geopandas                 0.14.0             pyhd8ed1ab_1    conda-forge
geopandas-base            0.14.0             pyha770c72_1    conda-forge
geos                      3.12.0               h1537add_0    conda-forge
geotiff                   1.7.1               hcf4a93f_14    conda-forge
gettext                   0.21.1               h5728263_0    conda-forge
glib                      2.78.0               h12be248_0    conda-forge
glib-tools                2.78.0               h12be248_0    conda-forge
gst-plugins-base          1.22.6               h001b923_2    conda-forge
gstreamer                 1.22.6               hb4038d2_2    conda-forge
hdf4                      4.2.15               h5557f11_7    conda-forge
hdf5                      1.14.2          nompi_h73e8ff5_100    conda-forge
icu                       73.2                 h63175ca_0    conda-forge
idna                      3.4                pyhd8ed1ab_0    conda-forge
importlib-metadata        6.8.0              pyha770c72_0    conda-forge
importlib-resources       6.1.0              pyhd8ed1ab_0    conda-forge
importlib_metadata        6.8.0                hd8ed1ab_0    conda-forge
importlib_resources       6.1.0              pyhd8ed1ab_0    conda-forge
iniconfig                 2.0.0              pyhd8ed1ab_0    conda-forge
intel-openmp              2023.2.0         h57928b3_50496    conda-forge
ipykernel                 6.25.2             pyh60829e3_0    conda-forge
ipython                   8.16.1             pyh5737063_0    conda-forge
jedi                      0.19.1             pyhd8ed1ab_0    conda-forge
jinja2                    3.1.2              pyhd8ed1ab_1    conda-forge
joblib                    1.3.2              pyhd8ed1ab_0    conda-forge
jupyter_client            8.4.0              pyhd8ed1ab_0    conda-forge
jupyter_core              5.4.0            py39hcbf5309_0    conda-forge
kealib                    1.5.2                ha10e780_1    conda-forge
kiwisolver                1.4.5            py39h1f6ef14_1    conda-forge
krb5                      1.21.2               heb0366b_0    conda-forge
lcms2                     2.15                 h67d730c_3    conda-forge
lerc                      4.0.0                h63175ca_0    conda-forge
libabseil                 20230802.1      cxx17_h63175ca_0    conda-forge
libaec                    1.1.2                h63175ca_1    conda-forge
libarchive                3.7.2                h6f8411a_0    conda-forge
libarrow                  13.0.0          h64a251c_10_cpu    conda-forge
libblas                   3.9.0              19_win64_mkl    conda-forge
libboost-headers          1.82.0               h57928b3_6    conda-forge
libbrotlicommon           1.1.0                hcfcfb64_1    conda-forge
libbrotlidec              1.1.0                hcfcfb64_1    conda-forge
libbrotlienc              1.1.0                hcfcfb64_1    conda-forge
libcblas                  3.9.0              19_win64_mkl    conda-forge
libclang                  15.0.7          default_h77d9078_3    conda-forge
libclang13                15.0.7          default_h77d9078_3    conda-forge
libcrc32c                 1.1.2                h0e60522_0    conda-forge
libcurl                   8.4.0                hd5e4a3a_0    conda-forge
libdeflate                1.19                 hcfcfb64_0    conda-forge
libevent                  2.1.12               h3671451_1    conda-forge
libexpat                  2.5.0                h63175ca_1    conda-forge
libffi                    3.4.2                h8ffe710_5    conda-forge
libgdal                   3.7.2                h3217549_7    conda-forge
libglib                   2.78.0               he8f3873_0    conda-forge
libgoogle-cloud           2.12.0               ha74b051_3    conda-forge
libgrpc                   1.58.1               h2a9c87f_2    conda-forge
libhwloc                  2.9.3           default_haede6df_1009    conda-forge
libiconv                  1.17                 h8ffe710_0    conda-forge
libjpeg-turbo             3.0.0                hcfcfb64_1    conda-forge
libkml                    1.3.0             haf3e7a6_1018    conda-forge
liblapack                 3.9.0              19_win64_mkl    conda-forge
libnetcdf                 4.9.2           nompi_h8284064_112    conda-forge
libogg                    1.3.4                h8ffe710_1    conda-forge
libpng                    1.6.39               h19919ed_0    conda-forge
libpq                     16.0                 h43585b0_1    conda-forge
libprotobuf               4.24.3               hb8276f3_1    conda-forge
libre2-11                 2023.06.02           h8c5ae5e_0    conda-forge
librttopo                 1.1.0               h92c5fdb_14    conda-forge
libsodium                 1.0.18               h8d14728_1    conda-forge
libspatialindex           1.9.3                h39d44d4_4    conda-forge
libspatialite             5.1.0                hbf340bc_0    conda-forge
libsqlite                 3.43.2               hcfcfb64_0    conda-forge
libssh2                   1.11.0               h7dfc565_0    conda-forge
libthrift                 0.19.0               ha2b3283_1    conda-forge
libtiff                   4.6.0                h6e2ebb7_2    conda-forge
libutf8proc               2.8.0                h82a8f57_0    conda-forge
libvorbis                 1.3.7                h0e60522_0    conda-forge
libwebp-base              1.3.2                hcfcfb64_0    conda-forge
libxcb                    1.15                 hcd874cb_0    conda-forge
libxml2                   2.11.5               hc3477c8_1    conda-forge
libzip                    1.10.1               h1d365fa_3    conda-forge
libzlib                   1.2.13               hcfcfb64_5    conda-forge
locket                    1.0.0              pyhd8ed1ab_0    conda-forge
lz4                       4.3.2            py39hf617134_1    conda-forge
lz4-c                     1.9.4                hcfcfb64_0    conda-forge
lzo                       2.10              he774522_1000    conda-forge
m2w64-gcc-libgfortran     5.3.0                         6    conda-forge
m2w64-gcc-libs            5.3.0                         7    conda-forge
m2w64-gcc-libs-core       5.3.0                         7    conda-forge
m2w64-gmp                 6.1.0                         2    conda-forge
m2w64-libwinpthread-git   5.0.0.4634.697f757               2    conda-forge
mapclassify               2.6.1              pyhd8ed1ab_0    conda-forge
markupsafe                2.1.3            py39ha55989b_1    conda-forge
matplotlib                3.8.0            py39hcbf5309_2    conda-forge
matplotlib-base           3.8.0            py39hf19769e_2    conda-forge
matplotlib-inline         0.1.6              pyhd8ed1ab_0    conda-forge
minizip                   4.0.1                h5bed578_5    conda-forge
mkl                       2023.2.0         h6a75c08_50496    conda-forge
mkl-service               2.4.0            py39h2bbff1b_1  
msgpack-python            1.0.6            py39h1f6ef14_0    conda-forge
msys2-conda-epoch         20160418                      1    conda-forge
munch                     4.0.0              pyhd8ed1ab_0    conda-forge
munkres                   1.1.4              pyh9f0ad1d_0    conda-forge
nest-asyncio              1.5.8              pyhd8ed1ab_0    conda-forge
netcdf4                   1.6.4           nompi_py39h9a3bb69_103    conda-forge
networkx                  3.1                pyhd8ed1ab_0    conda-forge
numexpr                   2.8.7            py39h2cd9be0_0  
numpy                     1.26.0           py39hddb5d58_0    conda-forge
openjpeg                  2.5.0                h3d672ee_3    conda-forge
openssl                   3.1.3                hcfcfb64_0    conda-forge
orc                       1.9.0                hd95f75e_3    conda-forge
packaging                 23.2               pyhd8ed1ab_0    conda-forge
pandas                    2.1.1            py39h32e6231_1    conda-forge
parso                     0.8.3              pyhd8ed1ab_0    conda-forge
partd                     1.4.1              pyhd8ed1ab_0    conda-forge
pcre2                     10.40                h17e33f8_0    conda-forge
pickleshare               0.7.5                   py_1003    conda-forge
pillow                    10.1.0           py39h368b509_0    conda-forge
pip                       23.3               pyhd8ed1ab_0    conda-forge
pixman                    0.42.2               h63175ca_0    conda-forge
platformdirs              3.11.0             pyhd8ed1ab_0    conda-forge
pluggy                    1.3.0              pyhd8ed1ab_0    conda-forge
ply                       3.11                       py_1    conda-forge
poppler                   23.10.0              hc2f3c52_0    conda-forge
poppler-data              0.4.12               hd8ed1ab_0    conda-forge
postgresql                16.0                 hc80876b_1    conda-forge
progressbar2              4.2.0              pyhd8ed1ab_0    conda-forge
proj                      9.3.0                he13c7e8_1    conda-forge
prompt-toolkit            3.0.39             pyha770c72_0    conda-forge
prompt_toolkit            3.0.39               hd8ed1ab_0    conda-forge
psutil                    5.9.5            py39ha55989b_1    conda-forge
pthread-stubs             0.4               hcd874cb_1001    conda-forge
pthreads-win32            2.9.1                hfa6e2cd_3    conda-forge
pure_eval                 0.2.2              pyhd8ed1ab_0    conda-forge
pyarrow                   13.0.0          py39h0a09291_10_cpu    conda-forge
pygments                  2.16.1             pyhd8ed1ab_0    conda-forge
pyparsing                 3.1.1              pyhd8ed1ab_0    conda-forge
pyproj                    3.6.1            py39h9e31499_2    conda-forge
pyqt                      5.15.9           py39hb77abff_5    conda-forge
pyqt5-sip                 12.12.2          py39h99910a6_5    conda-forge
pysocks                   1.7.1              pyh0701188_6    conda-forge
pytest                    7.4.2              pyhd8ed1ab_0    conda-forge
python                    3.9.18          h4de0772_0_cpython    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python-tzdata             2023.3             pyhd8ed1ab_0    conda-forge
python-utils              3.8.1              pyhd8ed1ab_0    conda-forge
python_abi                3.9                      4_cp39    conda-forge
pytz                      2023.3.post1       pyhd8ed1ab_0    conda-forge
pywin32                   306              py39h99910a6_2    conda-forge
pyyaml                    6.0.1            py39ha55989b_1    conda-forge
pyzmq                     25.1.1           py39hea35a22_1    conda-forge
qt-main                   5.15.8              h9e85ed6_17    conda-forge
rasterio                  1.3.8            py39hdc4e632_4    conda-forge
re2                       2023.06.02           hcbb65ff_0    conda-forge
requests                  2.31.0             pyhd8ed1ab_0    conda-forge
rioxarray                 0.15.0             pyhd8ed1ab_0    conda-forge
rtree                     1.1.0            py39h09fdee3_0    conda-forge
scikit-learn              1.3.1            py39h7c199eb_1    conda-forge
scipy                     1.11.3           py39hddb5d58_1    conda-forge
setuptools                68.2.2             pyhd8ed1ab_0    conda-forge
shapely                   2.0.2            py39hacc7841_0    conda-forge
sip                       6.7.12           py39h99910a6_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
snappy                    1.1.10               hfb803bf_0    conda-forge
snuggs                    1.4.7                      py_0    conda-forge
sortedcontainers          2.4.0              pyhd8ed1ab_0    conda-forge
sqlite                    3.43.2               hcfcfb64_0    conda-forge
stack_data                0.6.2              pyhd8ed1ab_0    conda-forge
tbb                       2021.10.0            h91493d7_1    conda-forge
tblib                     2.0.0              pyhd8ed1ab_0    conda-forge
threadpoolctl             3.2.0              pyha21a80b_0    conda-forge
tiledb                    2.16.3               h1ffc264_3    conda-forge
tk                        8.6.13               hcfcfb64_0    conda-forge
toml                      0.10.2             pyhd8ed1ab_0    conda-forge
tomli                     2.0.1              pyhd8ed1ab_0    conda-forge
toolz                     0.12.0             pyhd8ed1ab_0    conda-forge
tornado                   6.3.3            py39ha55989b_1    conda-forge
tqdm                      4.66.1             pyhd8ed1ab_0    conda-forge
traitlets                 5.11.2             pyhd8ed1ab_0    conda-forge
typing-extensions         4.8.0                hd8ed1ab_0    conda-forge
typing_extensions         4.8.0              pyha770c72_0    conda-forge
tzdata                    2023c                h71feb2d_0    conda-forge
ucrt                      10.0.22621.0         h57928b3_0    conda-forge
unicodedata2              15.1.0           py39ha55989b_0    conda-forge
uriparser                 0.9.7                h1537add_1    conda-forge
urllib3                   2.0.6              pyhd8ed1ab_0    conda-forge
vc                        14.3                h64f974e_17    conda-forge
vc14_runtime              14.36.32532         hdcecf7f_17    conda-forge
vs2015_runtime            14.36.32532         h05e6639_17    conda-forge
wcwidth                   0.2.8              pyhd8ed1ab_0    conda-forge
wheel                     0.41.2             pyhd8ed1ab_0    conda-forge
win_inet_pton             1.1.0              pyhd8ed1ab_6    conda-forge
xarray                    2023.9.0           pyhd8ed1ab_0    conda-forge
xerces-c                  3.2.4                h63175ca_3    conda-forge
xorg-libxau               1.0.11               hcd874cb_0    conda-forge
xorg-libxdmcp             1.1.3                hcd874cb_0    conda-forge
xyzservices               2023.10.0          pyhd8ed1ab_0    conda-forge
xz                        5.2.6                h8d14728_0    conda-forge
yaml                      0.2.5                h8ffe710_2    conda-forge
zeromq                    4.3.4                h0e60522_1    conda-forge
zict                      3.0.0              pyhd8ed1ab_0    conda-forge
zipp                      3.17.0             pyhd8ed1ab_0    conda-forge
zlib                      1.2.13               hcfcfb64_5    conda-forge
zstd                      1.5.5                h12be248_0    conda-forge

When trying to run building_stock_weather_aggregation.ipynb with the following cutout (spans 2 months)

cutout = atlite.Cutout(
    path="finland-2011-0607.nc",
    module="era5",
    x=slice(x1 - 0.2, x2 + 0.2),
    y=slice(y1 - 0.2, y2 + 0.2),
    time=slice("2011-06","2011-07"),
)

I get a new error:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
[c:\atlite\examples\building_stock_weather_aggregation.ipynb](file:///C:/atlite/examples/building_stock_weather_aggregation.ipynb) Cell 18 line 1
     [13](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=12) dirs = {"north": 0.0, "east": 90.0, "south": 180.0, "west": 270.0}
     [14](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=13) for name, lout in layouts.items():
---> [15](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=14)     irr_total[name] = {
     [16](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=15)         d: cutout.irradiation(
     [17](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=16)             orientation={"slope": 90.0, "azimuth": az}, layout=lout.fillna(0.0)
     [18](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=17)         )
     [19](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=18)         .squeeze()
     [20](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=19)         .to_series()
     [21](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=20)         for d, az in dirs.items()
     [22](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=21)     }
     [23](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=22)     irr_direct[name] = {
     [24](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=23)         d: cutout.irradiation(
     [25](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=24)             orientation={"slope": 90.0, "azimuth": az},
   (...)
     [31](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=30)         for d, az in dirs.items()
     [32](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=31)     }
     [33](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=32)     irr_diffuse[name] = {
     [34](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=33)         d: cutout.irradiation(
     [35](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=34)             orientation={"slope": 90.0, "azimuth": az},
   (...)
     [41](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=40)         for d, az in dirs.items()
     [42](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=41)     }

[c:\atlite\examples\building_stock_weather_aggregation.ipynb](file:///C:/atlite/examples/building_stock_weather_aggregation.ipynb) Cell 18 line 1
     [13](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=12) dirs = {"north": 0.0, "east": 90.0, "south": 180.0, "west": 270.0}
     [14](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=13) for name, lout in layouts.items():
     [15](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=14)     irr_total[name] = {
---> [16](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=15)         d: cutout.irradiation(
     [17](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=16)             orientation={"slope": 90.0, "azimuth": az}, layout=lout.fillna(0.0)
     [18](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=17)         )
     [19](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=18)         .squeeze()
     [20](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=19)         .to_series()
     [21](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=20)         for d, az in dirs.items()
     [22](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=21)     }
     [23](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=22)     irr_direct[name] = {
     [24](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=23)         d: cutout.irradiation(
     [25](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=24)             orientation={"slope": 90.0, "azimuth": az},
   (...)
     [31](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=30)         for d, az in dirs.items()
     [32](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=31)     }
     [33](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=32)     irr_diffuse[name] = {
     [34](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=33)         d: cutout.irradiation(
     [35](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=34)             orientation={"slope": 90.0, "azimuth": az},
   (...)
     [41](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=40)         for d, az in dirs.items()
     [42](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=41)     }

AttributeError: 'Cutout' object has no attribute 'irradiation'

As if the irradiation attribute is missing? However, when trying to run a 3-month cutout:

cutout = atlite.Cutout(
    path="finland-2011-0608.nc",
    module="era5",
    x=slice(x1 - 0.2, x2 + 0.2),
    y=slice(y1 - 0.2, y2 + 0.2),
    time=slice("2011-06","2011-08"),
)

I still get the same error as before:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
[c:\atlite\examples\building_stock_weather_aggregation.ipynb](file:///C:/atlite/examples/building_stock_weather_aggregation.ipynb) Cell 18 line 6
      [3](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=2) layouts = {"Weighted": layout, "Uniform": uniform, "Jyväskylä": jyvaskyla}
      [4](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=3) for name, lout in layouts.items():
      [5](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=4)     temperature_dict[name] = (
----> [6](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=5)         cutout.temperature(layout=lout.fillna(0.0)).squeeze().to_series()
      [7](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=6)     )
      [9](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=8) # Calculate radiation data into dictionaries
     [10](vscode-notebook-cell:/c%3A/atlite/examples/building_stock_weather_aggregation.ipynb#X23sZmlsZQ%3D%3D?line=9) irr_total = {}

File [c:\Users\trtopi\AppData\Local\miniconda3\envs\atlite\lib\site-packages\atlite\convert.py:216](file:///C:/Users/trtopi/AppData/Local/miniconda3/envs/atlite/lib/site-packages/atlite/convert.py:216), in temperature(cutout, **params)
    215 def temperature(cutout, **params):
--> 216     return cutout.convert_and_aggregate(convert_func=convert_temperature, **params)

File [c:\Users\trtopi\AppData\Local\miniconda3\envs\atlite\lib\site-packages\atlite\convert.py:174](file:///C:/Users/trtopi/AppData/Local/miniconda3/envs/atlite/lib/site-packages/atlite/convert.py:174), in convert_and_aggregate(cutout, convert_func, matrix, index, layout, shapes, shapes_crs, per_unit, return_capacity, capacity_factor, show_progress, dask_kwargs, **convert_kwds)
    171 if index is None:
    172     index = pd.RangeIndex(matrix.shape[0])
--> 174 results = aggregate_matrix(da, matrix=matrix, index=index)
    176 if per_unit or return_capacity:
    177     caps = matrix.sum(-1)

File [c:\Users\trtopi\AppData\Local\miniconda3\envs\atlite\lib\site-packages\atlite\aggregate.py:19](file:///C:/Users/trtopi/AppData/Local/miniconda3/envs/atlite/lib/site-packages/atlite/aggregate.py:19), in aggregate_matrix(da, matrix, index)
     17 if isinstance(da.data, dask.array.core.Array):
     18     da = da.stack(spatial=("y", "x"))
---> 19     return xr.apply_ufunc(
     20         lambda da: da * matrix.T,
     21         da,
     22         input_core_dims=[["spatial"]],
     23         output_core_dims=[[index.name]],
     24         dask="parallelized",
     25         output_dtypes=[da.dtype],
     26         dask_gufunc_kwargs=dict(output_sizes={index.name: index.size}),
     27     ).assign_coords(**{index.name: index})
     28 else:
     29     da = da.stack(spatial=("y", "x")).transpose("spatial", "time")

File [c:\Users\trtopi\AppData\Local\miniconda3\envs\atlite\lib\site-packages\xarray\core\computation.py:1249](file:///C:/Users/trtopi/AppData/Local/miniconda3/envs/atlite/lib/site-packages/xarray/core/computation.py:1249), in apply_ufunc(func, input_core_dims, output_core_dims, exclude_dims, vectorize, join, dataset_join, dataset_fill_value, keep_attrs, kwargs, dask, output_dtypes, output_sizes, meta, dask_gufunc_kwargs, on_missing_core_dim, *args)
   1247 # feed DataArray apply_variable_ufunc through apply_dataarray_vfunc
   1248 elif any(isinstance(a, DataArray) for a in args):
-> 1249     return apply_dataarray_vfunc(
   1250         variables_vfunc,
   1251         *args,
   1252         signature=signature,
   1253         join=join,
   1254         exclude_dims=exclude_dims,
   1255         keep_attrs=keep_attrs,
   1256     )
   1257 # feed Variables directly through apply_variable_ufunc
   1258 elif any(isinstance(a, Variable) for a in args):

File [c:\Users\trtopi\AppData\Local\miniconda3\envs\atlite\lib\site-packages\xarray\core\computation.py:308](file:///C:/Users/trtopi/AppData/Local/miniconda3/envs/atlite/lib/site-packages/xarray/core/computation.py:308), in apply_dataarray_vfunc(func, signature, join, exclude_dims, keep_attrs, *args)
    303 result_coords, result_indexes = build_output_coords_and_indexes(
    304     args, signature, exclude_dims, combine_attrs=keep_attrs
    305 )
    307 data_vars = [getattr(a, "variable", a) for a in args]
--> 308 result_var = func(*data_vars)
    310 out: tuple[DataArray, ...] | DataArray
    311 if signature.num_outputs > 1:

File [c:\Users\trtopi\AppData\Local\miniconda3\envs\atlite\lib\site-packages\xarray\core\computation.py:754](file:///C:/Users/trtopi/AppData/Local/miniconda3/envs/atlite/lib/site-packages/xarray/core/computation.py:754), in apply_variable_ufunc(func, signature, exclude_dims, dask, output_dtypes, vectorize, keep_attrs, dask_gufunc_kwargs, *args)
    752             for axis, dim in enumerate(core_dims, start=-len(core_dims)):
    753                 if len(data.chunks[axis]) != 1:
--> 754                     raise ValueError(
    755                         f"dimension {dim} on {n}th function argument to "
    756                         "apply_ufunc with dask='parallelized' consists of "
    757                         "multiple chunks, but is also a core dimension. To "
    758                         "fix, either rechunk into a single array chunk along "
    759                         f"this dimension, i.e., ``.chunk(dict({dim}=-1))``, or "
    760                         "pass ``allow_rechunk=True`` in ``dask_gufunc_kwargs`` "
    761                         "but beware that this may significantly increase memory usage."
    762                     )
    763     dask_gufunc_kwargs["allow_rechunk"] = True
    765 output_sizes = dask_gufunc_kwargs.pop("output_sizes", {})

ValueError: dimension spatial on 0th function argument to apply_ufunc with dask='parallelized' consists of multiple chunks, but is also a core dimension. To fix, either rechunk into a single array chunk along this dimension, i.e., ``.chunk(dict(spatial=-1))``, or pass ``allow_rechunk=True`` in ``dask_gufunc_kwargs`` but beware that this may significantly increase memory usage.

I deleted and re-downloaded the weather data .nc files just in case, but that didn't seem to make a difference. (Not sure why it would, but hey, worth a shot?)

I'll probably keep trying to mess with my environments to see if this goes away somehow...

[davide-f]

To add on this issue, I've encountered this issue as well, which may occur with large cutouts.
This is linked to pypsa-meets-earth/pypsa-earth#900 as well.

[Tasqu]

@davide-f thanks for the tip! Seems like downgrading to xarray=2023.8.0 also avoids this error, but as far as I can tell it results in similar performance than passing allow_rechunk=True in dask_gufunc_kwargs (as suggested by the error messages) for the aggregate_matrix function.

Both solutions allow me to bypass this issue, but the performance seems worse than I remember it being last time I was dealing with country-wide year-long weather data aggregation. It seems to me that every time the aggregate_matrix function is called, it does some re-chunking of the underlying data, losing time with repeated cutout.temperature(...) etc. calls.

[FabianHofmann]

@Tasqu could you do us a favour and test the implementation in #326?

[Tasqu]

@FabianHofmann sure thing, I'll give it a go.

[Tasqu]

@FabianHofmann yeah, #326 seems to work. As far as I can tell, it's working the same way as downgrading to xarray=2023.8.0 or passing the allow_rechunk=True. The performance for repeated cutout.temperature etc. still isn't what I remember it being for long timespans, but that might be related to #252.

Or maybe I'm just imagining things 😅 it has been several months since I last did full-year weather data aggregation over entire countries.

[coroa]

Completely different thing you can try with those dask heavy internals is to run with the dask.distributed TaskScheduler instead of the default dask local, which has slightly different characteristics, which makes it a bit more robust.

Concretely, this means in your script or notebook that you add:

from dask.distributed import Client
client = Client()  # set up local cluster on your laptop

as described at
https://distributed.dask.org/en/stable/quickstart.html#setup-dask-distributed-the-easy-way

before you open your cutout object.

[Tasqu]

@coroa Thanks for the tip! In my use case, this cuts the weather aggregation time roughly in half. Preparing the layout seems to take a little longer, but still seems like a worthwhile gain overall.

distributed throws a ton of garbage collection and worker unresponsiveness related warnings, but still worth investigating further 😅