How to use the BiasAdjustCXX to perform bias correction on the monthly scale temperature and precipitation data of some models in CMIP6.

[liming489]

I would like to use the BiasAdjustCXX tool you proposed to perform bias correction on the monthly scale temperature and precipitation data of some models in CMIP6. Do you have any good suggestions? Because the data examples you provided are all for daily scale data, Thank you.

[btschwertfeger]

Hello @liming489, BiasAdjustCXX is not designed to apply scaling-based techniques on data that does not provide 365 days per year (see here).

There are several ways to approach this:

• An example of bias correction using scaling methods involves preparing your input datasets by manually scaling the data. Imagine you have 30 years of monthly temperature or precipitation values. You could modify the data so that each day of the month has a consistent value. For instance, if January of the first year has an average value of 5°C, you can set every day of that January to this value. Once the data is prepared, you can apply the scaling methods as described in the documentation.

• Another option is to use the --no-group flag, that disables the adjustment of scaling-based techniques on 31-day intervals. This takes the whole input data as base for applying the mean adjustments. You can leverage these for example to split each of your input data set (reference, historical modeled, and scenario period) and split it into 12 individual data sets, each including only the data for one month over the whole time period, e.g. one data set containing all monthly data of all Januaries. Then you apply BiasAdjustCXX on each of those slices, meaning after that you will have 12 bias corrected data sets, each containing the data for one month. Then you have to merge these datasets into a single one and you're done. An example of this can be found at examples/example_all_methods.run.sh

• I'm not 100% sure but have you tried applying the distribution-based methods Quantile Mapping and Quantile Delta Mapping on your data? Did that worked for you? Scaling-based techniques, as implemented and documented are based on research that assumes the existence of daily data and works best with a lot of entries due to long-term monthly (and even interval-based) means. Imagine you have 30 years of daily data i.e. 10950 days => 912,5 days for each month over the whole time span in contrast to having 30 years of monthly data, meaning only 30 values for each month... guess which data results in more reliable results. Distribution based techniques on the other hand does not rely on monthly values, but as the name says modifies the data based on their distributional properties. These should suit better for monthly data.

• Did you see my other bias correction tool python-cmethods? It is implemented in Python and is way more flexible in terms of parameters, input data, shapes and more. For those who know Python at least a bit, this might be the better option, as it not only provides a command-line interface but also enables accessing its API in order to apply custom bias corrections.

I'm a bit out of climate research and maintain these projects in my spare time, so if you have any suggestions or could provide generic code examples that might help to reproduce faulty, or help understanding expected behavior, I'd much welcome it.

[liming489]

Dear Professor Schwertfeger,
Apologies for troubling you again. I have tried the second and third methods you provided, but the bias correction results were not satisfactory. Using quantile delta mapping as an example, the images below show the reference data, control data, scenario data, and the bias correction results. From the images, we can observe that the temperature range in the correction results is clearly incorrect. Do you have any good suggestions?The attachment contains the raw data and the code is as follows:
**The code **

#!/bin/bash

# enter the project directory, to access the input data more easily
cd /mnt/h/CMIP6_BC/bias_correction

# create an output directory (if not exist)
mkdir -p output 

BiasAdjustCXX \
    --ref CMIP6_data_resample_seldate_setmiss_sel/ERA5_monthly_mean_temperature_1950_2014_remap.nc                 \
    --contr CMIP6_data_resample_seldate_setmiss_sel/tas_Amon_ACCESS-ESM1-5_historical_r1i1p1f1_gn_195001-201412_remap_seldate.nc                   \
    --scen CMIP6_data_resample_seldate_setmiss_sel/tas_Amon_ACCESS-ESM1-5_ssp245_r1i1p1f1_gn_201501-210012_remap.nc                  \
    --output output/tas_Amon_ACCESS-ESM1-5_historical_r1i1p1f1_gn_195001-201412_remap_seldate_biascorrection.nc   \
    --variable "tas"                                 \
    --kind "+"                                       \
    --method quantile_delta_mapping   \
    --no-group                         \
    --processes 10        

### [raw_data.zip](https://github.com/user-attachments/files/17037376/raw_data.zip)

[btschwertfeger]

Hello @liming489, indeed that looks not got. I still don't open any files shared for security reasons.

I just tried it myself on my own data using the data sets provided by this repository and transformed the units into Kelvin, but I don't encounter strange values like yours.

I have no clue what could cause this behavior, but I see the following options:

• Provide a minimal reproducible example of the data as code so that I can just copy code and generate the faulty data sets by myself (e.g. https://github.com/btschwertfeger/python-cmethods/blob/v2.3.0/examples/examples.ipynb):

  import numpy as np
  import xarray as xr
  import random
  import matplotlib.pyplot as plt
  
  np.random.seed(0)
  random.seed(0)
  
  historical_time = xr.cftime_range("1971-01-01", "2000-12-31", freq="D", calendar="noleap")
  future_time = xr.cftime_range("2001-01-01", "2030-12-31", freq="D", calendar="noleap")
  
  get_hist_temp_for_lat = lambda val: 273.15 - (val * np.cos(2 * np.pi * historical_time.dayofyear 
  / 365) + 2 * np.random.random_sample((historical_time.size,)) + 273.15 + .1 * (historical_time - 
  historical_time[0]).days / 365)
  get_rand = lambda: np.random.rand() if np.random.rand() > .5 else  -np.random.rand()
  
  latitudes = np.arange(23,27,1)
  some_data = [get_hist_temp_for_lat(val) for val in latitudes]
  data = np.array([some_data, np.array(some_data)+1])
  
  obsh = xr.DataArray(
      data,
      dims=("lon", "lat", "time"),
      coords={"time": historical_time, "lat": latitudes, "lon": [0,1]},
      attrs=attrs
  ).transpose("time","lat","lon").to_dataset(name="tas")#.to_netcdf("observations.nc", )
  
  simh = xr.DataArray(
      data-2,
      dims=("lon", "lat", "time"),
      coords={"time": historical_time, "lat": latitudes, "lon": [0,1]},
      attrs=attrs
  ).transpose("time","lat","lon").to_dataset(name="tas")#.to_netcdf("control.nc", )
  
  simp = xr.DataArray(
      data-1,
      dims=("lon", "lat", "time"),
      coords={"time": future_time, "lat": latitudes, "lon": [0,1]},
      attrs=attrs
  ).transpose("time","lat","lon").to_dataset(name="tas")#.to_netcdf("scenario.nc", )
  
  obsp = xr.DataArray(
      data+1,
      dims=("lon", "lat", "time"),
      coords={"time": historical_time, "lat": latitudes, "lon": [0,1]},
      attrs=attrs
  ).transpose("time","lat","lon").to_dataset(name="tas")#.to_netcdf("observations_future.nc")

  ... then reproduce the faulty behavior and share the code here (you can also just use a single time series, for simplicity.)

• Strip down your data, just try to use a single time series and apply the bias correction on that. Analyze the results, check the documentation of the method and try to understand what could be wrong with the data.

• Try using python-cmethods as it provides an easy API to apply bias corrections in code, which helps tracking down the error. Additionally by applying python-cmethods successfully, you could verify that there must be a problem with BiasAdjustCXX. Here is a good example on how to use python-cmethods: https://github.com/btschwertfeger/python-cmethods/blob/v2.3.0/examples/examples.ipynb.

BTW: The --no-group flag only applies to scaling-based methods (delta method, linar and variance scaling).

[liming489]

Many thanks to Professor @btschwertfeger for the kind and helpful response. Although I still haven't pinpointed the exact issue, when I used the following CDO command to convert the temperature units of both ERA5 and CMIP6 from Kelvin to Celsius, and then applied the BiasAdjustCXX tool for bias correction, everything worked fine. It's quite strange.

The CDO code is as follows:

#!/bin/bash

## Goal: batch convert  Kelvin to Celsius

# enter the project directory, to access the input data more easily
cd /mnt/h/CMIP6_BC/bias_correction

# create an output directory (if not exist)
mkdir -p converted 

for file in raw/*.nc; do
    output="converted/$(basename "$file" .nc)_celsius.nc"
    cdo -b F32 subc,273.15 "$file" "$output"
done

[btschwertfeger]

These are great news!

So you confirm that Kelvin is a weak point of BiasAdjustCXX? If so, I'll document that in order to avoid confusion in the future.

[liming489]

Theoretically, I believe that a change in units should not affect the results of bias correction. Therefore, I attempted to use the experimental data provided with the BiasAdjustCXX tool, converting it into monthly scale data, and then applied the BiasAdjustCXX tool for bias correction without encountering any issues. Hence, I still suspect that there might be some problem with the format of my original data.