Personal Weather Station (PWS) quality control using python.

---

How to cite:

https://doi.org/10.5281/zenodo.4501919

Reference paper:

Bárdossy, A., Seidel, J., and El Hachem, A.: The use of personal weather station observations to improve precipitation estimation and interpolation, Hydrol. Earth Syst. Sci., 25, 583–601, https://doi.org/10.5194/hess-25-583-2021, 2021.

El Hachem, A., Seidel, J., O'Hara, T., Villalobos Herrera, R., Overeem, A., Uijlenhoet, R., Bárdossy, A., and de Vos, L.: Technical note: Overview and comparison of three quality control algorithms for rainfall data from personal weather stations, Hydrol. Earth Syst. Sci. Discuss. [preprint], https://doi.org/10.5194/hess-2023-195, in review, 2023.

https://www.researchgate.net/profile/Abbas-El-Hachem

---

Main Procedure

Flowchart from raw PWS data to filtered data for interpolation

The three main codes corresponding to the IBF, Bias correction and EBF are available in the python_code folder

** Jupyter notebook Indicator_corr_Ams

Demonstration of applying the filters to the PWS data set of Amsterdam (check the Opensene folder for the data)

https://hess.copernicus.org/preprints/hess-2023-195/

---

Indicator based Filter IBF

Corresponsing code _02_pws_indicator_correlation_IBF.py

Required Input

1. Hdf5 data file containing the PWS station data, the corresponding timestamps and their corresponding coordinates, in a metric coordinate system
2. Hdf5 data file containing the primary network station data, the corresponding timestamps and their corresponding coordinates, in a metric coordinate system (same as PWS)

Output

1. A dataframe containing mainly the correlation values between each PWS and the corresponding neighboring station data and the correlation between the neighboring primary network stations .
2. The final result is obtained by keeping all the PWS where the correlation between PWS and primary network is greater or equal to the correlation between the primary network stations.

Results: indicator filter applied to the test_data

Note: the present code is slighty different than the one in the original paper. A similar code will be uploaded soon.

---

Bias correction

Corresponsing code _02_pws_bias_correction_BC.py

Required Input

1. Hdf5 data file containing the filtered PWS station data, the corresponding timestamps and their corresponding coordinates, in a metric coordinate system
2. Hdf5 data file containing the primary network station data, the corresponding timestamps and their corresponding coordinates, in a metric coordinate system (same as PWS)

Output

1. A dataframe for each PWS with the new data, a 'complete' new timeseries, used later on (for example in the interpolation)

Results: example of corrected data of one PWS

---

Event based filter (EBF)

Corresponsing code _04_event_based_filter_EBF.py

Required Input

1. Hdf5 data file containing the filtered and bias corrected PWS station data, the corresponding timestamps and their corresponding coordinates, in a metric coordinate system
2. Hdf5 data file containing the primary network station data, the corresponding timestamps and their corresponding coordinates, in a metric coordinate system (same as PWS)

Output

1. A dataframe containing for every event (or timestamp) the PWS that should be flagged and not used for the interpolation of the corresponding event or timestep

Results PWS stations that do not fit in the spatial neighboring are flagged

Credit:

The precipitation data was downloaded from the German Weather Service (DWD) open data server which can be found under the following link: https://opendata.dwd.de/climate_environment/CDC/

The PWS data were downloaded using the Netatmo API: https://dev.netatmo.com/

Note:

In the bias correction and event based filter an Ordinary Kriging implementation is used. This has not been yet uploaded but could be easily subtituted by the PyKrige code (10.5281/zenodo.3738604).