ENRAM meeting Norrkoping

#Meeting minutes ENRAM workshop meeting at SMHI, Norrköping, 12-15 July 2016

Participants:

• AD = Adriaan Dokter
• AH = Anders Henja
• GH = Günther Haase
• CN = Cecilia Nilsson
• LV = Liesbeth Verlinden

These minutes provide on overview of discussion points, findings and actions taken during the meeting. In the meeting we combined short discussion sessions and actions to directly solve identified problems.

Main aim of the meeting was to discuss specifications and immediately implement a production pipeline that:

1. extracts bird profiles for the entire OPERA network
2. makes these bird products available on a server
3. stores the bird products in a long-term repository for further analysis.

We reached the milestone of set up and testing this production pipeline. Pending final permission from OPERA to run the production pipeline, it is currently running only for the data from Belgian and the Netherlands - which are the countries that already publish bird profile data via http://www.flysafe-birdtam.eu.

###1. Setting up a production pipeline of bird profiles

1. Bird algorithm (vol2bird) was installed both in the Baltrad system (nordic countries), as well to act on the OPERA data stream (these are two separate data streams at SMHI)
2. A Python wrapper for vol2bird and python scripts were installed on the Baltrad server to generate bird profiles for newly incoming radar data (AH, AD)
3. An FTP service was set up to make bird data available to the outside world (AH). We decided to have the data available there for two days, after that they are overwritten.
4. We decided on a prototype service to grab the data from the BALTRAD FTP server. AD has set up a repository at beehub, and installed a script at a server at the university of Amsterdam to grab these data, and store them on beehub. The pipeline was tested for 3 days for a total of 100 radars as a proof of concept. The data stream has then be stopped, pending official OPERA permission to use the data. The pipeline will be continued to be tested for data from the Netherlands and Belgium, for which the permission is already obtained.
5. The BALTRAD toolbox has been set up at the Swedish National Supercomputer Centre (NSC). At the NSC also the historical data of OPERA are archived. Using the python scripts (see item 1.2) archived data can be processed much more easily.
6. A critical technical issue in the Baltrad system was identified, which causes OPERA data to be overwritten after entering SMHI. As a result, currently only for 3 countries complete data from OPERA has been properly stored.
7. We discussed definition of nominal time in OPERA. There is unfortunately no convention in OPERA to define a nominal time. Most countries have this the start of the volume scan, except UK data, where it is a few minutes in the future. Currently the nominal time stored in the root /what attribute is the time at which the polar volume was created at Baltrad, i.e. the time of receiving the data, not the time of collecting the data. As a solution, AD added an option to vol2bird that reads start and end time attributes of all scans contained in the polar volume, and stores their extrema in the bird profile output.
8. AD implemented dialiasing of radial winds by the method of Haase et al., as available in the Baltrad toolbox. We found that many countries still provide only aliased data (e.g. Denmark, Finland), therefore proper dealiasing is a critical feature needed for large-scale analyses.
9. ACTION Evaluate the quality of dealiased VRAD in the case of bird migration (AD/LV)
10. ACTION As an intermediate solution, GH will attempt before 1 Aug to apply a fix that stores the incoming data, to secure data for the upcoming migration season.
11. ACTION A ticket has been made for the Swedish radar network to fix this issue, as it is more general than ENRAM alone (e.g., also the Numerical Weather Prediction community)
12. ACTION store nominal time instead of time of receiving the data (GH)

###2. Evaluation bird profile quality

1. Two 2x 1 week of data was selected (1 week spring and 1 week autumn) as a test (CN/GH/LV). Only for three countries (SE, NL, FR) both radial velocity and reflectivity data was available, for which bird profiles could be extracted
2. 2 weeks test case data was processed and visualised (LV/CN). We found that for many countries (meta)data was missing, and that radial velocity data was regularly folded (e.g. Denmark, Finland). A list has been compiled that will be forwarded to the OPERA user group, such that our findings can be discussed during the next OPERA meeting (Sep 2016).

###3. Kullaberg validation campaign / other papers

1. Cecilia provided an update of analyses so far. Correspondence between weather radar and tracking radar looks good, but weather radar suffers from a few short but intense rain contaminations.
2. Decided on rules to filter out residual rain in Kullaberg data. We applied an additional post-processing filtering step for the weather radar data, in which altitude profiles are removed when at least 80% of the altitude layers in the 0-2 km band contains rain. This improved the nightly accumulations of bird density as compared to the Birdscan and tracking radars.
3. ACTION LV will look at weather radar speed biases with altitude and density. Results will be shared with CN so that they can become part of the paper.
4. ACTION CN will share tracks and balloon winds with LV
5. ACTION Günther will check whether HIRLAM wind profiles for the Kullaberg site are readily available, and if so make them available for the project.
6. AD/CN brainstormed on swift paper, and set a deadline of 1 jan 2017 for a finished draft.

###4. Progress license OPERA - ENRAM

• ACTION AD has sent email to Elena Saltikoff with results of this meeting, and requested an update on the licensing process. Licenses are decided by the general assembly of EUMETNET, i.e. a decision level above OPERA. AD will discuss with Elena whether a license should include an option to give permissions for profiles, volumes, and composites separately.

###5. North Sea / western Europe dataset We defined the critical actions needed to compile a large-scale migration case, being a main output / goal of the ENRAM COST action.

Highest priority

• signing a data use license with OPERA
• Fix problem overwriting of DBZ/VRAD files (see actions under 1 above, SWERAD)

Second priority

1. ACTION AD, GH will report to OPERA on findings of the 2 weeks of test data, and data quality issues. This document can also be sent to the OPERA user group. Deadline is 1 Aug, for it to be ready before next OPERA meeting In this note we should list:
   • Push OPERA to store Nyquist velocity
   • Push OPERA to store wavelengths
   • Move to 2 weeks test case data to full autumn 2016, due to lack of data
   • Data quality/filtering
2. ACTION Make configuration options settable per radar (AD)
3. ACTION Add de-aliased VRAD to polar volumes generated in Baltrad by default (SVERAD)
4. Evaluate de-aliased wind data for bird migration cases.

The current biggest open technical issue is the overwriting of the DBZ/VRAD files within the OPERA data stream within SMHI. BALTRAD does not have the overwrite issue, so in principle it would be possible to get workable data from Nordic countries (Norway and Finland) via BALTRAD. We will have to ask these two countries explicitly if we can access their data via BALTRAD and not via OPERA, as the two are different data streams.

We decided we will update vol2bird on BALTRAD and start to run it already there. Once we got the permissions, we can access the data then. At BALTRAD data is only stored for limited (unknown) time, though this time period will be increased to make sure we will have data for the autumn migration of 2016.

GH will do his best to fix the overwrite issue before the start of autumn migration, at the earliest some time in September, but no hard guarantees on the timeline are possible. It depends on when people (Anders and colleagues at SMHI) will be back from holiday and their workload ob other higher priority projects.

End of August, beginning of September we will evaluate the progress of achieving the license. In case it seems this could take quite some time, we will approach the countries around the North sea and ask them individual permission to use data of a limited time interval, e.g. 2 months.

Holidays:

• AH until 2nd week August
• GH 1/8 - 4/9
• LV 4/8 - 18/8
• CN not
• AD starting in US 29/7

Appendix: evaluation 2 week test data

Under the data quality working group of OPERA, we got permission to evaluate two weeks of OPERA data (all countries) to evaluate the suitability of the data for extracting biological signals. Below a country-by-country summary of our findings during the meeting, focussing on technical issues with the data formats.

###Summary

• All countries but Sweden, Netherlands, France, Finland, Slovenia, and Croatia had reflectivity and/or radial velocity missing.
• Slovenia, Croatia and Finland had consistently the Nyquist velocity too low, which had also the same value for each scan.
• Most of the countries do not store the radar wavelength in the polar volume.
• Vol2bird could only produce a dataset for France, Sweden, and Netherlands, indicated in bold in the list below.

###Country-by-country evaluation

####FRANCE all scans:

Warning: radar wavelength not stored in polar volume. Using user-defined value of 5.300000 cm ...

####SPAIN for all scans:

Warning: radial velocity missing, dropping scan ...

####CZECK REPUBLIC for all scans:

Warning: requested reflectivity factor 'DBZH' missing, searching for alternatives ... 
Warning: reflectivity factor missing, dropping scan 0 ...
Warning: radar wavelength not stored in polar volume. Using user-defined value of 5.300000 cm ...

in autumn 2015 a lot also:

Warning: radial velocity missing, dropping scan ...

####CROATIA for all scans:

Warning: radial velocity Nyquist interval (1.0 m/s) too low, dropping scan ...

####GERMANY for all scans: Warning: requested reflectivity factor 'DBZH' missing, searching for alternatives ... Warning: reflectivity factor missing, dropping scan ... Warning: radial velocity missing, dropping scan ... Warning: radar wavelength not stored in polar volume. Using user-defined value of 5.300000 cm ...

####BELGIUM all scans:

Warning: radial velocity missing, dropping scan ...
Warning: requested reflectivity factor 'DBZH' missing, searching for alternatives ... 
Warning: reflectivity factor missing, dropping scan 0 ...

(more in regular way) some (autumn 2015, though I didn't go through all spring data):

Warning: radar wavelength not stored in polar volume. Using user-defined value of 5.300000 cm ...

####FINLAND for all scans:

Warning: using radar wavelength stored for scan 1 (5.330000 cm) for all scans ...
Warning: radial velocity Nyquist interval (7.7 m/s or 7.6 m/s) too low, dropping scan 0 ...

####POLAND for all scans:

Warning: requested reflectivity factor 'DBZH' missing, searching for alternatives ... 
Warning: reflectivity factor missing, dropping scan 9 ...
Warning: radial velocity missing, dropping scan 4 ...

####NETHERLANDS For each first scan:

Warning: radar wavelength not stored in polar volume. Using user-defined value of 5.300000 cm ...
Warning: radial velocity Nyquist interval (3.3 m/s) too low, dropping scan 0 ...

(always scan 0)

####NORWAY for all scans:

Warning: radar wavelength not stored in polar volume. Using user-defined value of 5.300000 cm ...
Warning: requested reflectivity factor 'DBZH' missing, searching for alternatives ... Warning: reflectivity factor missing, dropping scan 0 ...
Warning: radial velocity missing, dropping scan 4 ...

####UK for all scans:

Warning: requested reflectivity factor 'DBZH' missing, searching for alternatives ... 
Warning: reflectivity factor missing, dropping scan ...
Warning: using radar wavelength stored for scan 1 (5.346740 cm) for all scans 
Warning: radial velocity missing, dropping scan  ...

####SWITZERLAND (only spring 2016) for all scans:

Warning: radar wavelength not stored in polar volume. Using user-defined value of 5.300000 cm ...
Warning: radial velocity missing, dropping scan ...

####SLOVENIA for all scans:

Warning: radial velocity Nyquist interval (1.0 m/s) too low, dropping scan ...

####SWEDEN for all scans:

Warning: using radar wavelength stored for scan 1 (5.350000 cm) for all scans ...

or:

Warning: radar wavelength not stored in polar volume. Using user-defined value of 5.300000 cm ...