-
Notifications
You must be signed in to change notification settings - Fork 19
/
CHANGES
1170 lines (801 loc) · 43.9 KB
/
CHANGES
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
1.24 2024-11-11
~~~~~~~~~~~~~~~
* Fixed appending non-float variables with different sub-dimensions when
using the Python harp.concatenate() function.
* Updates to product ingestions:
- ESACCI_GHG: fixed incorrect unit for CH4/CO variables
- GOME-2 L2: added support for O3MARS aerosol layer height and added
support for O3MARP V5
- S5P L2: added snow_ice_type/sea_ice_fraction ingestion for AAI/CO/CH4
- S5P L2 CH4: added 'corrected' option to ingest destriped columns and
fixed issue in NIR/SWIR selection of surface albedo
- S5P PAL L2: added support for SRON/SICOR HDO (L2__HDO__S) product
1.23 2024-07-30
~~~~~~~~~~~~~~~
* Fixed numpy 2.x compatibility issues in python interface.
* Added derivations to retrieve dry air mass/volume mixing ratio from H2O
mass/volume mixing ratio with regard to dry air.
* Fixed some issues with building the HARP R interface extension.
* Updates to product ingestions:
- S5P PAL L2 AOT: added support for cloud_fraction vs.
effective_cloud_fraction split
1.22 2024-06-12
~~~~~~~~~~~~~~~
* Updates to product ingestions:
- EARLINET: fixed memory issue when reading extinction data.
- EarthCARE: updates for new V11.2 format definition baseline
- GEOMS UVVIS.DOBSON: fixed issue with reading of int16 typed data.
- S5P PAL L2 SO2 COBRA: fixed cloud_fraction option behaviour and added
alternative qa value calculation for processor V02.00.00.
1.21 2024-03-15
~~~~~~~~~~~~~~~
* Fixed incorrect weight division for random uncertainties in spatial binning.
* Added check for unit equality when appending variables.
* Fixed CMake build for locating Python 3.12.
* Updates to product ingestions:
- EarthCARE: updates for new V11 format definition baseline
- ECMWF GRIB: added general support for new masterTablesVersion numbers
- S5P L2 CH4: added support for nir aerosol/cloud/surface property ingestion.
- S5P L2 CLOUD: added support for nir-band spatial grid data
- S5P PAL L2: added support for OClO
- S5P PAL L2 AOT: added support for V02.00.00 output format
- S5P PAL L2 BRO: fixed memory issue with reading total column and
uncertainty variables
- S5P PAL L2 TCWV: fixed wrong quantity for density variables
1.20.2 2023-11-13
~~~~~~~~~~~~~~~~~
* Fill values of floating point types are now explicitly set to NaN when
exporting to a netcdf or hdf5 file.
* Updates to product ingestions:
- EARLINET: fixed issue where no backscatter/extinction data was ingested.
1.20.1 2023-10-12
~~~~~~~~~~~~~~~~~
* Updates to product ingestions:
- IASI L1C: added missing angle variables
- S5P PAL L2 AER_OT: added missing wavelength variable
- S5P PAL L2 CHOCHO: fixed missing units for number density variables
1.20 2023-09-19
~~~~~~~~~~~~~~~
* Updates to product ingestions:
- GEOMS FTIR: added support for C2H3NO5/PAN and C2H4, added support for
template V3.
- GEOMS TCCON: added support for template V6.
- S5P L2 ALH: added support for revised surface_albedo variable and added
surface_albedo_uncertainty.
- S5P L2 O3: fixed custom V1 QA filter for NRTI products
- S5P PAL L2: fixed memory issue when reading aerosol_optical_depth_validity
of AER_OT product.
* Fixed missing documentation on weight factor handling in binning algorithm.
* Updated embedded expat library to version 2.5.0.
* Fixed issue where jpeg dll was missing from windows binary installer.
1.19 2023-07-24
~~~~~~~~~~~~~~~
* The python harp.import_product() function now allows HARP .pth files to be
provided as input.
* Reintroduced propagation of *_uncertainty_random variables when performing
spatial binning with area weighted averaging.
* Allowed regridding using a multi-dimensional source axis-variable as input.
e.g. convert from an 'altitude {latitude,longitude,vertical}' grid to
a fixed 'altitude {vertical}` grid.
* Added documentation for regridding/binning algorithms.
* Updates to product ingestions:
- ACE-FTS: fix ingestion of ASCII formatted products
- GOME-2 L2: added support for offline absorbing aerosol index (O3MARP)
- EARLINET: added support for new format (removed support for old format)
- EarthCARE: added support for AM__ACD_2B, AM__CTH_2B, BM__RAD_2B,
BMA_FLX_2B, MSI_AOT_2A, MSI_CM__2A, and MSI_COP_2A
- ECMWF GRIB: added support for new masterTablesVersion=30 data
- S5P PAL L2: added support for AER_OT, BRO, CHOCHO, SIF, SO2 COBRA, and
TCWV
1.18 2023-06-08
~~~~~~~~~~~~~~~
* Renamed all uses of 'water vapour' to 'water vapor'.
* Added =| bitfield operater to allow filtering for the case where only some
bits within the mask are set.
* The uncertainty propagation of *_uncertainty_random variables for the
temporal binning operation now always assumes no correlation. For spatial
binning the random component is removed for the area weighted average
approach and it is propagated using no correlation for the simple binning
based on center points.
* Added derivation for 'area {latitude,longitude}' and fixed area calculation
where South Pole was an edge point.
* Fixed reading of string arrays in HARP Python interface.
* Updates to product ingestions:
- ECMWF GRIB: added support for new masterTablesVersion=29 data
- S5P L2 CLOUD: fixed qa_value ingestion for CRB (now use qa_value_crb)
- S5P L2 SO2: added support for SO2 layer height
- OMI L3: added support for OMHCHOd, OMNO2d, and OMSO2e
1.17 2022-12-12
~~~~~~~~~~~~~~~
* Added additional conventions for variable names related to areosol/cloud
properties.
* Fixed issue with geographical area overlap calculation and improvided
performance of performing operations on geographical polygon data.
* Windows .msi installer now allows HARP component selection for core, Python,
IDL, MATLAB, and R.
* Updates to product ingestions:
- AEOLUS L2B: properly split datetime, datetime_start, and datetime_stop
- CCI L3 O3 LP: fixed wrong variable name
- EarthCARE: added additional product types and updated ACM_CAP_2B
- ECMWF GRIB: changed RWC/SWC/LWC/IWC variable names
- OSIRIS L2: fixed wrong variable names
- TES TL2ATMTN/TL2ATMTL: fixed wrong variable names
1.16 2022-10-31
~~~~~~~~~~~~~~~
* Improved algorithm to calculate center point from latitude/longitude bounds
polygon.
* Added derivation of time dependent from time indepent for datetime variables.
* Updates to product ingestions:
- GEOMS PANDORA: added systematic/random uncertainty for effective
temperature, renamed H2CO in harp variable names to HCHO.
- OCO: added support for ACOS/OCO-2/OCO-3 L2/L3 products.
- Ozone CCI L3 LP: fixed memory issue when reading
merged_ozone_concentration.
- S5P L2 HCHO: added amf=clear_sky ingestion option
1.15.1 2022-05-18
~~~~~~~~~~~~~~~~~
* Updates to product ingestions:
- CLOUDNET L2 classification: fixed fillvalue mapping of values to NaN.
- GEOMS PANDORA: fixes for template V003 support
- GEOMS FTIR: allow instrument lat/lon/alt variables to be one-dimensional
with replicated values.
- GEOMS UVVIS.BREWER: added random+systematic uncertainty variables.
- S5P L2 CO: fixed vertical ordering of new apriori profile.
1.15 2022-03-14
~~~~~~~~~~~~~~~
* Fixed issue where bin() operation was removing scalar enumeration variables.
* Updates to product ingestions:
- ECMWF GRIB: fixed parameter mapping for CAMS model data that uses
masterTablesVersion=27 data and added support for new
masterTablesVersion=28 data
- GEOMS UVVIS.BREWER: added support for template V001
- GEOMS PANDORA: added support for template V003
- S5P L2 AAI: added new 335/367 wavelength pair
- S5P L2 CO: added new apriori profile and support for new AVK unit
- S5P L2 O3 Profile: added surface_albedo and cloud_albedo
- S5P L2 HCHO/O3/SO2: added support for updated NRTI indication attribute.
1.14 2021-10-20
~~~~~~~~~~~~~~~
* Updated embedded udunits2 library to version 2.2.28.
* Improved CMake build system.
* Fixed issue when building against HDF5 1.12.
* Updates to product ingestions:
- AEOLUS L1B: fixed ingestion of Mie/measurement data; added sensor angles
- AEOLUS L2B: fixed ingestion of Mie data; added sensor angles
- CLOUDNET L2 classification: added support for new (netcdf4) format
- EARTHCARE: added very early support (limited version of ECA_ACM_CAP_2B)
- ECMWF GRIB: fixed unit of backscatter coefficient 1/(milli sr) -> 1/(m sr),
added support for new masterTablesVersion=27 data
- GEOMS UVVIS.DOAS: allow template V007 files to be stored in hdf5 format
- GHG CCI: added support for TROPOMI WFMD L2
- OMI L2: added sensor alt/lat/lon and surface altitude/pressure
- OMI L2 OMAERUV: fix ingestion issues
- QA4ECV HCHO: added amf=clear_sky ingestion option
- S5P L2 HCHO: renamed cloud_altitude variable to cloud_height
- S5P L2 NO2: added retrieval of O2-O2 cloud (O22CLD) data
- S5P L2 NO2+FRESCO: added scene_albedo and scene_pressure
- S5P L2 O3 Profile: fixed wrong concentration variables (vmr -> nd)
- S5P L2 SO2: updated SO2_type enumeration list
1.13 2021-03-12
~~~~~~~~~~~~~~~
* Added rebin() operation.
* Added harp_product_rebin_with_axis_bounds_variable() function.
* Added support for datetime_bounds variable.
* Fixed issue in =& operation (it returned true if any of the bits were 1
instead of all bits).
* Updates to product ingestions:
- GEOMS: renamed all site_name variables to location_name
- GEOMS FTIR H2CO: fixed issue in determining unit for H2O data (that would
fail the ingestion)
- S5P L2 CO: added co=corrected ingestion option
- SCIAMACHY L2: fixed averaging of cloud fraction for co-added pixels
1.12 2020-10-29
~~~~~~~~~~~~~~~
* The csv output of 'harpdump --dataset' can now be used as content of a .pth
file. This improves the performance of harpcollocate/harpmerge/etc. as they
no longer need to open and extract metadata from files themselves anymore.
* harpcollocate now stores differences as signed values in collocation result
file (instead of absolute differences).
* Added harp_dataset_prefilter(). This function is now also called within
harpmerge and harpcollocate to pre-filter the input dataset(s) for datetime
and collocate left/right filters that are at the start of the operations
string.
* Added format and history fields to harp_product_metadata.
* Added harp.import_product_metadata() to Python interface.
* harp_dataset_add_product() now ignores files/directories starting with '.'.
* Added uncertainty propagation to bin() and bin_spatial() operations
(using uncorrelated assumption for bin() and correlated for bin_spatial()).
* Added global option to determine whether uncertainty propagation should
use a correlated or uncorrelated approach. Currently only used for the bin()
operation.
* Added global option to create a collocation_datetime variable when performing
a collocate_left() or collocate_right() operation.
* Added wildcard pattern matching ('*'/'?') to keep() and exclude() operations.
* Added derivation to retrieve angstrom exponent from aerosol optical depth.
* Added derivation to retrieve tropospheric/stratospheric column avk from
total column avk and tropopause altitude.
* Fixed potential memory issues and improved overall code stability.
* Updates to product ingestions:
- S5P L2 CLOUD: added cloud_type and cloud_top_temperature
- S5P L2 O3: added qa_filter=custom ingestion option to set validity based
on official PRF recommendation for processor V1.x generated products
- MLS L2: fixed memory issue in reading of validity variable and updated
validity flag mapping to version 4.x-4.0 of the Data Quality document
1.11 2020-05-19
~~~~~~~~~~~~~~~
* Added derivations to retrieve column number density from column vmr.
* Fixed issue in bin_spatial() where the weightfactor could be too high when
oversampling the input data. Grid cells that intersected with the data
polygon could end up with a weightfactor >1.0 (but <2.0).
* Fixed issue where molar mass was incorrectly provided for the species with
the shortest matching name. e.g. the molar mass of CO was returned for CO2
and of NO for NO2. This impacts vmr <-> mmr conversions for BrO2, COF2, COS,
CO2, C2H2O2, H2O2, NOCl, NO2, N2O, N2O5, and all isotopologues (e.g. H2O_161).
* Removed irrelevant harp_str64() and harp_str64u() functions (HARP does not
support 64bit integers).
* Fixed issue where molar mass linked to variables with a 'surface',
'tropospheric', or 'stratospheric' prefix returned 0.
* Fixed error/crash in deriving dry air column vmr from dry air column mmr.
* Updates to product ingestions:
- GOME-2 L2: added support for SO2 columns at all plume heights
- OMI L2 CLDO2/CLDRR: added validity variable
1.10 2020-03-31
~~~~~~~~~~~~~~~
* Added support for -ar/--reduce-operations parameter in harpmerge and
for reduce_operations parameter in harp.import_product() Python function.
* Fixed incorrect "invalid variable" error on weight variables for angular
quantities when performing a harpmerge.
* Fixed issue where the udunits2 library could not be initialized if the
environment had a LC_NUMERIC locale that did not have '.' as decimal
separator.
* Fixed issue in name mangling of embedded expat library.
* Updates to product ingestions:
- GOMOS L1 Limb: fixed unit of wavelength_photon_radiance (and uncertainty)
and documented conversion algorithm
- S5P L2 O3 PR: support for IODD 12.0.0
- S5P L2 O3 TPR: removed support (future availability of product uncertain)
- GEOMS PANDORA: allow for trailing \0 in global attribute values
1.9.2 2019-12-01
~~~~~~~~~~~~~~~~
* Fixed issue where using the 'bin()' operation did not correctly create a
'<variable>_weight' variable from a 'weight' variable when it was missing.
* Added default units to documention on variable name conventions.
* Updates to product ingestions:
- Calipso L2: fixed unit of backscatter coefficient
- ERS GOME L2: fixed unit of cloud top height
- S5P L2 AER_LH: fixed unit of aerosol optical depth
- S5P L2 SO2: add scaled avk for 1km/7km/15km box profiles
1.9.1 2019-11-01
~~~~~~~~~~~~~~~~
* Fixed issue where results from deriving partial column densities from vmr and
pressure were a factor 10 too low.
* Fixed issue where operation following an index filter operation did not
get executed.
1.9 2019-10-21
~~~~~~~~~~~~~~
* Added operations to filter on dimension indices (e.g. 'index(vertical) < 5').
* Changed bin() and bin_spatial() operations to use/produce separate 'count'
and 'weight' variables. The counts/weights are used to allow the following
commands to now be equivalent:
harpmerge -a 'bin()' -ap 'bin()' a.nc b.nc merged.nc
harpmerge -ap 'bin()' a.nc b.nc merged.nc
As well as the following:
harpmerge -a 'bin_spatial(...)'
-ap 'squash(time, (latitude_bounds, longitude_bounds));bin()'
a.nc b.nc merged.nc
harpmerge -ap 'bin_spatial(...)' a.nc b.nc merged.nc
* Fixed issue with invalid bounding box optimization for spherical area
filter operations.
* Fixed missing error handling when using invalid operations in
harp.export_product() in Python.
* Fixed Python3 compatibility issue when exporting string data.
* Updates to ECMWF GRIB, IASI L1/L2, GOME-2 L1/L2, MIPAS L1, and S5P L2
product ingestions.
1.8 2019-07-31
~~~~~~~~~~~~~~
* Added R interface.
* Improved derivation for tropopause altitude/pressure (to not return NaN).
* Added derivations for gravity.
* Added derivations to convert between total air and dry air mixing ratios.
* Fixed valid() operation to properly use valid_min/valid_max attribute values.
* Fixed issue in adding *_count variable in append/merge when count variable
had more than one dimension.
* Updates to ECMWF GRIB, GEOMS MWR, and S5P L2 product ingestions.
1.7 2019-06-06
~~~~~~~~~~~~~~
* Added --no-history option to harpconvert, hardump, and harpmerge.
* Added harp.execute_operations() and harp.convert_unit() to Python interface.
* Added optional min_collocation_index/max_collocation_index arguments to
collocate_left()/collocate_right() operations.
* harp_product_set_history() and harp_product_set_source_product() now accept
NULL as argument to clear the current value.
* Appending/merging products now automatically adds missing *_count variables.
* Fixed potential crash during initialization of HARP ingestion module.
* Fixed invalid results when using bin_spatial with latitude_bounds {time,2}
and longitude_bounds {time,2} variables (i.e. range instead of polygon).
* Fixed array ordering in MATLAB interface. Multi-dimensional arrays are now
represented from fastest running to slowest (=fortran ordering) as intended.
* Update to S5P L2 product ingestion.
1.6 2019-04-29
~~~~~~~~~~~~~~
* Added clamp() and squash() operations.
* Allow bin() and sort() operations to be performed using a set of variables.
* harp_product_sort() now takes a list of variables.
* The rename() operation no longer fails if the source variable does not exist
but a variable with the target name already exists.
* The python harp.import_product() function now uses the C library to perform
the contactenation of products.
* Added post_operations argument to python harp.import_product() function.
* Besides using a polygon of >=3 vertices, a geospatial area can now also be
defined via a bounding rect. Use e.g. 'latitude_bounds {time,2}' and
'longitude_bounds {time,2}' to define the min/max of the bounding rect.
* Added derivation for tropospheric/stratospheric column from partial column
profile and tropopause.
* Corrected derivation for calculating tropopause altitude/pressure.
* Deriving a variable will now use the conversion that requires the minimum
amount of intermediate variables to be calculated.
* Added support for scene albedo/pressure/type variables.
* Fixed issue where regridding in the vertical dimension using pressure as axis
could end up with wrong pressure_bounds values.
* Fixed problem in matlab mex extension determination on linux when using the
configure script.
* Fixed setting of default CODA definition path and udunits2 xml path for
Python on Windows.
* Added format, variable count, and dimension sizes to harpcheck output.
* Added support for TCCON products in GEOMS format.
* Updates to AEOLUS L2A, ESACCI_OZONE_L2_TC, GEOMS FTIR, GOME2 L2, OMI L2 NO2,
QA4ECV, and S5P L2 product ingestions.
* Updated embedded expat library to version 2.2.6.
1.5 2018-12-17
~~~~~~~~~~~~~~
* Fixed issue where vertical smoothing/regridding provided invalid results if
input variables were available as float instead of double data.
* harp_product_get_derived_variable() no longer converts data to double if
no data type was provided and data already had the right unit.
* Fixed issue where listing derivations for a specific variable would not show
all possible conversions.
* Add derivation for cloud height from cloud base/top height.
* Added support for Dobson, Pandora and Sonde products in GEOMS format.
* Updates to ACE-FTS, GOME/GOME-2 L2, and S5P L2 product ingestions.
1.4 2018-09-28
~~~~~~~~~~~~~~
* Added -t/--target option to harpdump --list-derivations.
* Added variable_name parameter to harp_doc_list_conversions().
* Added support for meridional/zonal wind speed, and planetary boundary layer
height variables.
* Added support for sic (Shannon information content) variables for vertical
profiles.
* Aerosol optical depth and extinction coefficient variables will now
automatically use a log/log regridding in the spectral dimension.
This is equivalent to using the Angstrom Exponent approach.
* Fixed interpolation issue where points beyond the source grid could end up
having an arbitrary wrong value (instead of NaN). This is a regression of
harp 1.3.
* Add derivations between wind speed/direction and zonal/meridional velocity.
* Flattening a dimension now special cases a dimension length 1 (with less
modifications to variables/product).
* Fixed issue where a point distance of 0 could sometimes result in NaN.
* Updates to S5P L2, GOME/GOME-2 L2, GEOMS UVVIS-DOAS-OFFAXIS, ECMWF GRIB, and
AERONET product ingestions.
1.3 2018-07-05
~~~~~~~~~~~~~~
* Added bin_spatial() operation.
* Improved binning operation to automatically filter for NaN values and add
variable-specific count variables to specify the number of non-NaN values.
* Performing a binning operation on top of a binning operation will propagate
averages based on the available count variables (i.e. as if the final bin was
performed on the original data).
* Entries added to the global 'history' attribute now also include the current
utc time and harp version before the execution command.
* Allow NULL as argument for harp_variable_set_description() and
harp_variable_set_unit() to clear the current description/unit.
* Added harp_product_bin_spatial() and harp_variable_copy_attributes()
functions.
* Added optional parameter to regrid() operation to specify the axis boundaries.
* Python harp.import_product() now ignores empty products when ingesting a list
of products.
* Added derivations:
- column vmr/mmr from column density ratios
- total column mass density from surface pressure and pressure profile
- mass mixing ratio from mass density
- between dry air and total air mass (column) density
- between dry air, total air, and H2O number densities
* Fixed derivation of altitude from gph/latitude.
* Updates to S5P L2 and MIPAS L2 product ingestions.
1.2 2018-04-09
~~~~~~~~~~~~~~
* Allow data compression for HDF5 output:
- Added harp_set_option_hdf5_compression() and
harp_get_option_hdf5_compression() functions to C library.
- Added --hdf5-compression option to harpconvert and harpmerge.
- Added hdf5_compression parameter to harp.export_product() Python function.
* Allow regrid() operation to interpolate in the time dimension.
* Improved binning of angle values (average of 1 and 359 [deg] is now 0).
* Added bin() operation variant without parameters.
* The bin() operation using a collocation result file now retains the
collocation_index variable (with the first value for each bin).
* Changed smooth() operation such that each vertical level where the source
grid point was NaN the target grid point also remains NaN.
* Added -ap option to harpmerge.
* Added --dataset option to harpdump and changed output format of
harp_dataset_print() and harp_product_metadata_print() to csv format.
* Added 'operations' parameter to Python harp.export_product() function.
* Importing/exporting a product with operations in Python now automatically
updates the history attribute.
* Fixed issue where harpmerge did not introduce a time dimension for all
variables.
* Fixed issue where datetime_start and datetime_stop global attributes were not
created when exporting in HDF4 or HDF5 format.
* String attributes in HDF5 files can now be empty strings (using H5S_NULL).
* Allow HARP to import HDF5 files that are generated by the netCDF4 library
(allow numeric attributes to be 1-element arrays instead of scalars).
* harpcheck now returns with exit code 1 if any of the imports fail.
* harpcollocate --update now only allows one dataset as argument.
* Fixed performance issue with spatial filtering on systems with slow math lib.
* Fixed mapping description of GOME2 L1 product ingestion.
* Fixed wrong results of area overlap calculation.
* Replaced '_' by '-' in long versions of argument names of harpcollocate.
* Added support for dfs (degrees of freedom for signal) profiles.
* Updates to S5P L2 product ingestions.
1.1 2017-11-06
~~~~~~~~~~~~~~
* Parameters that required a file or a directory as dataset reference can now
also take a file with a .pth extension as input. This file should contain a
line-separated list of paths to files and/or directories of the dataset.
* Specifying a directory for a dataset will now recursively add all products
(including sub-directories and contents of .pth files).
* The variable that is binned on with a 'bin(variable)' operation is now
always kept in the resulting product.
* The bin() operation can now handle 'count' variables (summing it if present
and creating it otherwise).
* Fixed issue where collocate filter operation on a product with a single
sample could end in an infinite loop.
* Fixed issue in harpcollocate when providing a -nx option without a -ny.
* Fixed issue with wrap() operation for values < specified minimum.
* Fixed issue with collocate_left/collocate_right operations not including
last sample when applying it on products in an external format.
* harpcollocate now ignores empty products instead of raising errors for them.
* Added -oa/-ob and -aa/-ab options to harpcollocate matchup.
* Added solar_declination_angle and solar_hour_angle variables and derivations.
* Added orbit_index variable and added this to various product ingestions.
* Fixed derivations of solar azimuth/zenith angles from datetime and lat/lon.
* Added derivations for averaging kernel conversions.
* Added derivations for tropopause altitude/pressure.
* Added NH3 (amonia) as supported species.
* Updates to ECMWF-GRIB, GEOMS-TE-UVVIS-DOAS-OFFAXIS, GEOMS-TE-FTIR, and
S5P L2 product ingestions (and several others for the new orbit_index).
1.0 2017-09-25
~~~~~~~~~~~~~~
* The import and ingest functions have been combined into a single import
function that can both read HARP compliant products as well as ingest from
the supported external formats. This means:
- harp_import() and harp_ingest() are combined into a single harp_import()
- harp_ingest_test() has been renamed to harp_import_test()
- harpfilter has been removed (use harpconvert instead)
- '--test' option of harpconvert has been removed (use harpcheck instead)
* harpcollocate has been completely rewritten.
- any 1D time-dependent variable can now be a collocation criterium
- nearest neighbour filtering with weighted average is no longer supported
- collocation result files are not backward compatible due to a change in
the CSV header
- arguments to harp_collocation_result_new() and
harp_collocation_result_add_pair() have changed
* Added support for enumeration variables. Enumeration variables have a int8,
int16, or int32 type and provide an index into a fixed list of enum names.
Enumeration variables can be treated as string values in operations.
* Added MATLAB and IDL interfaces for HARP.
* Added -o option to harpdump and harpmerge.
* Added -a and -o options for 'harpdump --list-derivations'.
* '--list-derivations' option of harpconvert has been removed (use harpdump
instead).
* Added -l option to harpmerge.
* Renamed area/point operations:
- area_mask_covers_area -> area_covers_area
- area_mask_covers_point -> point_in_area
- area_mask_intersects_area -> area_intersects_area
- point_in_area -> area_covers_point
The first part of the operation name now reflects the item in the
product and the second part reflects the parameter that is provided.
* Improved performance of point/area vs. area overlap operations by first
comparing against the bounding box of a polygon.
* Added sort() and area_inside_area() operations.
* The derive() operation can now be used without dimension specification
to perform a unit and/or data type conversion on an existing variable.
* The derive_smoothed_column(), regrid() and smooth() operations can now
use a single merged collocated product (instead of a combination of a csv
collocation result file and a dataset directory of collocated products).
* An area parameter for operations can now either be a reference to an
area mask file or a single polygon provided as lat/lon array parameters.
* Added derivation to derive the area (size) of latitude/longitude bounds.
* Added derivation to derive degree of freedom for signal (DFS) from 2D
averaging kernels (AVK).
* Derivations of bound variables will now clamp to the edge midpoints when
the 'regrid_out_of_bounds' option is set to 'edge'.
* The 'data_type' parameter for harp_product_get_derived_variable() and
harp_product_add_derived_variable() is now a pointer to allow it to be
optional (i.e. pass NULL to use the default data type).
* Added harp_geometry_get_area(),
harp_product_get_smoothed_column_using_collocated_product(),
harp_product_smooth_vertical_with_collocated_product(),
harp_product_sort(), harp_product_regrid_with_collocated_product(),
harp_set_udunits2_xml_path(), harp_set_udunits2_xml_path_conditional(),
and harp_variable_set_enumeration_values() functions.
* harp_import_product_metadata() and harp_dataset_import() now have an
'options' parameter.
* Changed 'percentage' parameter of harp_geometry_has_area_overlap() to
'fraction'.
* harp_product_set_source_product() now has a product path as parameter
and stores the base name of this as source_product value.
* Added HARP_UNIT_AREA default unit.
* The C interface for collocation results was modified to support a
dynamic set of collocation criteria differences (based on variable names
for these differences).
* The Python interface now sets the udunits2 xml path dynamically,
relative to the location of the HARP C library.
* wrap() operation now properly updates valid_min/valid_max properties.
* codadef files are now installed in <prefix>/share/coda/definitions.
This allows a present CODA installation to use the same codadef files.
* Windows installation now installs data and documentation files in the
subdirectory <prefix>/share/harp.
* Fix issue in deriving total column from partial column profile if partial
column profile did not have {time,vertical} dimensions.
* Fix invalid results from binning operation/function.
* Improve speed of --list-derivations on a product.
* Split data format documentation into separate sections under a new
heading 'conventions'.
* Added product ingestions for Cloudnet L2a classification products.
* Updates to ECMWF-GRIB, GEOMS-TE-UVVIS-DOAS-OFFAXIS, MIPAS, QA4ECV, and
S5P product ingestions.
0.6 2017-04-23
~~~~~~~~~~~~~~
* Added ingestion support for ACE-FTS, CALIPSO, Cloud CCI, and NPP-Suomi.
* Added bin() and set() operations.
* Added optional data_type parameter to derive() operation.
* Major refactoring of internal implementation of operations (greatly
reducing and simplifying the code).
* Fix application of apriori for smoothed column derivation (for vertical
range where partial columns are NaN, don't subtract apriori*AVK, but just
set to apriori).
* Fix memory issue in partial column from vmr dry air derivation.
* Remove double definition of radiance conversions.
* Added global HARP option 'regrid_out_of_bounds'.
* Added harp_product_bin() function.
* Add data_type parameter to harp_product_add_derived_variable() and
harp_product_get_derived_variable() functions.
* Added data_type parameter to harp_product_get_derived_variable() and
harp_product_add_derived_variable() functions.
* Allow 'Conventions' global attribute to contain more than just a reference
to HARP.
* Renamed O3MSAF codadef to ACSAF. When updating HARP, you should remove
any existing O3SMAF-*.codadef files from your installation directory.
Otherwise you will get 'detection rule for ... is shadowed by ... ' errors.
* Fixed issue with TES TL2CH4L ingestion.
* Updates to Aeolus, GEOMS, S5P, QA4ECV, and Ozone CCI product ingestions.
0.5 2017-03-26
~~~~~~~~~~~~~~
* Changed Open Source licence of HARP from GPLv2 to BSD (3-Clause).
* Changed all '-' to '_' in operation function names to make them identifiers.
* Added harpmerge command line tool.
* Added regrid(), rename(), smooth(), derive_smoothed_column(),
point_in_area(), and wrap() operations.
* Removed the harpprofile command line tool. Its functionality is replaced by
the regrid() and smooth() operations.
* Renamed harp_product_regrid_vertical_with_collocated_dataset() to
harp_product_regrid_with_collocated_dataset() and
harp_product_smooth_vertical() to
harp_product_smooth_vertical_with_collocated_dataset()
* Added vertical_unit parameter to harp_product_smooth_vertical() and
harp_product_regrid_with_collocated_dataset() functions.
* Replaced harp_product_regrid_vertical_with_axis_variable() with
harp_product_regrid_with_axis_variable(). It can now be used for other
dimensions too (except the time dimension).
* Added harp_collocation_result_filter_for_collocation_indices(),
harp_product_get_smoothed_column(),
harp_product_get_smoothed_column_using_collocated_dataset(),
harp_product_remove_variable_by_name(), harp_variable_append(),
harp_product_append(), and harp_variable_smooth_vertical() functions.
* Renamed harp_product_get_variable_id_by_name() to
harp_product_get_variable_index_by_name().
* Python harp.import_product() and harp.ingest_product() functions now accept
a list of filenames or a globbing pattern as parameter (which will return
the result of harp.concatenate() on the imported/ingested products).
* Automatically set CODA definition path for HARP Python interface
(path can be overruled with CODA_DEFINITION environment variable).
* The parsing of the operations string has been rewritten using flex/yacc.
This has fixed several issues related to the parsing and execution of
operations.
* Fixed issue with flatten() operation providing wrong results.
* Fixed issue where vertical regridding to a common grid provided wrong
results.
* Fixed issue where interval interpolation for a pressure grid was not
using a log-linear interpolation.
* Aligned order of latitude/longitude arguments in all functions to have
latitude first and longitude second.
* Added unit 'ppv' to the included udunits2-common.xml file.
* Added variables for surface temperature, total water vapour content, and
total cloud cover to ECMWF GRIB ingestion.
* The default internal units have been changes to SI (except for molar mass).
All algorithms have been adapted accordingly.
* Added variable conversions for: apriori profiles, column density from
mass density (including pm1/pm2.5/pm10).
* Added support for AEOLUS L1b/L2a/L2b products.
* Added support for Aerosol and GHG CCI L2/L3 products.
* Added support for GEOMS FTIR C2H2 ingestion.
* Added support for OMI OMCHOCHO ingestion.
* Added support for additional TES L2 products.
* Added support for QA4ECV NO2 and HCHO products.
* Added support for GOME, GOME-2, GOMOS, MIPAS, IASI, SCIAMACHY L1 products.
* Added support for ODIN OSIRIS and SMR L2 products.
* Changed --list-conversions command line option to --list-derivations.
And added --list-derivations option to harpdump.
* Use NC_64BIT_OFFSET when storing data to netcdf larger than 1GB.
0.4 2016-10-16
~~~~~~~~~~~~~~
* Fixed wrong scaling factor in conversions related to the ideal gas law
(10^2 vs. 10^-2) and barometric formula (10^-9 vs. 10^-6).
* Fixed error in algorithm to derive gravity from latitude and height.
* Fixed invalid sorting of collocation result file (which resulted in wrong
results when performing a collocation resampling).
* Fixed wrong independent dimension length for altitude/pressure bounds for
SCIAMACHY Level2 ingestion.
* Added apriori number density variables to SCIAMACHY Level2 ingestion.
* Added variable conversion from latitude/longitude per vertical profile point
to single (mid-point) latitude/longitude for profile.
* Variable conversions for altitude and pressure bounds now also work when
they are dependend on latitude and longitude dimensions.
* Fixed crash in derived variable conversion for volume mixing ratio from
mass mixing ratio (for total air).
* Changed dimension argument of 'flatten()' operation from string to
identifier.
* Fixed issue where -inf/+inf valid_min/valid_max attributes could not be
exported as float values in the Python harp.export_product() interface.
* When deriving latitude/longitude bounds from mid points, latitude is now
clamped to [-90,90] and longitude is wrapped to [-180,180].
* Fixed volume mixing ratio unit for GEOMS-TE-UVVIS-DOAS-OFFAXIS ingestion.
0.3 2016-09-30
~~~~~~~~~~~~~~
* The changes between 0.2 and 0.3 are extensive. The list below provides the
most important aspects, but there are many more smaller changes that are not
listed.
* HARP 'actions' are now called 'operations' and are now executed in a strict
sequence. This means that you can now do things like:
'derive(var); var < 10; exclude(var)'
* Added harpprofile command line tool that allows regridding and smoothing
of vertical gridded data.
Also added harp_product_regrid_vertical_with_axis_variable(),
harp_product_regrid_vertical_with_collocated_dataset(), and
harp_product_smooth_vertical() to C interface.
* Added algorithm descriptions for the derived variable conversions to the
HARP documentation.
* Introduced a more strict definition of variable names for the HARP data
format. As part of this the list of supported species and associated naming
has also been revised.
* Introduced strict separation between dry air and total/wet air. The default
air density in HARP is now the total/wet air. Anything related to dry air
should now be annotated as such in the variable name. As a result, 'dry_air'
is now introduced as a species, and for mixing ratios the '_wet' postfix is
removed and the '_dry_air' postfix is introduced.
Derived variable conversions have been added for dry/wet conversion and for
retrieving the needed 'molar mass' of total air.
* Added functionality to deal with HARP Datasets and Product Metadata to the
public C library interface. The collocation result interface in the C
libraryinterface has also been reworked to now work with HARP Datasets.