Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Work on BGC from 2024 LOV visit #356

Merged
merged 112 commits into from
Sep 25, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
112 commits
Select commit Hold shift + click to select a range
025070e
[skip-ci]
gmaze Jun 4, 2024
fdb7068
raise datanotfound in read_params if search results is empty
gmaze Jun 5, 2024
ec814c2
Update xarray.py
gmaze Jun 5, 2024
229de71
facade to pass user-mode to erddap data fetcher
gmaze Jun 5, 2024
b9800a6
Update erddap_data.py
gmaze Jun 5, 2024
b378da1
Add support for the "aux" index file + bug fix
gmaze Jun 6, 2024
a9561e4
Update erddap_data.py
gmaze Jun 6, 2024
6696a73
Update xarray.py
gmaze Jun 6, 2024
861ff31
Merge branch 'master' into bgc-2024
gmaze Jun 7, 2024
5592efc
New xarray.argo.transform_data_mode
gmaze Jun 7, 2024
2265e62
breaking change: renamed argo.filter_data_mode into argo.transform_da…
gmaze Jun 7, 2024
e22be87
Update test data
gmaze Jun 7, 2024
fe59468
Update fetchers.py
gmaze Jun 10, 2024
930fc64
docstring
gmaze Jun 10, 2024
da89110
Update erddap_data.py
gmaze Jun 10, 2024
5a96ff6
Cleaner postprocessor definition in the facade
gmaze Jun 10, 2024
7e0bb4d
New filter_data_mode, not backward compatible
gmaze Jun 10, 2024
63d1272
Add user_mode properties to low-level fetchers
gmaze Jun 11, 2024
cf38c5b
Update proto.py
gmaze Jun 11, 2024
f047945
Update fetchers.py
gmaze Jun 11, 2024
a569adc
more BGC var/param listing
gmaze Jun 11, 2024
1657ed0
Improve filters
gmaze Jun 11, 2024
bac1387
fix bugs here and there
gmaze Jun 11, 2024
f552cb9
Add support BGC HDF5 netcdf file
gmaze Jun 11, 2024
afbe4d9
kick start
gmaze Jun 11, 2024
e6ade19
CANYON-MED NN weights
gmaze Jun 11, 2024
ce11e50
New register_argodataset_accessor
gmaze Jun 12, 2024
d3a9060
misc
gmaze Jun 12, 2024
cc7a869
Bring CanyonMED to argopy !
gmaze Jun 12, 2024
6a4633d
New 'extensions' submodule
gmaze Jun 12, 2024
79b98a7
Update documentation
gmaze Jun 12, 2024
2fa392e
Update extensions sub-module
gmaze Jun 12, 2024
d5bc78c
Add mask to ensure that input measurements are in the Mediterranean Sea
gmaze Jun 12, 2024
8d59c03
Update canyon_med.py
gmaze Jun 12, 2024
9bc1403
Fix docstrings
gmaze Jun 12, 2024
a7231f3
Merge branch 'bgc-2024-canyon' into bgc-2024
gmaze Jun 13, 2024
2dde63b
Update gdacftp_data.py
gmaze Jun 13, 2024
1ea391e
Update canyon_med.py
gmaze Jun 13, 2024
148beb0
New split_data_mode transform
gmaze Jun 13, 2024
c8d546d
Add new bgc-s and ngc-b dataset
gmaze Jun 13, 2024
9055e71
Fix bug in the facade
gmaze Jun 14, 2024
9a7c418
catching up
gmaze Jul 5, 2024
9aa1ba3
Fix missing lists from utils
gmaze Jul 5, 2024
83b9501
Add deprecation policy for filter_data_mode
gmaze Jul 8, 2024
361f12c
Merge branch 'master' into bgc-2024
gmaze Jul 8, 2024
3c4b355
Update fetchers to use correct filter_data_mode
gmaze Jul 8, 2024
e959d7f
Better docstrings
gmaze Jul 8, 2024
f5df4e7
Apply Black on xarray
gmaze Jul 8, 2024
84aeef2
removed unused raiseNoDataLeft
gmaze Jul 8, 2024
c4e8c9c
fix CI tests
gmaze Jul 8, 2024
29ddf87
misc
gmaze Jul 9, 2024
c79cb36
Update erddap_data.py
gmaze Jul 9, 2024
1676297
Add URL registry for erddap fetcher
gmaze Jul 9, 2024
8c38f13
Merge branch 'master' into bgc-2024
gmaze Jul 12, 2024
3655bc8
update test data
gmaze Jul 12, 2024
814ae7f
Update filesystems.py
gmaze Jul 12, 2024
02264fd
Update xarray.py
gmaze Jul 18, 2024
1aa3ed4
Update proto.py
gmaze Jul 18, 2024
044602a
Update tutorial.py
gmaze Jul 19, 2024
dc3ab40
Improve coverage
gmaze Jul 19, 2024
ca18b0d
Re-activate mock server with erddap tests
gmaze Jul 19, 2024
4ad0e78
Update erddap_data.py
gmaze Jul 19, 2024
d9746e5
Update erddap_data.py
gmaze Jul 19, 2024
b6af5f6
Update mocked_http.py
gmaze Jul 19, 2024
c9eb9a3
Update xarray.py
gmaze Jul 19, 2024
17c8fb7
Update erddap_data.py
gmaze Jul 19, 2024
6853f05
Add CLI folder to git
gmaze Jul 19, 2024
20ea26b
Merge branch 'master' into bgc-2024
gmaze Jul 19, 2024
de218b1
Update test data
gmaze Jul 19, 2024
cf1722f
Create citests_httpdata_manager_altim.json
gmaze Aug 21, 2024
e3c3ee2
Update test data
gmaze Aug 21, 2024
284a5d2
Clear test data
gmaze Aug 21, 2024
058ba9b
Merge branch 'master' into bgc-2024
gmaze Aug 21, 2024
b6c8486
Update test_data
gmaze Aug 22, 2024
7d1680b
Merge branch 'master' into bgc-2024
gmaze Aug 22, 2024
69cefb5
Merge branch 'master' into bgc-2024
gmaze Aug 23, 2024
9137beb
Clear test data
gmaze Aug 23, 2024
3db25db
update test data
gmaze Aug 23, 2024
97a20e7
pin xarray<2024.3
gmaze Aug 23, 2024
62ecdab
Remove filter_data_mode_new
gmaze Sep 4, 2024
a721999
Merge branch 'master' into bgc-2024
gmaze Sep 5, 2024
eac13c5
improve docstrings
gmaze Sep 5, 2024
58a8258
Update test data
gmaze Sep 5, 2024
9affe78
Refactored register_argodataset_accessor into register_argo_accessor
gmaze Sep 5, 2024
29052b3
Update xarray.py
gmaze Sep 5, 2024
f81842c
Add missing test data
gmaze Sep 5, 2024
a484e30
Improve docstrings
gmaze Sep 5, 2024
09d2699
Update utils.py
gmaze Sep 5, 2024
8681887
Update utils.py
gmaze Sep 6, 2024
925c155
Update erddap_data.py
gmaze Sep 10, 2024
e72cb63
fix bugs in variable and parameter lists usage and definition
gmaze Sep 10, 2024
2eec9a6
Update User mode documentation page
gmaze Sep 10, 2024
4faa1aa
Update format.py
gmaze Sep 10, 2024
245e4b0
Update xarray.py
gmaze Sep 10, 2024
dc448c8
improve logs
gmaze Sep 10, 2024
d40b769
Update test data
gmaze Sep 10, 2024
5669032
update doc [skip-ci]
gmaze Sep 10, 2024
3875306
s3fs > 2023.12.12
gmaze Sep 10, 2024
7d103b1
Update pytest.ini
gmaze Sep 10, 2024
62ece42
misc
gmaze Sep 10, 2024
6d9be49
Merge branch 'master' into bgc-2024
gmaze Sep 18, 2024
c9de8b9
Clear CI tests data before merge
gmaze Sep 23, 2024
4681d55
Clear CI tests for easier merge with master [skip-ci]
gmaze Sep 23, 2024
cf41ba4
Merge branch 'master' into bgc-2024
gmaze Sep 23, 2024
a73f727
Update CI tests data
gmaze Sep 24, 2024
2ff193f
Update argovis_data.py
gmaze Sep 24, 2024
809adc9
Update create_json_assets
gmaze Sep 24, 2024
51355c3
update CI tests data
gmaze Sep 24, 2024
f7ebc21
Update test_deprecated.py
gmaze Sep 25, 2024
82c20c8
Update CI tests data
gmaze Sep 25, 2024
1797037
Update CI tests data
gmaze Sep 25, 2024
2d4785d
Remove 45mins timeout for CI tests
gmaze Sep 25, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
4 changes: 2 additions & 2 deletions .github/workflows/pytests-upstream.yml
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ jobs:
defaults:
run:
shell: bash -l {0}
timeout-minutes: 45
# timeout-minutes: 45
strategy:
fail-fast: true
matrix:
Expand Down Expand Up @@ -196,7 +196,7 @@ jobs:
defaults:
run:
shell: bash -l {0}
timeout-minutes: 45
# timeout-minutes: 45
strategy:
fail-fast: true
matrix:
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/pytests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ jobs:
run:
shell: bash -l {0}
continue-on-error: ${{ matrix.experimental }}
timeout-minutes: 45
# timeout-minutes: 45
strategy:
max-parallel: 12
fail-fast: false
Expand Down Expand Up @@ -169,7 +169,7 @@ jobs:
run:
shell: bash -l {0}
continue-on-error: ${{ matrix.experimental }}
timeout-minutes: 45
# timeout-minutes: 45
strategy:
max-parallel: 12
fail-fast: false
Expand Down
5 changes: 5 additions & 0 deletions argopy/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@
from .utils import MonitoredThreadPoolExecutor # noqa: E402, F401
from .utils import monitor_status as status # noqa: E402
from .related import TopoFetcher, OceanOPSDeployments, ArgoNVSReferenceTables, ArgoDocs, ArgoDOI # noqa: E402
from .extensions import CanyonMED


#
Expand Down Expand Up @@ -77,6 +78,10 @@
"ArgoColors", # Class
"stores",
"tutorial",

# Argo xarray accessor extensions
"CanyonMED",

# Constants
"__version__"
)
32 changes: 27 additions & 5 deletions argopy/data_fetchers/argovis_data.py
100644 → 100755
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,7 @@ def __init__(
"""
self.definition = "Argovis Argo data fetcher"
self.dataset_id = OPTIONS["dataset"] if ds == "" else ds
self.user_mode = kwargs["mode"] if "mode" in kwargs else OPTIONS["mode"]
self.server = kwargs["server"] if "server" in kwargs else api_server
timeout = OPTIONS["api_timeout"] if api_timeout == 0 else api_timeout
self.store_opts = {
Expand Down Expand Up @@ -379,14 +380,20 @@ def to_xarray(self, errors: str = "ignore"):
ds = ds[np.sort(ds.data_vars)]
return ds

def transform_data_mode(self, ds: xr.Dataset, **kwargs):
# Argovis data are already curated !
if ds.argo._type == "point":
ds["N_POINTS"] = np.arange(0, len(ds["N_POINTS"]))
return ds

def filter_data_mode(self, ds: xr.Dataset, **kwargs):
# Argovis data already curated !
# Argovis data are already curated !
if ds.argo._type == "point":
ds["N_POINTS"] = np.arange(0, len(ds["N_POINTS"]))
return ds

def filter_qc(self, ds: xr.Dataset, **kwargs):
# Argovis data already curated !
# Argovis data are already curated !
if ds.argo._type == "point":
ds["N_POINTS"] = np.arange(0, len(ds["N_POINTS"]))
return ds
Expand All @@ -396,7 +403,7 @@ def filter_researchmode(self, ds: xr.Dataset, *args, **kwargs) -> xr.Dataset:

This filter will select only QC=1 delayed mode data with pressure errors smaller than 20db

Use this filter instead of filter_data_mode and filter_qc
Use this filter instead of transform_data_mode and filter_qc
"""
ds = ds.argo.filter_researchmode()
if ds.argo._type == "point":
Expand Down Expand Up @@ -521,7 +528,12 @@ def uri(self):
)
boxes = self.Chunker.fit_transform()
for box in boxes:
urls.append(Fetch_box(box=box, ds=self.dataset_id).get_url())
opts = {
"ds": self.dataset_id,
"fs": self.fs,
"server": self.server,
}
urls.append(Fetch_box(box=box, **opts).get_url())
else:
urls.append(self.get_url())
else:
Expand Down Expand Up @@ -549,6 +561,16 @@ def uri(self):
)
boxes = self.Chunker.fit_transform()
for box in boxes:
urls.append(Fetch_box(box=box, ds=self.dataset_id).get_url())
opts = {
"ds": self.dataset_id,
"fs": self.fs,
"server": self.server,
}
urls.append(
Fetch_box(
box=box,
**opts,
).get_url()
)

return self.url_encode(urls)
Loading
Loading