Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

t.stac.import: Add STAC API import functionality #802

Merged
merged 99 commits into from
Aug 19, 2024
Merged
Show file tree
Hide file tree
Changes from 43 commits
Commits
Show all changes
99 commits
Select commit Hold shift + click to select a range
e5a9b7e
r.in.stac: inital setup of module structure options and flags
cwhite911 Sep 30, 2022
fe59675
Fixed formatting issues
cwhite911 Sep 30, 2022
df8a3b3
r.in.stac: added queries for collections and items
Jun 6, 2023
7ca8170
Organized options
Jun 6, 2023
1ad5487
t.stac.import: Changed module name to t.stac.import
Sep 18, 2023
7cb2222
t.stac.import: Changed module name to t.stac.import
Sep 19, 2023
c3192ce
renamed grass to gs
Sep 19, 2023
98b82b9
added get items function
Sep 19, 2023
3172c51
Added filter by bbox and user region
Sep 19, 2023
83feb9c
Started the import and download fucntionallity
Sep 19, 2023
167412d
fixed typo
Sep 19, 2023
d39c6ce
Added download method for COGs via https, s3, and gs
Feb 20, 2024
c410e13
Added query docs
Feb 20, 2024
e16cf83
Updated multiprocessing options
Feb 20, 2024
f4c31e9
Started added static catalog functionality
Feb 21, 2024
c16bef9
Updated docs
Feb 21, 2024
1072943
Updated documentation
Feb 21, 2024
5af75ed
Added authentication methods
Feb 21, 2024
911afe5
Added planetary computer support
Feb 21, 2024
f75886c
Cleaned code formatting
Feb 22, 2024
675fbe7
Added ability to filter items assets by asset ids
Feb 22, 2024
588eb08
Improved collection outputs
Feb 22, 2024
bf27b55
Added additional file types
Feb 23, 2024
1d8ef0f
Started restructuring code into different modules
Mar 29, 2024
812c473
Merge branch 'grass8' into r.in.stac
cwhite911 Mar 29, 2024
f94b5a1
Update src/temporal/t.stac.import/t.stac.import.py
cwhite911 Mar 29, 2024
5fb176a
Update src/temporal/t.stac.import/t.stac.import.py
cwhite911 Mar 29, 2024
33e9b78
Update src/temporal/t.stac.import/t.stac.import.py
cwhite911 Mar 29, 2024
9fdcc50
Update src/temporal/t.stac.import/t.stac.import.py
cwhite911 Mar 29, 2024
f33217c
Merge remote-tracking branch 'origin/stac.import' into r.in.stac
Mar 29, 2024
bc72965
Added t.stac.item
Mar 29, 2024
a14d923
Updated dependencies
Mar 29, 2024
97ea4e1
Fixed tests
Mar 29, 2024
802fb27
Merge branch 'grass8' into r.in.stac
cwhite911 May 7, 2024
8636b0d
Merge branch 'grass8' into r.in.stac
cwhite911 May 13, 2024
b2fd8f6
Merge branch 'grass8' into r.in.stac
cwhite911 Jun 14, 2024
d7f23c9
Added catalog tool
Jun 14, 2024
b067bc0
Updted collection
Jun 14, 2024
46482d5
Updated docs
Jun 14, 2024
9ce5899
Updated options
Jun 15, 2024
43152e9
Updated copyright
Jun 15, 2024
3303671
Can import item vector boundary metadata
Jun 16, 2024
9e4d607
Updated docs
Jun 16, 2024
6751f3b
Added assest metadata vector
Jun 17, 2024
5118ff7
Set s3 download parameters
Jun 17, 2024
535e92c
added format option
Jun 17, 2024
c80a5cc
Fixed item asset metadata
Jun 17, 2024
8ac3113
Removed t.stac.import
Jun 17, 2024
0d10afa
Added delete to tmpfile creation
Jun 17, 2024
dd21b64
Update src/temporal/t.stac/README.md
cwhite911 Jun 17, 2024
7b8f933
Update src/temporal/t.stac/t.stac.catalog/t.stac.catalog.html
cwhite911 Jun 17, 2024
06f040c
Update src/temporal/t.stac/t.stac.catalog/t.stac.catalog.html
cwhite911 Jun 17, 2024
602a28b
Update src/temporal/t.stac/t.stac.catalog/t.stac.catalog.html
cwhite911 Jun 17, 2024
02a7690
Update src/temporal/t.stac/t.stac.catalog/t.stac.catalog.html
cwhite911 Jun 17, 2024
a3eab54
Update src/temporal/t.stac/t.stac.collection/t.stac.collection.py
cwhite911 Jun 17, 2024
f13ad97
Update src/temporal/t.stac/t.stac.item/t.stac.item.py
cwhite911 Jun 17, 2024
fdb7226
Update src/temporal/t.stac/README.md
cwhite911 Jun 17, 2024
5c3af66
Merge branch 'grass8' into r.in.stac
cwhite911 Jun 17, 2024
f516b91
Removed references to t.stac.import
Jun 17, 2024
53b829d
Added mocked api tests
Jun 17, 2024
c4d9140
Added checks for conformance classes
Jun 18, 2024
611fc2b
Removed auth options from GUI
Jun 18, 2024
50c755f
Remove username param
Jun 18, 2024
8d6dfa1
updated pystac and pystac-client versions, documenation and settings …
Jun 19, 2024
5ec01a2
Updated docs and basic output
Jun 19, 2024
bb87497
Added basic collection output
Jun 19, 2024
0e3ed40
Added option to create strd registry
Jun 19, 2024
210c9f2
Updated max_limit to no default value
Jun 19, 2024
0724464
Merge branch 'grass8' into r.in.stac
cwhite911 Jul 9, 2024
0b3a1f7
Merge branch 'grass8' into r.in.stac
cwhite911 Jul 16, 2024
1d85dee
Update src/temporal/t.stac/README.md
cwhite911 Jul 16, 2024
a623d3c
Update src/temporal/t.stac/README.md
cwhite911 Jul 16, 2024
7b5bec9
Update src/temporal/t.stac/README.md
cwhite911 Jul 16, 2024
fdcf36b
Update src/temporal/t.stac/README.md
cwhite911 Jul 16, 2024
3604e7e
Update src/temporal/t.stac/README.md
cwhite911 Jul 16, 2024
4d9b12a
Update src/temporal/t.stac/README.md
cwhite911 Jul 16, 2024
1b1149e
Update src/temporal/t.stac/README.md
cwhite911 Jul 16, 2024
f6856ca
Update src/temporal/t.stac/README.md
cwhite911 Jul 16, 2024
295ca8f
Update src/temporal/t.stac/README.md
cwhite911 Jul 16, 2024
bb9be00
Update src/temporal/t.stac/README.md
cwhite911 Jul 16, 2024
41ce5fa
Update src/temporal/t.stac/README.md
cwhite911 Jul 16, 2024
3d63bb6
Update src/temporal/t.stac/README.md
cwhite911 Jul 16, 2024
ef881d3
Update src/temporal/t.stac/t.stac.catalog/t.stac.catalog.html
cwhite911 Jul 16, 2024
44672dd
Update src/temporal/t.stac/t.stac.catalog/t.stac.catalog.py
cwhite911 Jul 16, 2024
43e58e5
Update src/temporal/t.stac/t.stac.collection/t.stac.collection.html
cwhite911 Jul 16, 2024
dd8714d
Update src/temporal/t.stac/t.stac.catalog/t.stac.catalog.py
cwhite911 Jul 16, 2024
0cb3061
Update src/temporal/t.stac/t.stac.collection/t.stac.collection.html
cwhite911 Jul 16, 2024
0b37d5c
Update src/temporal/t.stac/t.stac.collection/t.stac.collection.html
cwhite911 Jul 16, 2024
8c568d3
Update src/temporal/t.stac/t.stac.html
cwhite911 Jul 16, 2024
45dfd59
Update src/temporal/t.stac/t.stac.html
cwhite911 Jul 16, 2024
78d74d8
Update src/temporal/t.stac/t.stac.html
cwhite911 Jul 16, 2024
d91c6e6
Update src/temporal/t.stac/t.stac.html
cwhite911 Jul 16, 2024
3aca4df
Update src/temporal/t.stac/t.stac.html
cwhite911 Jul 16, 2024
61ea743
Updated max_items default value
Jul 16, 2024
2aac0ac
Merge branch 'grass8' into r.in.stac
cwhite911 Aug 2, 2024
1d6e195
Merge branch 'grass8' into r.in.stac
cwhite911 Aug 13, 2024
347bb5b
Removed broken mock api tests
Aug 13, 2024
04d89c2
Fixed formatting error
Aug 16, 2024
cb1052f
Merge branch 'grass8' into r.in.stac
cwhite911 Aug 16, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions src/temporal/t.stac/Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
MODULE_TOPDIR =../..

PGM = t.stac

SUBDIRS = libstac \
t.stac.catalog \
t.stac.collection \
t.stac.item \
t.stac.import \

include $(MODULE_TOPDIR)/include/Make/Dir.make

default: parsubdirs htmldir

install: installsubdirs
$(INSTALL_DATA) $(PGM).html $(INST_DIR)/docs/html/
158 changes: 158 additions & 0 deletions src/temporal/t.stac/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,158 @@
# (In-Development) t.stac

## Description

**t.stac.import** utilizes the
[pystac-client (v0.5.1)](https://github.com/stac-utils/pystac-client) to search
STAC APIs and import and import items into GRASS GIS.
cwhite911 marked this conversation as resolved.
Show resolved Hide resolved

### Item Search Parameters

[Offical Docs](https://pystac-client.readthedocs.io/en/stable/api.html#item-search)

**method** – The HTTP method to use when making a request to the service. This
must be either "GET", "POST", or None. If None, this will default to "POST".
If a "POST" request receives a 405 status for the response, it
will automatically retry with "GET" for all subsequent requests.

**max_items** – The maximum number of items to return from the search, even if
there are more matching results. This client to limit the total number of Items
cwhite911 marked this conversation as resolved.
Show resolved Hide resolved
returned from the items(), item_collections(), and items_as_dicts methods().
The client will continue to request pages of items until the number of max
items is reached. This parameter defaults to 100. Setting this to None will
allow iteration over a possibly very large number of results.

**limit** – A recommendation to the service as to the number of items to return
per page of results. Defaults to 100.

**ids** – List of one or more Item ids to filter on.
cwhite911 marked this conversation as resolved.
Show resolved Hide resolved

**collections** – List of one or more Collection IDs or pystac.Collection
instances. Only Items in one of the provided Collections will be searched.

**bbox** – A list, tuple, or iterator representing a bounding box of 2D
or 3D coordinates. Results will be filtered to only those intersecting the
bounding box.

**intersects** – A string or dictionary representing a GeoJSON geometry,
or an object that implements a ``__geo_interface__`` property, as supported
by several libraries including Shapely, ArcPy, PySAL, and geojson.
Results filtered to only those intersecting the geometry.

**datetime –**
Either a single datetime or datetime range used to filter results. You may
cwhite911 marked this conversation as resolved.
Show resolved Hide resolved
express a single datetime using a datetime.datetime instance, a
cwhite911 marked this conversation as resolved.
Show resolved Hide resolved
RFC 3339-compliant timestamp, or a simple date string (see below).
Instances of datetime.datetime may be either timezone aware or unaware.
Timezone aware instances will be converted to a UTC timestamp before being
passed to the endpoint. Timezone unaware instances are assumed to represent
UTC timestamps. You may represent a datetime range using a "/" separated
string as described in the spec, or a list, tuple, or iterator of 2 timestamps
or datetime instances. For open-ended ranges, use either".."
cwhite911 marked this conversation as resolved.
Show resolved Hide resolved
('2020-01-01:00:00:00Z/..', ['2020-01-01:00:00:00Z', '..']) or
a value of None (['2020-01-01:00:00:00Z', None]).
cwhite911 marked this conversation as resolved.
Show resolved Hide resolved

If using a simple date string, the datetime can be specified in YYYY-mm-dd
format, optionally truncating to YYYY-mm or just YYYY. Simple date strings
will be expanded to include the entire time period, for example:

* 2017 expands to 2017-01-01T00:00:00Z/2017-12-31T23:59:59Z
* 2017-06 expands to 2017-06-01T00:00:00Z/2017-06-30T23:59:59Z
* 2017-06-10 expands to 2017-06-10T00:00:00Z/2017-06-10T23:59:59Z

If used in a range, the end of the range expands to
the end of that day/month/year, for example:

* 2017/2018 expands to 2017-01-01T00:00:00Z/2018-12-31T23:59:59Z
* 2017-06/2017-07 expands to 2017-06-01T00:00:00Z/2017-07-31T23:59:59Z
* 2017-06-10/2017-06-11 expands to 2017-06-10T00:00:00Z/2017-06-11T23:59:59Z

**query** – List or JSON of query parameters as per the STAC API query extension

**filter** – JSON of query parameters as per the STAC API filter extension
cwhite911 marked this conversation as resolved.
Show resolved Hide resolved

**filter_lang** – Language variant used in the filter body. If filter is a
dictionary or not provided, defaults to ‘cql2-json’. If filter is a string,
defaults to cql2-text.
cwhite911 marked this conversation as resolved.
Show resolved Hide resolved

**sortby** – A single field or list of fields to sort the response by
cwhite911 marked this conversation as resolved.
Show resolved Hide resolved

**fields** – A list of fields to include in the response.
Note this may result in invalid STAC objects, as they may not have required
fields. Use items_as_dicts to avoid object unmarshalling errors.
cwhite911 marked this conversation as resolved.
Show resolved Hide resolved

### Dependancies
cwhite911 marked this conversation as resolved.
Show resolved Hide resolved

* [pystac-client (v0.5.1)](https://github.com/stac-utils/pystac-client)

#### Optional Query

* [STAC API - Query Extension Specification](https://github.com/stac-api-extensions/query)

### S3 and GCS Requester Pays Support

* [GDAL Docs](https://gdal.org/user/virtual_file_systems.html#introduction)
* [STAC API - S3 Requester Pays](https://gdal.org/user/virtual_file_systems.html#vsis3-aws-s3-files)
* [STAC API - GCS Requester Pays](https://gdal.org/user/virtual_file_systems.html#vsigs-google-cloud-storage-files)

## Basic Usage

1. Check the STAC API for the available collections.

```bash
t.stac.import -c urlhttps://earth-search.aws.element84.com/v1/
```

2. Search for items in the collection.

```bash
t.stac.import -i url=https://earth-search.aws.element84.com/v1/
collections=sentinel-2-l2a
```

3. Import (dry-run) the items into GRASS GIS.

```bash
t.stac.import -d url=https://earth-search.aws.element84.com/v1/
collections=sentinel-2-l2a
```

cwhite911 marked this conversation as resolved.
Show resolved Hide resolved
## Workflow example

```python


# Search for items in the collection
STAC_API_URL = "https://planetarycomputer.microsoft.com/api/stac/v1"

catalog = gs.run_command(
"t.stac.catalog",
url=STAC_API_URL,
format="json"
)
collection = gs.run_command(
"t.stac.collection",
url=STAC_API_URL,
collection_id="sentinel-2-l2a",
format="json"
)
search = gs.run_command(
"t.stac.item",
url=STAC_API_URL,
collections="sentinel-2-l2a",
format="json"
)


# Search for items in the collection
items = gs.run_command(
"t.stac.items",
url=STAC_API_URL,
collections="sentinel-2-l2a",
format="json"
)




cwhite911 marked this conversation as resolved.
Show resolved Hide resolved
```
22 changes: 22 additions & 0 deletions src/temporal/t.stac/libstac/Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
MODULE_TOPDIR = ../../..

include $(MODULE_TOPDIR)/include/Make/Other.make
include $(MODULE_TOPDIR)/include/Make/Python.make

MODULES = __init__ staclib

ETCDIR = $(ETC)/t.stac

PYFILES := $(patsubst %,$(ETCDIR)/%.py,$(MODULES))
PYCFILES := $(patsubst %,$(ETCDIR)/%.pyc,$(MODULES))

default: $(PYFILES) $(PYCFILES)

$(ETCDIR):
$(MKDIR) $@

$(ETCDIR)/%: % | $(ETCDIR)
$(INSTALL_DATA) $< $@

install:
cp -r $(ETCDIR) $(INST_DIR)/etc
Empty file.
Loading