Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use CMR metadata to get S3FS sessions and authenticate queries to CMR using top level APIs #296

Merged
merged 9 commits into from
Sep 18, 2023

Conversation

betolink
Copy link
Member

@betolink betolink commented Sep 12, 2023

This PR allows earthacces.search_data() and earthaccess.search_dataset() use the auth object to send bearer tokens to CMR, usually this is not required but in the case of restricted datasets or early release data CMR will only return results if we are in the access control lists for a given dataset.

with this PR earthaccess will be able to use the metadata from CMR to grab S3 credentials, perhaps we need to simplify this even more but for now this works.

import earthaccess

earthaccess.login()

results = earthaccess.search_data(
    short_name='SWOT_L2_HR_RIVERSP_1.0',
    cloud_hosted=True,
    bounding_box=(-10, 20, 10, 50),
    temporal=("1999-02", "2019-03"),
    count=10
)

# now earthaccess will use that the S3 credentials if they are present in the metadata if not it will grab the DAAC's
earrthaccess.open(results)

This code was failing because the SWOT mission has its own S3 credentials that are not from PODAAC.

another way of getting the authenticated sessions would be:

if len(results) > 0:
    print(f"Actual granules returned: {len(results)}")    
    fs_s3 = earthaccess.get_s3fs_session(results=results)

And then we could use the storage_options to create authenticated sessions, this is the part that can be simplified:

import fiona
from fiona.session import AWSSession

with fiona.Env(
    session=AWSSession(
        aws_access_key_id=fs_s3.storage_options["key"],
        aws_secret_access_key=fs_s3.storage_options["secret"],
        aws_session_token=fs_s3.storage_options["token"]
    )
):
    # some code  with gpd.read_file() etc

@github-actions
Copy link

github-actions bot commented Sep 12, 2023

Binder 👈 Launch a binder notebook on this branch for commit a85f531

I will automatically update this comment whenever this PR is modified

Binder 👈 Launch a binder notebook on this branch for commit ca6590f

Binder 👈 Launch a binder notebook on this branch for commit c4321c0

@betolink betolink marked this pull request as ready for review September 12, 2023 18:45
@betolink betolink changed the title WIP: use CMR metadata to get S3FS sessions and authenticate queries to CMR using top level APIs Use CMR metadata to get S3FS sessions and authenticate queries to CMR using top level APIs Sep 12, 2023
@betolink betolink merged commit 01a7da0 into main Sep 18, 2023
11 checks passed
@betolink betolink deleted the wip-s3credentials branch November 22, 2023 17:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant