Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixes 16652: add GCS storage service #16917

Merged

Conversation

MChamberlin
Copy link
Contributor

@MChamberlin MChamberlin commented Jul 3, 2024

Describe your changes:

Fixes 16652.

Add GCS (Google Cloud Storage) storage service connection. Documentation for this service (and for ADLS) already existed, but the actual implementation did not and the service is marked as unsupported in the UI code.

I assume this documentation mismatch is an oversight. The ADLS documentation should probably be removed because that connector implementation does not exist either.

This PR adds an implementation for the GCS storage service connector. It tries to follow the general pattern of the S3 connector. It supports multiple project ids and bucket metrics via the google-cloud-monitoring package.

Connector UI screenshots:

Screenshot 2024-07-03 at 11 07 18 AM

Screenshot 2024-07-03 at 11 08 52 AM

Screenshot of storage container with structured data (parsed from manifest file):
Screenshot 2024-07-03 at 11 23 59 AM

Type of change:

  • New feature

Checklist:

  • I have read the CONTRIBUTING document.
  • My PR title is Fixes <issue-number>: <short explanation>
  • I have commented on my code, particularly in hard-to-understand areas.
  • For JSON Schema changes: I updated the migration scripts or explained why it is not needed.

Copy link
Contributor

github-actions bot commented Jul 3, 2024

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

@pmbrull pmbrull added the safe to test Add this label to run secure Github workflows on PRs label Jul 4, 2024
Copy link
Contributor

github-actions bot commented Jul 4, 2024

The Python checkstyle failed.

Please run make py_format and py_format_check in the root of your repository and commit the changes to this PR.
You can also use pre-commit to automate the Python code formatting.

You can install the pre-commit hooks with make install_test precommit_install.

Copy link
Contributor

github-actions bot commented Jul 4, 2024

Jest test Coverage

UI tests summary

Lines Statements Branches Functions
Coverage: 65%
65.41% (36028/55081) 41.8% (14179/33924) 43.9% (4386/9991)

@pmbrull
Copy link
Collaborator

pmbrull commented Jul 4, 2024

Hi @MChamberlin thanks for the PR. Could you please check the python checkstyle? a make py_format might do the trick. Thanks

Copy link
Contributor

github-actions bot commented Jul 9, 2024

Phylum OSS Supply Chain Risk Analysis - SUCCESS

The Phylum risk analysis is complete and has passed the active policy.

View this project in the Phylum UI

pmbrull
pmbrull previously approved these changes Jul 9, 2024
Copy link
Collaborator

@pmbrull pmbrull left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great PR, thanks @MChamberlin

Copy link

sonarcloud bot commented Jul 10, 2024

Copy link

sonarcloud bot commented Jul 10, 2024

Copy link
Collaborator

@pmbrull pmbrull left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR

@pmbrull pmbrull merged commit d757aa9 into open-metadata:main Jul 10, 2024
25 of 27 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backend e2e:Integration Ingestion safe to test Add this label to run secure Github workflows on PRs UI UI specific issues
Projects
None yet
Development

Successfully merging this pull request may close these issues.

GCS Storage Container support
2 participants