Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix #12770 - Cleanup DL structure & Readers & Python 3.8 #12776

Merged
merged 12 commits into from
Aug 9, 2023
Merged

Fix #12770 - Cleanup DL structure & Readers & Python 3.8 #12776

merged 12 commits into from
Aug 9, 2023

Conversation

pmbrull
Copy link
Collaborator

@pmbrull pmbrull commented Aug 7, 2023 •
edited
Loading

Describe your changes:

Fixes #12770

  • Reorganize datalake methods
  • Centralize DF readers and unify signatures
  • Reuse methods when needed
  • Add tests

Moreover, since we are planning on removing 3.7 for 1.2 (#12082), here we are already bumping the minimum version to use singledispatchmethod, only available in 3.8. We'll close the deprecation issue in another PR after validating all the comments there.

Type of change:

  • Bug fix
  • Improvement
  • New feature
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation

Checklist:

  • I have read the CONTRIBUTING document.
  • My PR title is Fixes <issue-number>: <short explanation>
  • I have commented on my code, particularly in hard-to-understand areas.
  • For JSON Schema changes: I updated the migration scripts or explained why it is not needed.

@pmbrull pmbrull requested a review from a team as a code owner August 7, 2023 14:35
@github-actions github-actions bot added Ingestion backend safe to test Add this label to run secure Github workflows on PRs labels Aug 7, 2023
@pmbrull pmbrull changed the title Fix #12770 - Cleanup DL structure & Readers Fix #12770 - Cleanup DL structure & Readers & Python 3.8 Aug 7, 2023
@pmbrull pmbrull temporarily deployed to test August 7, 2023 14:58 — with GitHub Actions Inactive
@pmbrull pmbrull temporarily deployed to test August 7, 2023 14:58 — with GitHub Actions Inactive
@pmbrull pmbrull temporarily deployed to test August 7, 2023 14:58 — with GitHub Actions Inactive
@pmbrull pmbrull temporarily deployed to test August 7, 2023 14:58 — with GitHub Actions Inactive
@pmbrull pmbrull temporarily deployed to test August 8, 2023 08:01 — with GitHub Actions Inactive
@pmbrull pmbrull temporarily deployed to test August 8, 2023 08:01 — with GitHub Actions Inactive
@pmbrull pmbrull temporarily deployed to test August 8, 2023 08:01 — with GitHub Actions Inactive
@pmbrull pmbrull temporarily deployed to test August 8, 2023 08:01 — with GitHub Actions Inactive
@pmbrull pmbrull temporarily deployed to test August 8, 2023 08:01 — with GitHub Actions Inactive
@pmbrull pmbrull requested a review from a team August 8, 2023 08:06
@pmbrull pmbrull temporarily deployed to test August 8, 2023 09:06 — with GitHub Actions Inactive
@pmbrull pmbrull temporarily deployed to test August 8, 2023 09:06 — with GitHub Actions Inactive
@pmbrull pmbrull temporarily deployed to test August 8, 2023 09:06 — with GitHub Actions Inactive
@pmbrull pmbrull temporarily deployed to test August 8, 2023 09:06 — with GitHub Actions Inactive
@pmbrull pmbrull requested a review from harshach August 8, 2023 17:25
ayush-shah
ayush-shah previously approved these changes Aug 9, 2023
View reviewed changes
@ayush-shah ayush-shah temporarily deployed to test August 9, 2023 07:39 — with GitHub Actions Inactive
@ayush-shah ayush-shah temporarily deployed to test August 9, 2023 07:39 — with GitHub Actions Inactive
@ayush-shah ayush-shah temporarily deployed to test August 9, 2023 07:39 — with GitHub Actions Inactive
@github-actions github-actions bot added the devops label Aug 9, 2023
@sonarcloud
Copy link

sonarcloud bot commented Aug 9, 2023

[open-metadata-airflow-apis] Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 0 Code Smells

No Coverage information No Coverage information
No Duplication information No Duplication information

@sonarcloud
Copy link

sonarcloud bot commented Aug 9, 2023 •
edited
Loading

[open-metadata-ingestion] Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 0 Code Smells

79.9% 79.9% Coverage
0.0% 0.0% Duplication

@ayush-shah ayush-shah merged commit e97d4be into open-metadata:main Aug 9, 2023
21 of 25 checks passed
harshach pushed a commit that referenced this pull request Aug 9, 2023
@pmbrull @harshach
Fix #12770 - Cleanup DL structure & Readers & Python 3.8 (#12776)
54b3bc2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ayush-shah ayush-shah ayush-shah approved these changes

@harshach harshach Awaiting requested review from harshach harshach is a code owner

@akash-jain-10 akash-jain-10 Awaiting requested review from akash-jain-10 akash-jain-10 is a code owner

Assignees
No one assigned
Labels
backend Backward-Incompatible-Change devops Ingestion safe to test Add this label to run secure Github workflows on PRs
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Cleanup DL structure
3 participants
@pmbrull @harshach @ayush-shah