Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DQ base workflow #13454

Merged
merged 17 commits into from
Oct 6, 2023
Merged

DQ base workflow #13454

merged 17 commits into from
Oct 6, 2023

Conversation

pmbrull
Copy link
Collaborator

@pmbrull pmbrull commented Oct 5, 2023

Describe your changes:

Fixes

I worked on ... because ...

Type of change:

  • Bug fix
  • Improvement
  • New feature
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation

Checklist:

  • I have read the CONTRIBUTING document.
  • My PR title is Fixes <issue-number>: <short explanation>
  • I have commented on my code, particularly in hard-to-understand areas.
  • For JSON Schema changes: I updated the migration scripts or explained why it is not needed.

@pmbrull pmbrull requested a review from a team as a code owner October 5, 2023 14:03
@github-actions github-actions bot added Ingestion safe to test Add this label to run secure Github workflows on PRs labels Oct 5, 2023
@github-actions
Copy link
Contributor

github-actions bot commented Oct 5, 2023

The Python checkstyle failed.

Please run make py_format and py_format_check in the root of your repository and commit the changes to this PR.
You can also use pre-commit to automate the Python code formatting.

You can install the pre-commit hooks with make install_test precommit_install.

@github-actions
Copy link
Contributor

github-actions bot commented Oct 5, 2023

The Python checkstyle failed.

Please run make py_format and py_format_check in the root of your repository and commit the changes to this PR.
You can also use pre-commit to automate the Python code formatting.

You can install the pre-commit hooks with make install_test precommit_install.

@github-actions
Copy link
Contributor

github-actions bot commented Oct 5, 2023

The Python checkstyle failed.

Please run make py_format and py_format_check in the root of your repository and commit the changes to this PR.
You can also use pre-commit to automate the Python code formatting.

You can install the pre-commit hooks with make install_test precommit_install.

TeddyCr
TeddyCr reviewed Oct 6, 2023
View reviewed changes
ingestion/src/metadata/data_quality/processor/test_case_runner.py
Comment on lines +66 to +70
# First, create the executable test suite if it does not exist yet
# This could happen if the process is executed from YAML and not the UI
if record.executable_test_suite:
# We pass the test suite request to the sink
return Either(right=record.executable_test_suite)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wouldn't this be handle in source/test_suite.py ln 172? If we return this in this step wouldn't we be just passing the testSuite to the sink?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes exactly that. In the source we return 2 possibilities:

  1. First, if needed, the executable test suite to be created. In that case, the processor does not need to do anything, just pass it to the sink to get it created
  2. Then, pick up the test cases (if any) and return them with the table. Now it's where the processor needs to get the cases from the CLI, execute them etc.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so in the case where we have a table without executable test suite, the source will first return the suite to be created, and then the rest of the logic will happen

TeddyCr
TeddyCr reviewed Oct 6, 2023
View reviewed changes
Copy link
Contributor

@TeddyCr TeddyCr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left one comment. 😊

@sonarcloud
Copy link

sonarcloud bot commented Oct 6, 2023

[open-metadata-airflow-apis] Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 0 Code Smells

0.0% 0.0% Coverage
0.0% 0.0% Duplication

@sonarcloud
Copy link

sonarcloud bot commented Oct 6, 2023

[open-metadata-ingestion] Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 1 Code Smell

78.1% 78.1% Coverage
0.0% 0.0% Duplication

@pmbrull pmbrull merged commit aed9e38 into open-metadata:main Oct 6, 2023
17 of 21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@TeddyCr TeddyCr TeddyCr approved these changes

Assignees
No one assigned
Labels
Ingestion safe to test Add this label to run secure Github workflows on PRs
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@pmbrull @TeddyCr