feat: OpenLineage integration #15317

mgorsk1 · 2024-02-22T13:10:16Z

Describe your changes:

Adds connector for collecting lineage using OpenLineage standard. Co-authored with @dechoma

I worked on adding OpenLineage connector because it's a widely used standard for harvesting lineage information and OpenMetadata was missing such integration.

This connector is targeted to users:

who use (or plan to use) OpenLineage to emit lineage information to kafka topic (this is one of two ways OL recommends emmiting lineage information)
use OpenMetadata to store metadata within their projects and needs to enrich it with lineage information collected using OL

Type of change:

Checklist:

I have read the CONTRIBUTING document.
My PR title is Fixes <issue-number>: <short explanation>
I have commented on my code, particularly in hard-to-understand areas.
For JSON Schema changes: I updated the migration scripts or explained why it is not needed.

github-actions · 2024-02-22T13:10:46Z

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

github-actions · 2024-02-22T13:12:31Z

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

mgorsk1 · 2024-03-11T20:48:15Z

all checks are green, can you re-review @TeddyCr @ulixius9 ? I wouldn't want this to hang much longer considering how many rebases this already needed

sonarcloud · 2024-03-11T21:12:20Z

Quality Gate passed for 'open-metadata-ui'

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

sonarcloud · 2024-03-11T21:13:20Z

Quality Gate passed for 'open-metadata-ingestion'

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
70.7% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

pmbrull

Thanks @mgorsk1! I think there were a couple of comments pending from the last review. Left two quick ones and we'll be good to go.

pmbrull · 2024-03-12T07:12:59Z

ingestion/src/metadata/ingestion/source/pipeline/openlineage/metadata.py

+        namespace = run_facet["facets"]["parent"]["job"]["namespace"]
+        name = run_facet["facets"]["parent"]["job"]["name"]


+1 on this, will make maintaining and evolving things easier

From previous comment:

In the OpenLineageEvent we have defined run_facet of generic Dict type do you think we can be more specific and define the exact model also to avoid keyerror in case when we did not receive the response in expected format.

ingestion/src/metadata/ingestion/source/pipeline/openlineage/metadata.py

ingestion/src/metadata/ingestion/source/pipeline/openlineage/models.py

ingestion/src/metadata/ingestion/source/pipeline/openlineage/metadata.py

ulixius9 · 2024-03-12T07:35:57Z

ingestion/src/metadata/ingestion/source/pipeline/openlineage/metadata.py

+        pipeline_name = self.get_pipeline_name(pipeline_details)
+        try:
+            description = f"""```json
+            {json.dumps(pipeline_details.run_facet, indent=4).strip()}```"""


ulixius9 · 2024-03-12T07:37:01Z

ingestion/src/metadata/utils/fqn.py

@@ -192,17 +192,34 @@ def _(

 @fqn_build_registry.add(DatabaseSchema)
 def _(
-    _: Optional[OpenMetadata],  # ES Search not enabled for Schemas
+    metadata: Optional[OpenMetadata],  # ES Search not enabled for Schemas


review if this impacts other parts of code and if this change can be avoided!

I think it's in line with fqn for Table and in general tests did not fail after this change

ulixius9 · 2024-03-12T07:37:11Z

ingestion/src/metadata/utils/fqn.py

@@ -574,6 +591,43 @@ def build_es_fqn_search_string(
    return fqn_search_string


+def search_database_schema_from_es(


same for this

pmbrull · 2024-03-12T07:40:13Z

let's go over the pending topics on a separate PR

mgorsk1 temporarily deployed to test February 22, 2024 13:10 — with GitHub Actions Inactive

github-actions bot added e2e:Integration e2e:Observability e2e:DataAssets documentation Improvements or additions to documentation UI UI specific issues Ingestion backend devops and removed e2e:Integration e2e:Observability e2e:DataAssets labels Feb 22, 2024

mgorsk1 force-pushed the feat/openlineage branch from 826db91 to 51d908b Compare February 22, 2024 13:12

mgorsk1 temporarily deployed to test February 22, 2024 13:12 — with GitHub Actions Inactive

fix tests 2

442c470

mgorsk1 temporarily deployed to test March 11, 2024 18:36 — with GitHub Actions Inactive

mgorsk1 temporarily deployed to test March 11, 2024 18:37 — with GitHub Actions Inactive

Merge branch 'main' into feat/openlineage

a3cebd5

mgorsk1 temporarily deployed to test March 11, 2024 20:48 — with GitHub Actions Inactive

pmbrull reviewed Mar 12, 2024

View reviewed changes

ingestion/src/metadata/ingestion/source/pipeline/openlineage/metadata.py Show resolved Hide resolved

ulixius9 approved these changes Mar 12, 2024

View reviewed changes

pmbrull merged commit 98850ab into open-metadata:main Mar 12, 2024
32 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: OpenLineage integration #15317

feat: OpenLineage integration #15317

mgorsk1 commented Feb 22, 2024 •

edited

Loading

github-actions bot commented Feb 22, 2024

github-actions bot commented Feb 22, 2024

mgorsk1 commented Mar 11, 2024

sonarcloud bot commented Mar 11, 2024

sonarcloud bot commented Mar 11, 2024

pmbrull left a comment

pmbrull Mar 12, 2024 •

edited

Loading

ulixius9 Mar 12, 2024

ulixius9 Mar 12, 2024 •

edited

Loading

mgorsk1 Mar 12, 2024

ulixius9 Mar 12, 2024

pmbrull commented Mar 12, 2024

		namespace = run_facet["facets"]["parent"]["job"]["namespace"]
		name = run_facet["facets"]["parent"]["job"]["name"]

		@@ -574,6 +591,43 @@ def build_es_fqn_search_string(
		return fqn_search_string


		def search_database_schema_from_es(

feat: OpenLineage integration #15317

feat: OpenLineage integration #15317

Conversation

mgorsk1 commented Feb 22, 2024 • edited Loading

Describe your changes:

Type of change:

Checklist:

github-actions bot commented Feb 22, 2024

github-actions bot commented Feb 22, 2024

mgorsk1 commented Mar 11, 2024

sonarcloud bot commented Mar 11, 2024

Quality Gate passed for 'open-metadata-ui'

sonarcloud bot commented Mar 11, 2024

Quality Gate passed for 'open-metadata-ingestion'

pmbrull left a comment

Choose a reason for hiding this comment

pmbrull Mar 12, 2024 • edited Loading

Choose a reason for hiding this comment

ulixius9 Mar 12, 2024

Choose a reason for hiding this comment

ulixius9 Mar 12, 2024 • edited Loading

Choose a reason for hiding this comment

mgorsk1 Mar 12, 2024

Choose a reason for hiding this comment

ulixius9 Mar 12, 2024

Choose a reason for hiding this comment

pmbrull commented Mar 12, 2024

mgorsk1 commented Feb 22, 2024 •

edited

Loading

pmbrull Mar 12, 2024 •

edited

Loading

ulixius9 Mar 12, 2024 •

edited

Loading