-
Notifications
You must be signed in to change notification settings - Fork 1.1k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
1 parent
88c8fb4
commit e51900a
Showing
8 changed files
with
205 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
65 changes: 65 additions & 0 deletions
65
openmetadata-docs/content/v1.6.x-SNAPSHOT/connectors/pipeline/stitch/index.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,65 @@ | ||
--- | ||
title: Stitch | ||
slug: /connectors/pipeline/stitch | ||
collate: true | ||
--- | ||
|
||
{% connectorDetailsHeader | ||
name="Stitch" | ||
stage="PROD" | ||
platform="Collate" | ||
availableFeatures=["Pipelines", "Lineage"] | ||
unavailableFeatures=["Owners", "Tags", "Pipeline Status"] | ||
/ %} | ||
|
||
|
||
In this section, we provide guides and references to use the Stitch connector. | ||
|
||
Configure and schedule Stitch metadata and profiler workflows from the OpenMetadata UI: | ||
|
||
- [Requirements](#requirements) | ||
- [Metadata Ingestion](#metadata-ingestion) | ||
|
||
{% partial file="/v1.6/connectors/ingestion-modes-tiles.md" variables={yamlPath: "/connectors/pipeline/stitch/yaml"} /%} | ||
|
||
## Requirements | ||
|
||
To extract metadata from Stitch, User first need to crate API crednetials: | ||
- `Token`: Token to access Stitch metadata. | ||
|
||
|
||
## Metadata Ingestion | ||
|
||
{% partial | ||
file="/v1.6/connectors/metadata-ingestion-ui.md" | ||
variables={ | ||
connector: "Stitch", | ||
selectServicePath: "/images/v1.6/connectors/stitch/select-service.png", | ||
addNewServicePath: "/images/v1.6/connectors/stitch/add-new-service.png", | ||
serviceConnectionPath: "/images/v1.6/connectors/stitch/service-connection.png", | ||
} | ||
/%} | ||
|
||
{% stepsContainer %} | ||
{% extraContent parentTagName="stepsContainer" %} | ||
|
||
#### Connection Details | ||
|
||
- **hostPort**: This parameter specifies the network location where your Stitch instance is accessible, combining both the hostname. It is based on the account and region where user has hosted his/her data pipelines. More about this you can check [here](https://www.stitchdata.com/docs/developers/import-api/api#base-urls) | ||
|
||
- **token**: Token to get access to Stitch metadata. This token is created by user after logging into stitch console. More about this, please check [here](https://www.stitchdata.com/docs/developers/import-api/guides/quick-start#obtain-api-credentials) | ||
|
||
|
||
{% /extraContent %} | ||
|
||
{% partial file="/v1.6/connectors/test-connection.md" /%} | ||
|
||
{% partial file="/v1.6/connectors/pipeline/configure-ingestion.md" /%} | ||
|
||
{% partial file="/v1.6/connectors/ingestion-schedule-and-deploy.md" /%} | ||
|
||
{% /stepsContainer %} | ||
|
||
By successfully completing these steps, the lineage information for the service will be displayed. | ||
|
||
{% partial file="/v1.6/connectors/troubleshooting.md" /%} |
114 changes: 114 additions & 0 deletions
114
openmetadata-docs/content/v1.6.x-SNAPSHOT/connectors/pipeline/stitch/yaml.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,114 @@ | ||
--- | ||
title: Run the Stitch Connector Externally | ||
slug: /connectors/pipeline/stitch/yaml | ||
collate: true | ||
--- | ||
|
||
{% connectorDetailsHeader | ||
name="Stitch" | ||
stage="PROD" | ||
platform="Collate" | ||
availableFeatures=["Pipelines", "Lineage"] | ||
unavailableFeatures=["Owners", "Tags", "Pipeline Status"] | ||
/ %} | ||
|
||
|
||
|
||
In this section, we provide guides and references to use the Stitch connector. | ||
|
||
Configure and schedule Stitch metadata and profiler workflows from the OpenMetadata UI: | ||
|
||
- [Requirements](#requirements) | ||
- [Metadata Ingestion](#metadata-ingestion) | ||
|
||
{% partial file="/v1.6/connectors/ingestion-modes-tiles.md" variables={yamlPath: "/connectors/pipeline/stitch/yaml"} /%} | ||
|
||
## Requirements | ||
|
||
To extract metadata from Stitch, User first need to crate API crednetials: | ||
- `Token`: Token to access Stitch metadata. | ||
|
||
|
||
### Python Requirements | ||
|
||
{% partial file="/v1.6/connectors/python-requirements.md" /%} | ||
|
||
To run the Stitch ingestion, you will need to install: | ||
|
||
```bash | ||
pip3 install "openmetadata-ingestion[stitch]" | ||
``` | ||
|
||
## Metadata Ingestion | ||
|
||
All connectors are defined as JSON Schemas. | ||
[Here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/pipeline/stitchConnection.json) | ||
you can find the structure to create a connection to Stitch. | ||
|
||
In order to create and run a Metadata Ingestion workflow, we will follow | ||
the steps to create a YAML configuration able to connect to the source, | ||
process the Entities if needed, and reach the OpenMetadata server. | ||
|
||
The workflow is modeled around the following | ||
[JSON Schema](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/workflow.json) | ||
|
||
### 1. Define the YAML Config | ||
|
||
This is a sample config for Stitch: | ||
|
||
{% codePreview %} | ||
|
||
{% codeInfoContainer %} | ||
|
||
#### Source Configuration - Service Connection | ||
|
||
{% codeInfo srNumber=1 %} | ||
|
||
**hostPort**: The hostname or IP address with the REST API enabled eg.`https://api.stitchdata.com` | ||
|
||
{% /codeInfo %} | ||
|
||
{% codeInfo srNumber=2 %} | ||
|
||
**token**: Token to get access to Stitch metadata. | ||
|
||
{% /codeInfo %} | ||
|
||
|
||
{% partial file="/v1.6/connectors/yaml/pipeline/source-config-def.md" /%} | ||
|
||
{% partial file="/v1.6/connectors/yaml/ingestion-sink-def.md" /%} | ||
|
||
{% partial file="/v1.6/connectors/yaml/workflow-config-def.md" /%} | ||
|
||
{% /codeInfoContainer %} | ||
|
||
{% codeBlock fileName="filename.yaml" %} | ||
|
||
|
||
```yaml {% isCodeBlock=true %} | ||
source: | ||
type: stitch | ||
serviceName: stitch_data | ||
serviceConnection: | ||
config: | ||
type: Stitch | ||
``` | ||
```yaml {% srNumber=1 %} | ||
hostPort: "https://api.stitchdata.com" | ||
``` | ||
```yaml {% srNumber=2 %} | ||
token: "token" | ||
``` | ||
{% partial file="/v1.6/connectors/yaml/pipeline/source-config.md" /%} | ||
{% partial file="/v1.6/connectors/yaml/ingestion-sink.md" /%} | ||
{% partial file="/v1.6/connectors/yaml/workflow-config.md" /%} | ||
{% /codeBlock %} | ||
{% /codePreview %} | ||
{% partial file="/v1.6/connectors/yaml/ingestion-cli.md" /%} |
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+116 KB
openmetadata-docs/images/v1.6/connectors/stitch/service-connection.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
21 changes: 21 additions & 0 deletions
21
openmetadata-ui/src/main/resources/ui/public/locales/en-US/Pipeline/Stitch.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,21 @@ | ||
# Stitch | ||
|
||
In this section, we provide guides and references to use the Stitch connector. | ||
|
||
## Requirements: | ||
|
||
To extract metadata from Stitch, User first need to crate API crednetials: | ||
- `Token`: Token to access Stitch metadata. | ||
|
||
## Connection Details | ||
|
||
$$section | ||
### Host Port $(id="hostPort") | ||
This parameter specifies the network location where your Stitch instance is accessible, combining both the hostname. | ||
It is based on the account and region where user has hosted his/her data pipelines. More about this you can check [here](https://www.stitchdata.com/docs/developers/import-api/api#base-urls) | ||
$$ | ||
|
||
$$section | ||
### Token $(id="token") | ||
Token to get access to Stitch metadata. This token is created by user after logging into stitch console. More about this, please check [here](https://www.stitchdata.com/docs/developers/import-api/guides/quick-start#obtain-api-credentials) | ||
$$ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters