-
Notifications
You must be signed in to change notification settings - Fork 1.1k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge branch 'main' into feature-import-export-custom-property
- Loading branch information
Showing
42 changed files
with
901 additions
and
649 deletions.
There are no files selected for viewing
45 changes: 45 additions & 0 deletions
45
...cs/content/v1.5.x/how-to-guides/getting-started/day-1/database-service-setup.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,45 @@ | ||
--- | ||
title: Database service setup | ||
slug: /how-to-guides/getting-started/day-1/database-service-setup | ||
collate: true | ||
--- | ||
|
||
## Setting Up a Database Service for Metadata Extraction | ||
|
||
You can quickly set up a database service for metadata extraction in OpenMetadata SaaS. Below is an example of how to configure a connection using the `Snowflake` Connector: | ||
|
||
1. ### Log in to OpenMetadata SaaS | ||
- Navigate to **Settings > Services > Databases**. | ||
- Click on **Add New Service**. | ||
|
||
{% image | ||
src="/images/v1.5/getting-started/add-service.png" | ||
alt="Adding Database Service" | ||
caption="Adding Database Service" /%} | ||
|
||
2. ### Select Database Type | ||
|
||
- Choose the database type for your service. In this example, select `Snowflake`. | ||
- Enter the required details, such as the **Name** and **Description**, to identify the database. | ||
|
||
{% image | ||
src="/images/v1.5/getting-started/select-service.png" | ||
alt="Selecting Database Service" | ||
caption="Selecting Database Service" /%} | ||
|
||
3. ### Enter Connection Details | ||
- Provide the necessary connection parameters, such as hostname, port, credentials, etc. | ||
- The side panel offers guidance with available documentation, and you can also refer to the specific `Snowflake` Connector [documentation](/connectors)for more information. | ||
|
||
{% image | ||
src="/images/v1.5/getting-started/configure-connector.png" | ||
alt="Updating Connection Details" | ||
caption="Updating Connection Details" /%} | ||
|
||
4. ### Test the Connection | ||
Click Test Connection to verify the setup. This will check if OpenMetadata can reach the Snowflake service. | ||
|
||
{% image | ||
src="/images/v1.5/getting-started/test-connection.png" | ||
alt="Verifying the Test Connection" | ||
caption="Verifying the Test Connection" /%} |
181 changes: 181 additions & 0 deletions
181
openmetadata-docs/content/v1.5.x/how-to-guides/getting-started/day-1/index.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,181 @@ | ||
--- | ||
title: Day 1 | ||
slug: /how-to-guides/getting-started/day-1 | ||
collate: true | ||
--- | ||
|
||
# Getting Started: Day 1 | ||
|
||
Get started with your OpenMetadata service in a few simple steps: | ||
|
||
1. Set up a Data Connector: Connect your data sources to begin collecting metadata. | ||
2. Ingest Metadata: Run the metadata ingestion process to gather and push data insights. | ||
3. Invite Users: Add team members to collaborate and manage metadata together. | ||
4. Explore the Features: Dive into OpenMetadata's extensive feature set to unlock the full potential of your data. | ||
|
||
**Ready to begin? Let's get started!** | ||
|
||
## Requirements | ||
|
||
You should receive your initial OpenMetadata credentials from OpenMetadata support, or from your existing OpenMetadata admin. For any questions, please reach us via [slack](https://slack.open-metadata.org/). The following steps will provide initial set up information, with links to more detailed documentation. | ||
|
||
## Step 1: Set up a Data Connector | ||
|
||
Once you have logged into your OpenMetadata instance, set up a data connector to start ingesting metadata. OpenMetadata provides [80+ turnkey connectors](/connectors) for a wide range of services, including: | ||
|
||
- Databases | ||
- Dashboards | ||
- Messaging services | ||
- Pipelines | ||
- ML models | ||
- Storage services | ||
- Other metadata services | ||
|
||
For [custom data sources](/connectors/custom-connectors), metadata ingestion can also be set up via API. | ||
|
||
There's two options on how to set up a data connector: | ||
1. **Run the connector in OpenMetadata**: In this scenario, an IP will be provided when you add the service. You must allow access to this IP in your data sources. | ||
|
||
{% tilesContainer %} | ||
{% tile | ||
title="Run the connector in OpenMetadata" | ||
description="Guide to start ingesting metadata seamlessly from your data sources." | ||
link="/how-to-guides/getting-started/day-1/database-service-setup" | ||
icon="discovery" | ||
/%} | ||
{% /tilesContainer %} | ||
|
||
2. **Run the connector in your infrastructure or on a local machine:**: This hybrid model allows organizations to run metadata ingestion components within their own infrastructure. This ensures that OpenMetadata's managed service doesn't need direct access to the underlying data. Only metadata is collected locally and securely transmitted to the platform, maintaining data privacy and security. | ||
|
||
## Step 2: Ingest Metadata | ||
|
||
Once the connector is set up, configure a [metadata ingestion pipeline](/how-to-guides/admin-guide/how-to-ingest-metadata) | ||
to import metadata into OpenMetadata on a regular schedule. | ||
|
||
- Navigate to **Settings > Services > Databases** and select the service you added. | ||
- Go to the **Ingestion** tab and click **Add Metadata Ingestion**. | ||
|
||
{% image | ||
src="/images/v1.5/getting-started/add-ingestion.png" | ||
alt="Adding Ingestion" | ||
caption="Adding Ingestion" /%} | ||
|
||
- Configure any required settings or filters for the ingestion process. Documentation is available in the side panel for reference. | ||
|
||
{% image | ||
src="/images/v1.5/getting-started/ingestion-config.png" | ||
alt="Configure Ingestion" | ||
caption="Configure Ingestion" /%} | ||
|
||
- Schedule the pipeline to ingest metadata at regular intervals. | ||
|
||
{% image | ||
src="/images/v1.5/getting-started/schedule-ingesgtion.png" | ||
alt="Schedule Ingestion" | ||
caption="Schedule Ingestion" /%} | ||
|
||
- In addition to metadata ingestion, you can set up pipelines for lineage, profiler data, or dbt information. | ||
- Once the metadata ingestion is completed, the ingested data assets can be viewed under the **Explore** section in the main menu. | ||
|
||
{% image | ||
src="/images/v1.5/getting-started/explore-tab.png" | ||
alt="Ingested Data" | ||
caption="Ingested Data under Explore Tab" /%} | ||
|
||
- You can repeat these steps to ingest metadata from other data sources as needed. | ||
|
||
## Step 3: Invite Users | ||
|
||
After metadata has been ingested into OpenMetadata, you can [invite users](/how-to-guides/admin-guide/teams-and-users/invite-users) to collaborate on the data and assign different roles. | ||
|
||
- Go to **Settings > Team & User Management > Users**. | ||
|
||
{% image | ||
src="/images/v1.5/getting-started/users.png" | ||
alt="Users Navigation" | ||
caption="Users Navigation" /%} | ||
|
||
- Click **Add User**, then enter their email and other required details to grant access to the platform. | ||
|
||
{% image | ||
src="/images/v1.5/getting-started/add-users.png" | ||
alt="Adding New User" | ||
height="750px" | ||
caption="Adding New User" /%} | ||
|
||
- Organize users into different Teams, and assign them specific Roles. | ||
- Users inherit access permissions from their assigned teams and roles. | ||
- Admin access can be granted to users who need full access to all settings, including the ability to invite new users. | ||
|
||
{% image | ||
src="/images/v1.5/getting-started/update-user.png" | ||
alt="Users Profile" | ||
caption="Users Profile" /%} | ||
|
||
- New users will receive an email invitation to set up their accounts. | ||
|
||
## Step 4: Explore Features of OpenMetadata | ||
|
||
OpenMetadata offers a comprehensive platform for data teams to: | ||
- Break down data silos | ||
- Securely share data assets across various sources | ||
- Foster collaboration around trusted data | ||
- Establish a documentation-first data culture within your organization | ||
|
||
Explore these features and unlock the full potential of your data using OpenMetadata. | ||
|
||
{% tilesContainer %} | ||
{% tile | ||
title="Data Discovery" | ||
description="Discover the right data assets to make timely business decisions." | ||
link="/how-to-guides/data-discovery" | ||
icon="discovery" | ||
/%} | ||
{% tile | ||
title="Data Collaboration" | ||
description="Foster data team collaboration to enhance data understanding." | ||
link="/how-to-guides/data-collaboration" | ||
icon="collaboration" | ||
/%} | ||
{% tile | ||
title="Data Quality & Observability" | ||
description="Trust your data with quality tests & monitor the health of your data systems." | ||
link="/how-to-guides/data-quality-observability" | ||
icon="observability" | ||
/%} | ||
{% tile | ||
title="Data Lineage" | ||
description="Trace the path of data across tables, pipelines, and dashboards." | ||
link="/how-to-guides/data-lineage" | ||
icon="lineage" | ||
/%} | ||
{% tile | ||
title="Data Insights" | ||
description="Define KPIs and set goals to proactively hone the data culture of your company." | ||
link="/how-to-guides/data-insights" | ||
icon="discovery" | ||
/%} | ||
{% tile | ||
title="Data Governance" | ||
description="Enhance your data platform governance using OpenMetadata." | ||
link="/how-to-guides/data-governance" | ||
icon="governance" | ||
/%} | ||
{% /tilesContainer %} | ||
|
||
## Deep Dive into OpenMetadata: Guides for Admins and Data Users | ||
|
||
{% tilesContainer %} | ||
{% tile | ||
title="Admin Guide" | ||
description="Admin users can get started with OpenMetadata with just three quick and easy steps & know-it-all with the advanced guides." | ||
link="/how-to-guides/admin-guide" | ||
icon="administration" | ||
/%} | ||
{% tile | ||
title="Guide for Data Users" | ||
description="Get to know the basics of OpenMetadata and about the data assets that you can explore in the all-in-one platform." | ||
link="/how-to-guides/guide-for-data-users" | ||
icon="steward" | ||
/%} | ||
{% /tilesContainer %} |
71 changes: 71 additions & 0 deletions
71
openmetadata-docs/content/v1.5.x/how-to-guides/getting-started/index.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,71 @@ | ||
--- | ||
title: Getting Started with OpenMetadata for Data cataloging | ||
slug: /how-to-guides/getting-started | ||
--- | ||
|
||
# Getting Started | ||
|
||
Welcome to OpenMetadata's unified platform for data discovery, observability, and governance. Our platform centralizes all data context to help you build high-quality data and AI assets. This guide provides the necessary information to set up your OpenMetadata environment in 30 minutes. | ||
|
||
## How Does Collate Work? | ||
|
||
OpenMetadata is designed to support both technical and non-technical data practitioners across various use cases, including data discovery, lineage, observability, quality, collaboration, governance, and insights. | ||
|
||
The platform includes a library of 80+ turnkey connectors to easily ingest metadata from sources such as data warehouses, data lakes, streaming platforms, dashboards, and ML models. For custom data sources, APIs are available to streamline metadata ingestion. Metadata from these sources is organized into a Unified Metadata Graph, providing a single, comprehensive source of truth for your entire data estate. | ||
|
||
This centralized metadata is accessible through a unified user interface, eliminating the need for practitioners to switch between multiple catalogs, quality, or governance tools. OpenMetadata can also be extended with applications, such as AI-driven productivity tools like MetaPilot, or through custom-built workflows that integrate the platform with existing systems. | ||
The platform’s native collaboration features support shared workflows, enabling different teams—data platform engineers, governance professionals, data scientists/analysts, and business users—to collaborate effectively in a single environment. | ||
|
||
## Key Features of OpenMetadata | ||
|
||
Before we get started, here’s a quick summary of some of Collate’s main features: | ||
|
||
### Discovery | ||
- Integrated catalog, data quality, and glossary | ||
- Natural language search, filtering, and faceting | ||
- 80+ turnkey data connectors | ||
|
||
### Lineage | ||
- Table and column-level lineage | ||
- Automated data estate mapping with APIs | ||
- Lineage layers, search capabilities | ||
- Governance and PII automation, with manual customization | ||
|
||
### Observability | ||
- Alerting and notifications | ||
- Incident management and third-party notifications | ||
- Pipeline monitoring, root cause analysis, and anomaly detection | ||
- Data profiler for performance insights | ||
|
||
### Quality | ||
- Table and column-level test cases | ||
- No-code and SQL-based data quality tests | ||
- Test suites, reporting, and dashboards | ||
- Quality lineage maps and widgets for data insights | ||
|
||
### Collaboration | ||
- Announcements, tasks, and team conversations | ||
- Slack/Teams integration for communication | ||
- Activity feed and team dashboards for tracking progress | ||
|
||
### Governance | ||
- Business glossary and classification tags | ||
- Automated PII classification and description generation | ||
|
||
### Insights | ||
- Data asset analytics and app usage metrics | ||
- Coverage KPIs and ownership dashboards | ||
- Data health and governance reports | ||
|
||
## Onboarding Journey | ||
|
||
In this section, you will find a series of guides to help you get started with OpenMetadata. These guides cover both basic and advanced features of the platform, organized in a step-by-step format and divided into milestones that can be achieved over the course of your onboarding process. | ||
|
||
{% tilesContainer %} | ||
{% tile | ||
title="Day 1: Connect your Data Sources and invite users" | ||
description="Discover the right data assets to make timely business decisions." | ||
link="/how-to-guides/getting-started/day-1" | ||
icon="discovery" | ||
/%} | ||
{% /tilesContainer %} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
45 changes: 45 additions & 0 deletions
45
...t/v1.6.x-SNAPSHOT/how-to-guides/getting-started/day-1/database-service-setup.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,45 @@ | ||
--- | ||
title: Database service setup | ||
slug: /how-to-guides/getting-started/day-1/database-service-setup | ||
collate: true | ||
--- | ||
|
||
## Setting Up a Database Service for Metadata Extraction | ||
|
||
You can quickly set up a database service for metadata extraction in OpenMetadata SaaS. Below is an example of how to configure a connection using the `Snowflake` Connector: | ||
|
||
1. ### Log in to OpenMetadata SaaS | ||
- Navigate to **Settings > Services > Databases**. | ||
- Click on **Add New Service**. | ||
|
||
{% image | ||
src="/images/v1.6/getting-started/add-service.png" | ||
alt="Adding Database Service" | ||
caption="Adding Database Service" /%} | ||
|
||
2. ### Select Database Type | ||
|
||
- Choose the database type for your service. In this example, select `Snowflake`. | ||
- Enter the required details, such as the **Name** and **Description**, to identify the database. | ||
|
||
{% image | ||
src="/images/v1.6/getting-started/select-service.png" | ||
alt="Selecting Database Service" | ||
caption="Selecting Database Service" /%} | ||
|
||
3. ### Enter Connection Details | ||
- Provide the necessary connection parameters, such as hostname, port, credentials, etc. | ||
- The side panel offers guidance with available documentation, and you can also refer to the specific `Snowflake` Connector [documentation](/connectors)for more information. | ||
|
||
{% image | ||
src="/images/v1.6/getting-started/configure-connector.png" | ||
alt="Updating Connection Details" | ||
caption="Updating Connection Details" /%} | ||
|
||
4. ### Test the Connection | ||
Click Test Connection to verify the setup. This will check if OpenMetadata can reach the Snowflake service. | ||
|
||
{% image | ||
src="/images/v1.6/getting-started/test-connection.png" | ||
alt="Verifying the Test Connection" | ||
caption="Verifying the Test Connection" /%} |
Oops, something went wrong.