Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add short descriptions of the UC connectors #954

Merged
merged 2 commits into from
Jul 25, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,9 @@

# Template Catalog

The template catalog provides descriptions of the [templates](/concepts/template) provided by Egeria, through the [Content Packs](/content-packs).
The template catalog provides descriptions of the [templates](/concepts/template) provided by Egeria, through the [Content Packs](/content-packs).

* [Templates for PostgreSQL](/catalog-templates/postgres-server-templates)
* [Templates for Unity Catalog](/catalog-templates/unity-catalog-templates)

--8<-- "snippets/abbr.md"
Original file line number Diff line number Diff line change
Expand Up @@ -12,4 +12,7 @@ The postgreSQL Server templates are provided through the [Core Content Pack](/co

## The templates in use

![PostgreSQL template in use](postgres-server-template-in-use.svg)
![PostgreSQL template in use](postgres-server-template-in-use.svg)


--8<-- "snippets/abbr.md"
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,7 @@

# Unity Catalog Templates

The templates for Unity Catalog are supplied through the [Core Content Pack](/content-packs/core-content-pack/overview).
The templates for Unity Catalog are supplied through the [Core Content Pack](/content-packs/core-content-pack/overview).


--8<-- "snippets/abbr.md"
4 changes: 2 additions & 2 deletions site/docs/concepts/placeholder.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ Content-Type: application/json
```
Below is an outline of the default template for cataloguing a PostgreSQL Database Server. This is represented as a [SoftwareServer](/types/0/0040-Software-Servers) asset, in includes a linked [SoftwareCapability](/types/0/0042-Software-Capabilities) for the [database manager (DBMS)](/types/0/0050-Applications-and-Processes). There is also a [connection](/concepts/connection) linked from the asset to define how to create a connector to the PostgreSQL Database Server. Notice that throughout the template, there are placeholder properties:

![PostgreSQL Database Server Template](/templates/postgres-server-catalog-template.svg)
![PostgreSQL Database Server Template](/catalog-templates/postgres-server-catalog-template.svg)

Below is an example of a call to create an asset using the template:

Expand All @@ -99,7 +99,7 @@ Content-Type: application/json
The picture below shows the resulting asset elements, linked back to the elements from the template using the [SourcedFrom](/types/0/0011-Managing-Referenceables).
Notice how the `{{serverName}}` placeholder property is used in each element to create a unique qualifiedName.

![PostgreSQL Database Server Template in use](/templates/postgres-server-template-in-use.svg)
![PostgreSQL Database Server Template in use](/catalog-templates/postgres-server-template-in-use.svg)

The placeholder properties can be used to make the templates easy to use, removing much of the repetitive creation of property values. The result is a consistent set of elements for the asset.

Expand Down
12 changes: 6 additions & 6 deletions site/docs/connectors/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,12 +72,12 @@ Files provide storage for many types of data. They are organizes into folders (

The Unity Catalog connectors provide a suite of function that integrates a Unity Catalog server into the open metadata ecosystem.

The [Unity Catalog Resource Connector](/connectors/unity-catalog/resource-connector) is a [digital resource connector](/concepts/digital-resource-connector) that acts as a Java client to the Unity Catalog Server REST API. It is used by the other Unity Catalog connectors.
The [Unity Catalog Server Synchronizer](/connectors/unity-catalog/sync-server-connector) is an [integration connector](/concepts/integration-connector) that exchanges details about the catalogs defined for a Unity Catalog Server.
The [Unity Catalog Inside Catalog Synchronizer](/connectors/unity-catalog/sync-catalog-connector) is an [integration connector](/concepts/integration-connector) that exchanges details about the resources (schemas, tables, volumes and functions) defined within a Catalog found in a Unity Catalog Server.
The [Unity Catalog Server Survey Service](/connectors/unity-catalog/server-survey-sevice) is a [Survey Action Service](/concepts/survey-action-service) that surveys the resources defined in a Unity Catalog Server.
The [Unity Catalog Inside Catalog Survey Service](/connectors/unity-catalog/catalog-survey-sevice) is a [Survey Action Service](/concepts/survey-action-service) that surveys the resources defined in a Catalog inside a Unity Catalog Server.
The [Unity Catalog Inside Schema Survey Service](/connectors/unity-catalog/schema-survey-sevice) is a [Survey Action Service](/concepts/survey-action-service) that surveys the resources defined in a Schema inside a Unity Catalog Server.
* The [Unity Catalog Resource Connector](/connectors/unity-catalog/resource-connector) is a [digital resource connector](/concepts/digital-resource-connector) that acts as a Java client to the Unity Catalog Server REST API. It is used by the other Unity Catalog connectors.
* The [Unity Catalog Server Synchronizer](/connectors/unity-catalog/sync-server-connector) is an [integration connector](/concepts/integration-connector) that exchanges details about the catalogs defined for a Unity Catalog Server.
* The [Unity Catalog Inside Catalog Synchronizer](/connectors/unity-catalog/sync-catalog-connector) is an [integration connector](/concepts/integration-connector) that exchanges details about the resources (schemas, tables, volumes and functions) defined within a Catalog found in a Unity Catalog Server.
* The [Unity Catalog Server Survey Service](/connectors/unity-catalog/server-survey-service) is a [Survey Action Service](/concepts/survey-action-service) that surveys the resources defined in a Unity Catalog Server.
* The [Unity Catalog Inside Catalog Survey Service](/connectors/unity-catalog/catalog-survey-service) is a [Survey Action Service](/concepts/survey-action-service) that surveys the resources defined in a Catalog inside a Unity Catalog Server.
* The [Unity Catalog Inside Schema Survey Service](/connectors/unity-catalog/schema-survey-service) is a [Survey Action Service](/concepts/survey-action-service) that surveys the resources defined in a Schema inside a Unity Catalog Server.

### Apache Kafka

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,8 @@

# OSS Unity Catalog Inside Catalog Survey Service

Survey of a catalog within an OSS Unity Catalog Server.
The Catalog Survey Service extracts information about the schemas, tables, functions and volumes managed in a specific catalog within an OSS Unity Catalog Server.

![Connector Operation](catalog-survey-service.svg)

--8<-- "snippets/abbr.md"
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
14 changes: 7 additions & 7 deletions site/docs/connectors/unity-catalog/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -100,7 +100,7 @@ The diagram below illustrates the mapping of the Unity Catalog metadata resource

![Type Mapping](unity-catalog-type-mapping.svg)

The templates that implement this mapping are described in [Unity Catalog Templates](/templates/unity-catalog-templates).
The templates that implement this mapping are described in [Unity Catalog Templates](/catalog-templates/unity-catalog-templates).

### Anchor design for Unity Catalog

Expand All @@ -124,12 +124,12 @@ The connectors shipped with Egeria are as follows:

| Connector Name | Connector Type | Purpose |
|----------------------------------------------------------------------------------------------|--------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| [Unity Catalog Resource Connector](/connector/unity-catalog/resource-connector) | [Digital Resource Connector](/concepts/digital-resource-connector) | Provides wrapper around Unity Catalog's REST API. |
| [Unity Catalog Server Survey](/connector/unity-catalog/server-survey-service) | [Survey Action Service](/concepts/survey-action-service) | Surveys the contents of a Unity Catalog Server. |
| [Unity Catalog Catalog Survey](/connector/unity-catalog/catalog-survey-service) | [Survey Action Service](/concepts/survey-action-service) | Surveys the contents of a Unity Catalog Catalog. |
| [Unity Catalog Schema Survey](/connector/unity-catalog/schema-survey-service) | [Survey Action Service](/concepts/survey-action-service) | Surveys the contents of a Unity Catalog Schema. |
| [Unity Catalog Server Synchronizer](/connector/unity-catalog/sync-server-connector) | [Integration Connector](/concepts/integration-connector) | Bootstraps the cataloguing of a Unity Catalog Server by retrieving the catalogs and configuring the Inside Catalog Connector (below). |
| [Unity Catalog Inside Catalog Synchronizer](/connector/unity-catalog/sync-catalog-connector) | [Integration Connector](/concepts/integration-connector) | Synchronizes the metadata describing a Unity Catalog Server's catalogs, schemas, tables, functions and volumes between Unity Catalog and the Open Metadata Ecosystem. |
| [Unity Catalog Resource Connector](/connectors/unity-catalog/resource-connector) | [Digital Resource Connector](/concepts/digital-resource-connector) | Provides wrapper around Unity Catalog's REST API. |
| [Unity Catalog Server Survey](/connectors/unity-catalog/server-survey-service) | [Survey Action Service](/concepts/survey-action-service) | Surveys the contents of a Unity Catalog Server. |
| [Unity Catalog Catalog Survey](/connectors/unity-catalog/catalog-survey-service) | [Survey Action Service](/concepts/survey-action-service) | Surveys the contents of a Unity Catalog Catalog. |
| [Unity Catalog Schema Survey](/connectors/unity-catalog/schema-survey-service) | [Survey Action Service](/concepts/survey-action-service) | Surveys the contents of a Unity Catalog Schema. |
| [Unity Catalog Server Synchronizer](/connectors/unity-catalog/sync-server-connector) | [Integration Connector](/concepts/integration-connector) | Bootstraps the cataloguing of a Unity Catalog Server by retrieving the catalogs and configuring the Inside Catalog Connector (below). |
| [Unity Catalog Inside Catalog Synchronizer](/connectors/unity-catalog/sync-catalog-connector) | [Integration Connector](/concepts/integration-connector) | Synchronizes the metadata describing a Unity Catalog Server's catalogs, schemas, tables, functions and volumes between Unity Catalog and the Open Metadata Ecosystem. |


--8<-- "snippets/abbr.md"
1 change: 1 addition & 0 deletions site/docs/connectors/unity-catalog/resource-connector.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,4 @@

The *OSS Unity Catalog resource connector* provides a client wrapper to the Open Source version of Unity Catalog. It aims to provide a buffer to the integration connector and the survey service to protect them from changes to the unity catalog API (which are expected).

![Connector Operation](resource-connector.svg)
4 changes: 4 additions & 0 deletions site/docs/connectors/unity-catalog/resource-connector.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,8 @@

# OSS Unity Catalog Schema Survey Service

Survey of a schema within an OSS Unity Catalog Server.
The Schema Survey Service extracts information about the tables, functions and volumes managed in a specific schema within an OSS Unity Catalog Server.

![Connector Operation](schema-survey-service.svg)

--8<-- "snippets/abbr.md"
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,8 @@

# OSS Unity Catalog Server Survey Service

Survey for an instance of the OSS Unity Catalog Server to discover the list of catalogs it is managing.
The Server Survey Service extracts information about the catalogs, schemas, tables, functions and volumes managed in a specific catalog within an OSS Unity Catalog Server.

![Connector Operation](server-survey-service.svg)

--8<-- "snippets/abbr.md"
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Original file line number Diff line number Diff line change
@@ -1,4 +1,10 @@
<!-- SPDX-License-Identifier: CC-BY-4.0 -->
<!-- Copyright Contributors to the Egeria project. -->

# OSS Unity Catalog (UC) Inside a Catalog Synchronizing Connector
# OSS Unity Catalog (UC) Inside a Catalog Synchronizing Connector

The Unity Catalog Inside Catalog Synchronizer Connector is an [integration connector](/concepts/integration-connector) that synchronizes metadata between catalogs in Unity Catalog and the open metadata ecosystem. If an element originates in Unity Catalog, the copy in open metadata is kept in sync with the unity catalog version and vice versa.

![Connector Operation](sync-catalog-connector.svg)

--8<-- "snippets/abbr.md"
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
8 changes: 6 additions & 2 deletions site/docs/connectors/unity-catalog/sync-server-connector.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,11 @@

# OSS Unity Catalog (UC) Server Synchronizing Connector

The *Unity Catalog OSS Synchronization Connector* exchanges metadata between the open-source version of Unity Catalog and the Open Metadata Ecosystem.
The *Unity Catalog OSS Synchronization Connector* is an [integration connector](/concepts/integration-connector) that exchanges metadata between the open-source version of Unity Catalog and the Open Metadata Ecosystem.

Its work is scoped by the metadata collection that represents Unity Catalog. This is identified by the *metadataSourceQualifiedName* property in the connector's configuration.
It performs two passes each time it is called.
It performs two passes each time it is called.

![Connector Operation](sync-server-connector.svg)

--8<-- "snippets/abbr.md"
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Loading