From dcdeb616bf3fac048dc107cdec30d642dff4203a Mon Sep 17 00:00:00 2001 From: Mandy Chessell Date: Tue, 20 Feb 2024 11:37:14 +0000 Subject: [PATCH] Improve the description of the digital resource connector Signed-off-by: Mandy Chessell --- site/docs/connectors/connector-catalog.drawio | 48 ++++++++-- site/docs/connectors/index.md | 87 ++++--------------- .../digital-resource-connector-intro.md | 27 ++++-- .../resource/digital-resource-connector.svg | 4 +- .../engine-host-services-section.md | 2 + .../by-section/event-bus-config-section.md | 3 +- .../guides/admin/servers/by-section/index.md | 6 ++ .../integration-daemon-services-section.md | 5 ++ .../lineage-warehouse-services-section.md | 5 +- .../calls/get-server-type-classification.md | 34 ++++++++ ...onfiguring-omag-server-basic-properties.md | 5 +- ...figuring-the-lineage-warehouse-services.md | 2 + 12 files changed, 136 insertions(+), 92 deletions(-) create mode 100644 site/snippets/admin/calls/get-server-type-classification.md create mode 100644 site/snippets/admin/configuring-the-lineage-warehouse-services.md diff --git a/site/docs/connectors/connector-catalog.drawio b/site/docs/connectors/connector-catalog.drawio index f59e4989ee..ec62405289 100644 --- a/site/docs/connectors/connector-catalog.drawio +++ b/site/docs/connectors/connector-catalog.drawio @@ -1,4 +1,4 @@ - + @@ -924,22 +924,22 @@ - + - + - - + + - + - + - + @@ -1177,6 +1177,38 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/site/docs/connectors/index.md b/site/docs/connectors/index.md index 4c426db1fa..16590be29d 100644 --- a/site/docs/connectors/index.md +++ b/site/docs/connectors/index.md @@ -17,18 +17,18 @@ Connectors enable Egeria to operate in many environments and with many types of ## Metadata exchange and maintenance connectors -The connectors that support the exchange and maintenance of metadata help to accelerate the rollout of your open metadata ecosystem since they can be used to automate the extraction and distribution of metadata to the third party technologies. +The connectors that support the exchange and maintenance of metadata help to accelerate the rollout of your open metadata ecosystem, since they can be used to automate the extraction and distribution of metadata to the third party technologies. -* [Secrets Store connectors](#secrets-store-connectors) manage the retrieval of secrets (passwords, certificates, ...) from secured locations at runtime. | -* [File connectors](#files) work with different types of files +* [Secrets Store connectors](#secrets-store-connectors) manage the retrieval of secrets (passwords, certificates, ...) from secured locations at runtime. +* [File connectors](#files) work with different types of files. * [JDBC Database connectors](#jdbc-databases) make use of the JDBC standards to work with different types of relational databases. -* [Apache Kafka](#apache-kafka) work with the topics and/or events passing through the Apache Kafka event broker. -* [Apache Atlas](#apache-atlas) work with an Apache Atlas server. -* [Strimzi](#strimzi) work with the cloud-based Apache Kafka deployment called Strimzi. -* [Open API Specification](#open-api-specification) extract metadata about APIs through the Open API interfaces provided through the Swagger API. -* [Open Lineage Events](#open-lineage-events) works with the open lineage event standard. +* [Apache Kafka connectors](#apache-kafka) work with the topics and/or events passing through the Apache Kafka event broker. +* [Apache Atlas connectors](#apache-atlas) work with an Apache Atlas server. +* [Strimzi connector](#strimzi) works with the cloud-based Apache Kafka deployment called Strimzi. +* [Open API Specification connectors](#open-api-specification) extract metadata about APIs through the Open API interfaces provided through the Swagger API. +* [Open Lineage Event connectors](#open-lineage-events) works with the open lineage event standard. -### Secrets Store Connectors +### Secrets Stores * The [Environment Variables Secret Store connector](/connectors/secrets/environment-variable-secrets-store-connector) retrieves secret values from environment variables. @@ -36,14 +36,16 @@ The connectors that support the exchange and maintenance of metadata help to acc Files provide storage for many types of data. They are organizes into folders (also known as directories on some operating systems). Some connectors work with any type of file. Other connectors are able to understand the content of specific types of file formats and so these connectors are organized by file type. -#### Any type of file +#### Any type of File * The [Basic File Resource Connector](/connectors/resource/basic-file-resource-connector) provides support to read and write to a file using the Java File object. +* The [Move/Copy File Provisioning Governance Action Service](/connectors/governance-action/move-copy-file-provisioning-governance-action-service) moves or copies files from one location to another and maintains the lineage of the action. #### File Folders (Directories) * The [Basic Folder Resource Connector](/connectors/resource/basic-folder-resource-connector) is for accessing the files within a folder (directory). -* The [Data Files Monitor Integration Connector](/connectors/integration/data-files-monitor-integration-connector) maintains a `DataFile` asset for each file in the directory (or any subdirectory). When a new file is created, a new DataFile asset is created. If a file is modified, the lastModified property of the corresponding DataFile asset is updated. When a file is deleted, its corresponding DataFile asset is also deleted (or archived if it is still needed for lineage). +* The [Data Files Monitor Integration Connector](/connectors/integration/data-files-monitor-integration-connector) maintains a `DataFile` asset for each file in the directory (or any subdirectory). When a new file is created, a new DataFile asset is created. If a file is modified, the lastModified property of the corresponding DataFile asset is updated. When a file is deleted, its corresponding DataFile asset is also deleted (or archived if it is still needed for lineage). +* The [Generic Folder Watchdog Governance Action Service](/connectors/governance-action/generic-folder-watchdog-governance-action-service) listens for changing `DataFile` assets linked to a specified `FileFolder` element and initiates governance actions when specific events occur. This may be for files directly linked to the folder or located in sub-folders. #### Data Folders @@ -99,71 +101,14 @@ The open lineage connectors work with the [Open Lineage standard](/features/line * [File-based Open Lineage Log Store integration connector](/connectors/integration/file-based-open-lineage-log-store-integration-connector) stores the open lineage events that are passed to it through the OpenLineage listener that is registered with the Lineage Integrator OMIS. Each OpenLineage event is stored in its own file in JSON format. These files are organized according to the namespace and job name in the event. * [Open Lineage Cataloguer integration connector](/connectors/integration/open-lineage-cataloguer-integration-connector) registers an Open Lineage listener with the Lineage Integrator OMIS and to catalog any processes that are not already known to the open metadata ecosystem. -### Open Discovery Services - ----8<-- "docs/connectors/discovery/discovery-service-intro.md" - -| Connector | Description | -|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------| -| [Sequential Discovery Pipeline :material-github:](https://github.com/odpi/egeria/tree/main/open-metadata-implementation/adapters/open-connectors/discovery-service-connectors){ target=gh } | runs nested discovery services in a sequence ([more information on discovery pipelines](/frameworks/odf/#discovery-pipeline)). | -| [CSV Discovery Service :material-github:](https://github.com/odpi/egeria/tree/main/open-metadata-implementation/adapters/open-connectors/discovery-service-connectors){ target=gh } | extracts the column names from the first line of the file, counts up the number of records in the file and extracts its last modified time. | -| [Apache Atlas Discovery Service](/connectors/discovery/apache-atlas-discovery-service) | profiles the content of an Apache Atlas server. | -| [Validate Drop Foot Weekly Measurements Discovery Service :material-github:](https://github.com/odpi/egeria/tree/main/open-metadata-resources/open-metadata-samples/governance-services-sample){ target=gh } | runs nested discovery services in a sequence ([more information on discovery pipelines](/frameworks/odf/#discovery-pipeline)). | -| [Validate Patient Records :material-github:](https://github.com/odpi/egeria/tree/main/open-metadata-resources/open-metadata-samples/governance-services-sample){ target=gh } | runs nested discovery services in a sequence ([more information on discovery pipelines](/frameworks/odf/#discovery-pipeline)). | - -??? education "Further information relating to Open Discovery Services" - - - [Configuring an engine host](/guides/admin/servers/by-server-type/configuring-an-engine-host) to understand how to set up the [Engine Host](/concepts/engine-host) server where the open discovery services run. - - [Setting up a governance engine content pack](/guides/developer/open-metadata-archive/creating-governance-engine-content-packs) to create an [open discovery engine](/concepts/open-discovery-engine) definition to load into a [Metadata Access Store](/concepts/metadata-access-store). - - [Writing an open discovery service](/guides/developer/open-discovery-services/overview) for information on writing new open discovery services. - ## Open Metadata Governance Connectors -### Governance Action Services - ----8<-- "docs/connectors/governance-action/governance-action-service-intro.md" - -| Connector | Description | -|---|---| -| [Generic Element Watchdog Governance Action Service](/connectors/governance-action/generic-element-watchdog-governance-action-service) | listens for changing metadata elements and initiates governance action processes when certain events occur. | -| [Generic Folder Watchdog Governance Action Service](/connectors/governance-action/generic-folder-watchdog-governance-action-service) | listens for changing assets linked to a `DataFolder` element and initiates governance actions when specific events occur. This may be for files directly linked to the folder or located in sub-folders.| -| [Move/Copy File Provisioning Governance Action Service](/connectors/governance-action/move-copy-file-provisioning-governance-action-service) | moves or copies files from one location to another and maintains the lineage of the action. | -| [Origin Seeker Remediation Governance Action Service](/connectors/governance-action/origin-seeker-remediation-governance-action-service) | walks backwards through the lineage mappings to discover the origin of the data | - -??? education "Further information relating to Governance Action Services" - - - [Configuring an engine host](/guides/admin/servers/by-server-type/configuring-an-engine-host) to understand how to set up the [Engine Host](/concepts/engine-host) server where the governance action services run. - - [Setting up a governance engine content pack](/guides/developer/open-metadata-archive/creating-governance-engine-content-packs) to create a [governance action engine](/concepts/governance-action-engine) definition to load into a [Metadata Access Store](/concepts/metadata-access-store). - - [Writing a governance action service](/guides/developer/governance-action-services/overview) for information on writing new governance action services. - -### Event Action Services - ----8<-- "docs/connectors/event-action/event-action-service-service-intro.md" - -There are currently no event action services supplied by Egeria. - -??? education "Further information relating to Event Action Services" - - - [Configuring an engine host](/guides/admin/servers/by-server-type/configuring-an-engine-host) to understand how to set up the [Engine Host](/concepts/engine-host) server where the event action services run. - - [Setting up a governance engine content pack](/guides/developer/open-metadata-archive/creating-governance-engine-content-packs) to create an [event action engine](/concepts/event-action-engine) definition to load into a [Metadata Access Store](/concepts/metadata-access-store). - - [Writing an event action service](/guides/developer/event-action-services/overview) to understand how to write an event-action service. - -### Repository Governance Services - ----8<-- "docs/connectors/repository-governance/repository-governance-service-intro.md" - -There are currently no repository governance services supplied by Egeria. - -??? education "Further information relating to Repository Governance Services" - - - [Configuring an engine host](/guides/admin/servers/by-server-type/configuring-an-engine-host) to understand how to set up the [Engine Host](/concepts/engine-host) server where the repository governance services run. - - [Setting up a governance engine content pack](/guides/developer/open-metadata-archive/creating-governance-engine-content-packs) to create a [repository governance engine](/concepts/repository-governance-engine) definition to load into a [Metadata Access Store](/concepts/metadata-access-store). - - [Writing a repository governance service](/guides/developer/archive-services/overview) to understand how to write a repository governance service. - +* The [Generic Element Watchdog Governance Action Service](/connectors/governance-action/generic-element-watchdog-governance-action-service) listens for changing metadata elements and initiates governance action processes when certain events occur. +* The [Origin Seeker Remediation Governance Action Service](/connectors/governance-action/origin-seeker-remediation-governance-action-service) walks backwards through the lineage mappings to discover the origin of the data ## Runtime connectors -*Runtime* connectors enable Egeria's [OMAG Server Platform](/egeria docs/concepts/omag-server-platform) and its hosted [OMAG Servers](/concepts/omag-server) to operate in many environments by providing plug-in points for the runtime services it needs to operate. Most of the runtime connectors relate to persistent storage, or connections to distributed services. +*Runtime* connectors enable Egeria's [OMAG Server Platform](/concepts/omag-server-platform) and its hosted [OMAG Servers](/concepts/omag-server) to operate in many environments by providing plug-in points for the runtime services it needs to operate. Most of the runtime connectors relate to persistent storage, or connections to distributed services. The connectors are organized by type to allow you to choose the options available from the Egeria community. | Type | Description | |-------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------| diff --git a/site/docs/connectors/resource/digital-resource-connector-intro.md b/site/docs/connectors/resource/digital-resource-connector-intro.md index 880fb8d645..ed54ae9f90 100644 --- a/site/docs/connectors/resource/digital-resource-connector-intro.md +++ b/site/docs/connectors/resource/digital-resource-connector-intro.md @@ -1,15 +1,32 @@ -The [digital resource connectors](/concepts/digital-resource-connector) provide access to [digital resources](/concepts/resource) and their [asset metadata](/concepts/asset) that is stored in the open metadata ecosystem. These connectors are for use by external applications and tools to connect with resources and services in the digital landscape. The can also be used by other connectors, such as [integration connectors](/concepts/integration-connector), [Open Discovery Services](/concepts/open-discovery-service) and [Governance Action Services](/concept/governance-action-service), to access a third party technology. +A *digital resource connector* is used to call the API of a particular type of [digital resource](/concepts/resource) in order to access its content or metadata. They may also listen for events from the digital resource if it emits them. For example, you may have a digital resource connector to access a file, another to access a database, another to listen to an Apache Kafka topic, and another to access a system such as Apache Atlas. -These connectors also supply the Asset metadata from Egeria that describes these resources. +## What is the interface of a digital resource connector? -Instances of these connectors are created through the [Asset Consumer OMAS](/services/omas/asset-consumer/overview), [Asset Owner OMAS](/services/omas/asset-owner/overview) and [Discovery Engine OMAS](/services/omas/discovery-engine/overview) interfaces. They use the Connection linked to the corresponding asset in the open metadata ecosystem. If there are more than one connection associated with the asset, then a selection is made by the server [metadata security connector](/concepts/server-metadata-security-conector) running in the OMASs' server. +The interface of a digital resource connector is a combination of the standard methods of an [Open Connector Framework (OCF)](/frameworks/ocf/overview) connector, to enable Egeria to start and stop it, plus something similar to the standard API provided by the digital resource. The aim is to make its interface familiar to the developer who is coding the calls to the digital resource. -The returned connector is configured with the `connectedAssetProperties` from the open metadata ecosystem. +## Why develop and use a digital resource connector? -Connection objects are associated with assets in the metadata catalog using the Asset Owner OMAS, [Data Manager OMAS](/services/omas/data-manager/overview) and [Asset Manager OMAS](/services/omas/asset-manager/overview). +The real value of a digital resource connector over just calling the digital resource's API directly is that an instance of a digital resource connector can be created from the information contained in a [connection](/concepts/connection). The connection includes details of the digital resource connectors' implementation, the endpoint of the digital resource plus any security information needed to access the digital resource. A connection can be described as a Java object, as a JSON document or stored in open metadata as a collection of related entities starting with the [Connection entity](/types/2/0201-Connectors-and-Connections). The connection, therefore, provides a standard way to distribute details about how to access the content of a particular digital resource, regardless of its type, around the open metadata ecosystem. + +The security information supplied through the connection keeps secrets out of the caller's code. The additional use of [secret store connectors](/concepts/secrets-store-connector) within a digital resource connector helps to keep these secrets out of the connection object and stored within encrypted vaults until they are needed. Using a secrets store connector within a digital resource connector has no impact on the caller of the digital resource connector. + +## Cataloguing a digital resource + +When a digital resource is catalogued in open metadata, it is represented as an [Asset](/concepts/asset) entity. Ideally, the Asset entity is linked to a Connection entity that describes the appropriate digital resource connector, any security information and the digital resource's endpoint. + +Thus the combination of the asset and its connection allows the distribution of both descriptive information about the digital resource, plus the information needed to access it. + +Since different users may be granted different levels of access to the digital resource's contents, it is also possible to add multiple connections to the asset, each with different security information attached. When a particular user requests the digital resource connector for the asset, the [Metadata Access Server](/concepts/metadata-access-server) supplying the connection metadata calls its [server metadata security connector](/concepts/server-metadata-security-conector) to choose the appropriate connection for the user. + +## How are digital resource connectors used? + +Each Open Metadata Access Service (OMAS) has a client called the *ConnectedAssetClient* that can be used to create an instance of a digital resource connector for an asset. This connector can then be used in a Java application, or by other connectors, such as [Integration Connectors](/concepts/integration-connector) and [Governance Services](/concept/governance-service), to access the digital resource. ![Digital Resource Connector](/connectors/resource/digital-resource-connector.svg) +The digital resource connector also offers an method called `getConnectedAssetProperties()` to retrieve all the [known metadata](/concepts/connected-asset-properties) about the digital resource that is stored in the open metadata ecosystem. This makes it easy for the caller of the digital resource connector to combine information from the digital resource with open metadata. + + diff --git a/site/docs/connectors/resource/digital-resource-connector.svg b/site/docs/connectors/resource/digital-resource-connector.svg index c585aceac9..174b5051b4 100644 --- a/site/docs/connectors/resource/digital-resource-connector.svg +++ b/site/docs/connectors/resource/digital-resource-connector.svg @@ -1,4 +1,4 @@ - + -
Java Application
Java Application
Egeria Client
Egeria Client
Digital Resource
Connector
Digital Resource...
Digital
Resource
Digital...Viewer does not support full SVG 1.1 \ No newline at end of file +
Java Application
Open Metadata
Access Service (OMAS) client
Digital Resource
Connector
Digital
Resource
Integration Daemon
Open Metadata
Integration Service (OMIS)
Integration
Connector
Digital Resource
Connector
Engine Host
Open Metadata
Engine Service
(OMES)
Governance Service
Digital Resource
Connector
 \ No newline at end of file diff --git a/site/docs/guides/admin/servers/by-section/engine-host-services-section.md b/site/docs/guides/admin/servers/by-section/engine-host-services-section.md index 9af53e49ce..2380acb58d 100644 --- a/site/docs/guides/admin/servers/by-section/engine-host-services-section.md +++ b/site/docs/guides/admin/servers/by-section/engine-host-services-section.md @@ -4,5 +4,7 @@ # Engine Host Services Section +--8<-- "snippets/admin/configuring-the-engine-services.md" + --8<-- "snippets/abbr.md" diff --git a/site/docs/guides/admin/servers/by-section/event-bus-config-section.md b/site/docs/guides/admin/servers/by-section/event-bus-config-section.md index b80268d5ad..c97a6f3beb 100644 --- a/site/docs/guides/admin/servers/by-section/event-bus-config-section.md +++ b/site/docs/guides/admin/servers/by-section/event-bus-config-section.md @@ -3,6 +3,7 @@ # Event Bus Config Section - +--8<-- "snippets/admin/configuring-event-bus.md" --8<-- "snippets/abbr.md" + diff --git a/site/docs/guides/admin/servers/by-section/index.md b/site/docs/guides/admin/servers/by-section/index.md index 968af2284e..6a40a6f789 100644 --- a/site/docs/guides/admin/servers/by-section/index.md +++ b/site/docs/guides/admin/servers/by-section/index.md @@ -24,12 +24,18 @@ The configuration document is divided into sections. Some sections contain prop ## Local Server Id +The localServerId is a unique identifier for the server, it is used when accessing resource that require their callers to supply a unique identifier. For example, when an OMAG server is accessing an Apache Kafka topic, it needs to reliably identify itself with a callerId so that the Kafka server knows which events it has received and which it has not. The localServerId is used for this purpose when accessing the [open metadata repository cohort topics](/concepts/cohort-events) for example. ## Local Server Name +The localServerName is a unique name for the server. This is a name you choose, and the administration services maintains the name in the configuration document. ## Local Server Type +The local server type is a classification of the type of the server based on the sections of the configuration document that have been configured. Leave it blank and Egeria will fill it in on server start up. There is also an administration call to query the type of server using the following call: + + ## Audit Trail +--8<-- "snippets/abbr.md" diff --git a/site/docs/guides/admin/servers/by-section/integration-daemon-services-section.md b/site/docs/guides/admin/servers/by-section/integration-daemon-services-section.md index 04263c55ef..870546e607 100644 --- a/site/docs/guides/admin/servers/by-section/integration-daemon-services-section.md +++ b/site/docs/guides/admin/servers/by-section/integration-daemon-services-section.md @@ -3,6 +3,11 @@ # Integration Daemon Services Section +This section is only relevant when configuring an [integration daemon](/concepts/integration-daemon). + +--8<-- "snippets/admin/configuring-the-integration-groups.md" + +--8<-- "snippets/admin/configuring-the-integration-services.md" --8<-- "snippets/abbr.md" diff --git a/site/docs/guides/admin/servers/by-section/lineage-warehouse-services-section.md b/site/docs/guides/admin/servers/by-section/lineage-warehouse-services-section.md index 2b6976caf0..3c09705feb 100644 --- a/site/docs/guides/admin/servers/by-section/lineage-warehouse-services-section.md +++ b/site/docs/guides/admin/servers/by-section/lineage-warehouse-services-section.md @@ -1,8 +1,11 @@ -# Open Lineage Services Section +# Lineage Warehouse Services Section +This section is only relevant when configuring a [lineage warehouse](/concepts/lineage-warehouse). + +--8<-- "snippets/admin/configuring-the-lineage-warehouse-services.md" --8<-- "snippets/abbr.md" diff --git a/site/snippets/admin/calls/get-server-type-classification.md b/site/snippets/admin/calls/get-server-type-classification.md new file mode 100644 index 0000000000..cbb7bdd9a9 --- /dev/null +++ b/site/snippets/admin/calls/get-server-type-classification.md @@ -0,0 +1,34 @@ + + + +!!! get "getServerTypeClassification" + Return the derived server type that is created from the classification of the server configuration. + + === "Java" + + ```java + OMAGServerConfigurationClient configurationClient = new OMAGServerConfigurationClient(adminUserId, + serverName, + adminPlatformURLRoot); + + ServerTypeClassificationSummary serverTypeClassification = configurationClient.getServerTypeClassification(); + ``` + + === "Python" + + ```python + admin_user_id="garygeeke" + server_name="active-metadata-store" + admin_platform_url_root="https://127.0.0.1:9443" + + config_client=CoreServerConfig(server_name, + admin_platform_url_root, + admin_user_id) + + config_client.get_server_classification() + ``` + + === "REST" + ``` + GET {{platformURLRoot}}/open-metadata/admin-services/users/{{adminUserId}}/servers/{{serverName}}/server-type-classification" + ``` \ No newline at end of file diff --git a/site/snippets/admin/configuring-omag-server-basic-properties.md b/site/snippets/admin/configuring-omag-server-basic-properties.md index 8af557dd46..e3342a2d9a 100644 --- a/site/snippets/admin/configuring-omag-server-basic-properties.md +++ b/site/snippets/admin/configuring-omag-server-basic-properties.md @@ -120,7 +120,4 @@ If, however, you want to use your own values, then this is the command to set it If you are curious to see the automated server type that has been assigned to a server, it can be retrieved with the following command. -!!! get "GET - getServerClassification" - ``` - {{platformURLRoot}}/open-metadata/admin-services/users/{{adminUserId}}/servers/{{serverName}}/server-type-classification" - ``` \ No newline at end of file +--8<-- "snippets/admin/calls/get-server-type-classification.md" diff --git a/site/snippets/admin/configuring-the-lineage-warehouse-services.md b/site/snippets/admin/configuring-the-lineage-warehouse-services.md new file mode 100644 index 0000000000..26d1e7a170 --- /dev/null +++ b/site/snippets/admin/configuring-the-lineage-warehouse-services.md @@ -0,0 +1,2 @@ + + \ No newline at end of file