diff --git a/site/docs/concepts/catalog-target.md b/site/docs/concepts/catalog-target.md new file mode 100644 index 0000000000..0467f61b31 --- /dev/null +++ b/site/docs/concepts/catalog-target.md @@ -0,0 +1,60 @@ + + + +# Catalog Target + +A *catalog target* describes the third party technology ([resource](/concepts/resource)) that a specific [integration connector](/concept/integration-connector) is exchanging metadata with. + +## Catalog Target Types + +The developer of an integration connector is able to specify one or more types of technology supported by the connector. These are called *Catalog Target Types* and they are specified in the implementation of the integration connector's [connector provider](/concepts/connector-provider). Catalog Target Types include the following properties: + +* **typeName** - the [open metadata type name](/types) of the element that describes a specific catalog type. +* **deployedImplementationType** - the [deployed implementation type](/concepts/deployed-implementation-type) of the element that describes a specific catalog type. This allows a more fine-grained specification of the type of technology supported by the integration connector. +* **otherPropertyNames** - as the name suggests, this allows other properties that are useful to a person deploying the connector. + +The catalog target type is given a name called the **catalogTargetName**. Here is a code snippet from the [Data Files Monitor Integration Connector's](/connectors/integration/data-files-monitor-integration-connector) connector provider showing how the catalog target types are set up. This connector is configured with the name of a directory (folder) on a filesystem and it catalogues the files underneath this folder. Therefore, the catalog target type describes a directory. + +```java + + static public final String CATALOG_TARGET_NAME = "directoryToMonitor"; + + CatalogTargetType catalogTargetType = new CatalogTargetType(); + + catalogTargetType.setTypeName(OpenMetadataTypesMapper.DATA_FOLDER_TYPE_NAME); + catalogTargetType.setDeployedImplementationType(DeployedImplementationType.DATA_FOLDER.getDeployedImplementationType()); + + super.catalogTargetTypes.put(CATALOG_TARGET_NAME, catalogTargetType); +``` + +This information is then available to the person choosing the appropriate integration connector and setting it up to run. + +At runtime, the specific catalog target can be supplied to an integration connector via its [endpoint](/concepts/endpoint) which is connected to its entity's [connection](/concepts/connection), or via one or more *CatalogTarget* relationships. + +## CatalogTarget relationship + +The [*CatalogTarget*](/types/4/0464-Dynamic-Integration-Groups) relationship links an integration connector's entity to an entity that describes the specific third party technology that the integration connector should exchange metadata with. It is used when configuring an [integration daemon](/concepts/integration-daemon) using [integration groups](/concepts/integration-group). With this style of configuration, details of the integration connectors to run are described in the open metadata repositories that the integration daemon is connected to. The catalog target relationship is part of this open metadata description. + +The diagram below shows the open metadata elements describing an integration group. This integration group has two integration connectors. Each integration connector is linked to a [connection](/concepts/connection) that describes how to create a running instance of the integration connector. This connection may optionally include the endpoint information of the third party technology that the integration connector is to exchange metadata with. The *CatalogTarget* relationship provides an alternative way to supply details of one, or more third party technologies that it should connect to. These technologies are described, typically by an [asset](/concepts/asset) open metadata element (shown in yellow) and are linked to the integration connector's element using the *CatalogTarget* relationship. + +![Integration group](/services/omas/governance-engine/integration-group-example) + +The [connection](/concepts/connection) information associated with the third party technology can be used by the integration connector to create a [resource connector](/concepts/digital-resource-connector) that acts as a client to call the third party technology. + +The diagram below shows an example of a metadata description for an integration connector that catalogs files found in a directory (folder). The catalog target is the asset that describes the directory where the files are located. Linked to the asset is a connection for the data folder connector that will provide access to the files in the directory. + +![Data file cataloguing integration connector](/services/omas/governance-engine/integration-connector-catalog-target-data-folder-example) + +This style of configuration means that the connection to a specific third party technology only needs to be specified once and can be used for multiple purposes. This is important for technologies that need a complex connection structure that may need to be updated from time to time. For example, the next diagram shows an integration connector that is linked to a database asset. This database's connection is complex and includes an embedded secret store connector. If details of the secrets used to connect to the database need to be changed, it only needs to be changed in one connection. + +![Database cataloguing integration connector](/services/omas/governance-engine/integration-connector-catalog-target-database-example) + +The *CatalogTarget* relationship is similar to the [*TargetForAction*](/types/4/0463-Engine-Actions) relationships that links [engine actions](/concepts/engine-action) to the elements describing the technology that the associated [governance service](/concepts/governance-service) is to work on. + +From the perspective of the database element, these relationships make it possible to see the automated processing (agents) working with the database, as well as the metadata they are gathering/validating/updating. This is useful for either the owner or the consumer of the resource to know how the technology and its metadata are being maintained. + +![Database's agents and information](/services/omas/governance-engine/database-agents-and-information) + + + +--8<-- "snippets/abbr.md" diff --git a/site/docs/concepts/index.md b/site/docs/concepts/index.md index f88d8b2dfa..1931b177ec 100644 --- a/site/docs/concepts/index.md +++ b/site/docs/concepts/index.md @@ -42,6 +42,8 @@ ## C - [Catalog Integrator OMIS](/services/omis/catalog-integrator/overview) +- [Catalog Target](/concepts/catalog-target) +- [Catalog Target Type](/concepts/catalog-target) - [Cohort](/concepts/cohort-member) - [Cohort Event](/concepts/cohort-events) - [Cohort Member](/concepts/cohort-member) @@ -103,6 +105,7 @@ - [Discovery Server](/concepts/engine-host) - [Discovery Service](/concepts/open-discovery-service) - [Display Integrator OMIS](/services/omis/display-integrator/overview) +- [Dynamic Integration Group](/concepts/integration-group) ## E @@ -128,7 +131,7 @@ - [Favorite Things Collections](/concepts/favorite-things-collection) - [Feedback](/concepts/feedback) -- [File Type](/concepts/deployed-implementation-type/#file-type) +- [File Type](/concepts/file-type) - Files - [omag.server.servername.config](/concepts/configuration-document) diff --git a/site/docs/concepts/integration-group.md b/site/docs/concepts/integration-group.md index e106361ae7..30bb82b704 100644 --- a/site/docs/concepts/integration-group.md +++ b/site/docs/concepts/integration-group.md @@ -1,8 +1,3 @@ ---- -hide: -- toc ---- - @@ -10,9 +5,40 @@ hide: An *integration group* describes a collection of [integration connectors](/concepts/integration-connector) that are to run together in one or more [integration daemons](/concepts/integration-daemon). -![integration group example](/services/omas/governance-engine/integration-group-example.svg) +## Specification + +Each integration connector is represented by an entity of type [*IntegrationConnector*](/types/4/0464-Dynamic-Integration-Groups) linked to a [*Connection*](/types/2/0201-Connectors-and-Connections) via the [*ConnectionToAsset*](/types/2/0205-Connection-Linkage) relationship. This [connection](/concepts/connection) is used to instantiate the integration connector. It is also passed to the integration connector as it initializes, so it is used to pass configuration properties that describes its desired behaviour. The connection's optional endpoint describes the [catalog target](/concepts/catalog-target). Alternatively, the integration connector's catalog target(s) can be supplied via the [*CatalogTarget*](/types/4/0464-Dynamic-Integration-Groups) relationship. + +![integration group structure](/services/omas/governance-engine/integration-group-example.svg) + +## Examples of integration connector definitions + +The exact specification of an integration connector depends on how the integration connector is implemented. Details of the integration connectors supplied with Egeria, and how to configure them are available in the [connector catalog](/connectors/#integration-connectors). + +Some integration connectors use a simple connection structure. The [*ConnectorType*](/concepts/connector-type) linked to the connection identifies the implementation of the integration connector. The endpoint describes how to connect to the third party technology. + +![integration connector with a simple connection](/services/omas/governance-engine/integration-connector-simple.svg) + +Some connectors need a virtual connection with embedded connectors that describe different resources. + +![Integration connector with a virtual connection](/services/omas/governance-engine/integration-connector-virtual-connection.svg) -The integration daemon is configured with a list of qualified names for various integration groups. It retrieves the list of integration connectors in each of these groups from the open metadata ecosystem and starts to run them. It continues to monitor for changes to the integration groups and their linked integration connectors, changing the connectors it is running as needed. +For example, this integration connector has a virtual connection with an embedded secrets store for managing passwords ans certificates. + +![Example of integration connector with a virtual connection](/services/omas/governance-engine/integration-connector-virtual-connection-example.svg) + +Alternatively, using [catalog targets](/concepts/catalog-target) provides a more efficient way to supply information about the technology that the integration connector is to work with. + +The connection for the integration connector just needs the connector type to describe its implementation. +The *CatalogTarget* relationship links to an asset that describes the technology that the integration connector is to work with. + +![Example of integration connector with a virtual connection](/services/omas/governance-engine/integration-connector-catalog-target.svg) + +The integration connector can have multiple catalog log targets that can change over time. + +## Configuring integration groups + +An [integration daemon is configured](/guides/admin/servers/configuring-an-integration-daemon/) with a list of qualified names for various integration groups. It retrieves the list of integration connectors in each of these groups from the open metadata ecosystem and starts to run them. It continues to monitor for changes to the integration groups and their linked integration connectors, changing the connectors it is running as needed. ![integration group operation](/services/omas/governance-engine/integration-group-use.svg) @@ -24,4 +50,5 @@ The integration daemon is configured with a list of qualified names for various The open metadata types for this definition are in model [0464 - Dynamic Integration Groups](/types/4/0464-Dynamic-Integration-Groups) - specifically notice that *IntegrationGroup* and *IntegrationConnector* are linked by the *RegisteredIntegrationConnector* relationship. + --8<-- "snippets/abbr.md" \ No newline at end of file diff --git a/site/docs/guides/admin/servers/configuring-an-integration-daemon.md b/site/docs/guides/admin/servers/configuring-an-integration-daemon.md index 58f9db6b09..2cf50348aa 100644 --- a/site/docs/guides/admin/servers/configuring-an-integration-daemon.md +++ b/site/docs/guides/admin/servers/configuring-an-integration-daemon.md @@ -1,12 +1,28 @@ -# Configuring an [integration daemon](/concepts/integration-daemon) +# Configuring an integration daemon -An *Integration Daemon* is configured by creating a [configuration document](/concepts/configuration-document). Below is the outline structure of the integration daemon's configuration document. +An [*Integration Daemon*](/concepts/integration-daemon) runs [integration connectors](/concepts/integraton-connector) that are exchanging metadata with third party technologies. For example, an integration connector may catalog files from a file system, or the tables and columns in a database. Alternatively, it may be gathering lineage information or distributing security information to LDAP. + +Integration connectors are typically long-running. Their implementation calls one of the [Open Metadata Integration Services (OMISs)](/services/omis) to maintain open metadata and access other useful integration utilities. The integration daemon needs information about which integration connectors to run along with details of the services they need to operate successfully. The information needed to bootstrap the integration daemon is configured by creating a [configuration document](/concepts/configuration-document) which is passed to the integration daemon when it starts up. + +## Overview + +Below is the outline structure of the integration daemon's configuration document. ![Configuration for an integration daemon](integration-daemon-config.svg) +The *Local Server URL Root* shown in light grey can be optionally configured to provide the [platformURLRoot](/concepts/platform-url-root) for the default [OMAG Server Platform](/concepts/omag-server-platform) where this integration daemon is to be deployed to. It is only used if the deploy command does not include the destination platformURLRoot. + +The properties shown in blue are standard properties for an [OMAG Server](/concepts/omag-server). + +The properties shown in green are specific to integration daemons and they identify which integration connectors to run. There are two approaches. An integration daemon can be configured with either approach or both: + +* The *Integration Services* section provides details of the integration connectors that are statically configured into the integration daemon. They are organized by the integration services that each integration connector is using. These integration connectors will be automatically started with the integration daemon. Any changes to these integration connectors need to be made in the configuration document and then the integration daemon needs to be restarted for the changes to take effect. +* The *Dynamic Integration Groups* section points to lists of integration connectors defines using open metadata elements store in the open metadata repositories. The integration daemon monitors these lists to determine which integration connectors to run. It dynamically updates the connectors it is running based on the values in the open metadata elements. + + ??? info "Configuring the basic properties" --8<-- "docs/guides/admin/servers/configuring-omag-server-basic-properties.md" @@ -35,6 +51,7 @@ An *Integration Daemon* is configured by creating a [configuration document](/co ??? education "Further information" - For help in fixing any error you find using the integration daemon, visit the [integration daemon diagnostic guide](/guides/diagnostic/integration-daemon-diagnostic-guide). - Link to the [Egeria solutions](/patterns/metadata-integration) to see the integration daemon in action. - - Link to the [integration daemon services](/services/integration-daemon-services) to understand how the integration daemon is implemented. + - Link to the [integration daemon services](/services/integration-daemon-services) to understand how the integration daemon is implemented. It also describes the integration daemon's REST API calls to restart and refresh integration connectors; and to make temporary changes to an integration connector's configuration. These changes are only made in-memory and they are lost when the integration connector restarts. + --8<-- "snippets/abbr.md" diff --git a/site/docs/guides/admin/servers/configuring-the-integration-groups.md b/site/docs/guides/admin/servers/configuring-the-integration-groups.md index 79efed11a6..cb042881d5 100644 --- a/site/docs/guides/admin/servers/configuring-the-integration-groups.md +++ b/site/docs/guides/admin/servers/configuring-the-integration-groups.md @@ -2,13 +2,30 @@ -### Configure dynamic integration groups +### Configure integration groups -Each integration group hosts one or more [integration connectors](/concepts/integration-connector). An integration connector is responsible for the exchange of metadata with a specific deployment of a third party technology. For example, the [database integrator](/services/omis/database-integrator) integration service supports integration connectors that work with relational databases (RDBMS). +An [integration group](/concepts/integration-group) identifies a list of [integration connectors](/concepts/integration-connector) that an integration daemon should run. These integration connectors are defined and maintained as open metadata elements stored in the open metadata repositories. Changes can be made to these integration connectors at any time using the [Governance Engine OMAS](/services/omas/governance-engine/overview). The integration daemon is monitoring for changes and updates the integration connectors it is running in line with the changes it discovers. This approach allows changes to be made to the integration daemon's configuration without needing access to the administration commands, nor a restart of the integration daemon. #### Configure an integration group -Each integration group is configured with the network location of the [metadata access point](/concepts/metadata-access-point) / [metadata access store](/concepts/metadata-access-store) running the appropriate [OMAS](/services/omas) along with the qualified name of the [IntegrationGroup](/types/4/0464-Dynamic-Integration-Groups) entity that represents the integration group. +Each integration group is configured with the network location of a [metadata access server](/concepts/metadata-access-server) running the appropriate [Open Metadata Access Services (OMASs)](/services/omas) along with the qualified name of the [IntegrationGroup](/types/4/0464-Dynamic-Integration-Groups) open metadata entity that represents the integration group. + +The OMASs needed in the metadata access server depend on the [Open Metadata Integration Services (OMISs)](/services/omis) in use by the integration connectors. If you are unsure of which integration services will be used by your integration connectors, configure all of the listed OMASs in the metadata access store. + +| Open Metadata Integration Service (OMIS) | OMIS Description | Open Metadata Access Service (OMAS) running in the metadata access server | +|:-------------------------------------------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------|:--------------------------------------------------------------------------| +| [API Integrator](/services/omis/api-integrator/overview) | provides cataloguing for APIs. | [Data Manager OMAS](/services/omas/data-manager/overview) | +| [Catalog Integrator](/services/omis/catalog-integrator/overview) | provides a two-way synchronization for data catalogs. | [Asset Manager OMAS](/services/omas/asset-manager/overview) | +| [Database Integrator](/services/omis/database-integrator/overview) | provides metadata extraction from relational databases. | [Data Manager OMAS](/services/omas/data-manager/overview) | +| [Display Integrator](/services/omis/display-integrator/overview) | provides metadata extraction from systems that provide user displays and forms to capture new data values. | [Data Manager OMAS](/services/omas/data-manager/overview) | +| [Files Integrator](/services/omis/files-integrator/overview) | collects metadata about files stored in a filesystem or file manager. | [Data Manager OMAS](/services/omas/data-manager/overview) | +| [Infrastructure Integrator](/services/omis/infrastructure-integrator/overview) | supports the extraction of metadata from IT infrastructure artifacts as well as the use of metadata to maintain IT infrastructure artifacts. | [IT Infrastructure OMAS](/services/omas/it-infrastructure/overview) | +| [Lineage Integrator](/services/omis/lineage-integrator/overview) | collects metadata about processes, their internal logic and the data assets they work with. | [Asset Manager OMAS](/services/omas/asset-manager/overview) | +| [Organization Integrator](/services/omis/organization-integrator/overview) | imports details of an organization's structure - such as teams and departments. | [Community Profile OMAS](/services/omas/community-profile/overview) | +| [Search Integrator](/services/omis/search-integrator/overview) | supports the building of search indexes based on open metadata events. | [Asset Consumer OMAS](/services/omas/asset-consumer/overview) | +| [Security Integrator](/services/omis/security-integrator/overview) | distributes security properties to access control enforcement points. | [Security Manager OMAS](/services/omas/security-manager/overview) | +| [Stewardship Integrator](/services/omis/stewardship-integrator/overview) | exchanges requests for stewardship action (and results) with a human task manager. | [Stewardship Action OMAS](/services/omas/stewardship-action/overview) | +| [Topic Integrator](/services/omis/topic-integrator/overview) | provides cataloguing of topics and event schema for event brokers. | [Data Manager OMAS](/services/omas/data-manager/overview) | !!! post "POST - configure an integration group" ``` diff --git a/site/docs/release-notes/4-4.md b/site/docs/release-notes/4-4.md index e9a35a418e..10cebd6f92 100644 --- a/site/docs/release-notes/4-4.md +++ b/site/docs/release-notes/4-4.md @@ -19,8 +19,8 @@ _**These are DRAFT release notes. The notes will be updated until & when 4.4 is * There are new lineage relationships called [*UltimateSource*](/types/7/0755-Ultimate-Source-Destination) and [*UltimateDestination*](/types/7/0755-Ultimate-Source-Destination) can be used to capture the size and edges of an element's lineage graph have been added. * The *GroupedMedia* relationship is deprecated in favour of [*DataContentForDataSet*](/types/2/0210-Data-Stores). * The [*DataFile*](/types/2/0220-Files-and-Folders) type has a attribute called *fileExtension*. New types that inherit from *DataFile* have been defined for a wide range of file types: [*SpreadsheetFile*](/types/2/0220-Files-and-Folders), [*XMLFile*](/types/2/0220-Files-and-Folders), [*AudioFile*](/types/2/0221-Document-Stores), [*VideoFile*](/types/2/0221-Document-Stores), [*3DImageFile*](/types/2/0221-Document-Stores), [*RasterFile*](/types/2/0221-Document-Stores), [*VectorFile*](/types/2/0221-Document-Stores), [*SourceCodeFile*](/types/2/0280-Software-Development-Assets), [*ExecutableFile*](/types/2/0280-Software-Development-Assets), [*ScriptFile*](/types/2/0280-Software-Development-Assets), [*BuildInstructionFile*](/types/2/0280-Software-Development-Assets), [*YAMLFile*](/types/2/0280-Software-Development-Assets), [*PropertiesFile*](/types/2/0280-Software-Development-Assets), and [*ArchiveFile*](/types/2/0226-Archive-Files). There is also a new relationship called [*ArchiveFileContents*](/types/2/0226-Archive-Files) to identify the contents of an archive file. - * There are new attribute called *isCaseSensitive* and *category* for [*ValidValueDefinition*](/types/5/0545-Reference-Data) that provides additional information used to match valid values. - * There are new types to help automate the curation of assets. They are in a new model called [0207](/types/2/0207-Deployed-Implementation-Types) and are called *DeployedImplementationType*, *FileType* and *CatalogTarget*. + * There are new attribute called *isCaseSensitive* and *category* for [*ValidValueDefinition*](/types/5/0545-Reference-Data) that provides additional information used to match valid values. In addition, there are two new types of valid value relationship: *ValidValueAssociation* and *ConsistentValidValues*. + * The [*ConnectorType*](/types/2/0201-Connectors-and-Connections) entity has new attribute called *deployedImplementationType* to allow a connector writer to record the [deployed implementation type](/concepts/deployed-implementation-type) that the connector works with. ??? warning "XTDB repository connector are now in egeria.git" The XTDB repository connector is now located in the core Egeria git repository which means it is always available in the containers and distributions. There are three new administration commands to simplify the configuration of an XTDB repository. This change is in recognition that XTDB is our premier metadata repository. @@ -88,4 +88,25 @@ _**These are DRAFT release notes. The notes will be updated until & when 4.4 is ??? deprecated "Asset Catalog OMAS and Glossary View OMAS" The Glossary View OMAS and Asset Catalog OMAS function has now moved to Glossary Browser OMVS and Asset Catalog OMVS respectively and so Glossary View OMAS and Asset Catalog OMAS are no longer needed. They will remain in the codebase to support the Egeria UI Application and will be removed at the same time as this runtime. +??? functional "New file extensions to Egeria' standard files" + Files produced by Egeria supplied connectors now have specific file extensions to help identify them in the file system. Previously they used the file extension for their format 9such as `.json`. + + | File Extension | File Type Name | + |------------------|:-------------------------------------------------------------| + | omarchive | Open Metadata Archive File | + | config | OMAG Server Configuration Document | + | cohortregistry | OMAG Server Cohort Registry | + | omalrecord | Open Metadata Audit Log Record File | + | auditlog | Open Metadata Audit Log Folder | + | openlineageevent | Open Lineage Event File | + +??? functional "ConnectorTypes can now identify the deployed implementation type for their connector" + The developer for a [connector](/concepts/connector) can provide information about their connector's implementation in a [connector type](/concepts/connector-type) object that can be retrieved from its [connector provider](/concepts/connector-provider). The [Open Connector Framework (OCF)](/frameworks/ocf/overview) has been extended to allow the connector type to include the deployed implementation type in addition to the type name of the asset subtype that this connector represents. Assets of this type are linked to [connection](/concepts/connection) objects via the [*ConnectionToAsset*](/types/2/205-Connectionr-Linkage) relationship. + +??? functional "Integration Connector Providers can now list catalog target types" + The developer of an [integration connector](/concepts/integration-connector) can provide information about the third party technology it exchanges metadata with through [catalog target types](/concepts/catalog-target). + +??? functional "Extension to the OpenConnectorArchive" + The `OpenMetadataArchive.omarchive` (was `OpenMetadataArchive.json`) now contains reference data for many [file types](/concepts/file-type) and [deployed implementation types](/concepts/deployed-implementation-type). + --8<-- "snippets/abbr.md" diff --git a/site/docs/services/omas/governance-engine/database-agents-and-information.svg b/site/docs/services/omas/governance-engine/database-agents-and-information.svg new file mode 100644 index 0000000000..3643e54299 --- /dev/null +++ b/site/docs/services/omas/governance-engine/database-agents-and-information.svg @@ -0,0 +1,4 @@ + + + +
Integration Connector
to identify schemas within
a database asset
Integration Connector...
Database
(asset describing the database to catalog)
Database...
Catalog Target
Catalog Target
Connection (connection for the database) 
Connection (connecti...
Database Schema
(asset describing the database schema)
Database Schema...
Engine Action to add profile of database to Discovery Analysis Report
Engine Action to add profi...
TargetForAction
TargetForAction
Engine Action to identify lineage origin for database
Engine Action to identify...
TargetForAction
TargetForAction
Discovery Analysis Report
Discovery Analysis R...
Application that is Ultimate Source of database
Application that is...
UltimateSource
UltimateSource
Agents acting on the database resource
Agents acting on the database resource
Information being gathered about the database resource
Information being gathered about the database resource
DataContentForDataSet
DataContentForDataSet
AssetDiscoveryReport
AssetDiscoveryReport
ConnectionForAsset
ConnectionForAssetText is not SVG - cannot display \ No newline at end of file diff --git a/site/docs/services/omas/governance-engine/governance-engine-omas.drawio b/site/docs/services/omas/governance-engine/governance-engine-omas.drawio index 2a0290011c..97ba0e99f1 100644 --- a/site/docs/services/omas/governance-engine/governance-engine-omas.drawio +++ b/site/docs/services/omas/governance-engine/governance-engine-omas.drawio @@ -1,4 +1,4 @@ - + @@ -614,20 +614,8 @@ - - - - - - - - - - - - - + @@ -635,8 +623,8 @@ - - + + @@ -652,20 +640,8 @@ - - - - - - - - - - - - - + @@ -699,6 +675,362 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/site/docs/services/omas/governance-engine/integration-connector-catalog-target-data-folder-example.svg b/site/docs/services/omas/governance-engine/integration-connector-catalog-target-data-folder-example.svg new file mode 100644 index 0000000000..8dbc1713ef --- /dev/null +++ b/site/docs/services/omas/governance-engine/integration-connector-catalog-target-data-folder-example.svg @@ -0,0 +1,4 @@ + + + +
Integration Connector
(description of the integration connector)
Integration Connector...
DataFolder
(asset describing the directory)
DataFolder...
Connection
(for the integration connector)
Connection...
Connector Type
(for the integration connector)
Connector Type...
Connection (connection for the directory) 
Connection (connecti...
Connector Type
(to access a directory)
Connector Type...
Endpoint
(directory path name)
Endpoint...
Catalog Target
Catalog TargetText is not SVG - cannot display \ No newline at end of file diff --git a/site/docs/services/omas/governance-engine/integration-connector-catalog-target-database-example.svg b/site/docs/services/omas/governance-engine/integration-connector-catalog-target-database-example.svg new file mode 100644 index 0000000000..a4a589de6a --- /dev/null +++ b/site/docs/services/omas/governance-engine/integration-connector-catalog-target-database-example.svg @@ -0,0 +1,4 @@ + + + +
Integration Connector
(description of the integration connector)
Integration Connector...
Database
(asset describing the database to catalog)
Database...
Connection
(for the integration connector)
Connection...
Connector Type
(for the integration connector)
Connector Type...
VirtualConnection (connection for the database) 
VirtualConnection (c...
Connector Type
(to access a database)
Connector Type...
Endpoint
(url for database)
Endpoint...
Catalog Target
Catalog Target
Connection
(for the secret store)
Connection...
Connector Type
(for the secret store)
Connector Type...
Endpoint
(for the secret store)
Endpoint...
EmbeddedConnection
EmbeddedConnectionText is not SVG - cannot display \ No newline at end of file diff --git a/site/docs/services/omas/governance-engine/integration-connector-catalog-target.svg b/site/docs/services/omas/governance-engine/integration-connector-catalog-target.svg new file mode 100644 index 0000000000..8708834318 --- /dev/null +++ b/site/docs/services/omas/governance-engine/integration-connector-catalog-target.svg @@ -0,0 +1,4 @@ + + + +
Open Metadata
Element
Open Metadata...
Connection
Connection
Connector Type
Connector Type
Endpoint
Endpoint
Catalog Target
Catalog Target
Integration Connector
(description of the integration connector)
Integration Connector...
Connection
(for the integration connector)
Connection...
Connector Type
(for the integration connector)
Connector Type...Text is not SVG - cannot display \ No newline at end of file diff --git a/site/docs/services/omas/governance-engine/integration-connector-simple.svg b/site/docs/services/omas/governance-engine/integration-connector-simple.svg new file mode 100644 index 0000000000..21ecbf2320 --- /dev/null +++ b/site/docs/services/omas/governance-engine/integration-connector-simple.svg @@ -0,0 +1,4 @@ + + + +
Integration
Connector
Integration...
Connection
Connection
Connector Type
Connector Type
Endpoint
EndpointText is not SVG - cannot display \ No newline at end of file diff --git a/site/docs/services/omas/governance-engine/integration-connector-virtual-connection-example.svg b/site/docs/services/omas/governance-engine/integration-connector-virtual-connection-example.svg new file mode 100644 index 0000000000..56b9f024c9 --- /dev/null +++ b/site/docs/services/omas/governance-engine/integration-connector-virtual-connection-example.svg @@ -0,0 +1,4 @@ + + + +
Integration
Connector
Integration...
VirtualConnection
(for the integration connector)
VirtualConnection...
Connector Type
(for the integration connector)
Connector Type...
Endpoint
(for the integration connector)
Endpoint...
Connection
(for the secret store)
Connection...
Connector Type
(for the secret store)
Connector Type...
Endpoint
(for the secret store)
Endpoint...
EmbeddedConnection
EmbeddedConnectionText is not SVG - cannot display \ No newline at end of file diff --git a/site/docs/services/omas/governance-engine/integration-connector-virtual-connection.svg b/site/docs/services/omas/governance-engine/integration-connector-virtual-connection.svg new file mode 100644 index 0000000000..0af1b44d1b --- /dev/null +++ b/site/docs/services/omas/governance-engine/integration-connector-virtual-connection.svg @@ -0,0 +1,4 @@ + + + +
Integration
Connector
Integration...
VirtualConnection
VirtualConnection
Connector Type
Connector Type
Endpoint
Endpoint
Connection
Connection
Connector Type
Connector Type
Endpoint
Endpoint
EmbeddedConnection
EmbeddedConnectionText is not SVG - cannot display \ No newline at end of file diff --git a/site/docs/services/omas/governance-engine/integration-group-example.svg b/site/docs/services/omas/governance-engine/integration-group-example.svg index 0e74c99fd5..390219e6d4 100644 --- a/site/docs/services/omas/governance-engine/integration-group-example.svg +++ b/site/docs/services/omas/governance-engine/integration-group-example.svg @@ -1,4 +1,4 @@ - + -
Integration
Connector
Integration...
Open Metadata
Elements
Open Metadata...
Connection
Connection
Connector Type
Connector Type
Endpoint
Endpoint
Integration Group
Integration Group
Catalog Targets
Catalog Targets
Integration
Connector
Integration...
Open Metadata
Element
Open Metadata...
Connection
Connection
Connector Type
Connector Type
Endpoint
EndpointText is not SVG - cannot display \ No newline at end of file +
Integration
Connector
Integration...
Open Metadata
Elements
Open Metadata...
Connection
Connection
Integration Group
Integration Group
CatalogTarget
CatalogTarget
Integration
Connector
Integration...
Open Metadata
Element
Open Metadata...
Connection
Connection
ConnectionToAsset
ConnectionToAssetText is not SVG - cannot display \ No newline at end of file diff --git a/site/docs/types/0/0035-Complex-Hosts.svg b/site/docs/types/0/0035-Complex-Hosts.svg index 7ada9a71ed..85688f6ca3 100644 --- a/site/docs/types/0/0035-Complex-Hosts.svg +++ b/site/docs/types/0/0035-Complex-Hosts.svg @@ -1,4 +1,4 @@ -
0035 - Hosts
0035 - Hosts
«entity»
VirtualContainer
«entity»...
0..1
0..1
hostCluster
hostCluster
«entity»
HostCluster
«entity»...
«relationship»
HostClusterMember
«relationship»...
memberRole : string
additionalProperties : map<string, string>
memberRole : string...
«entity»
VirtualMachine
«entity»...
«entity»
BareMetalComputer
«entity»...
«entity»
DockerContainer
«entity»...
«entity»
HadoopCluster
«entity»...
«entity»
KubernetesCluster
«entity»...
«entity»
Host
«entity»...
*
*
deployedElement
deployedElement
«entity»
ITInfrastructure
«entity»...
«entity»
Infrastructure
«entity»...
«entity»
Asset
«entity»...
«relationship»
DeployedOn
«relationship»...
deploymentTime : date
deployer : string
deployerTypeName : string
deployerPropertyName ; string
deploymentStatus : OperationalStatus
deploymentTime : date...
*
*
deployedTo
deployedTo
*
*
managedHosts
managedHosts
«enumeration»
OperationalStatus
«enumeration»...
DISABLED = 0
ENABLED = 1
DISABLED = 0...Text is not SVG - cannot display \ No newline at end of file +
0035 - Hosts
0035 - Hosts
«entity»
VirtualContainer
«entity»...
0..1
0..1
hostCluster
hostCluster
«entity»
HostCluster
«entity»...
«relationship»
HostClusterMember
«relationship»...
memberRole : string
additionalProperties : map<string, string>
memberRole : string...
«entity»
VirtualMachine
«entity»...
«entity»
BareMetalComputer
«entity»...
«entity»
DockerContainer
«entity»...
«entity»
HadoopCluster
«entity»...
«entity»
KubernetesCluster
«entity»...
«entity»
Host
«entity»...
*
*
deployedElement
deployedElement
«entity»
ITInfrastructure
«entity»...
«entity»
Infrastructure
«entity»...
«entity»
Asset
«entity»...
«relationship»
DeployedOn
«relationship»...
deploymentTime : date
deployer : string
deployerTypeName : string
deployerPropertyName ; string
deploymentStatus : OperationalStatus
deploymentTime : date...
*
*
deployedTo
deployedTo
*
*
managedHosts
managedHosts
«enumeration»
OperationalStatus
«enumeration»...
DISABLED = 0
ENABLED = 1
DISABLED = 0...Text is not SVG - cannot display \ No newline at end of file diff --git a/site/docs/types/0/area-0-basic-types-and-infrastructure.drawio b/site/docs/types/0/area-0-basic-types-and-infrastructure.drawio index 862b4c7f60..43a47eb010 100644 --- a/site/docs/types/0/area-0-basic-types-and-infrastructure.drawio +++ b/site/docs/types/0/area-0-basic-types-and-infrastructure.drawio @@ -1,4 +1,4 @@ - + @@ -2567,7 +2567,7 @@ - + diff --git a/site/docs/types/2/0201-Connectors-and-Connections.svg b/site/docs/types/2/0201-Connectors-and-Connections.svg index 8837b92de0..aa4bfe738f 100644 --- a/site/docs/types/2/0201-Connectors-and-Connections.svg +++ b/site/docs/types/2/0201-Connectors-and-Connections.svg @@ -1,4 +1,4 @@ - + -
0201 - Connectors and Connections
0201 - Connectors and Connections
«entity»
ConnectorType
«entity»...
0..1
0..1
connections
connections
connectorType
connectorType
*
*
«entity»
Connection
«entity»...
«relationship»
ConnectionConnectorType
«relationship»...
«entity»
Endpoint
«entity»...
*
*
connections
connections
connectionEndpoint
connectionEndpoint
0..1
0..1
«relationship»
ConnectionEndpoint
«relationship»...
displayName : string
description : string
securedProperties : map <string, string>
configurationProperties : map <string, object>
userId : string
clearPassword : string
encryptedPassword : string
displayName : string...
displayName : string
description : string
supportedAssetTypeName : string
expectedDataFormat : string
connectorProviderClassName : string
connectorFrameworkName : string
connectorInterfaceLanguage : string
connectorInterfaces : array<string>
targetTechnologySource : string
targetTechnologyName : string
targetTechnologyInterfaces : array<string>
targetTechnologyVersions : array<string>
recognizedAdditionalProperties : array <string>
recognizedSecuredProperties : array <string>
recognizedConfigurationProperties : array <string>
displayName : string...
«entity»
Referenceable
«entity»...
«entity»
ConnectorCategory
«entity»...
displayName : string
description : string
targetTechnologySource : string
targetTechnologyName : string
recognizedAdditionalProperties : map <string, boolean>
recognizedSecuredProperties : map <string, boolean>
recognizedConfigurationProperties : map <string, boolean>
displayName : string...
«relationship»
ConnectorImplementationChoice
«relationship»...
connectorTypes
connectorTypes
connectorCategories
connectorCategories
*
*
*
*
«entity»
Referenceable
«entity»...
«classification»
ConnectorTypeDirectory
«classification»...
«entity»
Collection
«entity»...Text is not SVG - cannot display \ No newline at end of file +
0201 - Connectors and Connections
0201 - Connectors and Connections
«entity»
ConnectorType
«entity»...
0..1
0..1
connections
connections
connectorType
connectorType
*
*
«entity»
Connection
«entity»...
«relationship»
ConnectionConnectorType
«relationship»...
«entity»
Endpoint
«entity»...
*
*
connections
connections
connectionEndpoint
connectionEndpoint
0..1
0..1
«relationship»
ConnectionEndpoint
«relationship»...
displayName : string
description : string
securedProperties : map <string, string>
configurationProperties : map <string, object>
userId : string
clearPassword : string
encryptedPassword : string
displayName : string...
displayName : string
description : string
supportedAssetTypeName : string
deployedImplementationType : string
expectedDataFormat : string
connectorProviderClassName : string
connectorFrameworkName : string
connectorInterfaceLanguage : string
connectorInterfaces : array<string>
targetTechnologySource : string
targetTechnologyName : string
targetTechnologyInterfaces : array<string>
targetTechnologyVersions : array<string>
recognizedAdditionalProperties : array <string>
recognizedSecuredProperties : array <string>
recognizedConfigurationProperties : array <string>
displayName : string...
«entity»
Referenceable
«entity»...
«entity»
ConnectorCategory
«entity»...
displayName : string
description : string
targetTechnologySource : string
targetTechnologyName : string
recognizedAdditionalProperties : map <string, boolean>
recognizedSecuredProperties : map <string, boolean>
recognizedConfigurationProperties : map <string, boolean>
displayName : string...
«relationship»
ConnectorImplementationChoice
«relationship»...
connectorTypes
connectorTypes
connectorCategories
connectorCategories
*
*
*
*
«entity»
Referenceable
«entity»...
«classification»
ConnectorTypeDirectory
«classification»...
«entity»
Collection
«entity»...Text is not SVG - cannot display \ No newline at end of file diff --git a/site/docs/types/2/0207-Deployed-Implementation-Types.md b/site/docs/types/2/0207-Deployed-Implementation-Types.md deleted file mode 100644 index eccf224b76..0000000000 --- a/site/docs/types/2/0207-Deployed-Implementation-Types.md +++ /dev/null @@ -1,34 +0,0 @@ - - - -# 0207 Deployed Implementation Types - -A [deployed implementation type](/concepts/deployed-implementation-type) describes a class of technology that is deployed in an organization's IT landscape. It is represented in open metadata to allow defaults for cataloguing, management and governance to be managed for this class of technology. - - -![UML](0207-Deployed-Implementation-Types.svg) - - -## DeployedImplementationType entity - -The *DeployedImplementationType* entity is a [Referenceable](/types/0/0010-Base-Model) that describes a class of technology. Its *displayName* can be used for UI dropdown menus. The *deployedImplementationType* property is the value to use in the property of the same name found in concrete *Asset* entities: - -* [Infrastructure](/types/0/0010-Base-Model) -* [DataSet](/types/0/0010-Base-Model) -* [DataStore](/types/2/0210-Data-Stores) -* [DeployedAPI](/types/2/0212-Deployed-APIs) -* [DeployedSoftwareComponent](/types/2/0215-Software-Components) -* [DataFeed](/types/2/0223-Events-and-Logs) - -In addition, *deployedImplementationType* is found in [*SoftwareServerCapability*](/types/0/0042-Software-Capabilities) entities, [*CloudPlatform*](/types/0/0090-Cloud-Platforms-and-Services) classifications and [*ServerPurpose*](/types/0/0041-Server-Purpose) classifications. - -## FileType entity - -The *FileType* entity is a specialization of *DeployedImplementationType* that captures specifics about different types of files. - -??? education "Further information" - - [Automated Curation OMVS](/services/omvs/automated-curation/overview) - ---8<-- "snippets/abbr.md" - \ No newline at end of file diff --git a/site/docs/types/2/0207-Deployed-Implementation-Types.svg b/site/docs/types/2/0207-Deployed-Implementation-Types.svg deleted file mode 100644 index 6bfaf607f1..0000000000 --- a/site/docs/types/2/0207-Deployed-Implementation-Types.svg +++ /dev/null @@ -1,4 +0,0 @@ - - - -
0207 - Deployed Implementation Types
0207 - Deployed Implementation Types
«entity»
DeployedFileType
«entity»...
«entity»
DeployedImplementationType
«entity»...
fileExtension : string
fileNamePattern : string
isDirectory : boolean
fileExtension : string...
displayName : string
description : string
deployedImplementationType : string
typeName : string
displayName : string...
«entity»
Referenceable
«entity»...Text is not SVG - cannot display \ No newline at end of file diff --git a/site/docs/types/2/area-2-assets.drawio b/site/docs/types/2/area-2-assets.drawio index 2c41d25aed..89ef91695a 100644 --- a/site/docs/types/2/area-2-assets.drawio +++ b/site/docs/types/2/area-2-assets.drawio @@ -1,4 +1,4 @@ - + @@ -168,55 +168,50 @@ + + + - + - - + + - + - + - + - + - + + + + - + - - - - - - - - - - - - + - + - + @@ -237,13 +232,13 @@ - + - - + + - + diff --git a/site/docs/types/4/area-4-governance.drawio b/site/docs/types/4/area-4-governance.drawio index c7e757d671..7b6ab61eba 100644 --- a/site/docs/types/4/area-4-governance.drawio +++ b/site/docs/types/4/area-4-governance.drawio @@ -1,6 +1,6 @@ - + - + diff --git a/site/docs/types/5/0545-Reference-Data.md b/site/docs/types/5/0545-Reference-Data.md index b21e53c7bf..3597105e7a 100644 --- a/site/docs/types/5/0545-Reference-Data.md +++ b/site/docs/types/5/0545-Reference-Data.md @@ -74,6 +74,10 @@ The *ConsistentValidValues* relationship allows this mapping to be captured. ![Consistent Valid Values](/features/reference-data-management/consistent-valid-values.svg) +## ValidValueAssociation relationship + +The *ValidValueAssociation* relationship allows arbitrary relationships between valid value definitions to be recorded. + ## ValidValuesImplementation relationship and ReferenceData classification The implementation of a code table is represented in open metadata using an [Asset](/types/0/0010-Base-Model). This asset can be tagged as containing valid values using the *ReferenceData* classification. diff --git a/site/docs/types/5/0545-Reference-Data.svg b/site/docs/types/5/0545-Reference-Data.svg index 49702f793c..07592659ea 100644 --- a/site/docs/types/5/0545-Reference-Data.svg +++ b/site/docs/types/5/0545-Reference-Data.svg @@ -1,4 +1,4 @@ -
0545 - Reference Data
0545 - Reference Data
«entity»
ValidValueDefinition
«entity»...
name : string
description : string
category : string
usage : string
scope : string
preferredValue : string
isDeprecated : boolean
isCaseSensitive : boolean
name : string...
«entity»
Referenceable
«entity»...
«entity»
Asset
«entity»...
«relationship»
ValidValuesImplementation
«relationship»...
symbolicName : string
implementationValue : string
additionalValues : map<string, string>
symbolicName : string...
*
*
*
*
validValues
validValues
validValuesImplementation
validValuesImplementation
«classification»
ReferenceData
«classification»...
«entity»
ValidValuesSet
«entity»...
«relationship»
ValidValuesAssignment
«relationship»...
strictRequirement : boolean
strictRequirement : boolean
memberOfValidValuesSet
memberOfValidVa...
validValuesSet
validValuesSet
validValuesConsumer
validValuesCons...
validValues
validValues
*
*
*
*
*
*
0..1
0..1
«relationship»
ValidValueMember
«relationship»...
isDefaultValue : boolean
isDefaultValue : boolean
matchingValue
matchingValue
matchingValue
matchingValue
«relationship»
ValidValuesMapping
«relationship»...
associationDescription : string
confidence : int
steward : string
stewardTypeName : string
stewardPropertyName : string
notes : string
associationDescription : string...
*
*
*
*
*
*
*
*
«relationship»
ReferenceValueAssignment
«relationship»...
attributeName : string
confidence : int
steward : string
stewardTypeName : string
stewardPropertyName : string
notes : string
attributeName : string...
assignedItem
assignedItem
referenceValue
referenceVal...
consistentValue
consistentVa...
consistentValue
consistentValue
«relationship»
ConsistentValidValues
«relationship»...
*
*
*
*Text is not SVG - cannot display \ No newline at end of file +
0545 - Reference Data
0545 - Reference Data
«entity»
ValidValueDefinition
«entity»...
name : string
description : string
category : string
usage : string
scope : string
preferredValue : string
isDeprecated : boolean
isCaseSensitive : boolean
name : string...
«entity»
Referenceable
«entity»...
«entity»
Referenceable
«entity»...
«relationship»
ValidValuesImplementation
«relationship»...
symbolicName : string
implementationValue : string
additionalValues : map<string, string>
symbolicName : string...
*
*
*
*
validValues
validValues
validValueImplementations
validValueImplementations
«classification»
ReferenceData
«classification»...
«entity»
ValidValuesSet
«entity»...
«relationship»
ValidValuesAssignment
«relationship»...
strictRequirement : boolean
strictRequirement : boolean
memberOfValidValuesSet
memberOfValidVa...
validValuesSet
validValuesSet
validValuesConsumer
validValuesCons...
validValues
validValues
*
*
*
*
*
*
0..1
0..1
«relationship»
ValidValueMember
«relationship»...
isDefaultValue : boolean
isDefaultValue : boolean
matchingValue
matchingValue
matchingValue
matchingValue
«relationship»
ValidValuesMapping
«relationship»...
associationDescription : string
confidence : int
steward : string
stewardTypeName : string
stewardPropertyName : string
notes : string
associationDescription : string...
*
*
*
*
*
*
*
*
«relationship»
ReferenceValueAssignment
«relationship»...
attributeName : string
confidence : int
steward : string
stewardTypeName : string
stewardPropertyName : string
notes : string
attributeName : string...
assignedItem
assignedItem
referenceValue
referenceVal...
consistentValue
consistentVa...
consistentValue
consistentValue
«relationship»
ConsistentValidValues
«relationship»...
*
*
*
*
*
*
*
*
associatedValues1
associatedValues1
associatedValues2
associatedValues2
«relationship»
ValidValueAssociation
«relationship»...
associationName : string
associationType : string
additionalProperties : map<string, string>
associationName : string...
«entity»
Asset
«entity»...Text is not SVG - cannot display \ No newline at end of file diff --git a/site/docs/types/5/area-5-models-and-schema.drawio b/site/docs/types/5/area-5-models-and-schema.drawio index 354d8c039b..db9d2c692a 100644 --- a/site/docs/types/5/area-5-models-and-schema.drawio +++ b/site/docs/types/5/area-5-models-and-schema.drawio @@ -1,4 +1,4 @@ - + @@ -2701,7 +2701,7 @@ - + @@ -2709,186 +2709,185 @@ - + - + - + - + - - + + - + - - + + - + - + - + - + - - + + - - - - + - + - + - + - + - - + + - + - + - + - + - + - + - + - + - + - + - - + + - + - - + + + - + - + - + - - + + - - + + + - + - + - + - + - + - + - - + + - + - + - + - + - + - - + + @@ -2896,42 +2895,82 @@ - - + + - - - + + + - + - + - + - + - - + + - - + - + - + - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/site/mkdocs.yml b/site/mkdocs.yml index 08829a44ab..402d016ced 100644 --- a/site/mkdocs.yml +++ b/site/mkdocs.yml @@ -664,6 +664,7 @@ nav: - Asset Log Message: concepts/asset-log-message.md - Audit Log: concepts/audit-log.md - Audit Log Destination Connector: concepts/audit-log-destination-connector.md + - Catalog Target: concepts/catalog-target.md - Cohort Events: concepts/cohort-events.md - Cohort Member: concepts/cohort-member.md - Cohort Registry: concepts/cohort-registry.md