Merge pull request #878 from mandy-chessell/dev-dojo

Improve description of integration groups and catalog targets

site/docs/concepts/catalog-target.md

@@ -0,0 +1,60 @@
+<!-- SPDX-License-Identifier: CC-BY-4.0 -->
+<!-- Copyright Contributors to the Egeria project. -->
+
+# Catalog Target
+
+A *catalog target* describes the third party technology ([resource](/concepts/resource)) that a specific [integration connector](/concept/integration-connector) is exchanging metadata with.  
+
+## Catalog Target Types
+
+The developer of an integration connector is able to specify one or more types of technology supported by the connector.  These are called *Catalog Target Types* and they are specified in the implementation of the integration connector's [connector provider](/concepts/connector-provider). Catalog Target Types include the following properties:
+
+* **typeName** - the [open metadata type name](/types) of the element that describes a specific catalog type.
+* **deployedImplementationType** - the [deployed implementation type](/concepts/deployed-implementation-type) of the element that describes a specific catalog type.  This allows a more fine-grained specification of the type of technology supported by the integration connector.
+* **otherPropertyNames** - as the name suggests, this allows other properties that are useful to a person deploying the connector.
+
+The catalog target type is given a name called the **catalogTargetName**.  Here is a code snippet from the [Data Files Monitor Integration Connector's](/connectors/integration/data-files-monitor-integration-connector) connector provider showing how the catalog target types are set up.  This connector is configured with the name of a directory (folder) on a filesystem and it catalogues the files underneath this folder.  Therefore, the catalog target type describes a directory.
+
+```java
+
+    static public final String CATALOG_TARGET_NAME  = "directoryToMonitor";
+
+    CatalogTargetType catalogTargetType = new CatalogTargetType();
+
+    catalogTargetType.setTypeName(OpenMetadataTypesMapper.DATA_FOLDER_TYPE_NAME);
+    catalogTargetType.setDeployedImplementationType(DeployedImplementationType.DATA_FOLDER.getDeployedImplementationType());
+
+    super.catalogTargetTypes.put(CATALOG_TARGET_NAME, catalogTargetType);
+```
+
+This information is then available to the person choosing the appropriate integration connector and setting it up to run. 
+
+At runtime, the specific catalog target can be supplied to an integration connector via its [endpoint](/concepts/endpoint) which is connected to its entity's [connection](/concepts/connection), or via one or more *CatalogTarget* relationships. 
+
+## CatalogTarget relationship
+
+The [*CatalogTarget*](/types/4/0464-Dynamic-Integration-Groups) relationship links an integration connector's entity to an entity that describes the specific third party technology that the integration connector should exchange metadata with. It is used when configuring an [integration daemon](/concepts/integration-daemon) using [integration groups](/concepts/integration-group).  With this style of configuration, details of the integration connectors to run are described in the open metadata repositories that the integration daemon is connected to.  The catalog target relationship is part of this open metadata description.
+
+The diagram below shows the open metadata elements describing an integration group.  This integration group has two integration connectors.  Each integration connector is linked to a [connection](/concepts/connection) that describes how to create a running instance of the integration connector.  This connection may optionally include the endpoint information of the third party technology that the integration connector is to exchange metadata with. The *CatalogTarget* relationship provides an alternative way to supply details of one, or more third party technologies that it should connect to.  These technologies are described, typically by an [asset](/concepts/asset) open metadata element (shown in yellow) and are linked to the integration connector's element using the *CatalogTarget* relationship.
+
+![Integration group](/services/omas/governance-engine/integration-group-example)
+
+The [connection](/concepts/connection) information associated with the third party technology can be used by the integration connector to create a [resource connector](/concepts/digital-resource-connector) that acts as a client to call the third party technology.
+
+The diagram below shows an example of a metadata description for an integration connector that catalogs files found in a directory (folder).  The catalog target is the asset that describes the directory where the files are located.  Linked to the asset is a connection for the data folder connector that will provide access to the files in the directory.
+
+![Data file cataloguing integration connector](/services/omas/governance-engine/integration-connector-catalog-target-data-folder-example)
+
+This style of configuration means that the connection to a specific third party technology only needs to be specified once and can be used for multiple purposes.  This is important for technologies that need a complex connection structure that may need to be updated from time to time.  For example, the next diagram shows an integration connector that is linked to a database asset.  This database's connection is complex and includes an embedded secret store connector.  If details of the secrets used to connect to the database need to be changed, it only needs to be changed in one connection.
+
+![Database cataloguing integration connector](/services/omas/governance-engine/integration-connector-catalog-target-database-example)
+
+The *CatalogTarget* relationship is similar to the [*TargetForAction*](/types/4/0463-Engine-Actions) relationships that links [engine actions](/concepts/engine-action) to the elements describing the technology that the associated [governance service](/concepts/governance-service) is to work on.
+
+From the perspective of the database element, these relationships make it possible to see the automated processing (agents) working with the database, as well as the metadata they are gathering/validating/updating.  This is useful for either the owner or the consumer of the resource to know how the technology and its metadata are being maintained.
+
+![Database's agents and information](/services/omas/governance-engine/database-agents-and-information)
+
+
+
+--8<-- "snippets/abbr.md"

site/docs/concepts/index.md

@@ -42,6 +42,8 @@
 ## C
 
 - [Catalog Integrator OMIS](/services/omis/catalog-integrator/overview)
+- [Catalog Target](/concepts/catalog-target)
+- [Catalog Target Type](/concepts/catalog-target)
 - [Cohort](/concepts/cohort-member)
 - [Cohort Event](/concepts/cohort-events)
 - [Cohort Member](/concepts/cohort-member)
@@ -103,6 +105,7 @@
 - [Discovery Server](/concepts/engine-host)
 - [Discovery Service](/concepts/open-discovery-service)
 - [Display Integrator OMIS](/services/omis/display-integrator/overview)
+- [Dynamic Integration Group](/concepts/integration-group)
 
 ## E
 
@@ -128,7 +131,7 @@
 
 - [Favorite Things Collections](/concepts/favorite-things-collection)
 - [Feedback](/concepts/feedback)
-- [File Type](/concepts/deployed-implementation-type/#file-type)
+- [File Type](/concepts/file-type)
 - Files
 
     - [omag.server.servername.config](/concepts/configuration-document)

site/docs/concepts/integration-group.md

@@ -1,18 +1,44 @@
----
-hide:
-- toc
----
-
 <!-- SPDX-License-Identifier: CC-BY-4.0 -->
 <!-- Copyright Contributors to the ODPi Egeria project. -->
 
 # Integration Group
 
 An *integration group* describes a collection of [integration connectors](/concepts/integration-connector) that are to run together in one or more [integration daemons](/concepts/integration-daemon).  
 
-![integration group example](/services/omas/governance-engine/integration-group-example.svg)
+## Specification
+
+Each integration connector is represented by an entity of type [*IntegrationConnector*](/types/4/0464-Dynamic-Integration-Groups) linked to a [*Connection*](/types/2/0201-Connectors-and-Connections) via the [*ConnectionToAsset*](/types/2/0205-Connection-Linkage) relationship.  This [connection](/concepts/connection) is used to instantiate the integration connector.  It is also passed to the integration connector as it initializes, so it is used to pass configuration properties that describes its desired behaviour.   The connection's optional endpoint describes the [catalog target](/concepts/catalog-target).  Alternatively, the integration connector's catalog target(s) can be supplied via the [*CatalogTarget*](/types/4/0464-Dynamic-Integration-Groups) relationship.  
+
+![integration group structure](/services/omas/governance-engine/integration-group-example.svg)
+
+## Examples of integration connector definitions
+
+The exact specification of an integration connector depends on how the integration connector is implemented.  Details of the integration connectors supplied with Egeria, and how to configure them are available in the [connector catalog](/connectors/#integration-connectors).
+
+Some integration connectors use a simple connection structure. The [*ConnectorType*](/concepts/connector-type) linked to the connection identifies the implementation of the integration connector.  The endpoint describes how to connect to the third party technology.
+
+![integration connector with a simple connection](/services/omas/governance-engine/integration-connector-simple.svg)
+
+Some connectors need a virtual connection with embedded connectors that describe different resources.
+
+![Integration connector with a virtual connection](/services/omas/governance-engine/integration-connector-virtual-connection.svg)
 
-The integration daemon is configured with a list of qualified names for various integration groups.  It retrieves the list of integration connectors in each of these groups from the open metadata ecosystem and starts to run them.  It continues to monitor for changes to the integration groups and their linked integration connectors, changing the connectors it is running as needed.  
+For example, this integration connector has a virtual connection with an embedded secrets store for managing passwords ans certificates.
+
+![Example of integration connector with a virtual connection](/services/omas/governance-engine/integration-connector-virtual-connection-example.svg)
+
+Alternatively, using [catalog targets](/concepts/catalog-target) provides a more efficient way to supply information about the technology that the integration connector is to work with.
+
+The connection for the integration connector just needs the connector type to describe its implementation.
+The *CatalogTarget* relationship links to an asset that describes the technology that the integration connector is to work with.
+
+![Example of integration connector with a virtual connection](/services/omas/governance-engine/integration-connector-catalog-target.svg)
+
+The integration connector can have multiple catalog log targets that can change over time.
+
+## Configuring integration groups
+
+An [integration daemon is configured](/guides/admin/servers/configuring-an-integration-daemon/) with a list of qualified names for various integration groups.  It retrieves the list of integration connectors in each of these groups from the open metadata ecosystem and starts to run them.  It continues to monitor for changes to the integration groups and their linked integration connectors, changing the connectors it is running as needed.  
 
 ![integration group operation](/services/omas/governance-engine/integration-group-use.svg)
 
@@ -24,4 +50,5 @@ The integration daemon is configured with a list of qualified names for various
 
     The open metadata types for this definition are in model [0464 - Dynamic Integration Groups](/types/4/0464-Dynamic-Integration-Groups) - specifically notice that *IntegrationGroup* and *IntegrationConnector* are linked by the *RegisteredIntegrationConnector* relationship.
 
+
 --8<-- "snippets/abbr.md"

site/docs/guides/admin/servers/configuring-an-integration-daemon.md

@@ -1,12 +1,28 @@
 <!-- SPDX-License-Identifier: CC-BY-4.0 -->
 <!-- Copyright Contributors to the Egeria project. -->
 
-# Configuring an [integration daemon](/concepts/integration-daemon)
+# Configuring an integration daemon
 
-An *Integration Daemon* is configured by creating a [configuration document](/concepts/configuration-document).  Below is the outline structure of the integration daemon's configuration document.
+An [*Integration Daemon*](/concepts/integration-daemon) runs [integration connectors](/concepts/integraton-connector) that are exchanging metadata with third party technologies.  For example, an integration connector may catalog files from a file system, or the tables and columns in a database.  Alternatively, it may be gathering lineage information or distributing security information to LDAP.
+
+Integration connectors are typically long-running.  Their implementation calls one of the [Open Metadata Integration Services (OMISs)](/services/omis) to maintain open metadata and access other useful integration utilities.  The integration daemon needs information about which integration connectors to run along with details of the services they need to operate successfully.  The information needed to bootstrap the integration daemon is configured by creating a [configuration document](/concepts/configuration-document) which is passed to the integration daemon when it starts up.
+
+## Overview
+
+Below is the outline structure of the integration daemon's configuration document.
 
 ![Configuration for an integration daemon](integration-daemon-config.svg)
 
+The *Local Server URL Root* shown in light grey can be optionally configured to provide the [platformURLRoot](/concepts/platform-url-root) for the default [OMAG Server Platform](/concepts/omag-server-platform) where this integration daemon is to be deployed to.  It is only used if the deploy command does not include the destination platformURLRoot.
+
+The properties shown in blue are standard properties for an [OMAG Server](/concepts/omag-server).
+
+The properties shown in green are specific to integration daemons and they identify which integration connectors to run.  There are two approaches.   An integration daemon can be configured with either approach or both: 
+
+* The *Integration Services* section provides details of the integration connectors that are statically configured into the integration daemon.  They are organized by the integration services that each integration connector is using.  These integration connectors will be automatically started with the integration daemon.  Any changes to these integration connectors need to be made in the configuration document and then the integration daemon needs to be restarted for the changes to take effect.
+* The *Dynamic Integration Groups* section points to lists of integration connectors defines using open metadata elements store in the open metadata repositories.  The integration daemon monitors these lists to determine which integration connectors to run.  It dynamically updates the connectors it is running based on the values in the open metadata elements.
+
+
 ??? info "Configuring the basic properties"
     --8<-- "docs/guides/admin/servers/configuring-omag-server-basic-properties.md"
 
@@ -35,6 +51,7 @@ An *Integration Daemon* is configured by creating a [configuration document](/co
 ??? education "Further information"
     - For help in fixing any error you find using the integration daemon, visit the [integration daemon diagnostic guide](/guides/diagnostic/integration-daemon-diagnostic-guide).
     - Link to the [Egeria solutions](/patterns/metadata-integration) to see the integration daemon in action.
-    - Link to the [integration daemon services](/services/integration-daemon-services) to understand how the integration daemon is implemented.
+    - Link to the [integration daemon services](/services/integration-daemon-services) to understand how the integration daemon is implemented.  It also describes the integration daemon's REST API calls to restart and refresh integration connectors; and to make temporary changes to an integration connector's configuration.  These changes are only made in-memory and they are lost when the integration connector restarts. 
+
 
 --8<-- "snippets/abbr.md"