Merge pull request #796 from popa-raluca/docs

DE OMAS - update documentation & move component to tech preview

site/docs/services/omas/data-engine/add-data-flows.md

@@ -66,8 +66,8 @@ POST {serverURLRoot}/servers/{serverName}/open-metadata/access-services/data-eng
 }
 ```
 `externalSourceName` - qualifiedName of the external data engine tool.
-Note that you need to register the data engine tool with [register-data-engine-tool](register-data-engine-tool.md)
-before adding data flows
+ Note that you need to register the data engine tool with [register-data-engine-tool](register-data-engine-tool.md) 
+ before adding data flows
 `VoidReponse` with status and error message if failing.
 
 

site/docs/services/omas/data-engine/create-data-file.md

@@ -6,7 +6,9 @@
 Create a data file, with the associated schema, columns, folder hierarchy and a connection. The schema is generated
 automatically, the columns according to the payload, the folder structure is based on the value of file.pathName and 
 the connection, which also includes an endpoint, is based on the value of optional fields file.protocol and
-file.networkAddress
+file.networkAddress. A connector type is linked to the connection indicating which implementation the connection uses.
+By default, the connector type is a OCF one, and it's chosen based on the file type. The data file can also be marked as incomplete,
+which means that it is missing data to be completely described.
 
 A csv file can be created by changing the value of field file.type from "DataFile" into "CSVFile"
 
@@ -23,6 +25,7 @@ POST {serverURLRoot}/servers/{serverName}/open-metadata/access-services/data-eng
         "type" : "DataFile", 
         "qualifiedName" : "(file)=DATAFILE.DAT",
         "displayName" : "dataFile.dat",
+        "incomplete" : false,
         "pathName" : "/home/files/dataFile.dat",
         "protocol" : "ftp",
         "networkAddress" : "localhost",
@@ -46,8 +49,8 @@ POST {serverURLRoot}/servers/{serverName}/open-metadata/access-services/data-eng
 `GUIDResponse` - response containing the data file GUID, with status and error message if failing  
 `type` - type of file to be created, either a DataFile or a CSVFile  
 `pathName` - file system location of the data file  
-`protocol` - used to create connection and endpoint  
-`networkAddress` - used to create connection and endpoint  
+`protocol` - used when creating the connection, endpoint and the relationship to the proper connector type
+`networkAddress` - used to create connection, endpoint and the relationship to the proper connector type
 
 
 ---8<-- "snippets/abbr.md"

site/docs/services/omas/data-engine/create-database-schema.md

@@ -0,0 +1,47 @@
+<!-- SPDX-License-Identifier: CC-BY-4.0 -->
+<!-- Copyright Contributors to the ODPi Egeria project. -->
+
+# Create database schema
+
+Create a database schema, linking it to a given database, if any is provided. The database schema can also be marked as incomplete,
+which means that it is missing data to be completely described, like the database to which it is linked.
+
+More examples with all available properties for a database can be found in the
+[sample collection](samples/collections/DataEngine-asset_endpoints.postman_collection.json)
+
+```
+POST {serverURLRoot}/servers/{serverName}/open-metadata/access-services/data-engine/users/{userId}/database-schemas
+
+{
+    "databaseQualifiedName": "(host)=HOST::(database)=MINIMAL",
+    "databaseSchema": {
+        "qualifiedName": "(host)=HOST::(database)=MINIMAL::(database_schema)=DB2INST1",
+        "displayName": "DB2INST1",
+        "description": "DB2INST1 database schema description",
+        "owner": "Administrator IIS",
+        "ownerType": "USER_ID",
+        "zoneMembership": [
+            "default"
+        ]
+    },
+    "externalSourceName": "(organization)=Company::(project)=ExternalDataPlatform",
+    "incomplete": false
+}
+```
+
+`databaseQualifiedName` - name of the database to which the database schema will be linked
+`externalSourceName` - qualifiedName of the external data engine tool.
+ Note that you need to register the data engine tool with [register-data-engine-tool](register-data-engine-tool.md) 
+ before creating any process or entity
+`incomplete` - determines if the database schema has all needed information to be described. If the value is `true`,
+then the database schema is not connected to a database
+<br><br>
+`GUIDResponse` - response containing the database GUID, with status and error message if failing 
+
+
+---8<-- "snippets/abbr.md"
+
+
+
+
+

site/docs/services/omas/data-engine/create-database.md

@@ -3,8 +3,11 @@
 
 # Create database
 
-Create a database, with the associated schema type. If no schema is provided, one will be created automatically.
-If provided, the fields protocol and networkAddress will trigger the creation of a connection and an endpoint
+Create a database and all the entities inside it, if any. These entities are a database schema and relational tables.
+If provided, the fields protocol and networkAddress will trigger the creation of a connection, an endpoint and a relationship
+to a proper connector type. A connector type is linked to the connection indicating which implementation the connection uses.
+By default, the connector type is a OCF one, but for the moment there is no current implementation for it.
+This means none of these three entities will be created (connection, connector type, endpoint).
 
 More examples with all available properties for a database can be found in the
 [sample collection](samples/collections/DataEngine-asset_endpoints.postman_collection.json)
@@ -31,8 +34,8 @@ POST {serverURLRoot}/servers/{serverName}/open-metadata/access-services/data-eng
  Note that you need to register the data engine tool with [register-data-engine-tool](register-data-engine-tool.md) 
  before creating any process
 `GUIDResponse` - response containing the database GUID, with status and error message if failing  
-`protocol` - used to create connection and endpoint  
-`networkAddress` - used to create connection and endpoint
+`protocol` - used to create connection, endpoint and the relationship to the proper connector type
+`networkAddress` - used to create connection, endpoint and the relationship to the proper connector type
 
 ---8<-- "snippets/abbr.md"
 

site/docs/services/omas/data-engine/create-event-type.md

@@ -0,0 +1,39 @@
+<!-- SPDX-License-Identifier: CC-BY-4.0 -->
+<!-- Copyright Contributors to the ODPi Egeria project. -->
+
+# Create an event type
+
+Create an event type, with the associated event schema attributes.
+More examples with all available properties for an event type can be found in the
+[sample collection](samples/collections/DataEngine-topics-lineage_examples.postman_collection.json)
+
+```
+POST {serverURLRoot}/servers/{serverName}/open-metadata/access-services/data-engine/users/{userId}/event-types
+
+{
+    "externalSourceName": "(organization)=Company::(project)=ExternalDataPlatform",
+    "topicQualifiedName": "(topic)=test-topic",
+    "eventType": {
+        "qualifiedName": "(topic)=test-topic::(eventType)=test-event",
+        "displayName": "test-event",
+        "eventSchemaAttributes": [
+            {
+                "qualifiedName": "(topic)=test-topic::(eventType)=test-event::(eventAttribute)=event-attribute",
+                "displayName": "event-attribute"
+            }
+        ]
+    }
+}
+```
+
+`externalSourceName` - qualifiedName of the external data engine tool.
+ Note that you need to register the data engine tool with [register-data-engine-tool](register-data-engine-tool.md) 
+ before creating any process
+`GUIDResponse` - response containing the topic GUID, with status and error message if failing  
+
+
+
+
+
+
+

site/docs/services/omas/data-engine/create-process.md

@@ -3,9 +3,11 @@
 
 # Create a process
 
-Create a Process with port implementations. For each port, it creates the associated schema type and columns.
+Create a Process, with port implementations.
+For each port, it creates the associated schema type and columns.
 
-Check [create-port-implementation](create-port-implementation.md) and [create-schema-types](create-schema-type.md) for examples of the more granular payloads.
+Check [create-port-implementation](create-port-implementation.md) 
+and [create-schema-types](create-schema-type.md) for examples of the more granular payloads.
 
 More examples can be found in the
 [sample collection](samples/collections/DataEngine-process_endpoints.postman_collection.json)

site/docs/services/omas/data-engine/create-topic.md

@@ -0,0 +1,44 @@
+<!-- SPDX-License-Identifier: CC-BY-4.0 -->
+<!-- Copyright Contributors to the ODPi Egeria project. -->
+
+# Create topic
+
+Create a topic, with the associated event types. For each event type it creates the associated event schema attributes
+More examples with all available properties for a topic can be found in the
+[sample collection](samples/collections/DataEngine-topics-lineage_examples.postman_collection.json)
+
+```
+POST {serverURLRoot}/servers/{serverName}/open-metadata/access-services/data-engine/users/{userId}/topics
+
+{
+    "externalSourceName": "(organization)=Company::(project)=ExternalDataPlatform",
+    "topic": {
+        "qualifiedName": "(topic)=test-topic",
+        "displayName": "test-topic",
+        "eventTypes": [
+            {
+                "qualifiedName": "(topic)=test-topic::(eventType)=test-event",
+                "displayName": "test-event",
+                "eventSchemaAttributes": [
+                    {
+                        "qualifiedName": "(topic)=test-topic::(eventType)=test-event::(eventAttribute)=event-attribute",
+                        "displayName": "event-attribute"
+                    }
+                ]
+            }
+        ]
+    }
+}
+```
+
+`externalSourceName` - qualifiedName of the external data engine tool.
+ Note that you need to register the data engine tool with [register-data-engine-tool](register-data-engine-tool.md) 
+ before creating any process
+`GUIDResponse` - response containing the topic GUID, with status and error message if failing  
+
+
+
+
+
+
+

site/docs/services/omas/data-engine/de-client.md

@@ -0,0 +1,87 @@
+<!-- SPDX-License-Identifier: CC-BY-4.0 -->
+<!-- Copyright Contributors to the ODPi Egeria project.  -->
+
+# Data Engine Open Metadata Access Service (OMAS) Client
+
+There are different client implementations available:
+
+* `DataEngineRESTClient` - Default client implementation interacting via HTTP transport mechanism and default OCF REST client.
+* `DataEngineRESTConfigurationClient` - An extension to the REST interface that can be used as helper to automate configuration of the event based client by retrieving connection details used to build the client topic connector.
+* `DataEngineEventClient` - Client implementation interacting via Events transport mechanism. It is using `DataEngineInTopicClientConnector` that produces events to the input topic of Data Engine access service.
+
+
+HTTP based REST clients can be constructed in following way:
+
+* No authentication embedded in the HTTP request - for test systems.
+* Basic authentication using a userId and password embedded in the HTTP request.
+
+Both constructors take the [URL root for the server platform](https://egeria-project.org/concepts/platform-url-root/)
+where the Data Engine OMAS is running and its [server name](https://egeria-project.org/concepts/server-name/).
+
+Here is a code example with the user id and password specified:
+
+```java
+DataEngineClient client = new DataEngineRESTClient   ("cocoMDS1",
+                                           "https://localhost:9444",
+                                           "cocoUI",
+                                           "cocoUIPassword");
+
+```
+This client is set up to call the `cocoMDS1` server running on the `https://localhost:9444`
+OMAG Server Platform.  The userId and password is for the application
+where the client is running.  The userId of the real end user is passed
+on each request.
+
+Similarly, extended REST configuration client can help retrieving additional configuration details from the remote server can be constructed in following way:
+
+```java
+DataEngineRESTConfigurationClient client = new DataEngineRESTConfigurationClient("remoteServerName",
+                                        "https://localhost:9444",
+                                        "remoteUserName",
+                                        "remoteUserPassword");
+ConnectionResponse connectionResponse = client.getInTopicConnection("remoteServerName","remoteUserName");
+```
+
+Event based client can be constructed by injecting instance of the DataEngineInTopicClientConnector.
+In the example below we can create the connector with remote configuration retrieved previously and pass it to the event based client:
+
+```java
+
+DataEngineInTopicClientConnector topicConnector = (DataEngineInTopicClientConnector) connectorBroker.getConnector(connectionResponse.getConnection());
+
+DataEngineClient client = new DataEngineEventClient(dataEngineInTopicClientConnector);
+```
+
+## Client operations
+
+Once you have an instance of the client, you can use it to create, update or delete the open metadata entities for the ETL jobs.
+
+* [Create External Data Engine](https://odpi.github.io/egeria/org/odpi/openmetadata/accessservices/dataengine/client/DataEngineClient.html#createExternalDataEngine(java.lang.String,org.odpi.openmetadata.accessservices.dataengine.model.Engine))
+* [Get external source name](https://odpi.github.io/egeria/org/odpi/openmetadata/accessservices/dataengine/client/DataEngineClient.html#getExternalSourceName())
+* [Set external source name](https://odpi.github.io/egeria/org/odpi/openmetadata/accessservices/dataengine/client/DataEngineClient.html#setExternalSourceName(java.lang.String))
+* [Upsert Process](https://odpi.github.io/egeria/org/odpi/openmetadata/accessservices/dataengine/client/DataEngineClient.html#createOrUpdateProcess(java.lang.String,org.odpi.openmetadata.accessservices.dataengine.model.Process))
+* [Upsert PortImplementation](https://odpi.github.io/egeria/org/odpi/openmetadata/accessservices/dataengine/client/DataEngineClient.html#createOrUpdatePortImplementation(java.lang.String,org.odpi.openmetadata.accessservices.dataengine.model.PortImplementation,java.lang.String))
+* [Upsert Schema Type](https://odpi.github.io/egeria/org/odpi/openmetadata/accessservices/dataengine/client/DataEngineClient.html#createOrUpdateSchemaType(java.lang.String,org.odpi.openmetadata.accessservices.dataengine.model.SchemaType))
+* [Upsert Database](https://odpi.github.io/egeria/org/odpi/openmetadata/accessservices/dataengine/client/DataEngineClient.html#upsertDatabase(java.lang.String,org.odpi.openmetadata.accessservices.dataengine.model.Database))
+* [Upsert Database Schema](https://odpi.github.io/egeria/org/odpi/openmetadata/accessservices/dataengine/client/DataEngineClient.html#upsertDatabaseSchema(java.lang.String,org.odpi.openmetadata.accessservices.dataengine.model.DatabaseSchema,java.lang.String))
+* [Upsert Data File](https://odpi.github.io/egeria/org/odpi/openmetadata/accessservices/dataengine/client/DataEngineClient.html#upsertDataFile(java.lang.String,org.odpi.openmetadata.accessservices.dataengine.model.DataFile))
+* [Upsert Event Type](https://odpi.github.io/egeria/org/odpi/openmetadata/accessservices/dataengine/client/DataEngineClient.html#upsertEventType(java.lang.String,org.odpi.openmetadata.accessservices.dataengine.model.EventType,java.lang.String))
+* [Upsert Relational Table](https://odpi.github.io/egeria/org/odpi/openmetadata/accessservices/dataengine/client/DataEngineClient.html#upsertRelationalTable(java.lang.String,org.odpi.openmetadata.accessservices.dataengine.model.RelationalTable,java.lang.String))
+* [Upsert Topic](https://odpi.github.io/egeria/org/odpi/openmetadata/accessservices/dataengine/client/DataEngineClient.html#upsertTopic(java.lang.String,org.odpi.openmetadata.accessservices.dataengine.model.Topic))
+* [Delete Connection](https://odpi.github.io/egeria/org/odpi/openmetadata/accessservices/dataengine/client/DataEngineClient.html#deleteConnection(java.lang.String,java.lang.String,java.lang.String))
+* [Delete Database](https://odpi.github.io/egeria/org/odpi/openmetadata/accessservices/dataengine/client/DataEngineClient.html#deleteDatabase(java.lang.String,java.lang.String,java.lang.String))
+* [Delete Database Schema](https://odpi.github.io/egeria/org/odpi/openmetadata/accessservices/dataengine/client/DataEngineClient.html#deleteDatabaseSchema(java.lang.String,java.lang.String,java.lang.String))
+* [Delete Data File](https://odpi.github.io/egeria/org/odpi/openmetadata/accessservices/dataengine/client/DataEngineClient.html#deleteDataFile(java.lang.String,java.lang.String,java.lang.String))
+* [Delete Endpoint](https://odpi.github.io/egeria/org/odpi/openmetadata/accessservices/dataengine/client/DataEngineClient.html#deleteEndpoint(java.lang.String,java.lang.String,java.lang.String))
+* [Delete Event Type](https://odpi.github.io/egeria/org/odpi/openmetadata/accessservices/dataengine/client/DataEngineClient.html#deleteEventType(java.lang.String,java.lang.String,java.lang.String))
+* [Delete Folder](https://odpi.github.io/egeria/org/odpi/openmetadata/accessservices/dataengine/client/DataEngineClient.html#deleteFolder(java.lang.String,java.lang.String,java.lang.String))
+* [Delete Port Implementation](https://odpi.github.io/egeria/org/odpi/openmetadata/accessservices/dataengine/client/DataEngineClient.html#deletePortImplementation(java.lang.String,java.lang.String,java.lang.String))
+* [Delete Process](https://odpi.github.io/egeria/org/odpi/openmetadata/accessservices/dataengine/client/DataEngineClient.html#deleteProcess(java.lang.String,java.lang.String,java.lang.String))
+* [Delete Relational Table](https://odpi.github.io/egeria/org/odpi/openmetadata/accessservices/dataengine/client/DataEngineClient.html#deleteRelationalTable(java.lang.String,java.lang.String,java.lang.String))
+* [Delete Schema Type](https://odpi.github.io/egeria/org/odpi/openmetadata/accessservices/dataengine/client/DataEngineClient.html#deleteSchemaType(java.lang.String,java.lang.String,java.lang.String))
+* [Delete Topic](https://odpi.github.io/egeria/org/odpi/openmetadata/accessservices/dataengine/client/DataEngineClient.html#deleteTopic(java.lang.String,java.lang.String,java.lang.String))
+* [Find an entity](https://odpi.github.io/egeria/org/odpi/openmetadata/accessservices/dataengine/client/DataEngineClient.html#find(java.lang.String,org.odpi.openmetadata.accessservices.dataengine.rest.FindRequestBody))
+* [Upsert Processing State](https://odpi.github.io/egeria/org/odpi/openmetadata/accessservices/dataengine/client/DataEngineClient.html#upsertProcessingState(java.lang.String,java.util.Map))
+* [Get Engine's Processing State](https://odpi.github.io/egeria/org/odpi/openmetadata/accessservices/dataengine/client/DataEngineClient.html#getProcessingState(java.lang.String))
+* [Add data flows](https://odpi.github.io/egeria/org/odpi/openmetadata/accessservices/dataengine/client/DataEngineClient.html)
+* [Add process hierarchy](https://odpi.github.io/egeria/org/odpi/openmetadata/accessservices/dataengine/client/DataEngineClient.html#addProcessHierarchy(java.lang.String,org.odpi.openmetadata.accessservices.dataengine.model.ProcessHierarchy))

site/docs/services/omas/data-engine/delete-data-file.md

@@ -23,7 +23,10 @@ DELETE {serverURLRoot}/servers/{serverName}/open-metadata/access-services/data-e
 Note that you must provide either the qualifiedName, or the guid of the data file <br>
 `VoidRespone` - void response with status and error message if failing
 
----8<-- "snippets/abbr.md"
+
+----
+License: [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/),
+Copyright Contributors to the ODPi Egeria project.
 
 
 

site/docs/services/omas/data-engine/delete-database-schema.md

@@ -0,0 +1,31 @@
+<!-- SPDX-License-Identifier: CC-BY-4.0 -->
+<!-- Copyright Contributors to the ODPi Egeria project. -->
+
+# Delete database schema
+
+Delete a Database Schema
+
+More examples can be found in the
+[sample collection](samples/collections/DataEngine-asset_endpoints.postman_collection.json)
+
+```
+DELETE {serverURLRoot}/servers/{serverName}/open-metadata/access-services/data-engine/users/{userId}/database-schemas/
+
+{
+    "guid": "databaseGUID",
+    "qualifiedName": "(host)=HOST::(database)=MINIMAL::(database_schema)=DB2INST1",
+    "externalSourceName": "(organization)=Company::(project)=ExternalDataPlatform"
+}
+```
+`externalSourceName` - qualifiedName of the external data engine tool.<br>
+`guid` - optional property describing the unique identifier of the database schema to be deleted
+`qualifiedName` - optional property describing the qualifiedName of the database schema to be deleted<br>
+Note that you must provide either the qualifiedName, or the guid of the database schema <br>
+`VoidRespone` - void response with status and error message if failing
+
+
+
+
+
+
+