Skip to content

Commit

Permalink
Merge pull request #366 from cmgrote/master
Browse files Browse the repository at this point in the history 
Event mapper deprecation and caching addition
  • Loading branch information
@cmgrote
cmgrote authored Aug 5, 2020
2 parents 02293d8 + 126c105 commit 9565601
Show file tree
Hide file tree
Showing 45 changed files with 657 additions and 297 deletions.
48 changes: 29 additions & 19 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -76,30 +76,16 @@ The quick version:
```
1. In another shell / command-line window, run the following commands to configure Egeria and startup its services --
making sure to replace the hostnames and port numbers with those relevant to your own environment (`localhost:9092`
for your own Kafka bus, `infosvr:59092` with the Information Server-embedded Kafka host and port, `infosvr` with
the hostname of your Information Server domain (services) tier, `9446` with the port number of your Information
Server domain (services) tier, `isadmin` with the username for your Information Server environment, and `isadmin`
with the password for your Information Server environment):
for your own Kafka bus, `infosvr` with the hostname of your Information Server domain (services) tier, `9446` with
the port number of your Information Server domain (services) tier, `isadmin` with the username for your Information
Server environment, and `isadmin` with the password for your Information Server environment):
```bash
$ curl -k -X POST -H "Content-Type: application/json" --data '{"producer":{"bootstrap.servers":"localhost:9092"},"consumer":{"bootstrap.servers":"localhost:9092"}}' "https://localhost:9443/open-metadata/admin-services/users/admin/servers/myserver/event-bus?connectorProvider=org.odpi.openmetadata.adapters.eventbus.topic.kafka.KafkaOpenMetadataTopicProvider&topicURLRoot=OMRSTopic"
$ curl -k -X POST "https://localhost:9443/open-metadata/admin-services/users/admin/servers/myserver/cohorts/mycohort"
$ curl -k -X POST -H "Content-Type: application/json" --data '{"class":"Connection","connectorType":{"class":"ConnectorType","connectorProviderClassName":"org.odpi.egeria.connectors.ibm.igc.repositoryconnector.IGCOMRSRepositoryConnectorProvider"},"endpoint":{"class":"Endpoint","address":"infosvr:9446","protocol":"https"},"userId":"isadmin","clearPassword":"isadmin","configurationProperties":{"defaultZones":["default"]}}' "https://localhost:9443/open-metadata/admin-services/users/admin/servers/myserver/local-repository/mode/repository-proxy/connection"
$ curl -k -X POST "https://localhost:9443/open-metadata/admin-services/users/admin/servers/myserver/local-repository/event-mapper-details?connectorProvider=org.odpi.egeria.connectors.ibm.igc.eventmapper.IGCOMRSRepositoryEventMapperProvider&eventSource=infosvr:59092"
$ curl -k -X POST "https://localhost:9443/open-metadata/admin-services/users/admin/servers/myserver/instance"
```

### Enable IGC's events

To start using the connector, you will need an IGC environment, running either version 11.5 or 11.7 of the software.
(The connector will automatically detect which version as part of its initialization.) You will need to first enable event
notification in your IGC environment:

1. Navigate to "Administration": !["Administration"](docs/ibm-igc-setup1.png)
1. Navigate to "Event Notification" within the "Setup" heading: !["Event Notification"](docs/ibm-igc-setup2.png)
1. Toggle "Enable" and then "Save and Close": !["Enable" and "Save and Close"](docs/ibm-igc-setup3.png)

There should not be any need to restart the environment after enabling the event notification.

### Obtain the connector

You can either download the latest released or snapshot version of the connector directly from ODPi, or build the
Expand Down Expand Up @@ -217,13 +203,29 @@ For example payloads and endpoints, see the [Postman samples](samples).
POST https://localhost:9443/open-metadata/admin-services/users/admin/servers/myserver/local-repository/mode/repository-proxy/connection
```

To operate, the IGC user credentials must have (at a minimum) the following roles within Information Server:
`Suite User` and `Information Governance Catalog User`. (These are both read-only, non-administrative roles.)

You can optionally also provide a list of zone names that will be used as default zones for all Assets retrieved
from IGC through the proxy (in the example above this is a single zone called `default`).

Note that you also need to provide the `connectorProvider` parameter, set to the name of the IGC
connectorProvider class (value as given above).

1. Configure the event mapper for IGC, by POSTing something like the following:
1. (Optional and _experimental_) Configure the event mapper for IGC, by POSTing something like the following:

The _experimental_ event mapper consumes events that are generated by IGC. By default this is disabled,
and it should be treated as experimental even when enabled. When enabled, the IGC user credentials provided for the
connector must have administrative authority to be able to automatically create the objects used by this
experimental capability.

Before enabling it, you will need to first enable event notification in your IGC environment:

1. Navigate to "Administration": !["Administration"](docs/ibm-igc-setup1.png)
1. Navigate to "Event Notification" within the "Setup" heading: !["Event Notification"](docs/ibm-igc-setup2.png)
1. Toggle "Enable" and then "Save and Close": !["Enable" and "Save and Close"](docs/ibm-igc-setup3.png)

There should not be any need to restart the environment after enabling the event notification.

```
POST https://localhost:9443/open-metadata/admin-services/users/admin/servers/myserver/local-repository/event-mapper-details?connectorProvider=org.odpi.egeria.connectors.ibm.igc.eventmapper.IGCOMRSRepositoryEventMapperProvider&eventSource=my.igc.services.host.com:59092
Expand All @@ -235,7 +237,7 @@ For example payloads and endpoints, see the [Postman samples](samples).
running on your Unified Governance / Enterprise Search tier, on port `9092`. In both cases the port will need to be
network-accessible by the host where you are running Egeria itself for any events to be picked up by Egeria.)

1. The connector and event mapper should now be configured, and you should now be able
1. The connector (and optionally the event mapper) should now be configured, and you should now be able
to start the instance by POSTing something like the following:

```
Expand Down Expand Up @@ -338,6 +340,14 @@ For example payloads and endpoints, see the [Postman samples](samples).
POST https://localhost:9443/open-metadata/admin-services/users/admin/servers/datastage_proxy/data-engine-proxy-service/configuration
```

To operate, the Information Server user credentials must have (at a minimum) the following roles:
`Suite User`, `Information Governance Catalog User`, and `Information Governance Catalog Glossary Author`.
(The first two are both read-only, non-administrative roles, while the last allows synchronization objects
to be created to track the last synchronization point of the DataStage job information.) Finally, if using the
connector option to include virtual assets (`"includeVirtualAssets": true`), the user will also need the
`Information Governance Catalog Information Asset Author` role, as this role is needed to be able to retrieve the
full details of virtual assets.

Note that you need to provide the `connectorProvider` parameter, set to the name of the DataStage
connectorProvider class (value as given above).

Expand Down
6 changes: 3 additions & 3 deletions ...ava/org/odpi/egeria/connectors/ibm/datastage/dataengineconnector/mapping/BaseMapping.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ String getDescription(InformationAsset igcObj) {
*/
String getFullyQualifiedName(Reference igcObj) {
if (igcObj != null) {
Identity identity = igcObj.getIdentity(igcRestClient);
Identity identity = igcObj.getIdentity(igcRestClient, cache.getIgcCache());
if (identity != null) {
return identity.toString();
}
Expand All @@ -58,7 +58,7 @@ String getFullyQualifiedName(Reference igcObj) {
String getParentQualifiedName(Reference igcObj) {
String parentQN = null;
if (igcObj != null) {
Identity thisObjIdentity = igcObj.getIdentity(igcRestClient);
Identity thisObjIdentity = igcObj.getIdentity(igcRestClient, cache.getIgcCache());
Identity parentObjIdentity = thisObjIdentity.getParentIdentity();
if (parentObjIdentity != null) {
parentQN = parentObjIdentity.toString();
Expand All @@ -76,7 +76,7 @@ String getParentQualifiedName(Reference igcObj) {
String getParentDisplayName(Reference igcObj) {
String parentDN = null;
if (igcObj != null) {
Identity thisObjIdentity = igcObj.getIdentity(igcRestClient);
Identity thisObjIdentity = igcObj.getIdentity(igcRestClient, cache.getIgcCache());
Identity parentObjIdentity = thisObjIdentity.getParentIdentity();
if (parentObjIdentity != null) {
parentDN = parentObjIdentity.getName();
Expand Down
2 changes: 1 addition & 1 deletion ...g/odpi/egeria/connectors/ibm/datastage/dataengineconnector/mapping/SchemaTypeMapping.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ public SchemaTypeMapping(DataStageCache cache, Identity storeIdentity) {
schemaType = null;
if (link != null) {
schemaType = new SchemaType();
schemaType.setQualifiedName(link.getIdentity(igcRestClient).toString() + stageNameSuffix);
schemaType.setQualifiedName(link.getIdentity(igcRestClient, cache.getIgcCache()).toString() + stageNameSuffix);
schemaType.setDisplayName(link.getId());
schemaType.setAuthor(link.getModifiedBy());
AttributeMapping attributeMapping = new AttributeMapping(cache, job, link, stageNameSuffix);
Expand Down
11 changes: 10 additions & 1 deletion ...va/org/odpi/egeria/connectors/ibm/datastage/dataengineconnector/model/DataStageCache.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
import org.odpi.egeria.connectors.ibm.datastage.dataengineconnector.DataStageConstants;
import org.odpi.egeria.connectors.ibm.datastage.dataengineconnector.mapping.ProcessMapping;
import org.odpi.egeria.connectors.ibm.igc.clientlibrary.IGCRestClient;
import org.odpi.egeria.connectors.ibm.igc.clientlibrary.cache.ObjectCache;
import org.odpi.egeria.connectors.ibm.igc.clientlibrary.model.base.*;
import org.odpi.egeria.connectors.ibm.igc.clientlibrary.model.common.Identity;
import org.odpi.egeria.connectors.ibm.igc.clientlibrary.model.common.ItemList;
Expand Down Expand Up @@ -32,6 +33,7 @@ public class DataStageCache {
private Map<String, List<Classificationenabledgroup>> storeToColumns;

private IGCRestClient igcRestClient;
private ObjectCache igcCache;
private Date from;
private Date to;
private List<String> limitToProjects;
Expand All @@ -44,6 +46,7 @@ public class DataStageCache {
* @param limitToProjects limit the cached jobs to only those in the provided list of projects
*/
public DataStageCache(Date from, Date to, List<String> limitToProjects) {
this.igcCache = new ObjectCache();
this.ridToJob = new HashMap<>();
this.ridToProcess = new HashMap<>();
this.storeToIdentity = new HashMap<>();
Expand Down Expand Up @@ -75,6 +78,12 @@ public void initialize(IGCRestClient igcRestClient) {
*/
public Date getTo() { return to; }

/**
* Retrieve the embedded cache of IGC objects.
* @return ObjectCache
*/
public ObjectCache getIgcCache() { return igcCache; }

/**
* {@inheritDoc}
*/
Expand Down Expand Up @@ -232,7 +241,7 @@ public List<Classificationenabledgroup> getFieldsForStore(InformationAsset store
if (fields != null) {
storeToColumns.put(rid, fields);
if (!fields.isEmpty()) {
Identity storeIdentity = fields.get(0).getIdentity(igcRestClient).getParentIdentity();
Identity storeIdentity = fields.get(0).getIdentity(igcRestClient, igcCache).getParentIdentity();
String storeId = storeIdentity.getRid();
storeToIdentity.put(storeId, storeIdentity);
}
Expand Down
Loading

0 comments on commit 9565601

Please sign in to comment.