From 88974028f853adcbd1b7cec380bb5b39cd02ebae Mon Sep 17 00:00:00 2001 From: Mandy Chessell Date: Mon, 8 Jan 2024 17:11:20 +0000 Subject: [PATCH] Add new pages for survey actions. Signed-off-by: Mandy Chessell --- site/docs/concepts/index.md | 5 + site/docs/concepts/survey-action-engine.md | 27 + site/docs/concepts/survey-action-service.md | 22 + site/docs/concepts/survey-report.md | 27 + site/docs/connectors/connector-catalog.drawio | 774 +++++++++++++++- ...e-atlas-survey-action-service-analysis.svg | 4 + ...tlas-survey-action-service-graph-links.svg | 4 + ...he-atlas-survey-action-service-profile.svg | 4 + ...he-atlas-survey-action-service-reports.svg | 4 + ...ache-atlas-survey-action-service-setup.svg | 4 + .../apache-atlas-survey-action-service.md | 131 +++ .../apache-atlas-survey-action-service.svg | 4 + .../survey-action-service-intro.md | 26 + .../survey-action/survey-action-service.svg | 4 + site/docs/frameworks/index.md | 2 +- site/docs/frameworks/saf/overview.md | 87 ++ site/docs/frameworks/saf/survey-action.drawio | 180 ++++ site/docs/frameworks/saf/survey-context.svg | 4 + .../annotation-intro.md | 28 + .../asset-to-survey-reports.svg | 4 + .../operation-of-a-survey-action-service.md | 15 + .../survey-action-services/overview.md | 216 +++++ ...survey-action-classification-discovery.svg | 4 + .../survey-action-data-class-discovery.svg | 4 + .../survey-action-data-profiling.svg | 4 + .../survey-action-data-source-measurement.svg | 4 + .../survey-action-pipeline-example.svg | 4 + .../survey-action-pipeline-intro.md | 13 + .../survey-action-quality-scores.svg | 4 + .../survey-action-relationship-discovery.svg | 4 + .../survey-action-request-for-action.svg | 4 + ...on-schema-extraction-with-schema-links.svg | 4 + .../survey-action-schema-extraction.svg | 4 + .../survey-action-semantic-discovery.svg | 4 + .../survey-action-service-operation.svg | 4 + .../survey-action.drawio | 874 ++++++++++++++++++ .../survey-report-structure.svg | 4 + ...ine-services-survey-action-server-side.svg | 4 + .../docs/services/omes/engine-services.drawio | 240 ++++- .../services/omes/survey-action/overview.md | 25 + site/docs/types/4/0461-Governance-Engines.md | 8 +- site/docs/types/4/0461-Governance-Engines.svg | 2 +- site/docs/types/4/area-4-governance.drawio | 307 ++---- site/docs/types/5/0533-Graph-Schemas.svg | 4 +- .../types/5/area-5-models-and-schema.drawio | 33 +- .../types/6/0601-Open-Discovery-Engine.svg | 3 +- site/docs/types/6/0603-Survey-Reports.md | 36 + site/docs/types/6/0603-Survey-Reports.svg | 4 + .../0605-Open-Discovery-Analysis-Reports.svg | 4 +- site/docs/types/6/0610-Annotations.md | 60 +- site/docs/types/6/0610-Annotations.svg | 4 +- site/docs/types/6/0612-Annotation-Reviews.md | 28 +- site/docs/types/6/0612-Annotation-Reviews.svg | 4 +- site/docs/types/6/0615-Schema-Extraction.md | 9 +- site/docs/types/6/0615-Schema-Extraction.svg | 2 +- .../docs/types/6/0616-Data-Field-Discovery.md | 16 + .../types/6/0616-Data-Field-Discovery.svg | 4 + .../docs/types/6/0617-Data-Field-Analysis.svg | 4 +- site/docs/types/6/0630-Semantic-Discovery.md | 2 +- .../types/6/0635-Classification-Discovery.md | 1 - site/docs/types/6/0640-Quality-Scores.md | 4 +- .../types/6/0650-Relationship-Discovery.svg | 4 +- site/docs/types/6/0690-Request-for-Action.md | 2 +- site/docs/types/6/0690-Request-for-Action.svg | 3 +- .../types/6/area-6-discovery-overview.svg | 3 +- site/docs/types/6/area-6-discovery.drawio | 834 +++++++++-------- site/docs/types/6/index.md | 28 +- site/mkdocs.yml | 9 +- .../survey-action-service-config-summary.md | 4 + ...survey-action-service-operation-summary.md | 12 + 70 files changed, 3464 insertions(+), 758 deletions(-) create mode 100644 site/docs/concepts/survey-action-engine.md create mode 100644 site/docs/concepts/survey-action-service.md create mode 100644 site/docs/concepts/survey-report.md create mode 100644 site/docs/connectors/survey-action/apache-atlas-survey-action-service-analysis.svg create mode 100644 site/docs/connectors/survey-action/apache-atlas-survey-action-service-graph-links.svg create mode 100644 site/docs/connectors/survey-action/apache-atlas-survey-action-service-profile.svg create mode 100644 site/docs/connectors/survey-action/apache-atlas-survey-action-service-reports.svg create mode 100644 site/docs/connectors/survey-action/apache-atlas-survey-action-service-setup.svg create mode 100644 site/docs/connectors/survey-action/apache-atlas-survey-action-service.md create mode 100644 site/docs/connectors/survey-action/apache-atlas-survey-action-service.svg create mode 100644 site/docs/connectors/survey-action/survey-action-service-intro.md create mode 100644 site/docs/connectors/survey-action/survey-action-service.svg create mode 100644 site/docs/frameworks/saf/overview.md create mode 100644 site/docs/frameworks/saf/survey-action.drawio create mode 100644 site/docs/frameworks/saf/survey-context.svg create mode 100644 site/docs/guides/developer/survey-action-services/annotation-intro.md create mode 100644 site/docs/guides/developer/survey-action-services/asset-to-survey-reports.svg create mode 100644 site/docs/guides/developer/survey-action-services/operation-of-a-survey-action-service.md create mode 100644 site/docs/guides/developer/survey-action-services/overview.md create mode 100644 site/docs/guides/developer/survey-action-services/survey-action-classification-discovery.svg create mode 100644 site/docs/guides/developer/survey-action-services/survey-action-data-class-discovery.svg create mode 100644 site/docs/guides/developer/survey-action-services/survey-action-data-profiling.svg create mode 100644 site/docs/guides/developer/survey-action-services/survey-action-data-source-measurement.svg create mode 100644 site/docs/guides/developer/survey-action-services/survey-action-pipeline-example.svg create mode 100644 site/docs/guides/developer/survey-action-services/survey-action-pipeline-intro.md create mode 100644 site/docs/guides/developer/survey-action-services/survey-action-quality-scores.svg create mode 100644 site/docs/guides/developer/survey-action-services/survey-action-relationship-discovery.svg create mode 100644 site/docs/guides/developer/survey-action-services/survey-action-request-for-action.svg create mode 100644 site/docs/guides/developer/survey-action-services/survey-action-schema-extraction-with-schema-links.svg create mode 100644 site/docs/guides/developer/survey-action-services/survey-action-schema-extraction.svg create mode 100644 site/docs/guides/developer/survey-action-services/survey-action-semantic-discovery.svg create mode 100644 site/docs/guides/developer/survey-action-services/survey-action-service-operation.svg create mode 100644 site/docs/guides/developer/survey-action-services/survey-action.drawio create mode 100644 site/docs/guides/developer/survey-action-services/survey-report-structure.svg create mode 100644 site/docs/services/omes/engine-services-survey-action-server-side.svg create mode 100644 site/docs/services/omes/survey-action/overview.md create mode 100644 site/docs/types/6/0603-Survey-Reports.md create mode 100644 site/docs/types/6/0603-Survey-Reports.svg create mode 100644 site/docs/types/6/0616-Data-Field-Discovery.md create mode 100644 site/docs/types/6/0616-Data-Field-Discovery.svg create mode 100644 site/snippets/survey-action-services/survey-action-service-config-summary.md create mode 100644 site/snippets/survey-action-services/survey-action-service-operation-summary.md diff --git a/site/docs/concepts/index.md b/site/docs/concepts/index.md index 1931b177ec..cc429d7527 100644 --- a/site/docs/concepts/index.md +++ b/site/docs/concepts/index.md @@ -148,6 +148,7 @@ - [Open Discovery Framework (ODF)](/frameworks/odf/overview) - [Governance Action Framework (GAF)](/frameworks/gaf/overview) - [Event Action Framework (EAF)](/frameworks/eaf/overview) + - [Survey Action Framework (EAF)](/frameworks/saf/overview) ## G @@ -379,6 +380,10 @@ - [Subject Area Domain](/concepts/subject-area) - [Subject Area OMAS](/services/omas/subject-area/overview) - [Subsystem](/concepts/omag-subsystem) +- [Survey Report](/concepts/survey-report) +- [Survey Action Engine](/concepts/survey-action-engine) +- [Survey Action Framework (SAF)](/frameworks/saf/overview) +- [Survey Action Service](/concepts/survey-action-service) - [Supported Zones](/concepts/governance-zone) diff --git a/site/docs/concepts/survey-action-engine.md b/site/docs/concepts/survey-action-engine.md new file mode 100644 index 0000000000..10cd1c63b7 --- /dev/null +++ b/site/docs/concepts/survey-action-engine.md @@ -0,0 +1,27 @@ +--- +hide: +- toc +--- + + + + +# Survey Action Engines + +A *survey action engine* describes a set of related [service action services](/guides/developer/survey-action-services/overview). Each survey action service implements a specific type of analysis. This analysis is looking into the content of a [digital resource](/concepts/resource) and storing its findings in annotations within a [survey report](/concepts/survey-report) that is linked of the resource's [Asset](/concepts/asset) in the metadata repository. + +A survey action is hosted in the [Survey Action Open Metadata Engine Service (OMES)](/services/omes/survey-action/overview) running on one or more [Engine Host OMAG Servers](/concepts/engine-host). + +![Survey Action Engine](/connectors/survey-action/survey-action-service.svg) + +When an open discovery engine is called, it is passed a governance request type and request parameters. This is mapped to a call to an open discovery service through the [open discovery engine definition](/concepts/governance-engine-definition). + +![Survey Action Engine Definition Structure](/guides/developer/open-metadata-archives/survey-action-engine-definition.svg) +> Logical structure of a survey action engine definition showing how the governance request types map to the survey action service definitions + +!!! education "Further information" + + The Open Metadata Types used to define the survey action engines are located in [model 0461 Governance Engines](/types/4/0461-Governance-Engines). + + +---8<-- "snippets/abbr.md" \ No newline at end of file diff --git a/site/docs/concepts/survey-action-service.md b/site/docs/concepts/survey-action-service.md new file mode 100644 index 0000000000..8c952bd0f1 --- /dev/null +++ b/site/docs/concepts/survey-action-service.md @@ -0,0 +1,22 @@ +--- +hide: +- toc +--- + + + + +# Survey Action Service + +---8<-- "docs/connectors/survey-action/survey-action-service-intro.md" + +## Survey Action Pipeline + +---8<-- "docs/guides/developer/open-discovery-services/discovery-pipeline-intro.md" + +!!! education "Further information" + + * [Writing Survey Action Services](/guides/developer/survey-action-services/overview) + * [Configuring Survey Action Services into a Governance Engine](/guides/developer/open-metadata-archives/creating-governance-engine-packs) + +--8<-- "snippets/abbr.md" \ No newline at end of file diff --git a/site/docs/concepts/survey-report.md b/site/docs/concepts/survey-report.md new file mode 100644 index 0000000000..d3a532c483 --- /dev/null +++ b/site/docs/concepts/survey-report.md @@ -0,0 +1,27 @@ +--- +hide: +- toc +--- + + + + +# Survey Report + +A *survey report* contains one or more sets of related properties that a [survey action service](/concepts/survey-action-service) has discovered about a resource, its metadata, structure and/or content. These are stored in a set of [annotations](#annotations) linked off of the survey report. + +The survey report is attached to the [asset](/concepts/asset) for the [digital resource](/concepts/resource) that was analysed. Over time, the survey reports show how the digital resource's contents are changing. + +![Asset with discovery analysis reports](/guides/developer/survey-action-services/asset-to-survey-reports.svg) + +The survey report is created in the open metadata repository by the [Survey Action OMES](/services/omes/survey-action/overview) when it creates an new survey action service instance. The survey action service can retrieve information about the survey report through the [survey context](/guides/developer/open-discovery-services/overview/#discovery-analysis-report-store). + +## Annotations + +---8<-- "docs/guides/developer/survey-action-services/annotation-intro.md" + + +!!! education "Further information" + [Metadata Discovery and Stewardship](/features/discovery-and-stewardship/overview) + +--8<-- "snippets/abbr.md" diff --git a/site/docs/connectors/connector-catalog.drawio b/site/docs/connectors/connector-catalog.drawio index a6e6bf9146..43f64242b8 100644 --- a/site/docs/connectors/connector-catalog.drawio +++ b/site/docs/connectors/connector-catalog.drawio @@ -1,6 +1,6 @@ - + - + @@ -64,7 +64,7 @@ - + @@ -1059,65 +1059,124 @@ - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + @@ -3703,7 +3762,7 @@ - + @@ -4156,7 +4215,7 @@ - + @@ -4225,7 +4284,7 @@ - + @@ -4317,15 +4376,676 @@ - + - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/site/docs/connectors/survey-action/apache-atlas-survey-action-service-analysis.svg b/site/docs/connectors/survey-action/apache-atlas-survey-action-service-analysis.svg new file mode 100644 index 0000000000..90e94c3ca9 --- /dev/null +++ b/site/docs/connectors/survey-action/apache-atlas-survey-action-service-analysis.svg @@ -0,0 +1,4 @@ + + + +
Data Profile
Annotation
Data Profile...
Data Profile
Annotation
Data Profile...
Business Metadata
Data Field
Business Metadata...
Relationship
Data Field
Relationship...
Business Metadata
Type
Data Field
Business Metadata...
Data Profile
Annotation
Data Profile...
Data Profile
Annotation
Data Profile...
Data Profile
Annotation
Data Profile...
Data Profile
Annotation
Data Profile...
Data Profile
Annotation
Data Profile...
Data Profile
Annotation
Data Profile...
Data Profile
Annotation
Data Profile...
Data Profile
Annotation
Data Profile...
Data Profile
Annotation
Data Profile...
Data Profile
Annotation
Data Profile...
Data Profile
Annotation
Data Profile...
Survey
Report
Survey...
DataSource
Measurements Annotation
DataSource...
Schema Analysis Annotation
Schema Analysis Anno...
Entity Data Field
Entity Data Field
Relationship Type
Data Field
Relationship Type...
Entity Data Field
Entity Data Field
Classification
Data Field
Classification...
Entity Data Field
Entity Data Field
Classification Type
Data Field
Classification Type...
Relationship
Data Field
Relationship...
Entity Data Field
Entity Data Field
Relationship Type
Data Field
Relationship Type...
Entity Data Field
Entity Data Field
Classification
Data Field
Classification...
Classification Type
Data Field
Classification Type...
Relationship Type
Data Field
Relationship Type...
Entity Type
Data Field
Entity Type...
Data Profile
Annotation
Data Profile...
Data Profile
Annotation
Data Profile...
Data Profile
Annotation
Data Profile...
Data Profile
Annotation
Data Profile...
STATS
STATS
SCHEMA
SCHEMA
PROFILE
PROFILE
1
1
1
1
2
2
2
2
3
3
3
3
Analysis Steps
Analysis StepsText is not SVG - cannot display \ No newline at end of file diff --git a/site/docs/connectors/survey-action/apache-atlas-survey-action-service-graph-links.svg b/site/docs/connectors/survey-action/apache-atlas-survey-action-service-graph-links.svg new file mode 100644 index 0000000000..fb4c9cd83a --- /dev/null +++ b/site/docs/connectors/survey-action/apache-atlas-survey-action-service-graph-links.svg @@ -0,0 +1,4 @@ + + + +
Entity Data Field
Entity Data Field
Classification Type
Data Field
Classification Type...
Relationship
Data Field
Relationship...
Entity Type
Data Field
Entity Type...
Business Metadata
Type
Data Field
Business Metadata...
end1
end1
end2
end2Text is not SVG - cannot display \ No newline at end of file diff --git a/site/docs/connectors/survey-action/apache-atlas-survey-action-service-profile.svg b/site/docs/connectors/survey-action/apache-atlas-survey-action-service-profile.svg new file mode 100644 index 0000000000..594589833d --- /dev/null +++ b/site/docs/connectors/survey-action/apache-atlas-survey-action-service-profile.svg @@ -0,0 +1,4 @@ + + + +
Attached
Entity Types
Attached...
Classification Type
Data Field
Classification Type...
Relationship Type
Data Field
Relationship Type...
Entity Type
Data Field
Entity Type...
Attached ClassificationTypes
Attached Classificat...
End 1 Attached
Entity Types
End 1 Attached...
Data Profile
Annotation
Data Profile...
End 1 Attached
Relationship Types
End 1 Attached...
End 2 Attached
Entity Types
End 2 Attached...
Data Profile
Annotation
Data Profile...
Data Profile
Annotation
Data Profile...
Data Profile
Annotation
Data Profile...
Data Profile
Annotation
Data Profile...
Attached
Entity Types
Attached...
Business Metadata Type
Data Field
Business Metadata Ty...
Data Profile
Annotation
Data Profile...
End 2 Attached
Relationship Types
End 2 Attached...
Attached Labels
Attached Labels
Data Profile
Annotation
Data Profile...
Data Profile
Annotation
Data Profile...
Attached Business Metadata Types
Attached Business Me...
Data Profile
Annotation
Data Profile...
Attached
Entity Type Pairs
Attached...
Data Profile
Annotation
Data Profile...Text is not SVG - cannot display \ No newline at end of file diff --git a/site/docs/connectors/survey-action/apache-atlas-survey-action-service-reports.svg b/site/docs/connectors/survey-action/apache-atlas-survey-action-service-reports.svg new file mode 100644 index 0000000000..d42fabf1a1 --- /dev/null +++ b/site/docs/connectors/survey-action/apache-atlas-survey-action-service-reports.svg @@ -0,0 +1,4 @@ + + + +
Survey
Report
Survey...
SoftwareServer
SoftwareServer
Apache Atlas
Apache Atlas
Connection
Connection
ApacheAtlas
RESTResource Connection
ApacheAtlas...
ConectorType
ConectorType
Endpoint
Endpoint
AssetConnection
AssetConnection
Survey
Report
Survey...Text is not SVG - cannot display \ No newline at end of file diff --git a/site/docs/connectors/survey-action/apache-atlas-survey-action-service-setup.svg b/site/docs/connectors/survey-action/apache-atlas-survey-action-service-setup.svg new file mode 100644 index 0000000000..4c5a1a25c2 --- /dev/null +++ b/site/docs/connectors/survey-action/apache-atlas-survey-action-service-setup.svg @@ -0,0 +1,4 @@ + + + +
SoftwareServer
SoftwareServer
Apache Atlas
Apache Atlas
Connection
Connection
ApacheAtlas
RESTResource Connection
ApacheAtlas...
ConectorType
ConectorType
Endpoint
Endpoint
AssetConnection
AssetConnectionText is not SVG - cannot display \ No newline at end of file diff --git a/site/docs/connectors/survey-action/apache-atlas-survey-action-service.md b/site/docs/connectors/survey-action/apache-atlas-survey-action-service.md new file mode 100644 index 0000000000..6650ebcb5d --- /dev/null +++ b/site/docs/connectors/survey-action/apache-atlas-survey-action-service.md @@ -0,0 +1,131 @@ + + + +--8<-- "snippets/content-status/tech-preview.md" + +# Apache Atlas Survey Action Service + +???+ info "Connector summary" + - Connector Category: [Survey Action Service](/concepts/survey-action-service) + - Hosting Service: [Asset Analysis OMES](/services/omes/asset-analysis/overview) + - Hosting Server: [Engine Host](/concepts/engine-host) + - Source Module: [discovery-service-connectors :material-github:](https://github.com/odpi/egeria/tree/main/open-metadata-implementation/adapters/open-connectors/discovery-service-connectors){ target=gh } + - Jar File Name: `discovery-service-connectors.jar` + - Connector Provider Class Name: `org.odpi.openmetadata.adapters.connectors.discoveryservices.discoveratlas.DiscoverApacheAtlasProvider` + +## Overview + +---8<-- "snippets/systems/apache-atlas-intro.md" + +The Apache Atlas Survey Action Service builds a [survey report](/concepts/survey-report) that describes the types defined in the Apache Atlas server and the numbers of instances that are found for these types. + + +---8<-- "snippets/discovery-services/discovery-service-config-summary.md" + + +![Figure 1](apache-atlas-survey-action-service.svg) +> **Figure 1:** Operation of the Apache Atlas Survey Action Service + +---8<-- "snippets/discovery-services/discovery-service-operation-summary.md" + +## Discovery service function + +The Apache Atlas Discovery Service provides a summary of the contents of the Apache Atlas repository found at the time it was run. + +It has three analysis steps: + +1. STATS - Retrieves the overall metrics from the Apache Atlas server. These are stored in a [*DataSourceMeasurementAnnotation*](/types/0660-Data-Source-Measurement) entity linked to the [*SurveyReport*](/types/6/0605-Open-Discovery-Analysis-Reports) entity generated for each run of the Apache Atlas Discovery Service. +2. SCHEMA - Retrieves the types from Apache Atlas and organizes them in a linked graph of [*DataField*](/types/6/0615-Schema-Extraction) entities. All the data fields are linked to a [*SchemaAnalysisAnnotation*](/types/6/0615-Schema-Extraction) entity which is in turn linked to the [*SurveyReport*](/types/6/0605-Open-Discovery-Analysis-Reports) entity +3. PROFILE - Retrieves each entity in the Apache Atlas server and adds the following counts to [*DataProfileAnnotation*](/types/6/0620-Data-Profiling) entities linked from the appropriate data field entities: + + * The number of instances of each entity type. + * The number of classifications of a particular type attached to each type of entity. + * The number of relationships of a particular type attached to each type of entity. + * The number of each type of label attached to each type of entity. + * The number of business metadata properties of a particular type attached to each type of entity. + +Each analysis step builds on the work of its predecessor. The processing requirements increase with each step, so you can choose to stop the processing after any step using the `finalAnalysisStep` property. This can be set as a configuration property in the connection object for this discovery service, or as an analysis parameter passed when the Apache Atlas Discovery Service is run. + +The default value for `finalAnalysisStep` is `PROFILE`. + +## Metadata Setup + +Prior to running the Apache Atlas Survey Service, an [asset](/concepts/asset) and [connection](/concepts/connection) must be created for the Apache Atlas server that is to be analysed. + +Typically, the asset for Apache Atlas is of type [*SoftwareServer*](/types/0/0040-Software-Server). The `networkAddress` in the connection's [endpoint](/concepts/endpoint) is the hostname and port of the Apache Atlas server. For example, `http://localhost:21000`. + +![Figure 2](apache-atlas-survey-action-service-setup.svg) +> **Figure 2:** Metadata added to the open metadata repository + +## Survey Reports + +Each time the Apache Atlas Survey Service runs, there is a new [survey report](/concepts/discovery-analysis-report) created. + +![Figure 3](apache-atlas-survey-action-service-reports.svg) +> **Figure 3:** Survey reports linked from Apache Atlas's asset + +Figure 4 shows the structure of the survey report. The annotations are labelled with the analysis steps that create them. + +![Figure 4](apache-atlas-survey-action-service-analysis.svg) +> **Figure 4:** Analysis stages performed by the survey action service + +### Data Source Measurements Annotation + +The data source measurements annotation is created in the STATS analysis step. It sets up the following properties in the *dataSourceProperties* map: + +* entityInstanceCount - number of active entity instances +* entityInstanceCount:*typeName* - number of active entity instance of this type +* entityWithSubtypesInstanceCount:*typeName* - number of active entity instances of this type and all subtypes. +* classificationCount - number of classifications added to entity instances. +* typeCount - number of defined types (and their versions). +* typeUnusedCount - number of types with no instances. + +This analysis is achieved using two REST API calls and so has minimum impact on the Apache Atlas Server. + +### Schema Analysis Annotation + +The schema analysis annotation is created in the SCHEMA analysis step. It is the parent entity for a set of [data fields](#data-fields). These data fields represent the types defined in the Apache Atlas server. + +### Data Fields + +In the SCHEMA analysis step, a *DataField* is created for each Apache Atlas entity type, relationship type, business metadata type and classification type. + +The classification type data fields are linked to the entity type data field to indicate which type of entity it can be attached to. +The relationship type data fields are each attached to two entity type data fields: one for the type of entity that can be attached at end 1 of the relationship; the other for the type of entity that can be attached at end 2. + +![Figure 5](apache-atlas-survey-action-service-graph-links.svg) +> **Figure 5:** Linkage of data fields based on Apache Atlas type + +The data fields are linked together using the [*DiscoveredLinkedDataField*](/types/6/0615-Schema-Extraction) relationship. + +### Data Profile Annotation + +This discovery service attaches multiple data profile annotations to each data field depending on their category (entity, relationship, classification or business metadata). + +![Figure 6](apache-atlas-survey-action-service-profile.svg) +> **Figure 6:** Details of the data profile annotations attached to each type of data field + +It sets up the following fields in each data profile annotation: + +* *analysisStep* - this is always set to "PROFILE". +* *annotationType* - this identifies the type of values that the annotation contains. +* *explanation* - this provides more information about the annotation type. +* *valueCount* - this is a map of typeName to count. For example, if this annotation was counting the classifications attached to the *DataSet* entity type, then the map would include an entry for each type of classification attached to this type of entity and a count of how many times it is used. +* *additionalProperties* - contains the count of instances for the particular type that the data field represents. + +The table summarizes the values in each of the data profile annotations depending on the category of the data field it is attached to. + +| Data Field Category | Annotation Type | Explanation | Value Count | Instance count in AdditionalProperties | +|---------------------|------------------------------------------------|-----------------------------------------------------------------------------------------------------------|--------------------------------------|-------------------------------------------| +| Entity | Apache Atlas Attached Classification Types | Count of classification types attached to this type of entity. | Classification Name to Count | Entity instances for this type | +| Entity | Apache Atlas End 1 Attached Relationship Types | Count of different types of relationships attached to this type of entity at End 1. | Relationship Name to Count | Entity instances for this type | +| Entity | Apache Atlas End 2 Attached Relationship Types | Count of different types of relationships attached to this type of entity at End 2. | Relationship Name to Count | Entity instances for this type | +| Entity | Apache Atlas Attached Labels | Count of the different labels attached to this type of entity. | Label Name to Count | Entity Instances for this type | +| Entity | Apache Atlas Attached Business Metadata Types | Count of the different types of business metadata properties attached to this type of entity. | Business Metadata Type Name to Count | Entity instances for this type | +| Classification | Apache Atlas Attached Entity Types | Count of entities where this classification is attached, organized by entity type. | Entity Type Name to Count | Classification Instances for this type | +| Business Metadata | Apache Atlas Attached Entity Types | Count of entities where this type of business metadata properties are attached, organized by entity type. | Entity Type Name to Count | Business metadata instances for this type | +| Relationship | Apache Atlas Attached End 1 Entity Types | Count of entity types attached at end 1 of this type of relationship. | Entity Type Name to Count | Relationship instances for this type | +| Relationship | Apache Atlas Attached End 2 Entity Types | Count of entity types attached at end 2 of this type of relationship. | Entity Type Name to Count | Relationship instances for this type | +| Relationship | Apache Atlas Attached Entity Type Pairs | Count of entity type pairs for this type of relationship. | Entity Type Name to Count | Relationship instances for this type | + +---8<-- "snippets/abbr.md" diff --git a/site/docs/connectors/survey-action/apache-atlas-survey-action-service.svg b/site/docs/connectors/survey-action/apache-atlas-survey-action-service.svg new file mode 100644 index 0000000000..ec70b9e53f --- /dev/null +++ b/site/docs/connectors/survey-action/apache-atlas-survey-action-service.svg @@ -0,0 +1,4 @@ + + + +
Engine Host
Engine Host
Asset Analysis OMES
Asset Analysis OMES
Survey Action Engine
Survey Action Engine
Apache Atlas
 Survey Service
Apache Atlas...
Apache Atlas
Apache Atlas
Metadata Access Server
Metadata Access Server
Stewardship Actions OMAS
Stewardship Actions OMAS
Read
Apache Atlas
metadata
Read...
Read asset and
connection metadata
Read asset and...
Build Survey
Report
Build Survey...Text is not SVG - cannot display \ No newline at end of file diff --git a/site/docs/connectors/survey-action/survey-action-service-intro.md b/site/docs/connectors/survey-action/survey-action-service-intro.md new file mode 100644 index 0000000000..ad3c5eb6df --- /dev/null +++ b/site/docs/connectors/survey-action/survey-action-service-intro.md @@ -0,0 +1,26 @@ + + + +An *survey-action service* is a component that performs specific analysis of the contents of a [digital resource](/concepts/resource) on request. The aim of the survey-action service is to enable a detailed picture of the properties of a resource to be built up. + +Each time an survey-action service runs, it creates a new [survey report](/concepts/survey-report) linked off of the digital resource's [Asset](/concepts/asset) metadata element that records the results of the analysis. + +![Asset with survey reports](/guides/developer/survey-action-services/asset-to-survey-reports.svg) +> Each time an survey-action service runs to analyse a digital resource, a new survey report is created and attached to the resource's asset. If the survey-action service is run regularly, it is possible to track how the contents are changing over time. + +The *survey report* contains one or more sets of related properties that the survey action service has discovered about the resource, its metadata, structure and/or content. These are stored in a set of [*annotations*](/concepts/survey-report/#annotations) linked off of the survey report. + +An survey action service is designed to run at regular intervals to gather a detailed perspective on the contents of the digital resource and how they are changing over time. Each time it runs, it is given access to the results of previously run survey-action services, along with a review of these findings made by individuals responsible for the digital resource (such as stewards, owners, custodians). + +??? info "Operation of an survey action service" + --8<-- "docs/guides/developer/survey-action-services/operation-of-a-survey-action-service.md" + +??? info "Runtime for an survey action service" + Survey action services are packaged into [Survey Action Engines](/concepts/survey-action-engine) that run in the [Survey Action OMES](/services/omes/survey-action/overview) hosted in an [Engine Host](/concepts/engine-host). + + The metadata repository interface for metadata discovery tools is implemented by the [Stewardship Actions OMAS](/services/omas/stewardship-action/overview) that runs in a [Metadata Access Server](/concepts/metadata-access-server). + + An survey action service may be triggered via an [Engine Action](/concepts/engine-action) or as part of a [governance action process](/concepts/governance-action-process). + + ![Survey Action Service](/connectors/survey-action/survey-action-service.svg) + diff --git a/site/docs/connectors/survey-action/survey-action-service.svg b/site/docs/connectors/survey-action/survey-action-service.svg new file mode 100644 index 0000000000..08c7832613 --- /dev/null +++ b/site/docs/connectors/survey-action/survey-action-service.svg @@ -0,0 +1,4 @@ + + + +
Engine Host
Engine Host
Survey Action
Engine Service
Survey Action...
Survey Action Service
Survey Action S...
Digital Resource
Digital Resource
Open Metadata
Ecosystem
Open Metadata...
     Data
     Data
Survey
Report
Survey...Text is not SVG - cannot display \ No newline at end of file diff --git a/site/docs/frameworks/index.md b/site/docs/frameworks/index.md index 78651a96ba..7510e2dc97 100644 --- a/site/docs/frameworks/index.md +++ b/site/docs/frameworks/index.md @@ -14,7 +14,7 @@ The open metadata and governance frameworks supported by Egeria are as follows: * *[Open Integration Framework (OIF)](/frameworks/oif/overview)* provides the interfaces and base implementations for connectors (called [integration connectors](/concepts/integration-connector)) that synchronize metadata between Egeria's open metadata ecosystem and third party technologies. The integration connectors run in an [integration daemon](/concepts/integration-daemon). -* *[Open Discovery Framework (ODF)](/frameworks/odf/overview)* provides the interfaces and base implementations for connectors (called [open discovery services](/concepts/open-discovery-service)) that access data-related assets and extract characteristics about the data that can be stored in an open metadata repository. The open discovery services run in an [open discovery engine](/concepts/open-discovery-engine) hosted on the [Engine Host](/concepts/engine-host) OMAG Server. +* *[Survey Action Framework (SAF)](/frameworks/saf/overview)* provides the interfaces and base implementations for connectors (called [survey action services](/concepts/survey-action-service)) that survey and extract characteristics about the real-world resources and stores them in an open metadata repository. * *[Governance Action Framework (GAF)](/frameworks/gaf/overview)* provides the interfaces and base implementations for connectors (called [governance action services](/concepts/governance-action-service)) that take action to assess and correct a situation that is harmful to the data, or the organization in some way. The governance action services run in a [governance action engine](/concepts/governance-action-engine) hosted on the [Engine Host](/concepts/engine-host) OMAG Server. diff --git a/site/docs/frameworks/saf/overview.md b/site/docs/frameworks/saf/overview.md new file mode 100644 index 0000000000..231d10046e --- /dev/null +++ b/site/docs/frameworks/saf/overview.md @@ -0,0 +1,87 @@ + + + +--8<-- "snippets/content-status/tech-preview.md" + +# Survey Action Framework (SAF) + +The *Survey Action Framework (SAF)* enables [metadata survey tools](/features/discovery-and-stewardship) to integrate with open metadata repositories by defining the interfaces for surveyor components (called [survey action services](#survey-action-service)) to: + +- Access survey request configuration. +- Search for [assets](/concepts/asset) and related metadata in the metadata repository. +- Extract all the metadata known about a specific asset. +- Record the results of the analysis in the open metadata repository and attach it to the asset's metadata for later processing. + + +## Survey action service + +A *survey action service* provides specific analysis of the real-world resources associated with an [asset](/concepts/asset) on request. It may provide the complete implementation of the function, or invoke the services of an external tool. + +The results of the analysis are stored in an [*survey report*](#survey-report). + +The result is one or more sets of related properties that the survey action service has discovered about the asset, its metadata, structure and/or content. These are stored in a set of [annotations](#annotations) linked off of a [survey report](#survey-report). The survey report is linked off of the asset definition in the open metadata repository. + + +### Survey action pipeline + +A *survey action pipeline* is a specialized implementation of a [survey action service](#survey-action-service) that runs a set of survey action services against a single asset. The implementation of the survey action pipeline determines the order that these survey action services are run. + +The aim of the survey action pipeline is to enable a detailed picture of the properties of an asset to be built up by the nested survey action services it calls. Each survey action service is able to access the results of the survey action services that have run before it. + + +## Survey report + +The *survey report* lists the [annotations](#annotations) that were created during the execution of a [survey action service](#survey-action-service). + +The survey report is created automatically in the open metadata repository when the survey action service is started. Annotations are published to report as soon as they are created by the survey action service so it is possible to process the annotations from a long-running survey before it has completed. + +### Annotations + +An *annotation* describes one or more related properties about an [asset](/concepts/asset) that has been surveyed by a [survey action service](#survey-action-service). + +Different types of annotations provide different types of information. + +| Annotation type | Description | +|---|---| +| Classification annotation | Captures a recommendation of which classifications to attach to this asset. It can be made at the asset or data field level. | +| Data class annotation | Captures a recommendation of which data class this data field closely represents. | +| Data profile annotation | Capture the characteristics of the data values stored in a specific data field in a data source. | +| Data profile log annotation | Capture the names of the log files where profile characteristics of the data values stored in a specific data field. This is used when the profile results are too large to store in open metadata. | +| Data source measurement annotation | Collect arbitrary properties about a data source. | +| Data source physical status annotation | Documents the physical characteristics of a data source asset. | +| Relationship advice annotation | Document a recommended relationship that should be established with the asset. | +| Quality annotation | Document calculated quality scores on different dimensions. | +| Semantic annotation | Documents suggested meanings for this data based on the values and name of the field. | +| Suspect duplicate annotation | Identifies other asset definitions that seem to point to the same physical asset. | + +## Survey context + +A survey action service is implemented as a specialized [connector](/concepts/connector) that is passed a *survey context* as part of its initialization process. + +The *survey context* provides the [survey action service](#survey-action-service) with access to information about the survey request along with the open metadata repository interfaces. + +![Survey context](survey-context.svg) + +The survey context provides parameters used by a survey action service to locate and analyze an asset and then record the results. + +### Annotation store + +The annotation store provides a survey action service with methods to write [annotations](#annotation) to an open metadata repository. These annotations describe the results of the analysis performed on an asset by the survey action service. + +The annotations are automatically linked to the survey survey report that is in turn linked off of the analysed asset. Each annotation can also be linked to other metadata elements that is describes. + +## Survey asset store + +A survey action service is able to create a [resource connector](/concepts/digital-resource-connector) for the asset it is to analyze using the *survey asset store*. + +## Open Metadata store + +The *open metadata store* provides a search interface that enables a survey action service to locate assets and related metadata that are described in the open metadata repository. It is also able to make updates to these elements. + +## Implementation in Egeria + +Survey action services run under the [Survey Action OMES](/services/omes/survey-action/overview) hosted in an [Engine Host](/concepts/engine-host). + +The open metadata types for a annotations are described in [area 6](/types/6) of the model. The main entity type is called [*Annotation*](/types/6/0610-Annotations). It is extended by *DataFieldAnnotation* to distinguish annotations that refer, primarily to a data field rather than the whole asset. Other more specialist annotations extend these two basic annotation types. + +--8<-- "snippets/abbr.md" diff --git a/site/docs/frameworks/saf/survey-action.drawio b/site/docs/frameworks/saf/survey-action.drawio new file mode 100644 index 0000000000..dce76952bc --- /dev/null +++ b/site/docs/frameworks/saf/survey-action.drawio @@ -0,0 +1,180 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/site/docs/frameworks/saf/survey-context.svg b/site/docs/frameworks/saf/survey-context.svg new file mode 100644 index 0000000000..7ae647d92d --- /dev/null +++ b/site/docs/frameworks/saf/survey-context.svg @@ -0,0 +1,4 @@ + + + +
Survey Action Service
Survey Action Service
requestType
requestParameters
assetGUID
assetStore
annotationStore
openMetadataStore
surveyReport
requestType...
SurveyContext
SurveyContext

  1. Choreographer of external survey services
  2. Full implementation of a survey 
Choreographer of external survey servicesFull im...
setContext()
setContext()
OpenMetadataStore
OpenMetadataStore
(Comprehensive open metadata
services from the
Governance Action Framework (GAF))
(Comprehensive open metadata...
SurveyAssetStore
SurveyAssetStore
getConnectionForAsset
getConnectorByConnection
getAssetProperties
logAssetAuditMessage
getConnectionForAsset...
AnnotationStore
AnnotationStore
getTypesOfAnnotation
getTypesOfAnnotationWithDescriptions
getPreviousAnnotationsForElement
getNewAnnotationsForElement
getExtendedAnnotations
getAnnotation
addAnnotation
addAnnotationExtension
updateAnnotation
deleteAnnotation
getTypesOfAnnotation...Text is not SVG - cannot display \ No newline at end of file diff --git a/site/docs/guides/developer/survey-action-services/annotation-intro.md b/site/docs/guides/developer/survey-action-services/annotation-intro.md new file mode 100644 index 0000000000..1dd6d147f5 --- /dev/null +++ b/site/docs/guides/developer/survey-action-services/annotation-intro.md @@ -0,0 +1,28 @@ + + + +An *annotation* describes one or more related properties about a [digital resource](/concepts/resource). Some annotations refer to the entire digital resource and others refer to a data field within the digital resource. The annotations that describe a single data field are called *data field annotations*. + +![Structure of survey reports](/guides/developer/open-discovery-services/survey-report-structure.svg) + +The annotation types defined in the [Survey Action Framework (SAF)](/frameworks/saf/overview) are as follows: + +* [Classification Annotation](/features/discovery-and-stewardship/overview/#classification-discovery) - Captures a recommendation of which classifications to attach to this asset. It can be made at the asset or data field level. +* [Data Class Annotation](/features/discovery-and-stewardship/overview/#data-class-discovery) - Captures a recommendation of which data class this data field closely represents. +* [Data Profile Annotation](/features/discovery-and-stewardship/overview/#data-profiling) - Capture the characteristics of the data values stored in a specific data field in a data source. +* [Data Profile Log Annotation](/features/discovery-and-stewardship/overview/#data-profiling) - Capture the named of the log files where profile characteristics of the data values stored in a specific data field. This is used when the profile results are too large to store in open metadata. +* [Data Source Measurement Annotation](/features/discovery-and-stewardship/overview/#capturing-measurements) - collect arbitrary properties about a digital resource. +* [Data Source Physical Status Annotation](/features/discovery-and-stewardship/overview/#capturing-measurements) - documents the physical characteristics of a data source asset. +* [Fingerprint Annotation](/features/discovery-and-stewardship/overview/#data-profiling) - Capture the characteristics of the data values stored in a specific data field or the whole digital resource and express it as a single value. +* [Request for Action Annotation](/features/discovery-and-stewardship/overview/#requesting-stewardship-action) - used to trigger governance and stewardship actions. +* [Relationship Advice Annotation](/features/discovery-and-stewardship/overview/#relationship-discovery) - document a recommended relationship that should be established with the asset. +* [Quality Annotation](/features/discovery-and-stewardship/overview/#calculating-quality-scores) - document calculated quality scores on different dimensions. +* [Schema Analysis Annotation](/features/discovery-and-stewardship/overview/#schema-extraction) - document the structure of the data (schema) inside the asset. +* [Semantic Annotation](/features/discovery-and-stewardship/overview/#semantic-discovery) - documents suggested meanings for this data based on the values and name of the field. + +??? education "Open Metadata Types for Annotations" + + The open metadata types for annotations are describe in [Area 6](/types/6). + + The main entity type is called [Annotation](/types/6/0610-Annotations). It is extended by *DataFieldAnnotation* to distinguish annotations that refer, primarily to a data field. Other more specialist annotations extend these two basic annotation types. + diff --git a/site/docs/guides/developer/survey-action-services/asset-to-survey-reports.svg b/site/docs/guides/developer/survey-action-services/asset-to-survey-reports.svg new file mode 100644 index 0000000000..414c78e3d7 --- /dev/null +++ b/site/docs/guides/developer/survey-action-services/asset-to-survey-reports.svg @@ -0,0 +1,4 @@ + + + +
Asset
Asset
Discovery Analysis Report
Discovery Analysis R...
Discovery Analysis Report
Discovery Analysis R...Text is not SVG - cannot display \ No newline at end of file diff --git a/site/docs/guides/developer/survey-action-services/operation-of-a-survey-action-service.md b/site/docs/guides/developer/survey-action-services/operation-of-a-survey-action-service.md new file mode 100644 index 0000000000..987b3468eb --- /dev/null +++ b/site/docs/guides/developer/survey-action-services/operation-of-a-survey-action-service.md @@ -0,0 +1,15 @@ + + + +![Operation of an open discovery service](/guides/developer/open-discovery-services/open-discovery-service-operation.svg) + +1. Each time an open discovery service runs, Egeria creates a discovery analysis report to describe the status and results of the open discovery service's execution. The open discovery service is passed a [discovery context](/guides/developer/open-discovery-services/overview/#discovery-context) that provides access to metadata. +2. The *discovery context* is able to supply metadata about the asset and create a connector to the digital resource using the connection information linked to the asset. The discovery service uses the connector to access the digital resource's contents in order to perform the analysis. +3. The discovery service creates *discovery annotations* to record the results of its analysis. It adds them to the discovery context which stores them in open metadata attached to the discovery analysis report. +4. The discovery annotations can be reviewed and commented on through an external stewardship process. This means choices from, for example, a list of potential options proposed by the discovery services, can be verified and the best one selected by an individual expert. The resulting choices are added to annotation reviews attached to the appropriate annotations. +5. The next time the open discovery service runs, a new discovery analysis report is created to link new attachments. +6. The discovery context provides access to the existing attachments for that asset along with any annotation reviews. The discovery services is able to link its new annotations to the existing annotations as an annotation extension. This means that the stewards can see the history associated with the new information. + + + + diff --git a/site/docs/guides/developer/survey-action-services/overview.md b/site/docs/guides/developer/survey-action-services/overview.md new file mode 100644 index 0000000000..799ed8e937 --- /dev/null +++ b/site/docs/guides/developer/survey-action-services/overview.md @@ -0,0 +1,216 @@ + + + +# Writing a Survey Action Service + +---8<-- "docs/connectors/survey-action/survey-action-service-intro.md" +---8<-- "docs/connectors/survey-action/survey-action-pipeline-intro.md" + +## Implementing survey action services + +A survey action service is implemented as a specialized [Open Connector Framework (OCF)](/frameworks/ocf/overview) connector. This means it has two parts to it: a *connector provider* that is responsible for creating a new instance of the survey action service and the *survey action service implementation* (that is the *connector implementation* in the OCF terminology.) + +### Dependencies + +The interfaces and base classes of the connector provider and survey action service implementation are provided by the [Survey Action Framework (ODF)](/frameworks/odf/overview). This means you need to include the following Egeria modules in your build script: + +* Open Connector Framework +* Survey Action Framework +* Governance Action Framework for access to open metadata definitions. +* Audit Log Framework for audit logging and exceptions. +* The digital resource connector(s) used to gain access to the digital resource being analysed. + +=== "Maven dependencies" + ```xml + + + + + org.odpi.egeria + audit-log-framework + + + + org.odpi.egeria + open-connector-framework + + + + org.odpi.egeria + survey-action-framework + + + + org.odpi.egeria + governance-action-framework + + + + : + Digital resource connectors need to be included + : + + + + + ``` +=== "Gradle dependencies" + ``` + dependencies { + implementation project(':open-metadata-implementation:frameworks:audit-log-framework') + implementation project(':open-metadata-implementation:frameworks:open-connector-framework') + implementation project(':open-metadata-implementation:frameworks:survey-action-framework') + implementation project(':open-metadata-implementation:frameworks:governance-action-framework') + implementation project('...data store connector...') + } + ``` + + +### Connector provider for a survey action service + +Each connector provider for a survey action service extends the following base class: + +``` +org.odpi.openmetadata.frameworks.surveyaction.SurveyActionServiceProvider +``` +This assumes: + +- The survey action service implementation class is instantiated with the default constructor. This means all of its configuration information is contained in the [Connection](/concepts/connection) object supplied on the `initialize()` method. + +If your survey action service implementation matches these requirements, its connector provider implementation need only implement a constructor to configure the base class's function with details of itself and the Java class of the connector it needs using: + +- a GUID for the [connector type](/concepts/connector-type) +- a name for the connector type. +- a description of what the connector is for and how to configure it. +- the connector class it instantiates. +- a list of the additional properties, configuration properties and secured properties needed to configure instances of the connector. +- a description of the connector for its audit log (if the connector implements `AuditLoggingComponent`). + +```java +/** + * XXXSurveyActionProvider is the OCF connector provider for the XXX survey action service. + */ +public class XXXSurveyActionProvider extends SurveyActionProviderBase +{ + /* + * Unique identifier of the connector for the audit log. + */ + private static final int connectorComponentId = 10001; /* Add unique number here - Egeria uses numbers under 1000 */ + + /* + * Unique identifier for the connector type. + */ + private static final String connectorTypeGUID = "Add unique GUID here"; + + /* + * Descriptive information about the connector for the connector type and audit log. + */ + private static final String connectorQualifiedName = "MyOrg:XXXSurveyActionService"; + private static final String connectorDisplayName = "XXX Survey Action Service"; + private static final String connectorDescription = "A survey action service that supports ... add details here."; + private static final String connectorWikiPage = "Add url to documentation here"; + + + /* + * Define the name of the connector implementation. + */ + private static final String connectorClassName = "packagename.XXXSurveyActionService"; + + /* + * Define the name of configuration properties (optional). + */ + public static final String REQUIRED_ANNOTATION_TYPES_CONFIGURATION_PROPERTY = "requiredAnnotationTypes"; + + /** + * Constructor used to initialize the ConnectorProviderBase class. + */ + public XXXSurveyActionProvider() + { + super(); + + /* + * Set up the class name of the survey action service that this provider creates. + */ + super.setConnectorClassName(connectorClassName); + + /* + * Set up the connector type that should be included in a connection used to configure this survey action service. + */ + ConnectorType connectorType = new ConnectorType(); + connectorType.setType(ConnectorType.getConnectorTypeType()); + connectorType.setGUID(connectorTypeGUID); + connectorType.setQualifiedName(connectorQualifiedName); + connectorType.setDisplayName(connectorDisplayName); + connectorType.setDescription(connectorDescription); + connectorType.setConnectorProviderClassName(this.getClass().getName()); + + List recognizedConfigurationProperties = new ArrayList<>(); + recognizedConfigurationProperties.add(REQUIRED_ANNOTATION_TYPES_CONFIGURATION_PROPERTY); + connectorType.setRecognizedConfigurationProperties(recognizedConfigurationProperties); + + super.connectorTypeBean = connectorType; + + /* + * Set up the component description used in the survey action service's audit log messages. + */ + AuditLogReportingComponent componentDescription = new AuditLogReportingComponent(); + + componentDescription.setComponentId(connectorComponentId); + componentDescription.setComponentName(connectorQualifiedName); + componentDescription.setComponentDescription(connectorDescription); + componentDescription.setComponentWikiURL(connectorWikiPage); + + super.setConnectorComponentDescription(componentDescription); + } +} +``` + +### Survey action service implementation + +The survey action service implementation inherits from: + +``` +org.odpi.openmetadata.frameworks.surveyaction.SurveyActionServiceConnector +``` +This base class handles the survey context, any nested connectors and the audit log. The following variables are set up: + +* `surveyServiceName` - the qualified name of the survey action service from the open metadata definition. + +* `surveyContext` - the survey context that provides methods for querying and storing survey metadata. + +* `auditLog` - audit logging destination. + +* `embeddedConnectors` - connectors created from connections nested in the survey action service connection. These connectors are typically survey action services that are embedded because this survey action service is an [survey action pipeline](#survey-action-pipelines). + + +There are two methods that you need to implement: + +* `start()` - this method is where all the work takes place. It retrieves any configuration properties from the connection, uses the survey context to access the asset and its [connector to the digital resource](/concepts/digital-resource-connector). As it analyses the content of the digital resource, it adds its results to the annotation store retrieved from the survey context. The annotations are immediately stored in one of the open metadata repositories. + +* `disconnect()` - this method is called either because the survey action service has completed running, or because its server is shutting down. The survey action service implementation should release any resources it is using - for example by calling the disconnect method on the digital resource connector it is using. + +### Survey Context + +The survey context provides the *survey action service* with access to information about the survey request along with the open metadata repository interfaces. + +![Structure of the survey context](/frameworks/saf/survey-context.svg) + +#### Survey Asset Catalog Store + +The survey asset catalog store provides a search interface that enables a survey action service to locate assets that are described in the open metadata repository. + +#### Annotation Store + +The survey annotation store provides a survey action service with access to the [annotations](/concepts/survey-report/#annotations) created about a specific asset both from past invocations of survey action services and those created during the current execution. + +#### Open Metadata Store + +Provides access to all types of open metadata through the Governance Action Framework (GAF). + +## Example implementations + +The definition of the connector interfaces for survey action services is defined in the [survey-action-services :material-github:](https://github.com/odpi/egeria/tree/main/open-metadata-implementation/frameworks/survey-action-framework/src/main/java/org/odpi/openmetadata/frameworks/survey){ target=gh } module. + +There are implementations of survey action services in the [survey-service-connectors](/connectors/#survey-action-services). + +---8<-- "snippets/abbr.md" diff --git a/site/docs/guides/developer/survey-action-services/survey-action-classification-discovery.svg b/site/docs/guides/developer/survey-action-services/survey-action-classification-discovery.svg new file mode 100644 index 0000000000..f0ed949cfe --- /dev/null +++ b/site/docs/guides/developer/survey-action-services/survey-action-classification-discovery.svg @@ -0,0 +1,4 @@ + + + +
Discovery Analysis Report
Discovery Analysis R...
Annotation
Annotation
Classification
Annotation
Classification...
Schema Analysis
Annotation
Schema Analysis...
Data Field
Data Field
Classification
Annotation
Classification...Text is not SVG - cannot display \ No newline at end of file diff --git a/site/docs/guides/developer/survey-action-services/survey-action-data-class-discovery.svg b/site/docs/guides/developer/survey-action-services/survey-action-data-class-discovery.svg new file mode 100644 index 0000000000..fffa43446b --- /dev/null +++ b/site/docs/guides/developer/survey-action-services/survey-action-data-class-discovery.svg @@ -0,0 +1,4 @@ + + + +
Discovery Analysis Report
Discovery Analysis R...
Annotation
Annotation
Fingerprint
Annotation
Fingerprint...
Schema Analysis
Annotation
Schema Analysis...
Data Field
Data Field
Data Class
Annotation
Data Class...
Schema Attribute
Schema Attribute
Schema Type
Schema TypeText is not SVG - cannot display \ No newline at end of file diff --git a/site/docs/guides/developer/survey-action-services/survey-action-data-profiling.svg b/site/docs/guides/developer/survey-action-services/survey-action-data-profiling.svg new file mode 100644 index 0000000000..7a6bfc0cf6 --- /dev/null +++ b/site/docs/guides/developer/survey-action-services/survey-action-data-profiling.svg @@ -0,0 +1,4 @@ + + + +
Discovery Analysis Report
Discovery Analysis R...
Annotation
Annotation
Fingerprint
Annotation
Fingerprint...
Schema Analysis
Annotation
Schema Analysis...
Data Field
Data Field
Data Profile
Annotation
Data Profile...
Fingerprint
Annotation
Fingerprint...
Schema Attribute
Schema Attribute
Schema Type
Schema Type
Data Profile
Log Annotation
Data Profile...
Log File Asset
Log File AssetText is not SVG - cannot display \ No newline at end of file diff --git a/site/docs/guides/developer/survey-action-services/survey-action-data-source-measurement.svg b/site/docs/guides/developer/survey-action-services/survey-action-data-source-measurement.svg new file mode 100644 index 0000000000..3c1f8896df --- /dev/null +++ b/site/docs/guides/developer/survey-action-services/survey-action-data-source-measurement.svg @@ -0,0 +1,4 @@ + + + +
Discovery Analysis Report
Discovery Analysis R...
Annotation
Annotation
Annotation
Annotation
Data Source
Measurement
Annotation
Data Source...
Data Source
Physical Status Annotation
Data Source...Text is not SVG - cannot display \ No newline at end of file diff --git a/site/docs/guides/developer/survey-action-services/survey-action-pipeline-example.svg b/site/docs/guides/developer/survey-action-services/survey-action-pipeline-example.svg new file mode 100644 index 0000000000..b1a27c7d11 --- /dev/null +++ b/site/docs/guides/developer/survey-action-services/survey-action-pipeline-example.svg @@ -0,0 +1,4 @@ + + + +
Schema Extraction
Schema Extraction
Data Profiling
Data Profiling
Data Source Measurement
Data Source Measureme...
Classification
Analysis
Classification...Text is not SVG - cannot display \ No newline at end of file diff --git a/site/docs/guides/developer/survey-action-services/survey-action-pipeline-intro.md b/site/docs/guides/developer/survey-action-services/survey-action-pipeline-intro.md new file mode 100644 index 0000000000..5c8f9b2cd3 --- /dev/null +++ b/site/docs/guides/developer/survey-action-services/survey-action-pipeline-intro.md @@ -0,0 +1,13 @@ + + + +There is a lot of common functions that are used repeatedly during the discovery process. + +An *open discovery pipeline* is a specialized implementation of an [open discovery service](/concepts/open-discovery-service) that runs a set of open discovery services against a single [digital resource](/concepts/resource). The implementation of the open discovery pipeline determines the order that these open discovery services are run. + +![Open discovery pipeline example](/guides/developer/open-discovery-services/open-discovery-pipeline-example.svg) + +Each open discovery service in the pipeline is able to access the results of the open discovery services that have run before it through the [discovery context](/guides/developer/open-discovery-services/overview/#discovery-context). The combined results of the open discovery pipeline are grouped into a single [discovery analysis report](/concepts/discovery-analysis-report) linked off of the asset. + +The aim of the open discovery pipeline is to enable reusable open discovery service implementations to be choreographed together for different types of digital resource. + diff --git a/site/docs/guides/developer/survey-action-services/survey-action-quality-scores.svg b/site/docs/guides/developer/survey-action-services/survey-action-quality-scores.svg new file mode 100644 index 0000000000..7f7eb73162 --- /dev/null +++ b/site/docs/guides/developer/survey-action-services/survey-action-quality-scores.svg @@ -0,0 +1,4 @@ + + + +
Discovery Analysis Report
Discovery Analysis R...
Annotation
Annotation
Quality
Annotation
Quality...
Schema Analysis
Annotation
Schema Analysis...
Data Field
Data Field
Quality
Annotation
Quality...Text is not SVG - cannot display \ No newline at end of file diff --git a/site/docs/guides/developer/survey-action-services/survey-action-relationship-discovery.svg b/site/docs/guides/developer/survey-action-services/survey-action-relationship-discovery.svg new file mode 100644 index 0000000000..e5071c0f71 --- /dev/null +++ b/site/docs/guides/developer/survey-action-services/survey-action-relationship-discovery.svg @@ -0,0 +1,4 @@ + + + +
Discovery Analysis Report
Discovery Analysis R...
Annotation
Annotation
Relationship Advice
Annotation
Relationship Advice...
Schema Analysis
Annotation
Schema Analysis...
Data Field
Data Field
Relationship Advice
Annotation
Relationship Advice...
Relationship
Annotation
Relationship...Text is not SVG - cannot display \ No newline at end of file diff --git a/site/docs/guides/developer/survey-action-services/survey-action-request-for-action.svg b/site/docs/guides/developer/survey-action-services/survey-action-request-for-action.svg new file mode 100644 index 0000000000..0a3a720776 --- /dev/null +++ b/site/docs/guides/developer/survey-action-services/survey-action-request-for-action.svg @@ -0,0 +1,4 @@ + + + +
Discovery Analysis Report
Discovery Analysis R...
Annotation
Annotation
RequestForAction
RequestForAction
Schema Analysis
Annotation
Schema Analysis...
Data Field
Data Field
RequestForAction
RequestForActionText is not SVG - cannot display \ No newline at end of file diff --git a/site/docs/guides/developer/survey-action-services/survey-action-schema-extraction-with-schema-links.svg b/site/docs/guides/developer/survey-action-services/survey-action-schema-extraction-with-schema-links.svg new file mode 100644 index 0000000000..1d621b6b0d --- /dev/null +++ b/site/docs/guides/developer/survey-action-services/survey-action-schema-extraction-with-schema-links.svg @@ -0,0 +1,4 @@ + + + +
Discovery Analysis Report
Discovery Analysis R...
Annotation
Annotation
Annotation
Annotation
Schema Analysis
Annotation
Schema Analysis...
Data Field
Data Field
Data Field
Data Field
Data Field
Data Field
Schema Attribute
Schema Attribute
Schema Type
Schema Type
Schema Attribute
Schema Attribute
Asset
Asset
AssetSchemaType relationship
AssetSchemaType relationship
Schema Attribute
Schema AttributeText is not SVG - cannot display \ No newline at end of file diff --git a/site/docs/guides/developer/survey-action-services/survey-action-schema-extraction.svg b/site/docs/guides/developer/survey-action-services/survey-action-schema-extraction.svg new file mode 100644 index 0000000000..1bec70a427 --- /dev/null +++ b/site/docs/guides/developer/survey-action-services/survey-action-schema-extraction.svg @@ -0,0 +1,4 @@ + + + +
Discovery Analysis Report
Discovery Analysis R...
Annotation
Annotation
Annotation
Annotation
Schema Analysis
Annotation
Schema Analysis...
Data Field
Data Field
Data Field
Data Field
Data Field
Data FieldText is not SVG - cannot display \ No newline at end of file diff --git a/site/docs/guides/developer/survey-action-services/survey-action-semantic-discovery.svg b/site/docs/guides/developer/survey-action-services/survey-action-semantic-discovery.svg new file mode 100644 index 0000000000..7c4ea01c3e --- /dev/null +++ b/site/docs/guides/developer/survey-action-services/survey-action-semantic-discovery.svg @@ -0,0 +1,4 @@ + + + +
Discovery Analysis Report
Discovery Analysis R...
Annotation
Annotation
Fingerprint
Annotation
Fingerprint...
Schema Analysis
Annotation
Schema Analysis...
Data Field
Data Field
Semantic
Annotation
Semantic...
Schema Attribute
Schema AttributeText is not SVG - cannot display \ No newline at end of file diff --git a/site/docs/guides/developer/survey-action-services/survey-action-service-operation.svg b/site/docs/guides/developer/survey-action-services/survey-action-service-operation.svg new file mode 100644 index 0000000000..74e5469c66 --- /dev/null +++ b/site/docs/guides/developer/survey-action-services/survey-action-service-operation.svg @@ -0,0 +1,4 @@ + + + +
Asset
Asset
Discovery Analysis Report
(Run 1)
Discovery Analysis R...
Discovery Analysis Report
(Run 2)
Discovery Analysis R...
Annotation
Annotation
Annotation
Annotation
Open Discovery
Service
Open Discovery...
Run 1
Run 1
Run 2
Run 2
Annotation
Annotation
Annotation
Annotation
Annotation
Annotation
Stewardship
Process
Stewardship...
Steward
Stewa...
Annotation
Review
Annotation...
Annotation
Review
Annotation...
1
1
2
2
Connection
Connection
ConnectorType
ConnectorType
Endpoint
Endpoint
Digital Resource
Digital Re...
3
3
4
4
5
5
AnnotationExtension
AnnotationExtension
6
6Text is not SVG - cannot display \ No newline at end of file diff --git a/site/docs/guides/developer/survey-action-services/survey-action.drawio b/site/docs/guides/developer/survey-action-services/survey-action.drawio new file mode 100644 index 0000000000..c4c23f0f61 --- /dev/null +++ b/site/docs/guides/developer/survey-action-services/survey-action.drawio @@ -0,0 +1,874 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/site/docs/guides/developer/survey-action-services/survey-report-structure.svg b/site/docs/guides/developer/survey-action-services/survey-report-structure.svg new file mode 100644 index 0000000000..2ff524c9de --- /dev/null +++ b/site/docs/guides/developer/survey-action-services/survey-report-structure.svg @@ -0,0 +1,4 @@ + + + +
Discovery Analysis Report
Discovery Analysis R...
Annotation
Annotation
Annotation
Annotation
Annotation
Annotation
Schema Analysis
Annotation
Schema Analysis...
Data Field
Data Field
Annotations describing
the whole digital resource
Annotations describing...
Annotation
Annotation
Data Field
Annotation
Data Field...
Data Field
Data Field
Data Field
Data Field
Annotations describing
a single data field
Annotations describing...Text is not SVG - cannot display \ No newline at end of file diff --git a/site/docs/services/omes/engine-services-survey-action-server-side.svg b/site/docs/services/omes/engine-services-survey-action-server-side.svg new file mode 100644 index 0000000000..96d92067b7 --- /dev/null +++ b/site/docs/services/omes/engine-services-survey-action-server-side.svg @@ -0,0 +1,4 @@ + + + +
Engine Host OMAG Server
Engine Host OMAG Server
Engine Host Services
Engine Host Services
Metadata Server
Metadata Server
Out
Topic
Out...
Listener
Listener
Request
Handler
Request...
Governance Engine
OMAS Client
Governance Engine...
REST API calls
REST API calls
OutTopic
Events
OutTopic...
Configuration
Handler
Configuration...
Governance Engine
OMAS
Governance Engine...
REST API
REST API
Stewardship Action OMAS
Stewardship Action...
REST API calls
REST API calls
Admin
Admin
REST API
REST API
Survey
Action
Engine
Service
Survey...
Admin
Admin
REST API
REST API
Survey Action Engine Handler
Survey Action Engine Handler
Governance Services Map
Governance Services Map
Stewardship
Action
OMAS
Client
Stewardship...
Governance
Service
Handler
Governance...
Governance
Service
Handler
Governance...
Survey Action
Service
Handler
Survey Action...
Audit Log (Repository Services)
Audit Log (Repository Services)Text is not SVG - cannot display \ No newline at end of file diff --git a/site/docs/services/omes/engine-services.drawio b/site/docs/services/omes/engine-services.drawio index 03d6fb3fa6..bf79875bbe 100644 --- a/site/docs/services/omes/engine-services.drawio +++ b/site/docs/services/omes/engine-services.drawio @@ -1,6 +1,6 @@ - + - + @@ -801,95 +801,95 @@ - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + @@ -898,57 +898,205 @@ - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/site/docs/services/omes/survey-action/overview.md b/site/docs/services/omes/survey-action/overview.md new file mode 100644 index 0000000000..5cc1b968e8 --- /dev/null +++ b/site/docs/services/omes/survey-action/overview.md @@ -0,0 +1,25 @@ +--- +hide: +- toc +--- + + + + +--8<-- "snippets/content-status/in-development.md" + +# Survey Action Open Metadata Engine Service (OMES) + +The Survey Action OMES provides support for [event action engines](/concepts/event-action-engine) that are part of the [Survey Action Framework (SAF)](/frameworks/saf/overview). + +![Survey Action OMES](/services/omes/engine-services-survey-action-server-side.svg) + +## Survey action engines + +A survey action engine hosts specialized governance services called [survey action services](/concepts/survey-action-services). + +The Survey Action OMES is capable of hosting one or more [survey action engines](/concepts/survey-action-engine). + +The survey action engine services call the [Stewardship Action Open Metadata Access Service (OMAS)](/services/omas/stewardship-action/overview) running in a [metadata access server](/concepts/metadata-access-server) to retrieve information about events and to store the results of the survey action services' work. + +--8<-- "snippets/abbr.md" diff --git a/site/docs/types/4/0461-Governance-Engines.md b/site/docs/types/4/0461-Governance-Engines.md index 1376030e59..6bd792bfa7 100644 --- a/site/docs/types/4/0461-Governance-Engines.md +++ b/site/docs/types/4/0461-Governance-Engines.md @@ -14,9 +14,10 @@ A [governance engine](/concepts/governance-engine) is a [software capability](/t Open metadata recognizes three types of governance engine: * *GovernanceActionEngine* - [Governance action engines and services](/concepts/governance-action-engine) support the active governance of metadata and the resources they represent. There are different types of governance action engines/services that are defined by the [Governance Action Framework (GAF)](/frameworks/gaf/overview). -* *[OpenDiscoveryEngine](/types/6/0601-Open-Discovery-Engine)* - [Discovery engines and services](/concepts/open-discovery-engine) support the analysis of [digital resources](/concepts/resource). The results of this analysis are stored in a [discovery analysis report](/types/6/0605-Open-Discovery-Analysis-Reports) chained off of the corresponding [Asset](/types/0/0010-Base-Model#asset) metadata element. The interfaces for discovery are found in the [Open Discovery Framework (ODF)](/frameworks/odf/overview). The types for the open discovery engines are shown on model . -* *RepositoryGovernanceEngine* - [Repository governance engines and services](/concepts/repository-governance-engine) support the maintenance of repository level concerns, such as monitoring audit logs and maintaining [open metadata archives](/concepts/open-metadata-archive) that are defined in the [Open Metadata Repository Services (OMRS)](/services/omrs). +* *OpenSurveyEngine* - * *EventActionEngine* - event action engines and services support the automated management of [context events](/types/4/0475-Context-Events) and their associated [actions](/types/1/0137-Actions). Note - this type of engine is still in development. +* *RepositoryGovernanceEngine* - [Repository governance engines and services](/concepts/repository-governance-engine) support the maintenance of repository level concerns, such as monitoring audit logs and maintaining [open metadata archives](/concepts/open-metadata-archive) that are defined in the [Open Metadata Repository Services (OMRS)](/services/omrs). +* *[OpenDiscoveryEngine](/types/6/0601-Open-Discovery-Engine)* - [Discovery engines and services](/concepts/open-discovery-engine) support the analysis of [digital resources](/concepts/resource). The results of this analysis are stored in a [discovery analysis report](/types/6/0605-Open-Discovery-Analysis-Reports) chained off of the corresponding [Asset](/types/0/0010-Base-Model#asset) metadata element. The interfaces for discovery are found in the [Open Discovery Framework (ODF)](/frameworks/odf/overview). The types for the open discovery engines are shown on model . ## SupportedGovernanceService relationship @@ -37,9 +38,10 @@ A governance service can be linked to multiple governance engines via the *Suppo The subtype of the governance service linked via the *SupportedGovernanceService* relationship should be consistent with the subtype of the associated governance engine. For example: * A *GovernanceActionService* is linked to a *GovernanceActionEngine*. -* An *OpenDiscoveryService* is linked to an *OpenDiscoveryService*. +* An *OpenSurveyService* is linked to an *OpenSurveyService*. * A *RepositoryGovernanceService* is linked to a *RepositoryGovernanceEngine*. * An *EventActionService* is linked to a *EventActionEngine*. +* An *OpenDiscoveryService* is linked to an *OpenDiscoveryService*. ??? education "Further information" diff --git a/site/docs/types/4/0461-Governance-Engines.svg b/site/docs/types/4/0461-Governance-Engines.svg index 0ff765c5b2..303c08fbdb 100644 --- a/site/docs/types/4/0461-Governance-Engines.svg +++ b/site/docs/types/4/0461-Governance-Engines.svg @@ -1,4 +1,4 @@ -
0461 - Governance Engines
0461 - Governance Engines
«entity»
GovernanceActionEngine
«entity»...
«entity»
SoftwareServerCapability
«entity»...
«entity»
GovernanceService
«entity»...
«relationship»
SupportedGovernanceService
«relationship»...
*
*
*
*
requestType : string
serviceRequestType : string
requestParameters : map<string, string>
requestType : string...
supportedGovernanceServices
supportedGovernanceServi...
calledFromGovernanceEngines
calledFromGovernanceEngines
«entity»
DeployedConnector
«entity»...
«entity»
GovernanceActionService
«entity»...
«entity»
GovernanceEngine
«entity»...
«entity»
RepositoryGovernanceEngine
«entity»...
«entity»
RepositoryGovernanceService
«entity»...
«entity»
EventActionEngine
«entity»...
«entity»
EventActionService
«entity»...Text is not SVG - cannot display \ No newline at end of file +
0461 - Governance Engines
0461 - Governance Engines
«entity»
GovernanceActionEngine
«entity»...
«entity»
SoftwareServerCapability
«entity»...
«entity»
GovernanceService
«entity»...
«relationship»
SupportedGovernanceService
«relationship»...
*
*
*
*
requestType : string
serviceRequestType : string
requestParameters : map<string, string>
requestType : string...
supportedGovernanceServices
supportedGovernanceServi...
calledFromGovernanceEngines
calledFromGovernanceEngines
«entity»
DeployedConnector
«entity»...
«entity»
GovernanceActionService
«entity»...
«entity»
GovernanceEngine
«entity»...
«entity»
RepositoryGovernanceEngine
«entity»...
«entity»
RepositoryGovernanceService
«entity»...
«entity»
EventActionEngine
«entity»...
«entity»
EventActionService
«entity»...
«entity»
SurveyActionEngine
«entity»...
«entity»
SurveyActionService
«entity»...Text is not SVG - cannot display \ No newline at end of file diff --git a/site/docs/types/4/area-4-governance.drawio b/site/docs/types/4/area-4-governance.drawio index 7b6ab61eba..328ceacb1a 100644 --- a/site/docs/types/4/area-4-governance.drawio +++ b/site/docs/types/4/area-4-governance.drawio @@ -1,6 +1,6 @@ - + - + @@ -2168,7 +2168,7 @@ - + @@ -2176,134 +2176,156 @@ - + - + - + - + - + - + - + - - + + - + - + - + - + - + - - + + - + - + - + - + - + - + - + - + - + - + - + - + - - + + - + - + - + - + - + - + + + + + + + + + + + + + + + + + + + + + + + - + @@ -2440,7 +2462,7 @@ - + @@ -2569,195 +2591,62 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + @@ -2765,61 +2654,61 @@ - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + @@ -2829,7 +2718,7 @@ - + diff --git a/site/docs/types/5/0533-Graph-Schemas.svg b/site/docs/types/5/0533-Graph-Schemas.svg index e1baf2e93d..799883ccd0 100644 --- a/site/docs/types/5/0533-Graph-Schemas.svg +++ b/site/docs/types/5/0533-Graph-Schemas.svg @@ -1,4 +1,4 @@ - + -
0533 - Graph Schemas
0533 - Graph Schemas
«entity»
SchemaAttribute
«entity»...
*
*
edges
edges
*
*
vertices
vertices
«entity»
RootSchemaType
«entity»...
«entity»
GraphSchemaType
«entity»...
«entity»
GraphVertex
«entity»...
«entity»
GraphEdge
«entity»...
«relationship»
GraphEdgeLink
«relationship»...

 Text is not SVG - cannot display \ No newline at end of file +
0533 - Graph Schemas
0533 - Graph Schemas
«entity»
SchemaAttribute
«entity»...
*
*
edges
edges
*
*
vertices
vertices
«entity»
RootSchemaType
«entity»...
«entity»
GraphSchemaType
«entity»...
«entity»
GraphVertex
«entity»...
«entity»
GraphEdge
«entity»...
«relationship»
GraphEdgeLink
«relationship»...
linkTypeName : string
relationshipEnd : int
relationshipEndName : string
description : string
minCardinality : int
maxCardinality : int
additionalProperties : map<string, string>
linkTypeName : string...Text is not SVG - cannot display \ No newline at end of file diff --git a/site/docs/types/5/area-5-models-and-schema.drawio b/site/docs/types/5/area-5-models-and-schema.drawio index db9d2c692a..1dde38e10b 100644 --- a/site/docs/types/5/area-5-models-and-schema.drawio +++ b/site/docs/types/5/area-5-models-and-schema.drawio @@ -1,6 +1,6 @@ - + - + @@ -501,7 +501,7 @@ - + @@ -1141,7 +1141,7 @@ - + @@ -1200,18 +1200,18 @@ - + - + - + - + @@ -1401,7 +1401,7 @@ - + @@ -1465,15 +1465,6 @@ - - - - - - - - - @@ -1488,6 +1479,12 @@ + + + + + + diff --git a/site/docs/types/6/0601-Open-Discovery-Engine.svg b/site/docs/types/6/0601-Open-Discovery-Engine.svg index 8af00c6253..66f528693c 100644 --- a/site/docs/types/6/0601-Open-Discovery-Engine.svg +++ b/site/docs/types/6/0601-Open-Discovery-Engine.svg @@ -1,3 +1,4 @@ + -
0601 - Open Discovery Engines and Services
0601 - Open Discovery Engines and Services
«entity»
OpenDiscoveryEngine
«entity»...
«entity»
GovernanceEngine
«entity»...
«entity»
OpenDiscoveryService
«entity»...
«entity»
GovernanceService
«entity»...
«entity»
OpenDiscoveryPipeline
«entity»...Viewer does not support full SVG 1.1 \ No newline at end of file +
0601 - Open Discovery Engines and Services
0601 - Open Discovery Engines and Services
«entity»
OpenDiscoveryEngine
«entity»...
«entity»
GovernanceEngine
«entity»...
«entity»
OpenDiscoveryService
«entity»...
«entity»
GovernanceService
«entity»...
«entity»
OpenDiscoveryPipeline
«entity»...
Deprecated
DeprecatedText is not SVG - cannot display \ No newline at end of file diff --git a/site/docs/types/6/0603-Survey-Reports.md b/site/docs/types/6/0603-Survey-Reports.md new file mode 100644 index 0000000000..15d39651ba --- /dev/null +++ b/site/docs/types/6/0603-Survey-Reports.md @@ -0,0 +1,36 @@ + + + +# 0603 Survey Reports + +A *SurveyReport* entity is created each time a [survey action service](/concepts/survey-action-service) runs. + +It acts as a parent entity to the [annotations](0610-Annotations.md) created by the execution of the survey action service. These annotations contain the findings of the survey. + +![UML](0603-Survey-Reports.svg) + +## SurveyReport entity + +The *SurveyReport* entity is a [*Referenceable*](/types/0/0010-Base-Model) entity that describes both the context behind the request to run the survey action service, and the execution status of the survey action service. + +* *displayName* is a short identifier that helps a consumer describe or choose a report. +* *description* gives more information about the analysis performed. +* *purpose* provides additional reasons, or expectations of the initiator of the survey request. +* *analysisParameters* describe the request parameters passed to the survey action service when it started. + +As the survey action services runs, the *SurveyReport* entity is updated as follows: + +* *startDate* - this date/time is filled out when the survey action service starts. If it is null, it means the survey action service is waiting to start. +* *analysisStep* - this is filled out by the survey action service as it runs to show where it is in its analysis. This is useful for long-running services. +* *completionDate* - this date/time is filled out when the survey action service disconnects (completes). If it is null, it means the survey action service has not completed. +* *completionMessage* is optionally filled in once the survey action service is completed. It is typically only used if the survey action service encountered an error or was unable to complete all steps. If the *completionDate* is set, but the *completionMessage* is not set, it is reasonable to assume that the survey action service completed successfully. + +## EngineActionSurveyReport relationship + +The *EngineActionSurveyReport* relationship links a *SurveyReport* entity to the [*EngineAction*](/types/4/0463/Engine-Actions) entity used to coordinate the execution of the survey action service. + +## AssetSurveyReport relationship + +The *AssetSurveyReport* relationship links a *SurveyReport* entity to the [*Asset*](/types/0/0010-Base-Model) entity that represents the real-world resource being analysed. + +--8<-- "snippets/abbr.md" \ No newline at end of file diff --git a/site/docs/types/6/0603-Survey-Reports.svg b/site/docs/types/6/0603-Survey-Reports.svg new file mode 100644 index 0000000000..0a5824b60b --- /dev/null +++ b/site/docs/types/6/0603-Survey-Reports.svg @@ -0,0 +1,4 @@ + + + +
0603 - Survey Reports
0603 - Survey Reports
«entity»
SurveyReport
«entity»...
displayName : string
description : string
purpose : string
analysisParameters : map<string, string>
analysisStep : string
startDate : date
completionDate : date
completionMessage : string
displayName : string...
«entity»
Referenceable
«entity»...
«relationship»
EngineActionSurveyReport
«relationship»...
*
*
0..1
0..1
surveyReports
surveyReports
engineAction
engineAction
«entity»
Asset
«entity»...
«relationship»
AssetSurveyReport
«relationship»...
*
*
0..1
0..1
surveyReports
surveyReports
surveyReportTarget
surveyReportTarget
«entity»
%3CmxGraphModel%3E%3Croot%3E%3CmxCell%20id%3D%220%22%2F%3E%3CmxCell%20id%3D%221%22%20parent%3D%220%22%2F%3E%3CmxCell%20id%3D%222%22%20style%3D%22edgeStyle%3DorthogonalEdgeStyle%3Brounded%3D0%3Bhtml%3D1%3BexitX%3D1%3BexitY%3D0.5%3BexitDx%3D0%3BexitDy%3D0%3BentryX%3D0%3BentryY%3D0.5%3BentryDx%3D0%3BentryDy%3D0%3BjettySize%3Dauto%3BorthogonalLoop%3D1%3BendArrow%3Dnone%3BendFill%3D0%3B%22%20edge%3D%221%22%20source%3D%223%22%20parent%3D%221%22%3E%3CmxGeometry%20relative%3D%221%22%20as%3D%22geometry%22%3E%3CmxPoint%20x%3D%22684%22%20y%3D%22384.5%22%20as%3D%22targetPoint%22%2F%3E%3C%2FmxGeometry%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%223%22%20value%3D%22%C2%ABentity%C2%BB%26lt%3Bbr%26gt%3B%26lt%3Bb%26gt%3BRequestForAction%26lt%3B%2Fb%26gt%3B%22%20style%3D%22html%3D1%3BstrokeColor%3D%23d79b00%3Balign%3Dcenter%3BfillColor%3D%23ffcd28%3BgradientColor%3D%23ffa500%3B%22%20vertex%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20x%3D%2291%22%20y%3D%22379%22%20width%3D%22249%22%20height%3D%2250%22%20as%3D%22geometry%22%2F%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%224%22%20value%3D%22%26lt%3Bspan%26gt%3BdiscoveryActivity%20%3A%20string%26lt%3Bbr%26gt%3B%26lt%3B%2Fspan%26gt%3B%26lt%3Bspan%26gt%3BactionRequested%20%3A%20string%26lt%3B%2Fspan%26gt%3B%26lt%3Bbr%26gt%3B%26lt%3Bspan%26gt%3BactionProperties%20%3A%20map%26amp%3Blt%3Bstring%2C%20string%26amp%3Bgt%3B%26lt%3Bbr%26gt%3B%26lt%3B%2Fspan%26gt%3B%22%20style%3D%22rounded%3D0%3BwhiteSpace%3Dwrap%3Bhtml%3D1%3BstrokeColor%3D%23000000%3BgradientColor%3D%23ffffff%3Balign%3Dleft%3B%22%20vertex%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20x%3D%2292%22%20y%3D%22429%22%20width%3D%22247%22%20height%3D%2259%22%20as%3D%22geometry%22%2F%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%225%22%20value%3D%22%22%20style%3D%22edgeStyle%3DorthogonalEdgeStyle%3Brounded%3D0%3Bhtml%3D1%3BendArrow%3Dnone%3BendFill%3D0%3BjettySize%3Dauto%3BorthogonalLoop%3D1%3BexitX%3D0%3BexitY%3D0.5%3B%22%20edge%3D%221%22%20source%3D%227%22%20target%3D%223%22%20parent%3D%221%22%3E%3CmxGeometry%20relative%3D%221%22%20as%3D%22geometry%22%2F%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%226%22%20value%3D%22%C2%ABentity%C2%BB%26lt%3Bbr%26gt%3B%26lt%3Bb%26gt%3BDataFieldAnnotation%26lt%3B%2Fb%26gt%3B%22%20style%3D%22html%3D1%3BstrokeColor%3D%23d79b00%3Balign%3Dcenter%3BfillColor%3D%23ffcd28%3BgradientColor%3D%23ffa500%3B%22%20vertex%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20x%3D%22136.5%22%20y%3D%22257.5%22%20width%3D%22160%22%20height%3D%2250%22%20as%3D%22geometry%22%2F%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%227%22%20value%3D%22%22%20style%3D%22triangle%3BwhiteSpace%3Dwrap%3Bhtml%3D1%3BstrokeColor%3D%23000000%3BgradientColor%3D%23ffffff%3Balign%3Dcenter%3Brotation%3D-90%3B%22%20vertex%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20x%3D%22207.5%22%20y%3D%22305.5%22%20width%3D%2217%22%20height%3D%2221%22%20as%3D%22geometry%22%2F%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%228%22%20value%3D%22%22%20style%3D%22endArrow%3Dnone%3Bdashed%3D1%3Bhtml%3D1%3BentryX%3D1%3BentryY%3D0.5%3BentryDx%3D0%3BentryDy%3D0%3B%22%20edge%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20width%3D%2250%22%20height%3D%2250%22%20relative%3D%221%22%20as%3D%22geometry%22%3E%3CmxPoint%20x%3D%22638%22%20y%3D%22234%22%20as%3D%22sourcePoint%22%2F%3E%3CmxPoint%20x%3D%22567.5%22%20y%3D%22234%22%20as%3D%22targetPoint%22%2F%3E%3C%2FmxGeometry%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%229%22%20value%3D%22*%22%20style%3D%22text%3Bhtml%3D1%3BstrokeColor%3Dnone%3BfillColor%3Dnone%3Balign%3Dcenter%3BverticalAlign%3Dmiddle%3BwhiteSpace%3Dwrap%3Brounded%3D0%3B%22%20vertex%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20x%3D%22636.5%22%20y%3D%22325.5%22%20width%3D%2240%22%20height%3D%2220%22%20as%3D%22geometry%22%2F%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%2210%22%20value%3D%22identifiedActions%22%20style%3D%22text%3Bhtml%3D1%3BstrokeColor%3Dnone%3BfillColor%3Dnone%3Balign%3Dcenter%3BverticalAlign%3Dmiddle%3BwhiteSpace%3Dwrap%3Brounded%3D0%3B%22%20vertex%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20x%3D%22543%22%20y%3D%22325.5%22%20width%3D%22104%22%20height%3D%2220%22%20as%3D%22geometry%22%2F%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%2211%22%20value%3D%22discoveryActivity%22%20style%3D%22text%3Bhtml%3D1%3BstrokeColor%3Dnone%3BfillColor%3Dnone%3Balign%3Dcenter%3BverticalAlign%3Dmiddle%3BwhiteSpace%3Dwrap%3Brounded%3D0%3B%22%20vertex%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20x%3D%22341.5%22%20y%3D%22381.5%22%20width%3D%22106%22%20height%3D%2220%22%20as%3D%22geometry%22%2F%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%2212%22%20value%3D%22requestForAction%22%20style%3D%22text%3Bhtml%3D1%3BstrokeColor%3Dnone%3BfillColor%3Dnone%3Balign%3Dcenter%3BverticalAlign%3Dmiddle%3BwhiteSpace%3Dwrap%3Brounded%3D0%3B%22%20vertex%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20x%3D%22571%22%20y%3D%22384%22%20width%3D%22113%22%20height%3D%2220%22%20as%3D%22geometry%22%2F%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%2213%22%20value%3D%22%C2%ABrelationship%C2%BB%26lt%3Bbr%26gt%3B%26lt%3Bb%26gt%3BDiscoveryActivity%26lt%3B%2Fb%26gt%3B%22%20style%3D%22html%3D1%3BstrokeColor%3D%23d79b00%3Balign%3Dcenter%3BfillColor%3D%23ffcd28%3BgradientColor%3D%23ffa500%3B%22%20vertex%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20x%3D%22393.5%22%20y%3D%22486%22%20width%3D%22206%22%20height%3D%2253%22%20as%3D%22geometry%22%2F%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%2214%22%20value%3D%22%22%20style%3D%22endArrow%3Dnone%3Bdashed%3D1%3Bhtml%3D1%3B%22%20edge%3D%221%22%20target%3D%2213%22%20parent%3D%221%22%3E%3CmxGeometry%20width%3D%2250%22%20height%3D%2250%22%20relative%3D%221%22%20as%3D%22geometry%22%3E%3CmxPoint%20x%3D%22496%22%20y%3D%22407%22%20as%3D%22sourcePoint%22%2F%3E%3CmxPoint%20x%3D%22543.5%22%20y%3D%22515%22%20as%3D%22targetPoint%22%2F%3E%3C%2FmxGeometry%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%2215%22%20value%3D%22description%20%3A%20string%22%20style%3D%22rounded%3D0%3BwhiteSpace%3Dwrap%3Bhtml%3D1%3BstrokeColor%3D%23000000%3BgradientColor%3D%23ffffff%3Balign%3Dleft%3B%22%20vertex%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20x%3D%22393.5%22%20y%3D%22538.5%22%20width%3D%22206%22%20height%3D%2231%22%20as%3D%22geometry%22%2F%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%2216%22%20value%3D%22*%22%20style%3D%22text%3Bhtml%3D1%3BstrokeColor%3Dnone%3BfillColor%3Dnone%3Balign%3Dcenter%3BverticalAlign%3Dmiddle%3BwhiteSpace%3Dwrap%3Brounded%3D0%3B%22%20vertex%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20x%3D%22643.5%22%20y%3D%22361.5%22%20width%3D%2240%22%20height%3D%2220%22%20as%3D%22geometry%22%2F%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%2217%22%20value%3D%22*%22%20style%3D%22text%3Bhtml%3D1%3BstrokeColor%3Dnone%3BfillColor%3Dnone%3Balign%3Dcenter%3BverticalAlign%3Dmiddle%3BwhiteSpace%3Dwrap%3Brounded%3D0%3B%22%20vertex%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20x%3D%22341.5%22%20y%3D%22405%22%20width%3D%2240%22%20height%3D%2220%22%20as%3D%22geometry%22%2F%3E%3C%2FmxCell%3E%3C%2Froot%3E%3C%2FmxGraphModel%3EngineAction
«entity»...Text is not SVG - cannot display \ No newline at end of file diff --git a/site/docs/types/6/0605-Open-Discovery-Analysis-Reports.svg b/site/docs/types/6/0605-Open-Discovery-Analysis-Reports.svg index 6217c90c26..1a15ddf01e 100644 --- a/site/docs/types/6/0605-Open-Discovery-Analysis-Reports.svg +++ b/site/docs/types/6/0605-Open-Discovery-Analysis-Reports.svg @@ -1,4 +1,4 @@ - + -
0605 - Open Discovery Analysis Reports
0605 - Open Discovery Analysis Reports
«entity»
OpenDiscoveryAnalysisReport
«entity»...

displayName : string
description : string
executionDate : date
analysisParameters : map<string, string>
discoveryAnalysisStep : string
discoveryServiceStatus : DiscoveryServiceRequestStatus

displayName : string...
«entity»
Referenceable
«entity»...
«entity»
OpenDiscoveryEngine
«entity»...
«relationship»
DiscoveryEngineReport
«relationship»...
«entity»
OpenDiscoveryService
«entity»...
«relationship»
DiscoveryInvocationReport
«relationship»...
*
*
0..1
0..1
*
*
0..1
0..1
sourceDiscoveryEngine
sourceDiscoveryEngine
serviceDiscoveryAnalysisReports
serviceDiscoveryAnalysisReports
sourceDiscoveryService
sourceDiscoveryService
discoveryEngineAnalysisReports
discoveryEngineAnalysisReports
«entity»
Asset
«entity»...
«relationship»
AssetDiscoveryReport
«relationship»...
*
*
0..1
0..1
assetDiscoveryAnalysisReports
assetDiscoveryAnalysisReports
discoveryReportTarget
discoveryReportTarget
«enumeration»
DiscoveryServiceRequestStatus
«enumeration»...
WAITING = 0
ACTIVATING = 1
IN_PROGRESS = 2
FAILED = 3
COMPLETED = 4
OTHER = 5
UNKNOWN = 99
WAITING = 0...Text is not SVG - cannot display \ No newline at end of file +
0605 - Open Discovery Analysis Reports
0605 - Open Discovery Analysis Reports
«entity»
OpenDiscoveryAnalysisReport
«entity»...

displayName : string
description : string
executionDate : date
analysisParameters : map<string, string>
discoveryAnalysisStep : string
discoveryServiceStatus : DiscoveryServiceRequestStatus

displayName : string...
«entity»
OpenDiscoveryEngine
«entity»...
«relationship»
DiscoveryEngineReport
«relationship»...
«entity»
OpenDiscoveryService
«entity»...
«relationship»
DiscoveryInvocationReport
«relationship»...
*
*
0..1
0..1
*
*
0..1
0..1
sourceDiscoveryEngine
sourceDiscoveryEngine
serviceDiscoveryAnalysisReports
serviceDiscoveryAnalysisReports
sourceDiscoveryService
sourceDiscoveryService
discoveryEngineAnalysisReports
discoveryEngineAnalysisReports
«entity»
Asset
«entity»...
«relationship»
AssetDiscoveryReport
«relationship»...
*
*
0..1
0..1
assetDiscoveryAnalysisReports
assetDiscoveryAnalysisReports
discoveryReportTarget
discoveryReportTarget
«enumeration»
DiscoveryServiceRequestStatus
«enumeration»...
WAITING = 0
ACTIVATING = 1
IN_PROGRESS = 2
FAILED = 3
COMPLETED = 4
OTHER = 5
UNKNOWN = 99
WAITING = 0...
«entity»
Referenceable
«entity»...Text is not SVG - cannot display \ No newline at end of file diff --git a/site/docs/types/6/0610-Annotations.md b/site/docs/types/6/0610-Annotations.md index c6c8803a27..459e9ae31b 100644 --- a/site/docs/types/6/0610-Annotations.md +++ b/site/docs/types/6/0610-Annotations.md @@ -1,37 +1,37 @@ ---- -hide: -- toc ---- - # 0610 Annotations -The **Annotation** entity captures the discovered characteristics of an asset. -Annotations are created by the analysis steps in the discovery service. -The attributes of the annotation capture the details of the discovery processing. -The subclasses of Annotation capture specific details of the discovered metadata. -Each annotation is linked to the discovery analysis report it was generated from. -It also links to each asset that the annotation relates to. +An annotation captures specific discovered characteristics of a real-world digital resource. + +Annotations are created by the analysis steps in the survey action service. +The attributes of the annotation capture the details of the analysis. +There are subclasses of Annotation to capture additional details. +Each annotation is linked to the report it was generated from. +It also links to other elements that the annotation relates to. Strings are used in many of the attributes to keep the model open -for discovery service developers and the tools that process them. +for capturing analysis from third-party tools. ![UML](0610-Annotations.svg) -* **annotationType** - descriptive string that acts as an identifier for the specific annotation type. This is a simple means to sub-type any one of the annotation subclasses. -* **summary** - a human-readable string to describe the annotation. -* **confidence** - an indicator of the certainty that the annotation is correct. -* **expression** - this attribute is used to provide more detail on how the asset is related to the annotation. -* **explanation** - another description field to assist human analysts reviewing the discovery results. -* **analysisStep** - identifier of the step in the discovery service that detected the annotation. -* **jsonProperties** - the properties that were used to initiate the discovery service. +## Annotation entity + +The *Annotation* entity inherits from [*OpenMetadataRoot*](/types/0/0010-Base_Model) and adds the following attributes: + +* *annotationType* - descriptive string that acts as an identifier for the specific annotation type. This is a simple means to sub-type any one of the annotation subclasses. +* *summary* - a human-readable string to describe the annotation. +* *confidence* - an indicator of the certainty that the annotation is correct. +* *expression* - this attribute is used to provide more detail on how the asset is related to the annotation. +* *explanation* - another description field to assist human analysts reviewing the analysis results. +* *analysisStep* - identifier of the step in the survey action service that detected the annotation. +* *jsonProperties* - the properties that were used to initiate the survey action service. +* *additionalProperties* - additional properties published by the survey action service. The types that follow provide more specialized annotations. * **[0615 Schema Extraction](0615-Schema-Extraction.md)** -* **[0617 Data Field Analysis](0617-Data-Field-Analysis.md)** * **[0620 Data Profiling](0620-Data-Profiling.md)** * **[0625 Data Class Discovery](0625-Data-Class-Discovery.md)** * **[0630 Semantic Discovery](0630-Semantic-Discovery.md)** @@ -41,4 +41,24 @@ The types that follow provide more specialized annotations. * **[0660 Measurements](0660-Data-Source-Measurements.md)** * **[0690 Request for Action](0690-Request-for-Action.md)** +## DataFieldAnnotation entity + +The *DataFieldAnnotation* type is used as a marker to indicate that the annotations that inherit from it will typically refer to a single schema attribute (data field) rather than the whole resource. + +## ReportedAnnotation relationship + +The *ReportedAnnotation* relationship links an annotation to the [*SurveyReport*](/types/6/0603-Survey-Reports) entity. It is used with [survey action services](/concepts/survey-action-service). + +## AssociatedAnnotation relationship + +The *AssociatedAnnotation* relationship links an annotation to an element that it describes. + +## AnnotationExtension relationship + +The *AnnotationExtension* relationship allows an annotation from a later run of the survey action service to add information to an annotation from a previous run. + +## DiscoveredAnnotation relationship + +The *DiscoveredAnnotation* relationship links an annotation to the [*OpenDiscoveryAnalysisReport*](0605-Open-Discovery-Analysis-Reports.md) entity. It is only used with [open discovery services](/concepts/open-discovery-service). + --8<-- "snippets/abbr.md" \ No newline at end of file diff --git a/site/docs/types/6/0610-Annotations.svg b/site/docs/types/6/0610-Annotations.svg index 52d190c741..33adc0a7b2 100644 --- a/site/docs/types/6/0610-Annotations.svg +++ b/site/docs/types/6/0610-Annotations.svg @@ -1,4 +1,4 @@ - + -
0610 - Annotations
0610 - Annotations
«entity»
Annotation
«entity»...
annotationType : string
summary : string
confidenceLevel : int
expression : string
explanation : string
analysisStep : string
jsonProperties : string
additionalProperties : map<string, string>
annotationType : string...
*
*
0..1
0..1
fromAnalysisReport
fromAnalysisReport
reportedAnnotations
reportedAnnotations
«entity»
OpenDiscoveryAnalysisReport
«entity»...
«relationship»
DiscoveredAnnotation
«relationship»...
extendedAnnotations
extendedAnnotations
*
*
*
*
«relationship»
AnnotationExtension
«relationship»...
annotationExtensions
annotationExtensions
«entity»
OpenMetadataRoot
«entity»...Text is not SVG - cannot display \ No newline at end of file +
0610 - Annotations
0610 - Annotations
«entity»
Annotation
«entity»...
annotationType : string
summary : string
confidenceLevel : int
expression : string
explanation : string
analysisStep : string
jsonProperties : string
additionalProperties : map<string, string>
annotationType : string...
*
*
0..1
0..1
fromSurveyReport
fromSurveyReport
reportedAnnotations
reportedAnnotations
«entity»
SurveyReport
«entity»...
«relationship»
ReportedAnnotation
«relationship»...
extendedAnnotations
extendedAnnotations
*
*
*
*
«relationship»
AnnotationExtension
«relationship»...
annotationExtensions
annotationExtensions
«entity»
OpenMetadataRoot
«entity»...
«entity»
OpenMetadataRoot
«entity»...
«relationship»
AssociatedAnnotation
«relationship»...
*
*
*
*
associatedAnnotations
associatedAnnotations
annotationSubject
annotationSubject
«entity»
DataFieldAnnotation
«entity»...
«entity»
Annotation
«entity»...
*
*
0..1
0..1
fromDiscoveryAnalysisReport
fromDiscoveryAnalysi...
reportedAnnotations
reportedAnnotations
«entity»
OpenDiscoveryAnalysisReport
«entity»...
«relationship»
DiscoveredAnnotation
«relationship»...Text is not SVG - cannot display \ No newline at end of file diff --git a/site/docs/types/6/0612-Annotation-Reviews.md b/site/docs/types/6/0612-Annotation-Reviews.md index 56ee1adf41..271209db19 100644 --- a/site/docs/types/6/0612-Annotation-Reviews.md +++ b/site/docs/types/6/0612-Annotation-Reviews.md @@ -1,26 +1,32 @@ ---- -hide: -- toc ---- - # 0612 Annotation Reviews The annotations associated with an asset can be seen by -people and tools querying the associated asset, servers or -discovery service. -However, often the analysis within a discovery service can -only make recommendations based on the information within the asset. +people and tools querying the associated asset, their elements or +the controlling engine action. +However, the analysis within a survey analysis service may +only be able to make recommendations based on the information within the resource. Where annotations refer to information that is used for governance, they need to be approved and converted into classifications, or related metadata. -The AnnotationReview entity records how the discovered annotations + + +![UML](0612-Annotation-Reviews.svg) + +## AnnotationReview entity + +The *AnnotationReview* entity records how the discovered annotations have been actioned in the metadata server and the steward that approved it. -![UML](0612-Annotation-Reviews.svg) +## AnnotationReviewLink relationship + +The *AnnotationReviewLink* relationship links an annotation to its review. + +## AnnotationStatus enumeration +The *AnnotationStatus* enumeration describes the status of an annotation review. --8<-- "snippets/abbr.md" \ No newline at end of file diff --git a/site/docs/types/6/0612-Annotation-Reviews.svg b/site/docs/types/6/0612-Annotation-Reviews.svg index 78fd63fc1d..a056f10ad7 100644 --- a/site/docs/types/6/0612-Annotation-Reviews.svg +++ b/site/docs/types/6/0612-Annotation-Reviews.svg @@ -1,4 +1,4 @@ - + -
0612 - Annotation Reviews
0612 - Annotation Reviews
«entity»
AnnotationReview
«entity»...
reviewDate : date
steward : string
comment : string
reviewDate : date...
«entity»
Annotation
«entity»...
«relationship»
AnnotationReviewLink
«relationship»...
annotationStatus : AnnotationStatus
annotationStatus : AnnotationStatus
*
*
*
*
reviewedAnnotations
reviewedAnnotatio...
annotationReviews
annotationReviews
«enumeration»
AnnotationStatus
«enumeration»...
NEW = 0
REVIEWED = 1
APPROVED = 2
ACTIONED = 3
INVALID = 4
IGNORE = 5
OTHER = 99
NEW = 0...
«entity»
OpenMetadataRoot
«entity»...Text is not SVG - cannot display \ No newline at end of file +
0612 - Annotation Reviews
0612 - Annotation Reviews
«entity»
AnnotationReview
«entity»...
reviewDate : date
steward : string
comment : string
reviewDate : date...
«entity»
Annotation
«entity»...
«relationship»
AnnotationReviewLink
«relationship»...
annotationStatus : AnnotationStatus
annotationStatus : AnnotationStatus
*
*
*
*
reviewedAnnotations
reviewedAnnotatio...
annotationReviews
annotationReviews
«enumeration»
AnnotationStatus
«enumeration»...
NEW = 0
REVIEWED = 1
APPROVED = 2
ACTIONED = 3
INVALID = 4
IGNORE = 5
OTHER = 99
NEW = 0...
«entity»
OpenMetadataRoot
«entity»...Text is not SVG - cannot display \ No newline at end of file diff --git a/site/docs/types/6/0615-Schema-Extraction.md b/site/docs/types/6/0615-Schema-Extraction.md index bed709855a..ae8dd80c9b 100644 --- a/site/docs/types/6/0615-Schema-Extraction.md +++ b/site/docs/types/6/0615-Schema-Extraction.md @@ -1,18 +1,17 @@ ---- -hide: -- toc ---- # 0615 Schema Extraction -Schema extraction creates a set of metadata entities that represent how the data is structured in the digital resource. +Schema extraction creates a set of [*SchemaElement*](/types/5/0501-Schema-Elements) entities that represent how the data is structured in the digital resource. This is then either matched with an existing schema or a new schema is created (see [Area 5](/types/5)). This may be completely automated, or with stewards' assistance. ![UML](0615-Schema-Extraction.svg) +## SchemaAnalysis entity + +The *SchemaAnalysis* entity identifies that the [survey action service](/concepts/survey-action-service) either created or verified the schema description for the analysed resource. It identifies which schema it analysed both in its attributes and, typically using the [*AssociatedAnnotation*](/types/6/0610-Annotations) relationship to link to the root schema type. --8<-- "snippets/abbr.md" \ No newline at end of file diff --git a/site/docs/types/6/0615-Schema-Extraction.svg b/site/docs/types/6/0615-Schema-Extraction.svg index 1a7cd28591..40946586d3 100644 --- a/site/docs/types/6/0615-Schema-Extraction.svg +++ b/site/docs/types/6/0615-Schema-Extraction.svg @@ -1,4 +1,4 @@ -
0615 - Schema Extraction
0615 - Schema Extraction
«entity»
SchemaAnalysisAnnotation
«entity»...
schemaName : string
schemaType : string
schemaName : string...
«entity»
Annotation
«entity»...
«entity»
DataField
«entity»...
«relationship»
DiscoveredDataField
«relationship»...
dataFieldPosition : int
dataFieldPosition : int
«relationship»
DiscoveredNestedDataField
«relationship»...
*
*
*
*
0..1
0..1
0..1
0..1
discoveredDataFields
discoveredDataFields
dataFieldPosition : int
dataFieldPosition : int
nestedDataFields
nestedDataFields
parentDataField
parentDataField
schemaAnalysisAnnotation
schemaAnalysisAnnotati...
dataFieldName : string
dataFieldType : string
dataFieldDescription : string
dataFieldNamespace : string
dataFieldAliases : array<string>
dataFieldSortOrder : DataItemSortOrder
minCardinality : int
maxCardinality : int
isNullable : int
minimumLength : int
length : int
precision : int
significantDigits : int
defaultValue : string
isDeprecated : boolean
version : long
versionIdentifier : string
additionalProperties : map<string, string>
dataFieldName : string...
«entity»
SchemaAttribute
«entity»...
schemaAttributeDefinition
schemaAttributeDefini...
*
*
0..1
0..1
«relationship»
SchemaAttributeDefinition
«relationship»...
assetGUID : string
assetGUID : string
deployedSchemaAttributes
deployedSchemaAttributes
«entity»
SchemaType
«entity»...
schemaTypeDefinition
schemaTypeDefinition
*
*
0..1
0..1
«relationship»
SchemaTypeDefinition
«relationship»...
deployedSchemaTypes
deployedSchemaTypes
«entity»
OpenMetadataRoot
«entity»...
«relationship»
DiscoveredLinkedDataField
«relationship»...
relationshipTypeName : string
relationshipEnd : int
name : string
description : string
minCardinality : int
maxCardinality : int
additionalProperties : map<string, string>
relationshipTypeName : string...
linkFromDataFields
linkFromDataFields
linkToDataFields
linkToDataFields
*
*
*
*Text is not SVG - cannot display \ No newline at end of file +
0615 - Schema Extraction
0615 - Schema Extraction
«entity»
SchemaAnalysisAnnotation
«entity»...
schemaName : string
schemaType : string
schemaName : string...
«entity»
Annotation
«entity»...Text is not SVG - cannot display \ No newline at end of file diff --git a/site/docs/types/6/0616-Data-Field-Discovery.md b/site/docs/types/6/0616-Data-Field-Discovery.md new file mode 100644 index 0000000000..48d299bb75 --- /dev/null +++ b/site/docs/types/6/0616-Data-Field-Discovery.md @@ -0,0 +1,16 @@ +--- +hide: +- toc +--- + + + + +# 0616 Data Field Discovery + +*DataField* entities are used by [*Open Discovery Services*](/concepts/open-discovery-service) to describe the schema structure discovered in a digital resource it is analysing. + +![UML](0616-Data-Field-Discovery.svg) + + +--8<-- "snippets/abbr.md" \ No newline at end of file diff --git a/site/docs/types/6/0616-Data-Field-Discovery.svg b/site/docs/types/6/0616-Data-Field-Discovery.svg new file mode 100644 index 0000000000..5813911043 --- /dev/null +++ b/site/docs/types/6/0616-Data-Field-Discovery.svg @@ -0,0 +1,4 @@ + + + +
0616 - Data Field Discovery
0616 - Data Field Discovery
«entity»
SchemaAnalysisAnnotation
«entity»...
«entity»
DataField
«entity»...
«relationship»
DiscoveredDataField
«relationship»...
dataFieldPosition : int
dataFieldPosition : int
«relationship»
DiscoveredNestedDataField
«relationship»...
*
*
*
*
0..1
0..1
0..1
0..1
discoveredDataFields
discoveredDataFields
dataFieldPosition : int
dataFieldPosition : int
nestedDataFields
nestedDataFields
parentDataField
parentDataField
schemaAnalysisAnnotation
schemaAnalysisAnnotati...
dataFieldName : string
dataFieldType : string
dataFieldDescription : string
dataFieldNamespace : string
dataFieldAliases : array<string>
dataFieldSortOrder : DataItemSortOrder
minCardinality : int
maxCardinality : int
isNullable : int
minimumLength : int
length : int
precision : int
significantDigits : int
defaultValue : string
isDeprecated : boolean
version : long
versionIdentifier : string
additionalProperties : map<string, string>
dataFieldName : string...
«entity»
SchemaAttribute
«entity»...
schemaAttributeDefinition
schemaAttributeDefini...
*
*
0..1
0..1
«relationship»
SchemaAttributeDefinition
«relationship»...
assetGUID : string
assetGUID : string
deployedSchemaAttributes
deployedSchemaAttributes
«entity»
SchemaType
«entity»...
schemaTypeDefinition
schemaTypeDefinition
*
*
0..1
0..1
«relationship»
SchemaTypeDefinition
«relationship»...
deployedSchemaTypes
deployedSchemaTypes
«entity»
OpenMetadataRoot
«entity»...
«relationship»
DiscoveredLinkedDataField
«relationship»...
relationshipTypeName : string
relationshipEnd : int
name : string
description : string
minCardinality : int
maxCardinality : int
additionalProperties : map<string, string>
relationshipTypeName : string...
linkFromDataFields
linkFromDataFields
linkToDataFields
linkToDataFields
*
*
*
*
Data Field is Deprecated
Data Field is DeprecatedText is not SVG - cannot display \ No newline at end of file diff --git a/site/docs/types/6/0617-Data-Field-Analysis.svg b/site/docs/types/6/0617-Data-Field-Analysis.svg index b2e7c27c42..3759e8ef20 100644 --- a/site/docs/types/6/0617-Data-Field-Analysis.svg +++ b/site/docs/types/6/0617-Data-Field-Analysis.svg @@ -1,4 +1,4 @@ - + -
0617 - Data Field Analysis
0617 - Data Field Analysis
«entity»
DataFieldAnnotation
«entity»...
«entity»
Annotation
«entity»...
«entity»
DataField
«entity»...
«relationship»
DataFieldAnalysis
«relationship»...
*
*
*
*
dataFieldAnnotations
dataFieldAnnotations
annotatedDataFields
annotatedDataFieldsText is not SVG - cannot display \ No newline at end of file +
0617 - Data Field Analysis
0617 - Data Field Analysis
«entity»
DataFieldAnnotation
«entity»...
«entity»
DataField
«entity»...
«relationship»
DataFieldAnalysis
«relationship»...
*
*
*
*
dataFieldAnnotations
dataFieldAnnotations
annotatedDataFields
annotatedDataFieldsText is not SVG - cannot display \ No newline at end of file diff --git a/site/docs/types/6/0630-Semantic-Discovery.md b/site/docs/types/6/0630-Semantic-Discovery.md index f4388a7c8c..a6abba9437 100644 --- a/site/docs/types/6/0630-Semantic-Discovery.md +++ b/site/docs/types/6/0630-Semantic-Discovery.md @@ -9,7 +9,7 @@ hide: # 0630 Semantic Discovery Semantic discovery is attempting to define the -meaning of the data values in the asset. +meaning of the data values in the resource. The result is a recommended glossary term stored as an annotation. diff --git a/site/docs/types/6/0635-Classification-Discovery.md b/site/docs/types/6/0635-Classification-Discovery.md index 09b20fddd3..0652634ed2 100644 --- a/site/docs/types/6/0635-Classification-Discovery.md +++ b/site/docs/types/6/0635-Classification-Discovery.md @@ -9,7 +9,6 @@ hide: # 0635 Classification Discovery Classification discovery adds suggestions for how the data could be classified. -These annotations are the discovery engine equivalent of the [Informal Tag](/types/1/0150-Feedback) shown in 0150 - Feedback in Area 1. ![UML](0635-Classification-Discovery.svg) diff --git a/site/docs/types/6/0640-Quality-Scores.md b/site/docs/types/6/0640-Quality-Scores.md index 58deaf4dea..1e083b5e01 100644 --- a/site/docs/types/6/0640-Quality-Scores.md +++ b/site/docs/types/6/0640-Quality-Scores.md @@ -12,7 +12,7 @@ Quality scores describe how well the data values, typically in a data field, con ![UML](0640-Quality-Scores.svg) -A data field can have multiple quality scores associated to it, +A schema attribute can have multiple quality scores associated to it, for different types of problems. Examples of quality problem types include: * Data class violations @@ -30,7 +30,7 @@ Each quality score for each type of problem could have additional properties ass confidence, date it was calculated, for example. The set of such properties may differ across discovery services and should be open. -A combined quality score for an entire Asset can be typically calculated as +A combined quality score for an entire resource can be typically calculated as the average of the scores for all its columns, but other factors may be taken into account by a given system and we should be able to associate a score to an Asset, independently of the way it is calculated. diff --git a/site/docs/types/6/0650-Relationship-Discovery.svg b/site/docs/types/6/0650-Relationship-Discovery.svg index 43daa787e3..6548649be0 100644 --- a/site/docs/types/6/0650-Relationship-Discovery.svg +++ b/site/docs/types/6/0650-Relationship-Discovery.svg @@ -1,4 +1,4 @@ - + -
0650 - Relationship Discovery
0650 - Relationship Discovery
«relationship»
RelationshipAnnotation
«relationship»...

«entity»
Referenceable
«entity»...
relatedFromObjectAnnotations
relatedFromObjectAnnotat...
relatedToObjectAnnotations
relatedToObjectAnnotations
*
*
*
*
«entity»
RelationshipAdviceAnnotation
«entity»...
relatedEntityGUID : string
relationshipTypeName : string
relationshipProperties : map<string, string>
relatedEntityGUID : string...
«entity»
DataFieldAnnotation
«entity»...
discoveryReportGUID : string
annotationType : string
summary : string
confidenceLevel : int
expression : string
explanation : string
analysisStep : string
jsonProperties : string
additionalProperties : map<string, string
annotationStatus : AnnotationStatus
discoveryReportGUID : string...Text is not SVG - cannot display \ No newline at end of file +
0650 - Relationship Discovery
0650 - Relationship Discovery
«entity»
RelationshipAdviceAnnotation
«entity»...
relatedEntityGUID : string
relationshipTypeName : string
relationshipProperties : map<string, string>
relatedEntityGUID : string...
«entity»
DataFieldAnnotation
«entity»...
«relationship»
RelationshipAnnotation
«relationship»...

«entity»
Referenceable
«entity»...
relatedFromObjectAnnotations
relatedFromObjectAnnotat...
relatedToObjectAnnotations
relatedToObjectAnnotations
*
*
*
*
discoveryReportGUID : string
annotationType : string
summary : string
confidenceLevel : int
expression : string
explanation : string
analysisStep : string
jsonProperties : string
additionalProperties : map<string, string
annotationStatus : AnnotationStatus
discoveryReportGUID : string...Text is not SVG - cannot display \ No newline at end of file diff --git a/site/docs/types/6/0690-Request-for-Action.md b/site/docs/types/6/0690-Request-for-Action.md index 90ecfcf05a..e6d5689f42 100644 --- a/site/docs/types/6/0690-Request-for-Action.md +++ b/site/docs/types/6/0690-Request-for-Action.md @@ -10,7 +10,7 @@ hide: A RequestForAction entity (RfA) is used to trigger the [Stewardship Action OMAS](/services/omas/stewardship-action/overview). -It is used when an [open discovery service](/concepts/open-discovery-service) performs a test on the data (such as a discovery rule) or has discovered an anomaly in the data landscape compared to its metadata that potentially needs a steward or a curator's action. +It is used when a [survey action service](/concepts/survey-action-service) performs a test on the data (such as a quality rule) or has discovered an anomaly in the data landscape compared to its metadata that potentially needs a steward or a curator's action. ![UML](0690-Request-for-Action.svg) diff --git a/site/docs/types/6/0690-Request-for-Action.svg b/site/docs/types/6/0690-Request-for-Action.svg index 26f5ca1d2c..eb6d2a51e9 100644 --- a/site/docs/types/6/0690-Request-for-Action.svg +++ b/site/docs/types/6/0690-Request-for-Action.svg @@ -1,3 +1,4 @@ + -
0690 - Request For Action
0690 - Request For Action
«entity»
RequestForAction
«entity»...
discoveryActivity : string
actionRequested : string
actionProperties : map<string, string>
discoveryActivity : string...
«entity»
DataFieldAnnotation
«entity»...Viewer does not support full SVG 1.1 \ No newline at end of file +
0690 - Request For Action
0690 - Request For Action
«entity»
RequestForAction
«entity»...
actionSourceName : string
actionRequested : string
actionProperties : map<string, string>
actionSourceName : string...
«entity»
DataFieldAnnotation
«entity»...Text is not SVG - cannot display \ No newline at end of file diff --git a/site/docs/types/6/area-6-discovery-overview.svg b/site/docs/types/6/area-6-discovery-overview.svg index aa716699c8..da786ae938 100644 --- a/site/docs/types/6/area-6-discovery-overview.svg +++ b/site/docs/types/6/area-6-discovery-overview.svg @@ -1,3 +1,4 @@ + -
Area 6 - Discovery
Area 6 - Discovery
0601
Open Discovery Engines and Services
0601...
0610
Annotations
0610...
0612
Annotation Reviews
0612...
0605
Open Discovery Analysis Reports
0605...
0617
Data Field Analysis
0617...
0620
Data Profiling
0620...
0625
Data Class Discovery
0625...
0630
Semantic Discovery
0630...
0635
Classification Discovery
0635...
0650
Relationship Discovery
0650...
0660
Data Source Measurements
0660...
0690
Request For Action
0690...
0640
Quality Scores
0640...
0615
Schema Extraction
0615...Viewer does not support full SVG 1.1 \ No newline at end of file +
Area 6 - Discovery Surveys and Analysis
Area 6 - Discovery Surveys and Analysis
0610
Annotations
0610...
0612
Annotation Reviews
0612...
0602
Survey Reports
0602...
0620
Data Profiling
0620...
0625
Data Class Discovery
0625...
0630
Semantic Discovery
0630...
0635
Classification Discovery
0635...
0650
Relationship Discovery
0650...
0660
Data Source Measurements
0660...
0690
Request For Action
0690...
0640
Quality Scores
0640...
0601
Discovery Engines and Services
0601...
0605
Discovery Analysis
Reports
0605...
0615
Schema Extraction
0615...
0616
Data Field Discovery
0616...
0617
Data Field Analysis
0617...Text is not SVG - cannot display \ No newline at end of file diff --git a/site/docs/types/6/area-6-discovery.drawio b/site/docs/types/6/area-6-discovery.drawio index b6a263663c..1f51908012 100644 --- a/site/docs/types/6/area-6-discovery.drawio +++ b/site/docs/types/6/area-6-discovery.drawio @@ -1,291 +1,402 @@ - + - + - + - - - - - - - + - + - + - + - - - - - - - + - - + + - + - + - + - + - + - + - + - + - + - - + + - + - + - + - + - + - + + + + + + + + + + + + + - - + + - - + + + + + + + + + + + + + + - - + + - - - + + + - + - + - + - + - + - + - + - + - + - + - + - + - + + + + - - + + - + - - - - - - - + - - + + + + + - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + + + + - + @@ -296,91 +407,169 @@ - + - + - + - + - - + + - + - - + + - - + + - + - - + + - + - - + + - - + + - + - + - + - + - - + + - + - + - + - + - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + @@ -430,10 +619,10 @@ - + - + @@ -442,7 +631,7 @@ - + @@ -453,143 +642,166 @@ - + - + - + - + + + + + + + + + + + + + + + + + + + + + + + + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - - + + - + - + - + - + - - + + - + - + - + - + - + @@ -599,44 +811,41 @@ - + - + - + - + - + - + - + - + - - - - - + + - + @@ -649,18 +858,6 @@ - - - - - - - - - - - - @@ -679,7 +876,7 @@ - + @@ -698,7 +895,7 @@ - + @@ -782,7 +979,7 @@ - + @@ -811,7 +1008,7 @@ - + @@ -840,7 +1037,7 @@ - + @@ -869,7 +1066,7 @@ - + @@ -898,7 +1095,7 @@ - + @@ -908,197 +1105,65 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + - + - + - + - - + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + - - + + - - - - - - - - - - - - - - - - - - - - - - + - - + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + - - - - - - - - - - - - - - + + - - + + - - + + - - + + - - + + - + @@ -1139,7 +1204,7 @@ - + @@ -1149,20 +1214,17 @@ + + + - + - - - - - - - + diff --git a/site/docs/types/6/index.md b/site/docs/types/6/index.md index e41138fda8..bcca3d7cce 100644 --- a/site/docs/types/6/index.md +++ b/site/docs/types/6/index.md @@ -6,22 +6,21 @@ hide: -# Area 6 Models - Metadata Discovery +# Area 6 Models - Discovery Surveys and Analysis -Area 6 provides structures for recording the results of automated metadata discovery as annotations to associated assets in the metadata repository. +Area 6 provides structures for recording the results of automated metadata surveys as annotations to associated assets in the metadata repository. -Metadata discovery requires different types of analysis. This analysis may run just once, say when the asset is created, on demand or based on an event or schedule. +Metadata surveys require different types of analysis. This analysis may run just once, say when the asset is created, on demand or based on an event or schedule. -A particular type of analysis is implemented in an [open discovery service](/concepts/open-discovery-service). Within the open discovery service are one to many discovery steps. Each step performs some sort of analysis that may result in an [annotation](/concepts/discovery-analysis-report/#discovery-annotations) for one or more assets. +A particular type of analysis is implemented in a [survey action service](/concepts/survey-action-service). Within the survey action service are one to many analysis steps. Each step performs some sort of analysis that may result in an [annotation](/concepts/survey-report/#annotations) for one or more assets. -The annotations from a particular run of an open discovery service are grouped together into a [discovery analysis report](/concepts/discovery-analysis-report). The annotations may be reviewed and approved by a steward. The steward may convert the annotation to a harden metadata type, or they may flag the annotation as invalid. When the discovery service is rerun, the new annotations can be matched to the annotations from the previous run. The steward's actions will impact how the new annotations are processed. +The annotations from a particular run of a survey action service are grouped together into a [survey report](/concepts/survey-report). The annotations may be reviewed and approved by a steward. The steward may convert the annotation to a harden metadata type, or they may flag the annotation as invalid. When the survey action service is rerun, the new annotations can be matched to the annotations from the previous run. The steward's actions will impact how the new annotations are processed. -* **[0601 Open Discovery Engines and Services](0601-Open-Discovery-Engine.md)** -* **[0605 Discovery Analysis Report](0605-Open-Discovery-Analysis-Reports.md)** +* **[0603 Survey Report](0603-Survey-Reports.md)** * **[0610 Annotation](0610-Annotations.md)** * **[0611 Annotation Reviews](0612-Annotation-Reviews.md)** * **[0615 Schema Extraction](0615-Schema-Extraction.md)** -* **[0617 Data Field Analysis](0617-Data-Field-Analysis.md)** + * **[0620 Data Profiling](0620-Data-Profiling.md)** * **[0625 Data Class Discovery](0625-Data-Class-Discovery.md)** * **[0630 Semantic Discovery](0630-Semantic-Discovery.md)** @@ -33,6 +32,17 @@ The annotations from a particular run of an open discovery service are grouped t ![UML Packages](area-6-discovery-overview.svg) -Egeria's [Open Discovery Framework (ODF)](/frameworks/odf/overview) supports the development and execution of open discovery services. The ODF runs in an [Engine Host](/concepts/engine-host) server. ODF discovery services use connectors from the [Open Connector Framework (OCF)](/frameworks/ocf/overview) to connect to the data assets and access the known metadata about them. +Egeria's [Survey Action Framework (SAF)](/frameworks/saf/overview) supports the development and execution of [survey action services](/concepts/survey-action-services). These services run in the [Survey Action OMES](/services/omes/survey-action/overview) hosted in an [Engine Host](/concepts/engine-host). Survey action services use connectors from the [Open Connector Framework (OCF)](/frameworks/ocf/overview) to connect to the data assets and access the known metadata about them. + +## A moment of transition + +You may notice that some of the types in the UML models are coloured grey. These types are exclusively for the [open discovery services](/concepts/open-discovery-service) which are now deprecated and replaced by the [survey action services](/concepts/survey-action-framework). The type descriptions remain in the models to help people migrate their open discovery services to the survey action framework. + +The main differences are that the [*OpenDiscoveryAnalysisReport*](0605-Open-Discovery-Analysis-Reports.md) becomes the [*SurveyReport*](0603-Survey-Reports.svg) and the schema extraction process now uses [*SchemaElement*](/type/5/0501-Schema-Elements) entities rather than [*DataField*](0616-Data-Field-Discovery.md) entities. + +* **[0601 Open Discovery Engines and Services](0601-Open-Discovery-Engine.md)** +* **[0605 Discovery Analysis Report](0605-Open-Discovery-Analysis-Reports.md)** +* **[0616 Data Field Discovery](0616-Data-Field-Discovery.md)** +* **[0617 Data Field Analysis](0617-Data-Field-Analysis.md)** --8<-- "snippets/abbr.md" diff --git a/site/mkdocs.yml b/site/mkdocs.yml index 402d016ced..3a83637f29 100644 --- a/site/mkdocs.yml +++ b/site/mkdocs.yml @@ -477,10 +477,12 @@ nav: - 6. Metadata Discovery: - Area 6 Metadata Discovery: types/6/index.md - Open Discovery Engines and Services: types/6/0601-Open-Discovery-Engine.md + - Survey Reports: types/6/0603-Survey-Reports.md - Discovery Analysis Report: types/6/0605-Open-Discovery-Analysis-Reports.md - Annotations: types/6/0610-Annotations.md - Annotation Reviews: types/6/0612-Annotation-Reviews.md - Schema Extraction: types/6/0615-Schema-Extraction.md + - Data Field Discovery: types/6/0616-Data-Field-Discovery.md - Data Field Analysis: types/6/0617-Data-Field-Analysis.md - Data Profiling: types/6/0620-Data-Profiling.md - Data Class Discovery: types/6/0625-Data-Class-Discovery.md @@ -559,6 +561,8 @@ nav: - Open Metadata Engine Services: services/omes/index.md - Asset Analysis: services/omes/asset-analysis/overview.md - Governance Action: services/omes/governance-action/overview.md + - Event Action: services/omes/event-action/overview.md + - Survey Action: services/omes/survey-action/overview.md - Repository Governance: services/omes/repository-governance/overview.md - Integration Services (OMIS): - Open Metadata Integration Services: services/omis/index.md @@ -595,9 +599,9 @@ nav: - Audit Logs (ALF): frameworks/alf/overview.md - Open Connectors (OCF): frameworks/ocf/overview.md - Open Integration (OIF): frameworks/oif/overview.md - - Open Discovery (ODF): frameworks/odf/overview.md - Governance Actions (GAF): frameworks/gaf/overview.md - Event Actions (EAF): frameworks/eaf/overview.md + - Survey Actions (EAF): frameworks/saf/overview.md - Conformance Test Suite: - CTS Overview: guides/cts/overview.md - Platform Profiles: @@ -792,6 +796,9 @@ nav: - Software Server Capability: concepts/software-server-capability.md - Star Rating: concepts/review.md - Subject Area: concepts/subject-area.md + - Survey Report: concepts/survey-report.md + - Survey Action Engine: concepts/survey-action-engine.md + - Survey Action Service: concepts/survey-action-service.md - Tag: concepts/informal-tag.md - To Do: concepts/to-do.md - Uni-link Relationships: concepts/uni-multi-link.md diff --git a/site/snippets/survey-action-services/survey-action-service-config-summary.md b/site/snippets/survey-action-services/survey-action-service-config-summary.md new file mode 100644 index 0000000000..a6ea6d3f8e --- /dev/null +++ b/site/snippets/survey-action-services/survey-action-service-config-summary.md @@ -0,0 +1,4 @@ + + + +This [open discovery service](/concepts/open-discovery-engine) is described in an [open discovery engine](/concepts/open-discovery-engine). The engine is configured to run in an [Engine Host](/concepts/engine-host) running the [Asset Analysis OMES](/services/omes/asset-analysis/overview) service. The Asset Analysis OMES is configured with the network address of a [Metadata Access Server](/concepts/metadata-access-server) running the [Discovery Engine OMAS](/services/omas/discovery-engine/overview) service. diff --git a/site/snippets/survey-action-services/survey-action-service-operation-summary.md b/site/snippets/survey-action-services/survey-action-service-operation-summary.md new file mode 100644 index 0000000000..82f9168c90 --- /dev/null +++ b/site/snippets/survey-action-services/survey-action-service-operation-summary.md @@ -0,0 +1,12 @@ + + + + +Once installed in the engine host, the survey action service can be called either by: + +* via an [engine action](/concepts/engine-action), or +* via a [governance action process](/concepts/governance-action-process). + +Each time the survey action service starts, the Survey Action OMES creates a new [Survey Report](/concepts/survey-report) via a call to the Stewardship Action OMAS. As the survey action service runs, it is retrieving metadata, and storing annotations, via its [survey context](/concepts/survey-context). The Survey Action OMES routes these requests to the Stewardship Action OMAS which has access to the open metadata repositories. + +