Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix typos and links in the SAF overview #884

Merged
merged 2 commits into from
Jan 10, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion site/docs/concepts/survey-action-service.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ hide:

## Survey Action Pipeline

---8<-- "docs/guides/developer/open-discovery-services/discovery-pipeline-intro.md"
---8<-- "docs/frameworks/saf/survey-action-pipeline-intro.md"

!!! education "Further information"

Expand Down
Original file line number Diff line number Diff line change
@@ -1,19 +1,19 @@
<!-- SPDX-License-Identifier: CC-BY-4.0 -->
<!-- Copyright Contributors to the ODPi Egeria project. -->

An *survey-action service* is a component that performs specific analysis of the contents of a [digital resource](/concepts/resource) on request. The aim of the survey-action service is to enable a detailed picture of the properties of a resource to be built up.
An *survey-action service* is a component that performs analysis of the contents of a [digital resource](/concepts/resource) on request. The aim of the survey action service is to enable a detailed picture of the properties of a resource to be built up.

Each time an survey-action service runs, it creates a new [survey report](/concepts/survey-report) linked off of the digital resource's [Asset](/concepts/asset) metadata element that records the results of the analysis.
Each time a survey action service runs, it creates a new [survey report](/concepts/survey-report) linked off of the digital resource's [Asset](/concepts/asset) metadata element that records the results of the analysis.

![Asset with survey reports](/guides/developer/survey-action-services/asset-to-survey-reports.svg)
> Each time an survey-action service runs to analyse a digital resource, a new survey report is created and attached to the resource's asset. If the survey-action service is run regularly, it is possible to track how the contents are changing over time.
![Asset with survey reports](/frameworks/saf/asset-to-survey-reports.svg)
> Each time an survey action service runs to analyse a digital resource, a new survey report is created and attached to the resource's asset. If the survey action service is run regularly, it is possible to track how the contents are changing over time.

The *survey report* contains one or more sets of related properties that the survey action service has discovered about the resource, its metadata, structure and/or content. These are stored in a set of [*annotations*](/concepts/survey-report/#annotations) linked off of the survey report.

An survey action service is designed to run at regular intervals to gather a detailed perspective on the contents of the digital resource and how they are changing over time. Each time it runs, it is given access to the results of previously run survey-action services, along with a review of these findings made by individuals responsible for the digital resource (such as stewards, owners, custodians).

??? info "Operation of an survey action service"
--8<-- "docs/guides/developer/survey-action-services/operation-of-a-survey-action-service.md"
--8<-- "docs/frameworks/saf/operation-of-a-survey-action-service.md"

??? info "Runtime for an survey action service"
Survey action services are packaged into [Survey Action Engines](/concepts/survey-action-engine) that run in the [Survey Action OMES](/services/omes/survey-action/overview) hosted in an [Engine Host](/concepts/engine-host).
Expand All @@ -24,3 +24,6 @@ An survey action service is designed to run at regular intervals to gather a det

![Survey Action Service](/connectors/survey-action/survey-action-service.svg)




60 changes: 25 additions & 35 deletions site/docs/frameworks/saf/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,57 +5,47 @@

# Survey Action Framework (SAF)

The *Survey Action Framework (SAF)* enables [metadata survey tools](/features/discovery-and-stewardship) to integrate with open metadata repositories by defining the interfaces for surveyor components (called [survey action services](#survey-action-service)) to:
The *Survey Action Framework (SAF)* enables [metadata survey tools](/features/discovery-and-stewardship) to integrate with open metadata repositories by defining the interfaces for surveyor components (called [survey action services](/concepts/survey-action-service)) to:

- Access survey request configuration.
- Search for [assets](/concepts/asset) and related metadata in the metadata repository.
- Extract all the metadata known about a specific asset.
- Record the results of the analysis in the open metadata repository and attach it to the asset's metadata for later processing.

The results of the analysis are stored in a [*survey report*](#survey-report).

## Survey action service

A *survey action service* provides specific analysis of the real-world resources associated with an [asset](/concepts/asset) on request. It may provide the complete implementation of the function, or invoke the services of an external tool.

The results of the analysis are stored in an [*survey report*](#survey-report).

The result is one or more sets of related properties that the survey action service has discovered about the asset, its metadata, structure and/or content. These are stored in a set of [annotations](#annotations) linked off of a [survey report](#survey-report). The survey report is linked off of the asset definition in the open metadata repository.


### Survey action pipeline

A *survey action pipeline* is a specialized implementation of a [survey action service](#survey-action-service) that runs a set of survey action services against a single asset. The implementation of the survey action pipeline determines the order that these survey action services are run.

The aim of the survey action pipeline is to enable a detailed picture of the properties of an asset to be built up by the nested survey action services it calls. Each survey action service is able to access the results of the survey action services that have run before it.

## Survey report

The survey report is created automatically in the open metadata repository when the survey action service is started. It is linked to the asset.
The survey report is created automatically in the open metadata repository when the survey action service is started. It is linked to the asset that describes the digital resource.

![Survey reports linked from the asset](asset-to-survey-reports.svg)

![]()
The results of the analysis are published by the survey acton service to the open metadata repository as [*annotations*](#annotations).
The *survey report* lists the [annotations](#annotations) that were created during the execution of a [survey action service](/concepts/survey-action-service). Each annotation may also link to a metadata entity that it describes. For example, if the annotation describes the data in a particular column of a CSV file, it may also be linked to the [Schema Attribute](/types/5/0505-Schema-Attributes) entity that describes that column.

The *survey report* lists the [annotations](#annotations) that were created during the execution of a [survey action service](#survey-action-service).
![Survey reports with their annotations](survey-report-structure.svg)

Annotations are published to report as soon as they are created by the survey action service so it is possible to process the annotations from a long-running survey before it has completed.
Annotations are published to report as soon as they are created by the survey action service, so it is possible to process the annotations from a long-running survey before it has completed.

### Annotations

An *annotation* describes one or more related properties about an [asset](/concepts/asset) that has been surveyed by a [survey action service](#survey-action-service).
An *annotation* describes one or more related properties about an [asset](/concepts/asset) that has been surveyed by a [survey action service](/concepts/survey-action-service).

Different types of annotations provide different types of information.

| Annotation type | Description |
|---|---|
| Classification annotation | Captures a recommendation of which classifications to attach to this asset. It can be made at the asset or data field level. |
| Data class annotation | Captures a recommendation of which data class this data field closely represents. |
| Data profile annotation | Capture the characteristics of the data values stored in a specific data field in a data source. |
| Data profile log annotation | Capture the names of the log files where profile characteristics of the data values stored in a specific data field. This is used when the profile results are too large to store in open metadata. |
| Data source measurement annotation | Collect arbitrary properties about a data source. |
| Data source physical status annotation | Documents the physical characteristics of a data source asset. |
| Relationship advice annotation | Document a recommended relationship that should be established with the asset. |
| Quality annotation | Document calculated quality scores on different dimensions. |
| Semantic annotation | Documents suggested meanings for this data based on the values and name of the field. |
| Suspect duplicate annotation | Identifies other asset definitions that seem to point to the same physical asset. |
| Annotation type | Description |
|----------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Classification annotation | Captures a recommendation of which classifications to attach to this asset. It can be made at the asset or data field level. |
| Data class annotation | Captures a recommendation of which data class this data field closely represents. |
| Data profile annotation | Capture the characteristics of the data values stored in a specific data field in a data source. |
| Data profile log annotation | Capture the names of the log files where profile characteristics of the data values stored in a specific data field. This is used when the profile results are too large to store in open metadata. |
| Data source measurement annotation | Collect arbitrary properties about a data source. |
| Data source physical status annotation | Documents the physical characteristics of a data source asset. |
| Relationship advice annotation | Document a recommended relationship that should be established with the asset. |
| Quality annotation | Document calculated quality scores on different dimensions. |
| Semantic annotation | Documents suggested meanings for this data based on the values and name of the field. |
| Suspect duplicate annotation | Identifies other asset definitions that seem to point to the same physical asset. |

#### Schema extraction

Expand Down Expand Up @@ -128,7 +118,7 @@ The measurement annotations capture a snapshot of the physical dimensions and ac

#### Requesting stewardship action

A RequestForAction entity (RfA) is used when an survey action service performs a test on the data (such as a [quality rule](/features/data-quality/overview)) or has discovered an anomaly in the data landscape compared to its metadata that potentially needs a steward or a curator's action.
A RequestForAction entity (RfA) is used when a survey action service performs a test on the data (such as a [quality rule](/features/data-quality/overview)) or has discovered an anomaly in the data landscape compared to its metadata that potentially needs a steward or a curator's action.

![Request for action](survey-action-request-for-action.svg)

Expand All @@ -148,7 +138,7 @@ The survey context provides parameters used by a survey action service to locate

The annotation store provides a survey action service with methods to write [annotations](#annotation) to an open metadata repository. These annotations describe the results of the analysis performed on an asset by the survey action service.

The annotations are automatically linked to the survey survey report that is in turn linked off of the analysed asset. Each annotation can also be linked to other metadata elements that is describes.
The annotations are automatically linked to the survey report that is in turn linked off of the analysed asset. Each annotation can also be linked to other metadata elements that is describes.

## Survey asset store

Expand All @@ -162,6 +152,6 @@ The *open metadata store* provides a search interface that enables a survey acti

Survey action services run under the [Survey Action OMES](/services/omes/survey-action/overview) hosted in an [Engine Host](/concepts/engine-host).

The open metadata types for a annotations are described in [area 6](/types/6) of the model. The main entity type is called [*Annotation*](/types/6/0610-Annotations). It is extended by *DataFieldAnnotation* to distinguish annotations that refer, primarily to a data field rather than the whole asset. Other more specialist annotations extend these two basic annotation types.
The open metadata types for annotations are described in [area 6](/types/6) of the model. The main entity type is called [*Annotation*](/types/6/0610-Annotations). It is extended by *DataFieldAnnotation* to distinguish annotations that refer, primarily to a data field rather than the whole asset. Other more specialist annotations extend these two basic annotation types.

--8<-- "snippets/abbr.md"
6 changes: 3 additions & 3 deletions site/docs/frameworks/saf/survey-action.drawio
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
<mxfile modified="2024-01-10T12:40:14.531Z" host="Electron" agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) draw.io/22.1.16 Chrome/120.0.6099.109 Electron/28.1.0 Safari/537.36" etag="RDdpeoc2ky7CB2RsTeG9" version="22.1.16" type="device" pages="15">
<mxfile modified="2024-01-10T13:28:42.327Z" host="Electron" agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) draw.io/22.1.16 Chrome/120.0.6099.109 Electron/28.1.0 Safari/537.36" etag="8rn7VCugJmf5JSsX02XN" version="22.1.16" type="device" pages="15">
<diagram id="nSDM-Rxb-zmBo6Db-v7M" name="survey-action-service-operation">
<mxGraphModel dx="1941" dy="812" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="827" pageHeight="1169" math="0" shadow="0">
<root>
Expand Down Expand Up @@ -872,7 +872,7 @@
</mxGraphModel>
</diagram>
<diagram id="80IOCSKDe2TH0Fwrc41k" name="file-system-discovery-process">
<mxGraphModel dx="1888" dy="773" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="827" pageHeight="1169" math="0" shadow="0">
<mxGraphModel dx="2142" dy="1048" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="827" pageHeight="1169" math="0" shadow="0">
<root>
<mxCell id="0" />
<mxCell id="1" parent="0" />
Expand Down Expand Up @@ -1671,7 +1671,7 @@
<mxGeometry x="-132" y="3185" width="170" height="40" as="geometry" />
</mxCell>
<mxCell id="qS_1YLQiRLYEo0A8dIZq-191" value="" style="rounded=0;whiteSpace=wrap;html=1;strokeColor=none;" vertex="1" parent="1">
<mxGeometry x="300" y="185" width="940" height="505" as="geometry" />
<mxGeometry x="-690" y="1740" width="940" height="505" as="geometry" />
</mxCell>
</root>
</mxGraphModel>
Expand Down
2 changes: 1 addition & 1 deletion site/mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -602,7 +602,7 @@ nav:
- Open Integration (OIF): frameworks/oif/overview.md
- Governance Actions (GAF): frameworks/gaf/overview.md
- Event Actions (EAF): frameworks/eaf/overview.md
- Survey Actions (EAF): frameworks/saf/overview.md
- Survey Actions (SAF): frameworks/saf/overview.md
- Conformance Test Suite:
- CTS Overview: guides/cts/overview.md
- Platform Profiles:
Expand Down