From 89e718d6d1b5b51cd0a3cd931cd9ef86aa140d30 Mon Sep 17 00:00:00 2001 From: Mandy Chessell Date: Wed, 22 Nov 2023 12:59:49 +0000 Subject: [PATCH] Update Atlas Discovery Service Signed-off-by: Mandy Chessell --- site/docs/connectors/connector-catalog.drawio | 205 ++++++++++++++---- ...pache-atlas-discovery-service-analysis.svg | 2 +- ...apache-atlas-discovery-service-profile.svg | 2 +- .../apache-atlas-discovery-service.md | 15 +- 4 files changed, 177 insertions(+), 47 deletions(-) diff --git a/site/docs/connectors/connector-catalog.drawio b/site/docs/connectors/connector-catalog.drawio index ad55db96ba..8f92faef4e 100644 --- a/site/docs/connectors/connector-catalog.drawio +++ b/site/docs/connectors/connector-catalog.drawio @@ -1,4 +1,4 @@ - + @@ -3648,6 +3648,103 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + @@ -4028,137 +4125,161 @@ - + - + - + - + - + - + - + - + - + - + - + - + - - + + - + - + - - + + - + - + - + - + - + - + - + - + - + - + - - + + - + - + - - + + - + - + - + - + - + - + - + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/site/docs/connectors/discovery/apache-atlas-discovery-service-analysis.svg b/site/docs/connectors/discovery/apache-atlas-discovery-service-analysis.svg index 2d8ba2ab48..dd40a32dd5 100644 --- a/site/docs/connectors/discovery/apache-atlas-discovery-service-analysis.svg +++ b/site/docs/connectors/discovery/apache-atlas-discovery-service-analysis.svg @@ -1,4 +1,4 @@ -
Data Profile
Annotation
Data Profile...
Data Profile
Annotation
Data Profile...
Data Profile
Annotation
Data Profile...
Data Profile
Annotation
Data Profile...
Data Profile
Annotation
Data Profile...
Data Profile
Annotation
Data Profile...
Data Profile
Annotation
Data Profile...
Data Profile
Annotation
Data Profile...
Data Profile
Annotation
Data Profile...
Data Profile
Annotation
Data Profile...
Discovery Analysis
Report
Discovery Analysis...
DataSource
Measurements Annotation
DataSource...
Schema Analysis Annotation
Schema Analysis Anno...
Entity Data Field
Entity Data Field
Relationship
Data Field
Relationship...
Entity Data Field
Entity Data Field
Classification
Data Field
Classification...
Entity Data Field
Entity Data Field
Classification
Data Field
Classification...
Relationship
Data Field
Relationship...
Entity Data Field
Entity Data Field
Relationship
Data Field
Relationship...
Entity Data Field
Entity Data Field
Classification
Data Field
Classification...
Classification
Data Field
Classification...
Relationship
Data Field
Relationship...
Entity Data Field
Entity Data Field
Data Profile
Annotation
Data Profile...
Data Profile
Annotation
Data Profile...
Data Profile
Annotation
Data Profile...
Data Profile
Annotation
Data Profile...
STATS
STATS
SCHEMA
SCHEMA
PROFILE
PROFILE
1
1
1
1
2
2
2
2
3
3
3
3
Analysis Steps
Analysis Steps
Text is not SVG - cannot display
\ No newline at end of file +
Data Profile
Annotation
Data Profile...
Relationship
Data Field
Relationship...
Label Data Field
Label Data Field
Label Data Field
Label Data Field
Data Profile
Annotation
Data Profile...
Data Profile
Annotation
Data Profile...
Data Profile
Annotation
Data Profile...
Data Profile
Annotation
Data Profile...
Business Metadata
Data Field
Business Metadata...
Relationship
Data Field
Relationship...
Business Metadata
Data Field
Business Metadata...
Data Profile
Annotation
Data Profile...
Data Profile
Annotation
Data Profile...
Data Profile
Annotation
Data Profile...
Data Profile
Annotation
Data Profile...
Data Profile
Annotation
Data Profile...
Data Profile
Annotation
Data Profile...
Data Profile
Annotation
Data Profile...
Data Profile
Annotation
Data Profile...
Data Profile
Annotation
Data Profile...
Data Profile
Annotation
Data Profile...
Data Profile
Annotation
Data Profile...
Discovery Analysis
Report
Discovery Analysis...
DataSource
Measurements Annotation
DataSource...
Schema Analysis Annotation
Schema Analysis Anno...
Entity Data Field
Entity Data Field
Relationship
Data Field
Relationship...
Entity Data Field
Entity Data Field
Classification
Data Field
Classification...
Entity Data Field
Entity Data Field
Classification
Data Field
Classification...
Relationship
Data Field
Relationship...
Entity Data Field
Entity Data Field
Relationship
Data Field
Relationship...
Entity Data Field
Entity Data Field
Classification
Data Field
Classification...
Classification
Data Field
Classification...
Relationship
Data Field
Relationship...
Entity Data Field
Entity Data Field
Data Profile
Annotation
Data Profile...
Data Profile
Annotation
Data Profile...
Data Profile
Annotation
Data Profile...
Data Profile
Annotation
Data Profile...
STATS
STATS
SCHEMA
SCHEMA
PROFILE
PROFILE
1
1
1
1
2
2
2
2
3
3
3
3
Analysis Steps
Analysis Steps
Apache Atlas 2.3+
Apache Atlas 2.3+
Text is not SVG - cannot display
\ No newline at end of file diff --git a/site/docs/connectors/discovery/apache-atlas-discovery-service-profile.svg b/site/docs/connectors/discovery/apache-atlas-discovery-service-profile.svg index aa426b4a8b..b6e299bb60 100644 --- a/site/docs/connectors/discovery/apache-atlas-discovery-service-profile.svg +++ b/site/docs/connectors/discovery/apache-atlas-discovery-service-profile.svg @@ -1,4 +1,4 @@ -
Attached
Entity Types
Attached...
Classification
Data Field
Classification...
Relationship
Data Field
Relationship...
Entity Data Field
Entity Data Field
Attached ClassificationTypes
Attached Classificat...
End 1 Attached
Entity Types
End 1 Attached...
Data Profile
Annotation
Data Profile...
Attached
Relationship Types
Attached...
End 2 Attached
Entity Types
End 2 Attached...
Data Profile
Annotation
Data Profile...
Data Profile
Annotation
Data Profile...
Data Profile
Annotation
Data Profile...
Data Profile
Annotation
Data Profile...
Text is not SVG - cannot display
\ No newline at end of file +
Attached
Entity Types
Attached...
Classification
Data Field
Classification...
Relationship
Data Field
Relationship...
Entity Data Field
Entity Data Field
Attached ClassificationTypes
Attached Classificat...
End 1 Attached
Entity Types
End 1 Attached...
Data Profile
Annotation
Data Profile...
Attached
Relationship Types
Attached...
End 2 Attached
Entity Types
End 2 Attached...
Data Profile
Annotation
Data Profile...
Data Profile
Annotation
Data Profile...
Data Profile
Annotation
Data Profile...
Data Profile
Annotation
Data Profile...
Attached
Entity Types
Attached...
Business Metadata
Data Field
Business Metadata...
Data Profile
Annotation
Data Profile...
Attached Labels
Attached Labels
Attached Business Metadata Types
Attached Business Me...
Data Profile
Annotation
Data Profile...
Data Profile
Annotation
Data Profile...
Attached
Entity Types
Attached...
Label
Data Field
Label...
Data Profile
Annotation
Data Profile...
Text is not SVG - cannot display
\ No newline at end of file diff --git a/site/docs/connectors/discovery/apache-atlas-discovery-service.md b/site/docs/connectors/discovery/apache-atlas-discovery-service.md index 4cfc3d3d80..eca0121af5 100644 --- a/site/docs/connectors/discovery/apache-atlas-discovery-service.md +++ b/site/docs/connectors/discovery/apache-atlas-discovery-service.md @@ -69,20 +69,29 @@ Figure 4 shows the structure of the discovery analysis report. The annotations ### Data Source Measurements Annotation - +The data source measurements annotation is created in the STATS analysis step. It has the following ### Schema Analysis Annotation -The schema analysis annotation is the parent entity for the data fields. It has the following properties: +The schema analysis annotation is created in the SCHEMA analysis step. It is the parent entity for the data fields. It sets up the following properties un the *dataSourceProperties* map: +* entityCount +* classificationCount +* typeUnusedCount +* typeCount +* activeEntityInstanceCount:*typeName* +* activeEntityWithSubtypesInstanceCount:*typeName* ### Data Field +In the SCHEMA analysis step, a *DataField* is created for each Apache Atlas entity type, relationship type and classification type. If the version of Apache Atlas is 2.3 or later, data fields are created for the Business Metadata Types. -### Data Profile Annotation +In the PROFILE analysis step, additional *DataFields* may be created for any labels discovered on the entity instances in Apache Atlas. +### Data Profile Annotation +The date profile annotations count the instances of each type. ---8<-- "snippets/abbr.md"