From ed113ac7b883324ae381fada0c1b25f57ea9993a Mon Sep 17 00:00:00 2001 From: D050513 Date: Thu, 29 Jun 2023 15:03:41 +0200 Subject: [PATCH 01/23] re @cap-js/audit-logging --- guides/data-privacy/audit-log.md | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/guides/data-privacy/audit-log.md b/guides/data-privacy/audit-log.md index 87a42aa22..3c7680ab5 100644 --- a/guides/data-privacy/audit-log.md +++ b/guides/data-privacy/audit-log.md @@ -21,7 +21,7 @@ This section deals with Audit Logging for reading sensitive data and changes to -In CAP, audit logging can be handled mostly automatically by adding certain annotations to your business entity definitions and adding some configuration to your project. +In CAP, audit logging can be handled mostly automatically by adding certain annotations to your business entity definitions and adding some configuration/ plugin to your project. ::: warning _❗ Data Subject and Data Object_
For each audit log on a data object (like a Sales Order) a valid data subject (like a Customer) is needed. @@ -89,6 +89,9 @@ The `@PersonalData.IsPotentiallySensitive` annotation induces an audit log for e --- Avoid unnecessary logging activities in your application. ::: +::: warning _Warning_ +Annotation `@AuditLog.Operation` is not applicable for the Node.js runtime. +::: ### Data Subject Details @@ -137,6 +140,10 @@ annotate bookshop.CustomerPostalAddress with @AuditLog.Operation : { Very similarly to the section on 'Data Subject' this entity is as well annotated in four levels. More details on these annotations can be found in the chapter [Indicate Personal Data in Your Domain Model](introduction#indicate-privacy). +::: warning _Warning_ +Annotation `@AuditLog.Operation` is not applicable for the Node.js runtime. +::: + ### Transactional Data In the section on 'Data Subject' and 'Data Subject Details' we have seen, how to annotate the master data entities carrying the semantical information of the 'Data Subject'. @@ -177,6 +184,10 @@ annotate bookshop.Orders with @AuditLog.Operation : { Finally, we annotate all standard operations (`Read`, `Insert`, `Update`, `Delete`) as relevant for the audit log - which should be the default case for most of the relevant business entities. +::: warning _Warning_ +Annotation `@AuditLog.Operation` is not applicable for the Node.js runtime. +::: +
\ No newline at end of file From e108bc57a0f3fef25b4d1802c0fdac684809f4b3 Mon Sep 17 00:00:00 2001 From: "Dr. Wolfgang Koch" <72808872+wmkoch@users.noreply.github.com> Date: Tue, 18 Jul 2023 15:03:17 +0200 Subject: [PATCH 02/23] Update audit-log.md --- guides/data-privacy/audit-log.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/guides/data-privacy/audit-log.md b/guides/data-privacy/audit-log.md index 3c7680ab5..61a3ca558 100644 --- a/guides/data-privacy/audit-log.md +++ b/guides/data-privacy/audit-log.md @@ -44,7 +44,7 @@ using { cuid, Country } from '@sap/cds/common'; email : String; firstName : String; lastName : String; - creditCardNo : String; +// creditCardNo : String; dateOfBirth : Date; postalAddress : Association to CustomerPostalAddress on postalAddress.Customer = $self; } @@ -190,4 +190,4 @@ Annotation `@AuditLog.Operation` is not applicable for the Node.js runtime.
- \ No newline at end of file + From 0c59db9336290cb9f94553faef2c8d825a38be4c Mon Sep 17 00:00:00 2001 From: "Dr. Wolfgang Koch" <72808872+wmkoch@users.noreply.github.com> Date: Tue, 18 Jul 2023 15:14:49 +0200 Subject: [PATCH 03/23] Update audit-log.md --- CustomerBillingData as separate entity --- guides/data-privacy/audit-log.md | 32 +++++++++++++++----------------- 1 file changed, 15 insertions(+), 17 deletions(-) diff --git a/guides/data-privacy/audit-log.md b/guides/data-privacy/audit-log.md index 61a3ca558..30d2721d1 100644 --- a/guides/data-privacy/audit-log.md +++ b/guides/data-privacy/audit-log.md @@ -44,7 +44,6 @@ using { cuid, Country } from '@sap/cds/common'; email : String; firstName : String; lastName : String; -// creditCardNo : String; dateOfBirth : Date; postalAddress : Association to CustomerPostalAddress on postalAddress.Customer = $self; } @@ -64,17 +63,9 @@ annotate bookshop.Customers with @PersonalData : { emailAddress @PersonalData.IsPotentiallyPersonal; firstName @PersonalData.IsPotentiallyPersonal; lastName @PersonalData.IsPotentiallyPersonal; - creditCardNo @PersonalData.IsPotentiallySensitive; dateOfBirth @PersonalData.IsPotentiallyPersonal; } -annotate bookshop.Customers with @AuditLog.Operation : { - Read : true, - Insert : true, - Update : true, - Delete : true -}; - ``` Here we again have the four levels of annotations as already described in the chapter [Indicate Personal Data in Your Domain Model](introduction#indicate-privacy). @@ -100,7 +91,7 @@ In the first example, the audited object was identical to the data subject, but In many cases you have additional master data describing more details of the data subject stored in a separate entity. In our terminology this has the semantics 'Data Subject Details'. -In our example we have the additional entity `CustomerPostalAddress` which contains additional master data belonging to a certain 'Customer', but which are stored in a separate entity, for better clarity or better separation of concerns. +In our example we have the additional entities `CustomerPostalAddress` and `CustomerBillingData` which contain additional master data belonging to a certain 'Customer', but which are stored in separate entities, for better clarity and better separation of concerns. ```cds @@ -112,9 +103,14 @@ entity CustomerPostalAddress : cuid, managed { someOtherField : String(128); }; +entity CustomerBillingData : cuid, managed { + Customer : Association to one Customers; + creditCardNo : String; +}; + ``` -This entity is annotated in the _db/data-privacy.cds_ file. +These entities are annotated in the _db/data-privacy.cds_ file. ```cds annotate bookshop.CustomerPostalAddress with @PersonalData : { @@ -128,12 +124,14 @@ annotate bookshop.CustomerPostalAddress with @PersonalData : { country @PersonalData.IsPotentiallyPersonal; } -annotate bookshop.CustomerPostalAddress with @AuditLog.Operation : { - Read : true, - Insert : true, - Update : true, - Delete : true -}; +annotate bookshop.CustomerBillingData with @PersonalData : { + DataSubjectRole : 'Customer', + EntitySemantics : 'DataSubjectDetails' +} +{ + Customer @PersonalData.FieldSemantics : 'DataSubjectID'; + creditCardNo @PersonalData.IsPotentiallySensitive; +} ``` From 3b9d919933942893f796b8eeadc7c5e7e5b8da1b Mon Sep 17 00:00:00 2001 From: "Dr. Wolfgang Koch" <72808872+wmkoch@users.noreply.github.com> Date: Tue, 18 Jul 2023 15:16:00 +0200 Subject: [PATCH 04/23] Update audit-log.md --- guides/data-privacy/audit-log.md | 4 ---- 1 file changed, 4 deletions(-) diff --git a/guides/data-privacy/audit-log.md b/guides/data-privacy/audit-log.md index 30d2721d1..e7a188dfc 100644 --- a/guides/data-privacy/audit-log.md +++ b/guides/data-privacy/audit-log.md @@ -80,10 +80,6 @@ The `@PersonalData.IsPotentiallySensitive` annotation induces an audit log for e --- Avoid unnecessary logging activities in your application. ::: -::: warning _Warning_ -Annotation `@AuditLog.Operation` is not applicable for the Node.js runtime. -::: - ### Data Subject Details In the first example, the audited object was identical to the data subject, but this is not always the case. From 59082ac356184929c2b0ba173e7a955921af8431 Mon Sep 17 00:00:00 2001 From: Rene Jeglinsky Date: Wed, 19 Jul 2023 15:49:34 +0200 Subject: [PATCH 05/23] moved text and graphic --- .../assets/Data-Privacy.drawio.svg | 127 ++++++++++++ guides/data-privacy/audit-log.md | 186 +++++++++++++++++- 2 files changed, 312 insertions(+), 1 deletion(-) create mode 100644 guides/data-privacy/assets/Data-Privacy.drawio.svg diff --git a/guides/data-privacy/assets/Data-Privacy.drawio.svg b/guides/data-privacy/assets/Data-Privacy.drawio.svg new file mode 100644 index 000000000..8ca0efacc --- /dev/null +++ b/guides/data-privacy/assets/Data-Privacy.drawio.svg @@ -0,0 +1,127 @@ + + + + + + + + + + + + + + +
+
+
+ + Annotate + +
+ Personal Data +
+
+
+ + + Annotate... + + + + + + + + + +
+
+
+ (Generic) +
+ Audit Logging +
+
+
+ + + (Generic)... + + + + + + + + + +
+
+
+ Personal Data Management +
+
+
+ + + Personal Data Manageme... + + + + + + + + + +
+
+
+ Data Retention Management +
+
+
+ + + Data Retention Managem... + + + + + + + + + + + + + + + + + +
+
+
+ in CDS +
+
+
+ + + in CDS + + + + + + + + + Text is not SVG - cannot display + + + + \ No newline at end of file diff --git a/guides/data-privacy/audit-log.md b/guides/data-privacy/audit-log.md index e7a188dfc..3212f522e 100644 --- a/guides/data-privacy/audit-log.md +++ b/guides/data-privacy/audit-log.md @@ -31,6 +31,8 @@ The application has to clarify that this link between data object and data subje Make sure that the data subject is a valid CAP entity, otherwise the metadata-driven automatism will not work. ::: +Data Privacy.drawio.svg + ## About the Audited Object ### Data Subject @@ -123,7 +125,7 @@ annotate bookshop.CustomerPostalAddress with @PersonalData : { annotate bookshop.CustomerBillingData with @PersonalData : { DataSubjectRole : 'Customer', EntitySemantics : 'DataSubjectDetails' -} +} { Customer @PersonalData.FieldSemantics : 'DataSubjectID'; creditCardNo @PersonalData.IsPotentiallySensitive; @@ -185,3 +187,185 @@ Annotation `@AuditLog.Operation` is not applicable for the Node.js runtime.
+ + +### Transactional Outbox + +By default all log messages are sent through a transactional outbox. This means, when sent, log messages are first stored in a local outbox table, which acts like a queue for outbound messages. Only when requests are fully and successfully processed, will these messages be forwarded to the audit log service. + +![Transactional Outbox.drawio](./assets/Transactional-Outbox.drawio.svg) + + + +This provides an ultimate level of resiliency, plus additional benefits: + +- **Audit log messages are guaranteed to be delivered** — even if the audit log service should be down for a longer time period. + +- **Asynchronous delivery of log messages** — the main thread doesn't wait for requests being sent and successfully processed by the audit log service. + +- **False log messages are avoided** — messages are forwarded to the audit log service on successfully committed requests; and skipped in case of rollbacks. + + + + +### Programmatic API + +In addition to the generic audit logging provided out of the box, applications can also log custom events with custom data using the programmatic API. + +Connecting to the service: + +```js +const audit = await cds.connect.to('audit-log') +``` + +Sending log messages: + +```js +await audit.log ('SomeEvent', { … }) +``` + +
+ + +::: tip +The Audit Log Service API is implemented as a CAP service, with the service API defined in CDS as shown below. In effect, the common patterns of [*CAP Service Consumption*](../../using-services) apply, as well as all the usual benefits like *mocking*, *late-cut µ services*, *resilience* and *extensibility*. +::: + +#### Basic Service API + +The basic service definition declares the generic `log` operation used for all kinds of events, along with type `LogMessage` declares the common fields of all log messages — these fields are filled in automatically if not provided by the caller. + +```cds +namespace sap.auditlog; + +service AuditLogService { + + action log (event: String, data: LogEntry); + action logSync (event: String, data: LogEntry); + +} + +/** Common fields, filled in automatically if missing */ +type LogEntry { + uuid : UUID; + tenant : String; + user : String; + timestamp : Timestamp; +} +``` + +::: warning _Warning_ +Only applicable for the Node.js runtime. Java still uses old model, I assume. +::: + +Usage is like that: + +```js +await audit.log ('SomeEvent', { + some_details: 'whatever' +}) +``` + +#### Personal Data-related Events + +In addition, pre-defined event payloads for personal data-related events are declared: + +```cds +namespace sap.auditlog; + +service AuditLogService { + // … as above + + event SensitiveDataRead : LogEntry { + // dataSubjects : many DataSubject; + dataSubject : DataSubject; + dataObject : DataObject; + // channel : String; + attributes : many { name: String }; + // attachments : many { name: String; id: String }; + } + + event PersonalDataChanged : LogEntry { + dataSubject : DataSubject; + dataObject : DataObject; + attributes : many { name: String; old: String; new: String; }; + } + +} + +type DataObject : { type: String; id: {} } +type DataSubject : DataObject { role: String } +``` + +Send corresponding log messages complying to these definitions like that: + +```js +await audit.log ('SensitiveDataRead', { + dataSubject: { + type: 'sap.capire.bookshop.Customers', + id: { ID: '1923bd11-b1d6-47b6-a91b-732e755fa976' }, + role: 'Customer', + }, + dataObject: { + type: 'sap.capire.bookshop.Customers', + id: { ID: '1923bd11-b1d6-47b6-a91b-732e755fa976' } + }, + attributes: [ + { name: 'creditCardNo' } + ] +}) +``` + + + +#### Config Change Events + +```cds +service AuditLogService { + // … as above + event ConfigChange : LogMessage { + object : DataObject; + attributes : ChangedAttributes; + } +} +``` + +::: warning _Warning_ +Not applicable for the Node.js runtime. +::: + + + +#### Security Events + +```cds +service AuditLogService { + // ... as above + event FailedLogin : LogMessage { + action : String; + data : String; + } +} +``` + +::: warning _Warning_ +Not applicable for the Node.js runtime. +::: + + + +### Service Providers + +In addition, everybody could provide new implementations in the same way as we implement the mock variant: + +```js +const cds = require('@sap/cds') + +class ConsoleAuditLogService extends cds.Service { + log (event, data) { + console.log (event, data) + } +} + +module.exports = ConsoleAuditLogService +``` From 7da081af5104ed8246e8dd4a4fe3ad06906ff6d6 Mon Sep 17 00:00:00 2001 From: Rene Jeglinsky Date: Wed, 19 Jul 2023 15:53:27 +0200 Subject: [PATCH 06/23] edit --- guides/data-privacy/audit-log.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/guides/data-privacy/audit-log.md b/guides/data-privacy/audit-log.md index 3212f522e..f9c0999ec 100644 --- a/guides/data-privacy/audit-log.md +++ b/guides/data-privacy/audit-log.md @@ -189,7 +189,7 @@ Annotation `@AuditLog.Operation` is not applicable for the Node.js runtime. -### Transactional Outbox +## Transactional Outbox By default all log messages are sent through a transactional outbox. This means, when sent, log messages are first stored in a local outbox table, which acts like a queue for outbound messages. Only when requests are fully and successfully processed, will these messages be forwarded to the audit log service. @@ -208,7 +208,7 @@ This provides an ultimate level of resiliency, plus additional benefits: -### Programmatic API +## Programmatic API In addition to the generic audit logging provided out of the box, applications can also log custom events with custom data using the programmatic API. @@ -231,7 +231,7 @@ await audit.log ('SomeEvent', { … }) The Audit Log Service API is implemented as a CAP service, with the service API defined in CDS as shown below. In effect, the common patterns of [*CAP Service Consumption*](../../using-services) apply, as well as all the usual benefits like *mocking*, *late-cut µ services*, *resilience* and *extensibility*. ::: -#### Basic Service API +### Basic Service API The basic service definition declares the generic `log` operation used for all kinds of events, along with type `LogMessage` declares the common fields of all log messages — these fields are filled in automatically if not provided by the caller. @@ -266,7 +266,7 @@ await audit.log ('SomeEvent', { }) ``` -#### Personal Data-related Events +### Personal Data-Related Events In addition, pre-defined event payloads for personal data-related events are declared: @@ -318,7 +318,7 @@ await audit.log ('SensitiveDataRead', { -#### Config Change Events +### Config Change Events ```cds service AuditLogService { @@ -336,7 +336,7 @@ Not applicable for the Node.js runtime. -#### Security Events +### Security Events ```cds service AuditLogService { @@ -354,7 +354,7 @@ Not applicable for the Node.js runtime. -### Service Providers +## Service Providers In addition, everybody could provide new implementations in the same way as we implement the mock variant: From c220261e44daacd623b615a1588236dc78c6c5ef Mon Sep 17 00:00:00 2001 From: D050513 Date: Thu, 20 Jul 2023 10:07:53 +0200 Subject: [PATCH 07/23] adding Setup on BTP --- guides/data-privacy/audit-log.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/guides/data-privacy/audit-log.md b/guides/data-privacy/audit-log.md index f9c0999ec..32ca1c894 100644 --- a/guides/data-privacy/audit-log.md +++ b/guides/data-privacy/audit-log.md @@ -369,3 +369,10 @@ class ConsoleAuditLogService extends cds.Service { module.exports = ConsoleAuditLogService ``` + +## Setup on BTP + +Your application needs to be bound to an instance of the SAP Audit Log Service. +In case you're providing a SaaS application, you additionally need to return the `xsappname` of the service instance's UAA instance (i.e., `cds.env.requires['audit-log'].credentials.uaa.xsappname`). +If you miss doing this, audit logs that are emitted via the persistent outbox (the default in production as it provides the ultimate resilience) will be lost in nirvana, as the sending effort will end in an unrecoverable error. +As with all dependencies: If you add them later on, you'll need to update all subscriptions. From 7f20890efb1b268d22284ee2d14139310efc5125 Mon Sep 17 00:00:00 2001 From: D050513 Date: Thu, 20 Jul 2023 10:19:36 +0200 Subject: [PATCH 08/23] more --- guides/data-privacy/audit-log.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/guides/data-privacy/audit-log.md b/guides/data-privacy/audit-log.md index 32ca1c894..2e945afd5 100644 --- a/guides/data-privacy/audit-log.md +++ b/guides/data-privacy/audit-log.md @@ -376,3 +376,19 @@ Your application needs to be bound to an instance of the SAP Audit Log Service. In case you're providing a SaaS application, you additionally need to return the `xsappname` of the service instance's UAA instance (i.e., `cds.env.requires['audit-log'].credentials.uaa.xsappname`). If you miss doing this, audit logs that are emitted via the persistent outbox (the default in production as it provides the ultimate resilience) will be lost in nirvana, as the sending effort will end in an unrecoverable error. As with all dependencies: If you add them later on, you'll need to update all subscriptions. + +TODO: should we add this to mtxs? +- something like `if (cds.env.requires?.['audit-log']?.credentials?.uaa.xsappname) dependencies.push({ xsappname: ... })` +- requires `@cap-js/audit-logging` to be added to mtxs sidecar +- NOTE: if we don't do this, then a single deploy of an mta would not suffice! the first deploy creates the service instance and the developer then would need to look up the `uaa.xsappname` and add to mtxs sidecar package.json as to-be-returned dependency + +TODO: add `@sap/audit-logging` manually if changing impl to `audit-log-to-library` (default is now `audit-log-to-restv2`)? + +## `cds add audit-logging` + +should we offer this? + +- add `@cap-js/audit-logging` + - also for mtxs sidecar, if dependency shall be returned automatically +- add audit log service with oauth2 plan in mta.yaml +- if dependency shall not be returned automatically, somehow tell mtxs to return it From 0df1b3a8cebbb33dacc0a2a7fd3c5993e97e64d5 Mon Sep 17 00:00:00 2001 From: D050513 Date: Fri, 21 Jul 2023 12:32:06 +0200 Subject: [PATCH 09/23] working session 7/21 --- .../assets/Transactional-Outbox.drawio.svg | 170 ++++++++++++++++++ .../{audit-log.md => audit-logging.md} | 19 +- guides/data-privacy/introduction.md | 71 +++++++- guides/data-privacy/pdm.md | 2 +- guides/data-privacy/sample-app.md | 16 ++ 5 files changed, 262 insertions(+), 16 deletions(-) create mode 100644 guides/data-privacy/assets/Transactional-Outbox.drawio.svg rename guides/data-privacy/{audit-log.md => audit-logging.md} (95%) create mode 100644 guides/data-privacy/sample-app.md diff --git a/guides/data-privacy/assets/Transactional-Outbox.drawio.svg b/guides/data-privacy/assets/Transactional-Outbox.drawio.svg new file mode 100644 index 000000000..a68eb9101 --- /dev/null +++ b/guides/data-privacy/assets/Transactional-Outbox.drawio.svg @@ -0,0 +1,170 @@ + + + + + + + + + + + + +
+
+
+ Request +
+
+
+ + + Request + + + + + + + + + + + + + + + + + + + +
+
+
+ Emitter +
+
+
+ + + Emitter + + + + + + + + + + + + + + + +
+
+
+ Outbox +
+
+
+ + + Outbox + + + + + + + + + +
+
+
+ Receiver +
+ /Broker +
+
+
+ + + Receiver... + + + + + + + + + + +
+
+
+ Database +
+
+
+ + + Database + + + + + + + +
+
+
+ Only When  Succeeded +
+
+
+ + + Only When  Su... + + + + + + + + + + + + + +
+
+
+ persisted in +
+
+
+ + + persisted... + + + + + + + + + Text is not SVG - cannot display + + + + \ No newline at end of file diff --git a/guides/data-privacy/audit-log.md b/guides/data-privacy/audit-logging.md similarity index 95% rename from guides/data-privacy/audit-log.md rename to guides/data-privacy/audit-logging.md index 2e945afd5..4b4ffbf6e 100644 --- a/guides/data-privacy/audit-log.md +++ b/guides/data-privacy/audit-logging.md @@ -1,8 +1,8 @@ --- layout: cookbook -shorty: Audit Log +shorty: Audit Logging synopsis: > - Enable and use audit-log capabilities with your CAP application. + Enable and use audit-logging capabilities with your CAP application. breadcrumbs: - Cookbook - Data Privacy @@ -11,7 +11,7 @@ status: released --- -# Audit Logging with CAP +# Audit Logging
@@ -43,11 +43,12 @@ In our case `Customers` is the main master data entity with the semantics of 'Da using { cuid, Country } from '@sap/cds/common'; entity Customers : cuid, managed { - email : String; - firstName : String; - lastName : String; - dateOfBirth : Date; - postalAddress : Association to CustomerPostalAddress on postalAddress.Customer = $self; + email : String; + firstName : String; + lastName : String; + dateOfBirth : Date; + postalAddresses : Composition of many CustomerPostalAddress on postalAddresses.Customer = $self; + billingData : Composition of many CustomerBillingData on billingData.Customer = $self; } ``` @@ -228,7 +229,7 @@ await audit.log ('SomeEvent', { … }) ::: tip -The Audit Log Service API is implemented as a CAP service, with the service API defined in CDS as shown below. In effect, the common patterns of [*CAP Service Consumption*](../../using-services) apply, as well as all the usual benefits like *mocking*, *late-cut µ services*, *resilience* and *extensibility*. +The Audit Log Service API is implemented as a CAP service, with the service API defined in CDS as shown below. In effect, the common patterns of [*CAP Service Consumption*](../using-services) apply, as well as all the usual benefits like *mocking*, *late-cut µ services*, *resilience* and *extensibility*. ::: ### Basic Service API diff --git a/guides/data-privacy/introduction.md b/guides/data-privacy/introduction.md index a42fedc31..a8ce93131 100644 --- a/guides/data-privacy/introduction.md +++ b/guides/data-privacy/introduction.md @@ -11,11 +11,12 @@ status: released --- -# Basics of Data Privacy with CAP +# Basics of Data Privacy
+ ## Introduction @@ -25,10 +26,62 @@ SAP provides specific features and functions to support compliance regarding the CAP supports applications in their obligations to comply to data privacy regulations, by automating tedious tasks as much as possible based on annotated models. Using annotations and configurations, CAP supports you using SAP BTP services, which enable you to fulfill specific data privacy requirements in your application. This means at first, personal data management, with the help of annotations and configurations and the SAP Personal Data Manager service. +ADDED FROM GUIDE #4: + +Compliance to data privacy regulations is an important requirement for all busines applications nowadays. CAP provides easy ways to designate personal data, as well as out-of-the-box integration with SAP BTP services, like SAP Personal Data Manager service. This greatly relieves application developers these tedious tasks and related efforts. + +Data Privacy.drawio.svg + +::: tip +DRM integration in progress +::: + + + + + +### Data Protection and Privacy Requirements + +EU regulation etc. -> [Personal Data](https://en.wikipedia.org/wiki/Personal_data) + +#### Right of access to personal data + +See [Right of access to personal data](https://en.wikipedia.org/wiki/Right_of_access_to_personal_data) -> SAP Personal Data Manager + +#### Right to be forgotten + +See [Right to be forgotten](https://en.wikipedia.org/wiki/Right_to_be_forgotten) -> SAP Data Rentention Manager + + + +### Addressed Requirements + +The most essential requests you have to answer are those in the table below, with the job to be done in response to that given on the right-hand side: + +| Question / Request | Discipline | +| ------------------------------------------- | -------------------------------------------------------- | +| *When was personal data stored/changed?* | → [Audit Logging](#audit-logging) | +| *What data about me do you have stored?* | → [Personal Data Management](#sap-personal-data-manager) | +| → "Right of access to personal data" | | +| *Please delete all personal data about me!* | → [Retention Management](#sap-data-retention-manager) | +| → "Right to be forgotten" | | + +
+ +::: warning +**PLEASE NOTE:** Full compliance is more than that!
+While CAP and SAP BTP services greatly facilitate fulfilling the obligations related to data privacy, there are usually numerous **additional regulations** you have comply to, such as from industry-specific legislation in different countries. +::: + + + + ## Indicate Personal Data in Your Domain Model { #indicate-privacy } Use `@PersonalData` annotations to indicate entities and elements in your domain model, which will contain personal data. @@ -49,27 +102,33 @@ using {sap.capire.bookshop} from './schema'; annotate bookshop.Customers with @PersonalData : { DataSubjectRole : 'Customer', EntitySemantics : 'DataSubject' -} -{ +} { ID @PersonalData.FieldSemantics : 'DataSubjectID'; emailAddress @PersonalData.IsPotentiallyPersonal; firstName @PersonalData.IsPotentiallyPersonal; lastName @PersonalData.IsPotentiallyPersonal; - creditCardNo @PersonalData.IsPotentiallySensitive; dateOfBirth @PersonalData.IsPotentiallyPersonal; } annotate bookshop.CustomerPostalAddress with @PersonalData : { DataSubjectRole : 'Customer', EntitySemantics : 'DataSubjectDetails' -} -{ +} { Customer @PersonalData.FieldSemantics : 'DataSubjectID'; street @PersonalData.IsPotentiallyPersonal; town @PersonalData.IsPotentiallyPersonal; country @PersonalData.IsPotentiallyPersonal; } + +annotate bookshop.CustomerBillingData with @PersonalData : { + DataSubjectRole : 'Customer', + EntitySemantics : 'DataSubjectDetails' +} { + Customer @PersonalData.FieldSemantics : 'DataSubjectID'; + creditCardNo @PersonalData.IsPotentiallySensitive; +} ``` + It is important to annotate the data privacy-relevant entities as `DataSubject`, `DataSubjectDetails`, or `Other`. diff --git a/guides/data-privacy/pdm.md b/guides/data-privacy/pdm.md index e4d9aebe4..a78546dfc 100644 --- a/guides/data-privacy/pdm.md +++ b/guides/data-privacy/pdm.md @@ -12,7 +12,7 @@ status: released --- -# Personal Data Management with CAP +# Personal Data Management
diff --git a/guides/data-privacy/sample-app.md b/guides/data-privacy/sample-app.md new file mode 100644 index 000000000..bb1efd025 --- /dev/null +++ b/guides/data-privacy/sample-app.md @@ -0,0 +1,16 @@ +--- +layout: cookbook +shorty: Sample App +synopsis: > + Learn how to build and deploy an app with PDM, DRM, and audit logging on SAP BTP CF. +breadcrumbs: + - Cookbook + - Data Privacy + - Sample App +#status: released +--- + +# Data Privacy Sample App for SAP BTP CF + +We should document how to build AND DEPLOY a SaaS app with PDM, audit logging, and DRM (when available). +There are many pitfalls such as returning all necessary xsappnames in a single deployment without manual copy & paste. From aa7cd64752edba4c97f596a0c3468de0964aa80f Mon Sep 17 00:00:00 2001 From: D050513 Date: Mon, 24 Jul 2023 22:51:09 +0200 Subject: [PATCH 10/23] more --- guides/data-privacy/drm.md | 38 +++++++ guides/data-privacy/introduction.md | 148 ++++++++++++++++++---------- guides/data-privacy/pdm.md | 73 ++++++++------ 3 files changed, 176 insertions(+), 83 deletions(-) create mode 100644 guides/data-privacy/drm.md diff --git a/guides/data-privacy/drm.md b/guides/data-privacy/drm.md new file mode 100644 index 000000000..3852f8153 --- /dev/null +++ b/guides/data-privacy/drm.md @@ -0,0 +1,38 @@ + diff --git a/guides/data-privacy/introduction.md b/guides/data-privacy/introduction.md index a8ce93131..026a32fc1 100644 --- a/guides/data-privacy/introduction.md +++ b/guides/data-privacy/introduction.md @@ -84,21 +84,66 @@ While CAP and SAP BTP services greatly facilitate fulfilling the obligations rel ## Indicate Personal Data in Your Domain Model { #indicate-privacy } +See [full sample](https://github.com/SAP-samples/cloud-cap-samples/tree/gdpr/gdpr). + +### Base Model + +In the remainder of this guide, we'll use this domain model as the base to add data privacy and audit logging. + +db/schema.cds +{.sample-label} + +```cds +using { Country, managed, cuid } from '@sap/cds/common'; +namespace sap.capire.bookshop; + +entity Customers : cuid, managed { + emailAddress : String; + firstName : String; + lastName : String; + dateOfBirth : Date; + addresses : Composition of Addresses on addresses.customer = $self; + billingData : Composition of BillingData on billingData.customer = $self; +} + +entity Addresses : cuid, managed { + customer : Association to Customers; + street : String(128); + town : String(128); + country : Country; + someOtherField : String(128); +} + +entity BillingData : cuid, managed { + customer : Association to Customers; + creditCardNo : String(16); +} + +entity Orders : cuid, managed { + orderNo : String(111); // human-readable key + customer : Association to Customers; + personalNote : String; + dateOfOrder : Date; + Items : Composition of many { … } +} +``` + +### Annotating Personal Data + +Let's annotate our data model to identify personal data. In essence, in all our entities we search for elements which carry personal data, such as person names, birth dates, etc., and tag them accordingly. All found entities are classified as either *Data Subjects*, *Data Subject Details* or *Other*. + Use `@PersonalData` annotations to indicate entities and elements in your domain model, which will contain personal data. -::: tip -The best practice is to do that in separate files.
-See also: [Using Aspects for Separation of Concerns](../../guides/domain-modeling#separation-of-concerns). -::: -Let's have a look at our [sample](https://github.com/SAP-samples/cloud-cap-samples/tree/gdpr/gdpr). +Data Subjects.drawio -Open the _db/data-privacy.cds_ file, which contains our data privacy-related annotations. +Following the [best practice of separation of concerns](../../domain-modeling#separation-of-concerns), we do that in a separate file `db/data-privacy.cds`: + +db/data-privacy.cds +{.sample-label} ```cds -// Proxy for importing schema from bookshop sample using {sap.capire.bookshop} from './schema'; -// annotations for Data Privacy annotate bookshop.Customers with @PersonalData : { DataSubjectRole : 'Customer', EntitySemantics : 'DataSubject' @@ -110,36 +155,49 @@ annotate bookshop.Customers with @PersonalData : { dateOfBirth @PersonalData.IsPotentiallyPersonal; } -annotate bookshop.CustomerPostalAddress with @PersonalData : { +annotate bookshop.Addresses with @PersonalData : { DataSubjectRole : 'Customer', EntitySemantics : 'DataSubjectDetails' } { - Customer @PersonalData.FieldSemantics : 'DataSubjectID'; + customer @PersonalData.FieldSemantics : 'DataSubjectID'; street @PersonalData.IsPotentiallyPersonal; town @PersonalData.IsPotentiallyPersonal; country @PersonalData.IsPotentiallyPersonal; } -annotate bookshop.CustomerBillingData with @PersonalData : { +annotate bookshop.BillingData with @PersonalData : { DataSubjectRole : 'Customer', EntitySemantics : 'DataSubjectDetails' } { - Customer @PersonalData.FieldSemantics : 'DataSubjectID'; + customer @PersonalData.FieldSemantics : 'DataSubjectID'; creditCardNo @PersonalData.IsPotentiallySensitive; } + +annotate bookshop.Orders with @PersonalData : { + DataSubjectRole : 'Customer', + EntitySemantics : 'Other' +} { + customer @PersonalData.FieldSemantics : 'DataSubjectID'; + personalNote @PersonalData.IsPotentiallyPersonal; +} ``` It is important to annotate the data privacy-relevant entities as `DataSubject`, `DataSubjectDetails`, or `Other`. - You can annotate different CDS artifacts, such as entities or fields. The data privacy annotations work on different levels - from the entity level to the field level, as described below. +- The **entity-level annotations** signify relevant entities as *Data Subject*, *Data Subject Details*, or *Other* in data privacy terms, as depicted in the graphic below. + +- The **key-level annotations** signify object primary keys, as well as references to data subjects (which have to be present on each object). + +- The **field-level annotations** identify elements containing personal data. ### Entity-Level Annotations -Entity-level annotations indicate which entities are relevant for data privacy. The most important annotations are: +#### EntitySemantics + +Entity-level annotations indicate which entities are relevant for data privacy. - ```cds @PersonalData.EntitySemantics: 'DataSubject' @PersonalData.EntitySemantics: 'DataSubjectDetails' @@ -160,6 +218,23 @@ The application has to clarify that this link between data object and data subje Make sure that the data subject is a valid CAP entity, otherwise the metadata-driven automatism will not work. ::: +#### DataSubjectRole + +```cds +@PersonalData.DataSubjectRole: '' +``` + +Can be added to `@PersonalData.EntitySemantics: 'DataSubject'`. User-chosen string designing the role name to use. Default is the entity name. + +Example: + +```cds +annotate Customers with @PersonalData: { + EntitySemantics: 'DataSubject', + DataSubjectRole: 'Customer' +}; +``` + ### Key-Level Annotations Key-level annotations indicate the corresponding key information. @@ -176,48 +251,13 @@ Field-level annotations tag which fields are relevant for data privacy in detail ```cds @PersonalData.IsPotentiallyPersonal +@PersonalData.IsPotentiallySensitive ``` This allows you to manage the data privacy-related actions on a fine granular level only using metadata definitions with annotations and without any need of implementation.
- +::: warning _Warning_ +Read access logs for sensitive data happen for each and every _Read_ access. Only use this annotation in [relevant cases](https://ec.europa.eu/info/law/law-topic/data-protection/reform/rules-business-and-organisations/legal-grounds-processing-data/sensitive-data/what-personal-data-considered-sensitive_en). Try to avoid reading sensitive data at all, for example, by obscuring credit card numbers as `**** **** **** 1234`. +::: diff --git a/guides/data-privacy/pdm.md b/guides/data-privacy/pdm.md index a78546dfc..f1e612640 100644 --- a/guides/data-privacy/pdm.md +++ b/guides/data-privacy/pdm.md @@ -20,12 +20,17 @@ status: released The SAP Personal Data Manager service is currently only available for [enterprise accounts](https://discovery-center.cloud.sap/missiondetail/3019/3297/). An entitlement in trial accounts is not possible. ::: +SAP BTP provides the [*SAP Personal Data Manager (PDM)*](https://help.sap.com/docs/PERSONAL_DATA_MANAGER) which allows administrators to respond to question "What data of me do you have?". To answer this question, the PDM service needs to fetch read all personal data via a respective OData endpoint, to be provided by the app as follows. + + + ## Provide a Service Interface to SAP Personal Data Manager SAP Personal Data Manager needs to call into your application to read personal data so you have to define a respective service endpoint, complying to the interface required by SAP Personal Data Manager. Following the CAP principles, we recommend adding a new dedicated CAP service that handles all the personal data manager requirements for you. This keeps the rest of your data model clean and enables reuse, just as CAP promotes it. + ### CAP Service Model for SAP Personal Data Manager Open the _srv/pdm-service.cds_ file, which contains the content for the Personal Data Manager service. @@ -38,44 +43,41 @@ using {sap.capire.bookshop.Orders} from '@capire/orders'; using {sap.capire.bookshop.OrderItems} from '@capire/orders'; @requires: 'PersonalDataManagerUser' // security check -service PDMService{ +service PDMService { - entity Customers as projection on db.Customers; - entity CustomerPostalAddress as projection on db.CustomerPostalAddress; + // Data Privacy annotations on 'Customers' and 'Addresses' are derived from original entity definitions. + entity Customers as projection on db.Customers; + entity Addresses as projection on db.Addresses; - // create view on Orders and Items as flat projection + // create view on Orders and Items as flat projection entity OrderItemView as select from Orders { ID, - key Items.ID as Item_ID, - OrderNo, - Customer.ID as Customer_ID, - Customer.email as Customer_Email, - Items.book.ID as Item_Book_ID, - Items.quantity as Item_Quantity, - Items.netQuantity as Item_NetQuantity + key items.ID as item_ID, + orderNo, + customer.ID as customer_ID, + customer.email as customer_email, + items.book.ID as item_book_ID, + items.quantity as item_quantity, + items.netQuantity as item_net_quantity }; - // annotate new view + // annotate new view annotate PDMService.OrderItemView with @(PersonalData.EntitySemantics : 'Other') { - Item_ID @PersonalData.FieldSemantics : 'ContractRelatedID'; - Customer_ID @PersonalData.FieldSemantics : 'DataSubjectID'; - Customer_Email @PersonalData.IsPotentiallyPersonal; + item_ID @PersonalData.FieldSemantics : 'ContractRelatedID'; + customer_ID @PersonalData.FieldSemantics : 'DataSubjectID'; + customer_email @PersonalData.IsPotentiallyPersonal; }; - // Data Privacy annotations on 'Customers' and 'CustomerPostalAddress' - // are derived from original entity definitions. - - -// annotations for Personal Data Manager - Search Fields -annotate bookshop.Customers with @Communication.Contact : { - n : - { - surname : lastName, - given : firstName - }, - bday : dateOfBirth -} + // annotations for Personal Data Manager - Search Fields + annotate bookshop.Customers with @Communication.Contact : { + n : + { + surname : lastName, + given : firstName + }, + bday : dateOfBirth + } }; ``` @@ -83,13 +85,16 @@ annotate bookshop.Customers with @Communication.Contact : { Make sure to have [indicated all relevant entities and elements in your domain model](introduction#indicate-privacy). ::: + + ### Provide Flat Projections As an additional step, you have to create flat projections on the additional business data, like transactional data. In our model, we have `Orders` and `OrderItems`, which are connected via a [composition](https://github.com/SAP-samples/cloud-cap-samples/blob/gdpr/orders/db/schema.cds). Since SAP Personal Data Manager needs flattened out structures, we define a helper view `OrderItemView` to flatten this out. -We have to then add data privacy-specific annotations to this new view as well. The `OrderItemView` as transactional data is marked as `Other`. In addition, it is important to tag the correct field, which defines the corresponding data subject, in our case that is `Customer_ID @PersonalData.FieldSemantics: 'DataSubjectID';` +We have to then add data privacy-specific annotations to this new view as well. The `OrderItemView` as transactional data is marked as `Other`. In addition, it is important to tag the correct field, which defines the corresponding data subject, in our case that is `customer_ID @PersonalData.FieldSemantics: 'DataSubjectID';` + ### Annotating Search Fields @@ -108,6 +113,7 @@ To restrict access to this sensitive data, the `PDMservice` is protected by the [Learn more about security configuration and the SAP Personal Data Manager.](https://help.sap.com/docs/PERSONAL_DATA_MANAGER/620a3ea6aaf64610accdd05cca9e3de2/4ee5705b8ded43e68bde610223722971.html#loio8eb6d9f889594a2d98f478bd57412ceb){.learn-more} + ### Activate Access Checks in _xs-security.json_ Because we protected the `PDMservice`, we need to establish the security check properly. In particular, you need the _xs-security.json_ file to make the security check active. The following _xs-security.json_ is from our sample. @@ -131,6 +137,7 @@ Because we protected the `PDMservice`, we need to establish the security check p Here you define that your personal data manager service instance, called `pdm`, is allowed to access your CAP application granting the `PersonalDataManagerUser` role. + ### Add `@sap/xssec` Library To make the authentication work, you have to enable the security strategy by installing the `@sap/xssec` package: @@ -171,6 +178,7 @@ cf create-service-push For multitenant-specific information, refer to our [Multitenancy Guide](../../guides/deployment/as-saas). + ### Subscribe to SAP Personal Data Manager Service [Subscribe to the service](https://help.sap.com/docs/PERSONAL_DATA_MANAGER/620a3ea6aaf64610accdd05cca9e3de2/ef10215655a540b6ba1c02a96e118d66.html) from the _Service Marketplace_ in the SAP BTP cockpit. @@ -181,6 +189,7 @@ For multitenant-specific information, refer to our [Multitenancy Guide](../../gu Follow the wizard to create your subscription. + ### Create Role Collections SAP Personal Data Manager comes with the following roles: @@ -198,6 +207,8 @@ Application identifiers with **!b** are needed for the UI, and identifiers with [Learn more about defining a role collection in SAP BTP cockpit](https://help.sap.com/products/BTP/65de2977205c403bbc107264b8eccf4b/4b20383efab341f181becf0a947a5498.html){.learn-more} + + ### Create a Service Instance You need a configuration file, like the following, to create a service instance for the Personal Data Manager. @@ -221,6 +232,8 @@ Create a service instance using the SAP BTP cockpit or execute the following com cf create-service personal-data-manager-service standard pdm -c ./.pdm/pdm-instance-config.json ``` + + ### Bind the Service Instance to Your Application. With both the application deployed and the SAP Personal Data Manger service set up, you can now bind the service instance of the Personal Data Manager to your application. Use the URL of your application in a configuration file, such as the following example, which you need when binding a service instance. @@ -258,6 +271,8 @@ You need two configuration files for the Personal Data Manager service. In our [sample](https://github.com/SAP-samples/cloud-cap-samples/tree/gdpr/gdpr), you can find the _.pdm/pdm-instance-config.json_ and _.pdm/pdm-config.json_ files. Use them in addition to the [reference documentation]( https://help.sap.com/docs/PERSONAL_DATA_MANAGER/620a3ea6aaf64610accdd05cca9e3de2/4ee5705b8ded43e68bde610223722971.html) to build your own files later on. + + ## Using the SAP Personal Data Manager Application Open the SAP Personal Data Manager application from the _Instances and Subscriptions_ page in the SAP BTP cockpit. From b7447b09371b918743f26d2c4f732d7a2605863d Mon Sep 17 00:00:00 2001 From: D050513 Date: Tue, 25 Jul 2023 12:42:27 +0200 Subject: [PATCH 11/23] more --- .../assets/Data-Subjects.drawio.svg | 137 ++++++++++++++++ guides/data-privacy/audit-logging.md | 152 ++++++++++++++---- guides/data-privacy/drm.md | 20 +++ guides/data-privacy/introduction.md | 22 ++- guides/data-privacy/pdm.md | 6 +- guides/data-privacy/sample-app.md | 2 + menu.md | 4 +- 7 files changed, 303 insertions(+), 40 deletions(-) create mode 100644 guides/data-privacy/assets/Data-Subjects.drawio.svg diff --git a/guides/data-privacy/assets/Data-Subjects.drawio.svg b/guides/data-privacy/assets/Data-Subjects.drawio.svg new file mode 100644 index 000000000..8d1083df5 --- /dev/null +++ b/guides/data-privacy/assets/Data-Subjects.drawio.svg @@ -0,0 +1,137 @@ + + + + + + + + + + + + + + + + + +
+
+
+ Customers +
+
+
+ + + Customers + + + + + + + + + + + + + + +
+
+
+ Addresses +
+
+
+ + + Addresses + + + + + + + + + +
+
+
+ Orders +
+
+
+ + + Orders + + + + + + + + +
+
+
+ Data Subject +
+
+
+ + + Data Subject + + + + + + + + +
+
+
+ Data Subject Details +
+
+
+ + + Data Subject Details + + + + + + + + +
+
+
+ Related Data +
+
+
+ + + Related Data + + + + + + + + + Text is not SVG - cannot display + + + + \ No newline at end of file diff --git a/guides/data-privacy/audit-logging.md b/guides/data-privacy/audit-logging.md index 4b4ffbf6e..1e58c6043 100644 --- a/guides/data-privacy/audit-logging.md +++ b/guides/data-privacy/audit-logging.md @@ -11,14 +11,21 @@ status: released --- + + # Audit Logging
+ ## Introduction This section deals with Audit Logging for reading sensitive data and changes to personal data. As a prerequisite, you have [indicated entities and elements in your domain model, which will contain personal data](introduction#indicate-privacy). + + + ## About the Audited Object + ### Data Subject In our case `Customers` is the main master data entity with the semantics of 'Data Subject'. @@ -42,7 +53,7 @@ In our case `Customers` is the main master data entity with the semantics of 'Da ```cds using { cuid, Country } from '@sap/cds/common'; - entity Customers : cuid, managed { +entity Customers : cuid, managed { email : String; firstName : String; lastName : String; @@ -50,30 +61,28 @@ using { cuid, Country } from '@sap/cds/common'; postalAddresses : Composition of many CustomerPostalAddress on postalAddresses.Customer = $self; billingData : Composition of many CustomerBillingData on billingData.Customer = $self; } - ``` This entity is annotated in the _db/data-privacy.cds_ file. ```cds - annotate bookshop.Customers with @PersonalData : { DataSubjectRole : 'Customer', EntitySemantics : 'DataSubject' +} { + ID @PersonalData.FieldSemantics : 'DataSubjectID'; + email @PersonalData.IsPotentiallyPersonal; + firstName @PersonalData.IsPotentiallyPersonal; + lastName @PersonalData.IsPotentiallyPersonal; + dateOfBirth @PersonalData.IsPotentiallyPersonal; } -{ - ID @PersonalData.FieldSemantics : 'DataSubjectID'; - emailAddress @PersonalData.IsPotentiallyPersonal; - firstName @PersonalData.IsPotentiallyPersonal; - lastName @PersonalData.IsPotentiallyPersonal; - dateOfBirth @PersonalData.IsPotentiallyPersonal; -} - ``` Here we again have the four levels of annotations as already described in the chapter [Indicate Personal Data in Your Domain Model](introduction#indicate-privacy). + In the context of audit logging, the `@PersonalData.IsPotentiallyPersonal` field-level annotation is relevant for inducing audit logs for _Insert_, _Update_, and _Delete_, whereas the `@PersonalData.IsPotentiallySensitive` annotation is relevant for _Read_ access audit logs. @@ -81,8 +90,17 @@ In the context of audit logging, the `@PersonalData.IsPotentiallyPersonal` field The `@PersonalData.IsPotentiallySensitive` annotation induces an audit log for each and every _Read_ access. --- Only use this annotation in [relevant cases](https://ec.europa.eu/info/law/law-topic/data-protection/reform/rules-business-and-organisations/legal-grounds-processing-data/sensitive-data/what-personal-data-considered-sensitive_en). --- Avoid unnecessary logging activities in your application. +For example, try to avoid reading sensitive data at all by obscuring credit card numbers as `**** **** **** 1234` +::: + +We recommend using fields with the `IsPotentiallySensitive` annotation only in detail view entities. This ensures that the audit log for reading sensitive data is only triggered after explicitly jumping to the detail view from the respective overview list view. + +::: warning _Warning_ +In `@cap-js/audit-logging`, out-of-the-box logging is configurable via `cds.requires['audit-log'].handle = [...]`, with possible values `READ` and `WRITE`, and a default of `['WRITE']`. +That is, data accesses are not logged by default, but must be opted into by overriding the default config. ::: + ### Data Subject Details In the first example, the audited object was identical to the data subject, but this is not always the case. @@ -93,7 +111,6 @@ In our terminology this has the semantics 'Data Subject Details'. In our example we have the additional entities `CustomerPostalAddress` and `CustomerBillingData` which contain additional master data belonging to a certain 'Customer', but which are stored in separate entities, for better clarity and better separation of concerns. ```cds - entity CustomerPostalAddress : cuid, managed { Customer : Association to one Customers; street : String(128); @@ -106,7 +123,6 @@ entity CustomerBillingData : cuid, managed { Customer : Association to one Customers; creditCardNo : String; }; - ``` These entities are annotated in the _db/data-privacy.cds_ file. @@ -115,8 +131,7 @@ These entities are annotated in the _db/data-privacy.cds_ file. annotate bookshop.CustomerPostalAddress with @PersonalData : { DataSubjectRole : 'Customer', EntitySemantics : 'DataSubjectDetails' -} -{ +} { Customer @PersonalData.FieldSemantics : 'DataSubjectID'; street @PersonalData.IsPotentiallyPersonal; town @PersonalData.IsPotentiallyPersonal; @@ -126,12 +141,10 @@ annotate bookshop.CustomerPostalAddress with @PersonalData : { annotate bookshop.CustomerBillingData with @PersonalData : { DataSubjectRole : 'Customer', EntitySemantics : 'DataSubjectDetails' -} -{ +} { Customer @PersonalData.FieldSemantics : 'DataSubjectID'; creditCardNo @PersonalData.IsPotentiallySensitive; } - ``` Very similarly to the section on 'Data Subject' this entity is as well annotated in four levels. @@ -141,6 +154,7 @@ More details on these annotations can be found in the chapter [Indicate Personal Annotation `@AuditLog.Operation` is not applicable for the Node.js runtime. ::: + ### Transactional Data In the section on 'Data Subject' and 'Data Subject Details' we have seen, how to annotate the master data entities carrying the semantical information of the 'Data Subject'. @@ -171,23 +185,25 @@ annotate bookshop.Orders with @PersonalData.EntitySemantics : 'Other' Customer @PersonalData.FieldSemantics : 'DataSubjectID'; personalComment @PersonalData.IsPotentiallyPersonal; } -annotate bookshop.Orders with @AuditLog.Operation : { - Read : true, - Insert : true, - Update : true, - Delete : true -}; ``` + +### Operation-Level Annotations + Finally, we annotate all standard operations (`Read`, `Insert`, `Update`, `Delete`) as relevant for the audit log - which should be the default case for most of the relevant business entities. -::: warning _Warning_ -Annotation `@AuditLog.Operation` is not applicable for the Node.js runtime. -::: +Operation-level annotations indicate which `@AuditLog.Operation` (_Read_, _Insert_, _Update_, _Delete_), related to data-privacy requirements, will be handled automatically in the audit log (read access or change log). This annotation is introduced to manage CAP Audit Logs on a fine granular basis, for three reasons: + + The first annotation on the entity level is also valid for the SAP Personal Data Manager service. + + Some entities do not need all Audit Log operations. + + Some entities manage their Audit Log operations themselves. -
+The default would be to switch on CAP Audit Logging for all standard operations. - +According to the information provided by annotations - written by the responsible developer or architect - the runtime will automatically write all the required read access and change logs by means of the audit log interface described in [Audit Log V2](https://github.wdf.sap.corp/xs-audit-log/audit-java-client/wiki/Audit-Log-V2). + +::: warning _Warning_ +`@AuditLog.Operation` is not applicable for `@cap-js/audit-logging` (i.e., the Node.js stack). +::: ## Transactional Outbox @@ -197,7 +213,6 @@ By default all log messages are sent through a transactional outbox. This means, ![Transactional Outbox.drawio](./assets/Transactional-Outbox.drawio.svg) - This provides an ultimate level of resiliency, plus additional benefits: - **Audit log messages are guaranteed to be delivered** — even if the audit log service should be down for a longer time period. @@ -207,10 +222,12 @@ This provides an ultimate level of resiliency, plus additional benefits: - **False log messages are avoided** — messages are forwarded to the audit log service on successfully committed requests; and skipped in case of rollbacks. - - ## Programmatic API +TODOs: +- add model +- add link to [Java's API](/java/auditlog#auditlog-service) + In addition to the generic audit logging provided out of the box, applications can also log custom events with custom data using the programmatic API. Connecting to the service: @@ -232,6 +249,77 @@ await audit.log ('SomeEvent', { … }) The Audit Log Service API is implemented as a CAP service, with the service API defined in CDS as shown below. In effect, the common patterns of [*CAP Service Consumption*](../using-services) apply, as well as all the usual benefits like *mocking*, *late-cut µ services*, *resilience* and *extensibility*. ::: + +## AuditLogService + +Reference modeling in `@cap-js/audit-logging`: + +```cds +namespace sap.auditlog; + +service AuditLogService { + + action log(event : String, data : LogEntry); + action logSync(event : String, data : LogEntry); + + event SensitiveDataRead : LogEntry { + data_subject : DataSubject; + object : DataObject; + attributes : many { + name : String; + }; + attachments : many { + id : String; + name : String; + }; + channel : String; + } + + event PersonalDataModified : LogEntry { + data_subject : DataSubject; + object : DataObject; + attributes : many Modification; + success : Boolean default true; + }; + + event ConfigurationModified : LogEntry { + object : DataObject; + attributes : many Modification; + }; + + event SecurityEvent : LogEntry { + data : {}; + ip : String; + }; + +} + +type LogEntry { + uuid : UUID; + tenant : String; + user : String; + time : Timestamp; +} + +type DataObject { + type : String; + id : {}; +} + +type DataSubject : DataObject { + role : String; +} + +type Modification { + name : String; + old : String; + new : String; +} +``` + +The content of aspect `LogEntry` are automatically applied and cannot be provided manually (i.e., are overwritten by the service implementation). + + ### Basic Service API The basic service definition declares the generic `log` operation used for all kinds of events, along with type `LogMessage` declares the common fields of all log messages — these fields are filled in automatically if not provided by the caller. diff --git a/guides/data-privacy/drm.md b/guides/data-privacy/drm.md index 3852f8153..f6f2920e5 100644 --- a/guides/data-privacy/drm.md +++ b/guides/data-privacy/drm.md @@ -1,3 +1,23 @@ +--- +layout: cookbook +shorty: Data Retention Management +synopsis: > + Use the SAP Data Retention Manager (DRM) with a CAP application. +breadcrumbs: + - Cookbook + - Data Privacy + - Data Retention +#status: released +--- + + + +# Data Retention Management + +Under construction. + + + + + # Personal Data Management
@@ -33,6 +35,8 @@ Following the CAP principles, we recommend adding a new dedicated CAP service th ### CAP Service Model for SAP Personal Data Manager +TODO: adjust modeling in cloud-cap-samples to this one or the other way around? + Open the _srv/pdm-service.cds_ file, which contains the content for the Personal Data Manager service. ```cds diff --git a/guides/data-privacy/sample-app.md b/guides/data-privacy/sample-app.md index bb1efd025..f3f8ab62f 100644 --- a/guides/data-privacy/sample-app.md +++ b/guides/data-privacy/sample-app.md @@ -10,6 +10,8 @@ breadcrumbs: #status: released --- + + # Data Privacy Sample App for SAP BTP CF We should document how to build AND DEPLOY a SaaS app with PDM, audit logging, and DRM (when available). diff --git a/menu.md b/menu.md index 36097af56..2a24bc199 100644 --- a/menu.md +++ b/menu.md @@ -47,7 +47,9 @@ - [Data Privacy](guides/data-privacy/) - [Basics](guides/data-privacy/introduction) - [Personal Data Management](guides/data-privacy/pdm) - - [Audit Log](guides/data-privacy/audit-log) + - [Data Retention Management](guides/data-privacy/drm) + - [Audit Logging](guides/data-privacy/audit-logging) + - [Sample App](guides/data-privacy/sample-app) ### [Advanced](advanced/) From 2e31e939850682e720051e9c3158b19f1ee68259 Mon Sep 17 00:00:00 2001 From: D050513 Date: Sun, 30 Jul 2023 23:32:59 +0200 Subject: [PATCH 12/23] audit logging api --- guides/data-privacy/audit-logging.md | 347 +++++++++++++++------------ guides/data-privacy/introduction.md | 18 +- guides/data-privacy/pdm.md | 49 ++-- guides/data-privacy/sample-app.md | 33 +++ 4 files changed, 265 insertions(+), 182 deletions(-) diff --git a/guides/data-privacy/audit-logging.md b/guides/data-privacy/audit-logging.md index 1e58c6043..acf0c635a 100644 --- a/guides/data-privacy/audit-logging.md +++ b/guides/data-privacy/audit-logging.md @@ -24,7 +24,9 @@ This section deals with Audit Logging for reading sensitive data and changes to In the context of audit logging, the `@PersonalData.IsPotentiallyPersonal` field-level annotation is relevant for inducing audit logs for _Insert_, _Update_, and _Delete_, whereas the `@PersonalData.IsPotentiallySensitive` annotation is relevant for _Read_ access audit logs. @@ -111,16 +111,16 @@ In our terminology this has the semantics 'Data Subject Details'. In our example we have the additional entities `CustomerPostalAddress` and `CustomerBillingData` which contain additional master data belonging to a certain 'Customer', but which are stored in separate entities, for better clarity and better separation of concerns. ```cds -entity CustomerPostalAddress : cuid, managed { - Customer : Association to one Customers; +entity Addresses : cuid, managed { + customer : Association to one Customers; street : String(128); town : String(128); country : Country; someOtherField : String(128); }; -entity CustomerBillingData : cuid, managed { - Customer : Association to one Customers; +entity BillingData : cuid, managed { + customer : Association to one Customers; creditCardNo : String; }; ``` @@ -128,21 +128,21 @@ entity CustomerBillingData : cuid, managed { These entities are annotated in the _db/data-privacy.cds_ file. ```cds -annotate bookshop.CustomerPostalAddress with @PersonalData : { +annotate bookshop.Addresses with @PersonalData : { DataSubjectRole : 'Customer', EntitySemantics : 'DataSubjectDetails' } { - Customer @PersonalData.FieldSemantics : 'DataSubjectID'; + customer @PersonalData.FieldSemantics : 'DataSubjectID'; street @PersonalData.IsPotentiallyPersonal; town @PersonalData.IsPotentiallyPersonal; country @PersonalData.IsPotentiallyPersonal; } -annotate bookshop.CustomerBillingData with @PersonalData : { +annotate bookshop.BillingData with @PersonalData : { DataSubjectRole : 'Customer', EntitySemantics : 'DataSubjectDetails' } { - Customer @PersonalData.FieldSemantics : 'DataSubjectID'; + customer @PersonalData.FieldSemantics : 'DataSubjectID'; creditCardNo @PersonalData.IsPotentiallySensitive; } ``` @@ -150,9 +150,7 @@ annotate bookshop.CustomerBillingData with @PersonalData : { Very similarly to the section on 'Data Subject' this entity is as well annotated in four levels. More details on these annotations can be found in the chapter [Indicate Personal Data in Your Domain Model](introduction#indicate-privacy). -::: warning _Warning_ -Annotation `@AuditLog.Operation` is not applicable for the Node.js runtime. -::: +You may have noticed property `someOtherField` was not annotated. Hence, no modification will be logged. ### Transactional Data @@ -168,10 +166,10 @@ In our example we have the entity 'Orders' ```cds entity Orders : cuid, managed { - OrderNo : String @title:'Order Number'; //> readable key - Items : Composition of many OrderItems on Items.parent = $self; + orderNo : String @title: 'Order Number'; //> readable key + items : Composition of many OrderItems on items.parent = $self; currency : Currency; - Customer : Association to Customers; + customer : Association to Customers; personalComment : String; } ``` @@ -182,7 +180,7 @@ To ensure proper audit logging we annotate using the usual four levels as descri annotate bookshop.Orders with @PersonalData.EntitySemantics : 'Other' { ID @PersonalData.FieldSemantics : 'ContractRelatedID'; - Customer @PersonalData.FieldSemantics : 'DataSubjectID'; + customer @PersonalData.FieldSemantics : 'DataSubjectID'; personalComment @PersonalData.IsPotentiallyPersonal; } ``` @@ -190,6 +188,10 @@ annotate bookshop.Orders with @PersonalData.EntitySemantics : 'Other' ### Operation-Level Annotations +::: danger _TODO_ +check if still needed for Java. In Node.js, this is no longer considered. +::: + Finally, we annotate all standard operations (`Read`, `Insert`, `Update`, `Delete`) as relevant for the audit log - which should be the default case for most of the relevant business entities. Operation-level annotations indicate which `@AuditLog.Operation` (_Read_, _Insert_, _Update_, _Delete_), related to data-privacy requirements, will be handled automatically in the audit log (read access or change log). This annotation is introduced to manage CAP Audit Logs on a fine granular basis, for three reasons: @@ -206,6 +208,7 @@ According to the information provided by annotations - written by the responsibl ::: + ## Transactional Outbox By default all log messages are sent through a transactional outbox. This means, when sent, log messages are first stored in a local outbox table, which acts like a queue for outbound messages. Only when requests are fully and successfully processed, will these messages be forwarded to the audit log service. @@ -224,12 +227,10 @@ This provides an ultimate level of resiliency, plus additional benefits: ## Programmatic API -TODOs: -- add model -- add link to [Java's API](/java/auditlog#auditlog-service) - In addition to the generic audit logging provided out of the box, applications can also log custom events with custom data using the programmatic API. +_The following is written from a Node.js perspective. For Java, please see [AuditLog Service](https://pages.github.tools.sap/cap/docs/java/auditlog)._ + Connecting to the service: ```js @@ -242,26 +243,53 @@ Sending log messages: await audit.log ('SomeEvent', { … }) ``` -
- - ::: tip The Audit Log Service API is implemented as a CAP service, with the service API defined in CDS as shown below. In effect, the common patterns of [*CAP Service Consumption*](../using-services) apply, as well as all the usual benefits like *mocking*, *late-cut µ services*, *resilience* and *extensibility*. ::: -## AuditLogService +### Basic Service API -Reference modeling in `@cap-js/audit-logging`: +The basic service definition declares the generic `log` operation used for all kinds of events, along with type `LogEntry` declares the common fields of all log messages — these fields are filled in automatically if not provided by the caller. ```cds namespace sap.auditlog; service AuditLogService { - action log(event : String, data : LogEntry); + action log (event : String, data : LogEntry); action logSync(event : String, data : LogEntry); +} + +/** Common fields, filled in automatically */ +type LogEntry { + uuid : UUID; + tenant : String; + user : String; + time : Timestamp; +} +``` + +Usage is like that: + +```js +await audit.log ('SomeEvent', { + some_details: 'whatever' +}) +``` + + +### Personal Data-Related Events + +In addition, pre-defined event payloads for personal data-related events are declared: + +```cds +namespace sap.auditlog; + +service AuditLogService { + // … as above + event SensitiveDataRead : LogEntry { data_subject : DataSubject; object : DataObject; @@ -280,27 +308,10 @@ service AuditLogService { object : DataObject; attributes : many Modification; success : Boolean default true; - }; - - event ConfigurationModified : LogEntry { - object : DataObject; - attributes : many Modification; - }; - - event SecurityEvent : LogEntry { - data : {}; - ip : String; - }; + } } -type LogEntry { - uuid : UUID; - tenant : String; - user : String; - time : Timestamp; -} - type DataObject { type : String; id : {}; @@ -317,129 +328,169 @@ type Modification { } ``` -The content of aspect `LogEntry` are automatically applied and cannot be provided manually (i.e., are overwritten by the service implementation). +Send corresponding log messages complying to these definitions like that: +```js +await audit.log ('SensitiveDataRead', { + data_subject: { + type: 'sap.capire.bookshop.Customers', + id: { ID: '1923bd11-b1d6-47b6-a91b-732e755fa976' }, + role: 'Customer', + }, + object: { + type: 'sap.capire.bookshop.BillingData', + id: { ID: '399a2704-3d2d-4fa1-9e7d-a4e45c67749b' } + }, + attributes: [ + { name: 'creditCardNo' } + ] +}) +``` -### Basic Service API +```js +await audit.log ('PersonalDataModified', { + data_subject: { + type: 'sap.capire.bookshop.Customers', + id: { ID: '1923bd11-b1d6-47b6-a91b-732e755fa976' }, + role: 'Customer', + }, + object: { + type: 'sap.capire.bookshop.Customers', + id: { ID: '1923bd11-b1d6-47b6-a91b-732e755fa976' } + }, + attributes: [ + { name: 'emailAddress', old: 'foo@example.com', new: 'bar@example.com' } + ] +}) +``` -The basic service definition declares the generic `log` operation used for all kinds of events, along with type `LogMessage` declares the common fields of all log messages — these fields are filled in automatically if not provided by the caller. -```cds -namespace sap.auditlog; -service AuditLogService { +### Configuration Modified Events - action log (event: String, data: LogEntry); - action logSync (event: String, data: LogEntry); +```cds +service AuditLogService { + // … as above -} + event ConfigurationModified : LogEntry { + object : DataObject; + attributes : many Modification; + } -/** Common fields, filled in automatically if missing */ -type LogEntry { - uuid : UUID; - tenant : String; - user : String; - timestamp : Timestamp; } ``` -::: warning _Warning_ -Only applicable for the Node.js runtime. Java still uses old model, I assume. -::: - -Usage is like that: +Send corresponding log messages complying to these definitions like that: ```js -await audit.log ('SomeEvent', { - some_details: 'whatever' +await audit.log ('ConfigurationModified', { + object: { + type: 'sap.common.Currencies', + id: { ID: 'f79ba248-c348-4962-9fef-680c3b88807c' } + }, + attributes: [ + { name: 'symbol', old: 'EUR', new: '€' } + ] }) ``` -### Personal Data-Related Events -In addition, pre-defined event payloads for personal data-related events are declared: +### Security Events ```cds -namespace sap.auditlog; - service AuditLogService { // … as above - event SensitiveDataRead : LogEntry { - // dataSubjects : many DataSubject; - dataSubject : DataSubject; - dataObject : DataObject; - // channel : String; - attributes : many { name: String }; - // attachments : many { name: String; id: String }; - } - - event PersonalDataChanged : LogEntry { - dataSubject : DataSubject; - dataObject : DataObject; - attributes : many { name: String; old: String; new: String; }; + event SecurityEvent : LogEntry { + data : {}; + ip : String; } } - -type DataObject : { type: String; id: {} } -type DataSubject : DataObject { role: String } ``` Send corresponding log messages complying to these definitions like that: ```js -await audit.log ('SensitiveDataRead', { - dataSubject: { - type: 'sap.capire.bookshop.Customers', - id: { ID: '1923bd11-b1d6-47b6-a91b-732e755fa976' }, - role: 'Customer', - }, - dataObject: { - type: 'sap.capire.bookshop.Customers', - id: { ID: '1923bd11-b1d6-47b6-a91b-732e755fa976' } +await audit.log ('SecurityEvent', { + data: { + user: 'alice', + action: 'Attempt to access restricted service "PDMService" with insufficient authority' }, - attributes: [ - { name: 'creditCardNo' } - ] + ip: '127.0.0.1' }) ``` +### AuditLogService -### Config Change Events +Here is the complete reference modeling as contained in `@cap-js/audit-logging`: ```cds +namespace sap.auditlog; + service AuditLogService { - // … as above - event ConfigChange : LogMessage { - object : DataObject; - attributes : ChangedAttributes; - } -} -``` -::: warning _Warning_ -Not applicable for the Node.js runtime. -::: + action log (event : String, data : LogEntry); + action logSync(event : String, data : LogEntry); + event SensitiveDataRead : LogEntry { + data_subject : DataSubject; + object : DataObject; + attributes : many { + name : String; + }; + attachments : many { + id : String; + name : String; + }; + channel : String; + }; + event PersonalDataModified : LogEntry { + data_subject : DataSubject; + object : DataObject; + attributes : many Modification; + success : Boolean default true; + }; -### Security Events + event ConfigurationModified : LogEntry { + object : DataObject; + attributes : many Modification; + }; + + event SecurityEvent : LogEntry { + data : {}; + ip : String; + }; -```cds -service AuditLogService { - // ... as above - event FailedLogin : LogMessage { - action : String; - data : String; - } +} + +/** Common fields, filled in automatically */ +type LogEntry { + uuid : UUID; + tenant : String; + user : String; + time : Timestamp; +} + +type DataObject { + type : String; + id : {}; +} + +type DataSubject : DataObject { + role : String; +} + +type Modification { + name : String; + old : String; + new : String; } ``` -::: warning _Warning_ -Not applicable for the Node.js runtime. -::: +The contents of aspect `LogEntry` are automatically applied and cannot be provided manually (i.e., are overwritten by the service implementation). @@ -450,34 +501,28 @@ In addition, everybody could provide new implementations in the same way as we i ```js const cds = require('@sap/cds') -class ConsoleAuditLogService extends cds.Service { +class MyAuditLogService extends cds.Service { log (event, data) { console.log (event, data) } + logSync (event, data) { + console.log (event, data) + } } -module.exports = ConsoleAuditLogService +module.exports = MyAuditLogService ``` -## Setup on BTP - -Your application needs to be bound to an instance of the SAP Audit Log Service. -In case you're providing a SaaS application, you additionally need to return the `xsappname` of the service instance's UAA instance (i.e., `cds.env.requires['audit-log'].credentials.uaa.xsappname`). -If you miss doing this, audit logs that are emitted via the persistent outbox (the default in production as it provides the ultimate resilience) will be lost in nirvana, as the sending effort will end in an unrecoverable error. -As with all dependencies: If you add them later on, you'll need to update all subscriptions. - -TODO: should we add this to mtxs? -- something like `if (cds.env.requires?.['audit-log']?.credentials?.uaa.xsappname) dependencies.push({ xsappname: ... })` -- requires `@cap-js/audit-logging` to be added to mtxs sidecar -- NOTE: if we don't do this, then a single deploy of an mta would not suffice! the first deploy creates the service instance and the developer then would need to look up the `uaa.xsappname` and add to mtxs sidecar package.json as to-be-returned dependency - -TODO: add `@sap/audit-logging` manually if changing impl to `audit-log-to-library` (default is now `audit-log-to-restv2`)? +As always, custom implementations need to be configured: -## `cds add audit-logging` - -should we offer this? - -- add `@cap-js/audit-logging` - - also for mtxs sidecar, if dependency shall be returned automatically -- add audit log service with oauth2 plan in mta.yaml -- if dependency shall not be returned automatically, somehow tell mtxs to return it +```json +{ + "cds": { + "requires": { + "audit-log": { + "impl": "./lib/MyAuditLogService.js" + } + } + } +} +``` diff --git a/guides/data-privacy/introduction.md b/guides/data-privacy/introduction.md index 087f620a7..2da840f64 100644 --- a/guides/data-privacy/introduction.md +++ b/guides/data-privacy/introduction.md @@ -30,7 +30,9 @@ ADDED FROM GUIDE #4: Compliance to data privacy regulations is an important requirement for all busines applications nowadays. CAP provides easy ways to designate personal data, as well as out-of-the-box integration with SAP BTP services, like SAP Personal Data Manager service. This greatly relieves application developers these tedious tasks and related efforts. -TODO: linie zu DRM sollte gestrichelt sein +::: danger _TODO_ +linie zu DRM sollte gestrichelt sein +::: Data Privacy.drawio.svg @@ -39,7 +41,10 @@ DRM integration in progress ::: @@ -90,8 +95,6 @@ See full sample in [cloud-cap-samples](https://github.com/SAP-samples/cloud-cap- ### Base Model -TODO: adjust modeling in cloud-cap-samples to this one or the other way around? - In the remainder of this guide, we'll use this domain model as the base to add data privacy and audit logging. db/schema.cds @@ -138,9 +141,12 @@ Let's annotate our data model to identify personal data. In essence, in all our Use `@PersonalData` annotations to indicate entities and elements in your domain model, which will contain personal data. -TODOs: +For more details on the `@PersonalData` vocabulary, see [this](https://github.com/SAP/odata-vocabularies/blob/main/vocabularies/PersonalData.md). + +::: danger _TODO_ - add BillingData to diagram -- fix types (i.e., `DataSubject`, `DataSubjectDetails`, and `Other`) +- adjust types to `DataSubject`, `DataSubjectDetails`, and `Other (e.g., Transactional Data)` +::: Data Subjects.drawio diff --git a/guides/data-privacy/pdm.md b/guides/data-privacy/pdm.md index d71a4f1c9..e9f3ef980 100644 --- a/guides/data-privacy/pdm.md +++ b/guides/data-privacy/pdm.md @@ -35,8 +35,6 @@ Following the CAP principles, we recommend adding a new dedicated CAP service th ### CAP Service Model for SAP Personal Data Manager -TODO: adjust modeling in cloud-cap-samples to this one or the other way around? - Open the _srv/pdm-service.cds_ file, which contains the content for the Personal Data Manager service. ```cds @@ -49,42 +47,43 @@ using {sap.capire.bookshop.OrderItems} from '@capire/orders'; @requires: 'PersonalDataManagerUser' // security check service PDMService { - // Data Privacy annotations on 'Customers' and 'Addresses' are derived from original entity definitions. - entity Customers as projection on db.Customers; - entity Addresses as projection on db.Addresses; + // Data Privacy annotations on 'Customers', 'Addresses', and 'BillingData' are derived from original entity definitions + entity Customers as projection on db.Customers; + entity Addresses as projection on db.Addresses; + entity BillingData as projection on db.BillingData; - // create view on Orders and Items as flat projection - entity OrderItemView as + // create view on Orders and Items as flat projection + entity OrderItemView as select from Orders { ID, - key items.ID as item_ID, - orderNo, - customer.ID as customer_ID, - customer.email as customer_email, - items.book.ID as item_book_ID, - items.quantity as item_quantity, - items.netQuantity as item_net_quantity + key Items.ID as item_ID, + OrderNo, + customer.ID as customer_ID, + customer.email as customer_email, + Items.book.ID as item_Book_ID, + Items.amount as item_Amount, + Items.netAmount as item_NetAmount }; - // annotate new view - annotate PDMService.OrderItemView with @(PersonalData.EntitySemantics : 'Other') { - item_ID @PersonalData.FieldSemantics : 'ContractRelatedID'; - customer_ID @PersonalData.FieldSemantics : 'DataSubjectID'; + // annotate new view + annotate PDMService.OrderItemView with @(PersonalData.EntitySemantics: 'Other') { + item_ID @PersonalData.FieldSemantics: 'ContractRelatedID'; + customer_ID @PersonalData.FieldSemantics: 'DataSubjectID'; customer_email @PersonalData.IsPotentiallyPersonal; }; // annotations for Personal Data Manager - Search Fields - annotate bookshop.Customers with @Communication.Contact : { - n : - { - surname : lastName, - given : firstName + annotate Customers with @(Communication.Contact: { + n : { + surname: lastName, + given : firstName }, - bday : dateOfBirth - } + bday: dateOfBirth + }); }; ``` + ::: tip Make sure to have [indicated all relevant entities and elements in your domain model](introduction#indicate-privacy). ::: diff --git a/guides/data-privacy/sample-app.md b/guides/data-privacy/sample-app.md index f3f8ab62f..09ae858a4 100644 --- a/guides/data-privacy/sample-app.md +++ b/guides/data-privacy/sample-app.md @@ -16,3 +16,36 @@ breadcrumbs: We should document how to build AND DEPLOY a SaaS app with PDM, audit logging, and DRM (when available). There are many pitfalls such as returning all necessary xsappnames in a single deployment without manual copy & paste. + + + +## Setup on BTP + +Your application needs to be bound to an instance of the SAP Audit Log Service. +In case you're providing a SaaS application, you additionally need to return the `xsappname` of the service instance's UAA instance (i.e., `cds.env.requires['audit-log'].credentials.uaa.xsappname`). +If you miss doing this, audit logs that are emitted via the persistent outbox (the default in production as it provides the ultimate resilience) will be lost in nirvana, as the sending effort will end in an unrecoverable error. +As with all dependencies: If you add them later on, you'll need to update all subscriptions. + +::: danger _TODO_ +should we add this to mtxs? +- something like `if (cds.env.requires?.['audit-log']?.credentials?.uaa.xsappname) dependencies.push({ xsappname: ... })` +- requires `@cap-js/audit-logging` to be added to mtxs sidecar +- NOTE: if we don't do this, then a single deploy of an mta would not suffice! the first deploy creates the service instance and the developer then would need to look up the `uaa.xsappname` and add to mtxs sidecar package.json as to-be-returned dependency +::: + +::: danger _TODO_ +add `@sap/audit-logging` manually if changing impl to `audit-log-to-library` (default is now `audit-log-to-restv2`)? +::: + + + +## `cds add audit-logging` + +::: danger _TODO_ +should we offer this? +::: + +- add `@cap-js/audit-logging` + - also for mtxs sidecar, if dependency shall be returned automatically +- add audit log service with oauth2 plan in mta.yaml +- if dependency shall not be returned automatically, somehow tell mtxs to return it From 15aeef31878a102bc618166aaf1928aa7f0fdfc5 Mon Sep 17 00:00:00 2001 From: D050513 Date: Sun, 30 Jul 2023 23:59:18 +0200 Subject: [PATCH 13/23] merge guides --- guides/data-privacy/audit-logging.md | 160 ++++++++++++++++++++++++--- guides/data-privacy/sample-app.md | 67 +++++++++++ 2 files changed, 210 insertions(+), 17 deletions(-) diff --git a/guides/data-privacy/audit-logging.md b/guides/data-privacy/audit-logging.md index acf0c635a..210b69106 100644 --- a/guides/data-privacy/audit-logging.md +++ b/guides/data-privacy/audit-logging.md @@ -17,32 +17,32 @@ status: released
+_The following is mainly written from a Node.js perspective. For Java's perspective, please see [Java - Audit Logging](https://pages.github.tools.sap/cap/docs/java/auditlog)._ -## Introduction -This section deals with Audit Logging for reading sensitive data and changes to personal data. As a prerequisite, you have [indicated entities and elements in your domain model, which will contain personal data](introduction#indicate-privacy). - ## About the Audited Object @@ -225,12 +225,138 @@ This provides an ultimate level of resiliency, plus additional benefits: - **False log messages are avoided** — messages are forwarded to the audit log service on successfully committed requests; and skipped in case of rollbacks. +## Setup & Configuration + +::: danger TODO +`cds add audit-logging` is not yet supported +::: + +Run this to enable audit logging + +```sh +cds add audit-logging +``` + +::: details Behind the Scenes… + +This CLI command is a convenient shortcut for… + +1. Installing required 3rd-party packages, e.g. + ```js + npm add @cap-js/audit-logging + ``` + +2. Which sets `cds.requires.audit-log = true` in `cds.env`, equivalent to: + ```json + {"cds":{ + "requires": { + "audit-log": true + } + }} + ``` + +3. Which in turn activates the `audit-log` configuration **preset**: + ```jsonc + { + "audit-log": { + "handle": ["WRITE"], + "[development]": { + "impl": "@cap-js/audit-logging/srv/audit-log-to-console", + "outbox": false + }, + "[production]": { + "impl": "@cap-js/audit-logging/srv/audit-log-to-restv2", + "outbox": true + } + } + } + ``` + +**The individual config options are:** + +- `impl` — the service implementation to use +- `outbox` — whether to use transactional outbox or not +- `handle` — which events (`READ` and/or `WRITE`) to intercept and generate log messages from + +**The preset uses profile-specific configs** for development and production. Use the `cds env` command to find out the effective configuration for your current environment: + +```sh +cds env requires.audit-log +``` + +```sh +cds env requires.audit-log --profile production +``` + +::: + +See the [Sample App](./sample-app.md) for more details. + + +## Generic Audit Logging + +[The @PersonalData annotations](introduction#indicate-privacy) are all we need to automatically log personal data-related events. Let's see that in action… + +1. **Start the server** as usual: + + ```sh + cds watch + ``` + +2. **Send an udate** request, changing personal data + + ```http + PATCH http://localhost:4004/admin/Customers(8e2f2640-6866-4dcf-8f4d-3027aa831cad) HTTP/1.1 + Authorization: Basic alice:in-wonderland + Content-Type: application/json + + { + "firstName": "Johnny", + "dateOfBirth": "2002-03-09" + } + ``` + +3. **See the audit logs** in the server's console output: + + ```js + { + data_subject: { + type: 'AdminService.Customers', + id: { ID: '8e2f2640-6866-4dcf-8f4d-3027aa831cad' }, + role: 'Customer', + }, + object: { + type: 'AdminService.Customers', + id: { ID: '8e2f2640-6866-4dcf-8f4d-3027aa831cad' } + }, + attributes: [ + { name: 'firstName', old: 'John', new: 'Johnny' }, + { name: 'dateOfBirth', old: '1970-01-01', new: '2002-03-09' } + ], + user: 'alice', + tenant: 't1', + uuid: '1391A703E2CBE52E817269EC7527368C', + time: '2023-02-26T08:13:48.287Z' + } + ``` + + + +**Behind the scenes** the generic audit logging implementation automatically cares for: + +- Intercepting all read operations potentially involving sensitive data and +- Intercepting all write operations potentially involving personal data +- Determining the affected fields containing personal data, if any +- Constructing log messages, and sending them to the connected audit log service +- All emitted log messages are sent through the [transactional outbox](#transactional-outbox) +- Applying resiliency mechanisms like retry with exponential backoff, etc. + + + ## Programmatic API In addition to the generic audit logging provided out of the box, applications can also log custom events with custom data using the programmatic API. -_The following is written from a Node.js perspective. For Java, please see [AuditLog Service](https://pages.github.tools.sap/cap/docs/java/auditlog)._ - Connecting to the service: ```js diff --git a/guides/data-privacy/sample-app.md b/guides/data-privacy/sample-app.md index 09ae858a4..1ac7d3717 100644 --- a/guides/data-privacy/sample-app.md +++ b/guides/data-privacy/sample-app.md @@ -14,6 +14,14 @@ breadcrumbs: # Data Privacy Sample App for SAP BTP CF + + +::: danger +Work in progress +::: + + + We should document how to build AND DEPLOY a SaaS app with PDM, audit logging, and DRM (when available). There are many pitfalls such as returning all necessary xsappnames in a single deployment without manual copy & paste. @@ -49,3 +57,62 @@ should we offer this? - also for mtxs sidecar, if dependency shall be returned automatically - add audit log service with oauth2 plan in mta.yaml - if dependency shall not be returned automatically, somehow tell mtxs to return it + + + +## Deploy for Production --- TODO: move to separate Sample App guide + +### Deploy your Application + +... for example, as documented in the [Deploy to Cloud Foundry](../../deployment/to-cf) guide. + +In essence, for a single-tenant app, run these commands in order: + +```sh +cds add approuter,xsuaa,hana,mta --for production +``` + +```sh +mbt build -t gen --mtar mta.tar +``` + +```sh +cf deploy gen/mta.tar +``` + +For a SaaS app, change first command to: + +```sh +cds add mtx,approuter,xsuaa,hana,mta --for production +``` + + + +### Bind to SAP Audit Log Service + +As shown in the configuration](#setup--configuration) section above, CAP provides out-of-the-box support for SAP Audit Log Service as the recommended target for collecting audit logs in production. + +TODO: Now I'd want us to say something like: + +1. **Given** you have access to instance of SAP Audit Log Service (→ in cockpit) +2. **Bind that to our app** using `cds bind` +3. Optionally: Test-drive it in hybrid setup +4. (Re-) Deploy your app ? + +> BTW: Can we do that without having local `.json` files? + + + +### Bind to SAP PDM Service + +TODO: Similar to above, I'd like us to say something like: + +1. **Given** you have access to instance of SAP Personal Data Manager (→ in cockpit) +2. **Bind that to our app** using `cds bind` + - care for `"authorities": ["$ACCEPT_GRANTED_AUTHORITIES"]` +3. Optionally: Test-drive it in hybrid setup +4. (Re-) Deploy your app ? + +> BTW: Creating and Asssigning Role Collections should be covered in their docs? + + From 9060788fdd549a4cd5a0b2426abd73e6a556b6ce Mon Sep 17 00:00:00 2001 From: D050513 Date: Tue, 1 Aug 2023 14:19:41 +0200 Subject: [PATCH 14/23] more --- guides/data-privacy/audit-logging.md | 6 +++++- guides/data-privacy/sample-app.md | 2 +- 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/guides/data-privacy/audit-logging.md b/guides/data-privacy/audit-logging.md index 210b69106..6cf446ce1 100644 --- a/guides/data-privacy/audit-logging.md +++ b/guides/data-privacy/audit-logging.md @@ -520,6 +520,8 @@ await audit.log ('ConfigurationModified', { }) ``` +Note: Configuration modified events are not (yet) logged out of the box. + ### Security Events @@ -547,6 +549,8 @@ await audit.log ('SecurityEvent', { }) ``` +Note: Security events are not (yet) logged out of the box. + ### AuditLogService @@ -620,7 +624,7 @@ The contents of aspect `LogEntry` are automatically applied and cannot be provid -## Service Providers +## Custom Implementation In addition, everybody could provide new implementations in the same way as we implement the mock variant: diff --git a/guides/data-privacy/sample-app.md b/guides/data-privacy/sample-app.md index 1ac7d3717..bb807278a 100644 --- a/guides/data-privacy/sample-app.md +++ b/guides/data-privacy/sample-app.md @@ -60,7 +60,7 @@ should we offer this? -## Deploy for Production --- TODO: move to separate Sample App guide +## Deploy for Production ### Deploy your Application From 587e7f58ae4b31442fc7e808d6a82fce444f1ca5 Mon Sep 17 00:00:00 2001 From: D050513 Date: Wed, 23 Aug 2023 11:01:26 +0200 Subject: [PATCH 15/23] some TODOs --- guides/data-privacy/index.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/guides/data-privacy/index.md b/guides/data-privacy/index.md index e751d445b..f3bcac3ae 100644 --- a/guides/data-privacy/index.md +++ b/guides/data-privacy/index.md @@ -8,6 +8,14 @@ permalink: guides/data-privacy/ status: released --- + + # Managing Data Privacy From 98a7c9d7dbeb9a12719b7207c959f687ada3477f Mon Sep 17 00:00:00 2001 From: D050513 Date: Wed, 23 Aug 2023 11:19:13 +0200 Subject: [PATCH 16/23] fix links --- guides/data-privacy/introduction.md | 2 +- node.js/audit-logging.md | 7 +++++++ 2 files changed, 8 insertions(+), 1 deletion(-) create mode 100644 node.js/audit-logging.md diff --git a/guides/data-privacy/introduction.md b/guides/data-privacy/introduction.md index 14d48aed5..8b66162c6 100644 --- a/guides/data-privacy/introduction.md +++ b/guides/data-privacy/introduction.md @@ -150,7 +150,7 @@ For more details on the `@PersonalData` vocabulary, see [this](https://github.co Data Subjects.drawio -Following the [best practice of separation of concerns](../../domain-modeling#separation-of-concerns), we do that in a separate file `db/data-privacy.cds`: +Following the [best practice of separation of concerns](../domain-modeling#separation-of-concerns), we do that in a separate file `db/data-privacy.cds`: db/data-privacy.cds {.sample-label} diff --git a/node.js/audit-logging.md b/node.js/audit-logging.md new file mode 100644 index 000000000..c7484dc77 --- /dev/null +++ b/node.js/audit-logging.md @@ -0,0 +1,7 @@ +--- +# status: released +--- + +# Audit Logging + +Under Construction. From 847c1b19ed903737e8261d3e7b1166b8bc32317d Mon Sep 17 00:00:00 2001 From: D050513 Date: Wed, 23 Aug 2023 11:47:44 +0200 Subject: [PATCH 17/23] update diagrams --- .../assets/Data-Privacy.drawio.svg | 131 +--------------- .../assets/Data-Subjects.drawio.svg | 141 +----------------- guides/data-privacy/introduction.md | 7 +- 3 files changed, 9 insertions(+), 270 deletions(-) diff --git a/guides/data-privacy/assets/Data-Privacy.drawio.svg b/guides/data-privacy/assets/Data-Privacy.drawio.svg index 8ca0efacc..9be4b07bc 100644 --- a/guides/data-privacy/assets/Data-Privacy.drawio.svg +++ b/guides/data-privacy/assets/Data-Privacy.drawio.svg @@ -1,127 +1,4 @@ - - - - - - - - - - - - - - -
-
-
- - Annotate - -
- Personal Data -
-
-
- - - Annotate... - - - - - - - - - -
-
-
- (Generic) -
- Audit Logging -
-
-
- - - (Generic)... - - - - - - - - - -
-
-
- Personal Data Management -
-
-
- - - Personal Data Manageme... - - - - - - - - - -
-
-
- Data Retention Management -
-
-
- - - Data Retention Managem... - - - - - - - - - - - - - - - - - -
-
-
- in CDS -
-
-
- - - in CDS - - - - - - - - - Text is not SVG - cannot display - - - - \ No newline at end of file + + + +
Annotate
Personal Data
Annotate...
(Generic)
Audit Logging
(Generic)...
Personal Data Management
Personal Data Manageme...
Data Retention Management
Data Retention Managem...
in CDS
in CDS
work in progress
work in pr...Text is not SVG - cannot display \ No newline at end of file diff --git a/guides/data-privacy/assets/Data-Subjects.drawio.svg b/guides/data-privacy/assets/Data-Subjects.drawio.svg index 8d1083df5..56d7075d7 100644 --- a/guides/data-privacy/assets/Data-Subjects.drawio.svg +++ b/guides/data-privacy/assets/Data-Subjects.drawio.svg @@ -1,137 +1,4 @@ - - - - - - - - - - - - - - - - - -
-
-
- Customers -
-
-
- - - Customers - - - - - - - - - - - - - - -
-
-
- Addresses -
-
-
- - - Addresses - - - - - - - - - -
-
-
- Orders -
-
-
- - - Orders - - - - - - - - -
-
-
- Data Subject -
-
-
- - - Data Subject - - - - - - - - -
-
-
- Data Subject Details -
-
-
- - - Data Subject Details - - - - - - - - -
-
-
- Related Data -
-
-
- - - Related Data - - - - - - - - - Text is not SVG - cannot display - - - - \ No newline at end of file + + + +
Customers
Customers
Addresses
Addresses
Orders
Orders
Data Subject
Data Subject
Data Subject Details
Data Subject Details
Other (e.g., Transactional Data)
Other (e.g., Transactional Data)
Billing Data
Billing DataText is not SVG - cannot display \ No newline at end of file diff --git a/guides/data-privacy/introduction.md b/guides/data-privacy/introduction.md index 8b66162c6..edbe09b15 100644 --- a/guides/data-privacy/introduction.md +++ b/guides/data-privacy/introduction.md @@ -31,7 +31,7 @@ ADDED FROM GUIDE #4: Compliance to data privacy regulations is an important requirement for all busines applications nowadays. CAP provides easy ways to designate personal data, as well as out-of-the-box integration with SAP BTP services, like SAP Personal Data Manager service. This greatly relieves application developers these tedious tasks and related efforts. ::: danger _TODO_ -linie zu DRM sollte gestrichelt sein +why is white not white? ::: Data Privacy.drawio.svg @@ -143,11 +143,6 @@ Use `@PersonalData` annotations to indicate entities and elements in your domain For more details on the `@PersonalData` vocabulary, see [this](https://github.com/SAP/odata-vocabularies/blob/main/vocabularies/PersonalData.md). -::: danger _TODO_ -- add BillingData to diagram -- adjust types to `DataSubject`, `DataSubjectDetails`, and `Other (e.g., Transactional Data)` -::: - Data Subjects.drawio Following the [best practice of separation of concerns](../domain-modeling#separation-of-concerns), we do that in a separate file `db/data-privacy.cds`: From 7c327a22701365aeb1399bf0492acb279c6a4bf7 Mon Sep 17 00:00:00 2001 From: D050513 Date: Wed, 23 Aug 2023 11:50:32 +0200 Subject: [PATCH 18/23] fix color --- guides/data-privacy/assets/Data-Privacy.drawio.svg | 2 +- guides/data-privacy/assets/Data-Subjects.drawio.svg | 2 +- guides/data-privacy/introduction.md | 4 ---- 3 files changed, 2 insertions(+), 6 deletions(-) diff --git a/guides/data-privacy/assets/Data-Privacy.drawio.svg b/guides/data-privacy/assets/Data-Privacy.drawio.svg index 9be4b07bc..30eda6c61 100644 --- a/guides/data-privacy/assets/Data-Privacy.drawio.svg +++ b/guides/data-privacy/assets/Data-Privacy.drawio.svg @@ -1,4 +1,4 @@ -
Annotate
Personal Data
Annotate...
(Generic)
Audit Logging
(Generic)...
Personal Data Management
Personal Data Manageme...
Data Retention Management
Data Retention Managem...
in CDS
in CDS
work in progress
work in pr...Text is not SVG - cannot display \ No newline at end of file +
Annotate
Personal Data
Annotate...
(Generic)
Audit Logging
(Generic)...
Personal Data Management
Personal Data Manageme...
Data Retention Management
Data Retention Managem...
in CDS
in CDS
work in progress
work in pr...Text is not SVG - cannot display \ No newline at end of file diff --git a/guides/data-privacy/assets/Data-Subjects.drawio.svg b/guides/data-privacy/assets/Data-Subjects.drawio.svg index 56d7075d7..7139076e3 100644 --- a/guides/data-privacy/assets/Data-Subjects.drawio.svg +++ b/guides/data-privacy/assets/Data-Subjects.drawio.svg @@ -1,4 +1,4 @@ -
Customers
Customers
Addresses
Addresses
Orders
Orders
Data Subject
Data Subject
Data Subject Details
Data Subject Details
Other (e.g., Transactional Data)
Other (e.g., Transactional Data)
Billing Data
Billing DataText is not SVG - cannot display \ No newline at end of file +
Customers
Customers
Addresses
Addresses
Orders
Orders
Data Subject
Data Subject
Data Subject Details
Data Subject Details
Other (e.g., Transactional Data)
Other (e.g., Transactional Data)
Billing Data
Billing DataText is not SVG - cannot display \ No newline at end of file diff --git a/guides/data-privacy/introduction.md b/guides/data-privacy/introduction.md index edbe09b15..83e31de26 100644 --- a/guides/data-privacy/introduction.md +++ b/guides/data-privacy/introduction.md @@ -30,10 +30,6 @@ ADDED FROM GUIDE #4: Compliance to data privacy regulations is an important requirement for all busines applications nowadays. CAP provides easy ways to designate personal data, as well as out-of-the-box integration with SAP BTP services, like SAP Personal Data Manager service. This greatly relieves application developers these tedious tasks and related efforts. -::: danger _TODO_ -why is white not white? -::: - Data Privacy.drawio.svg ::: tip From dad6175cef9d47b5ba52fac0b9f056618bc269a8 Mon Sep 17 00:00:00 2001 From: D050513 Date: Wed, 23 Aug 2023 11:52:24 +0200 Subject: [PATCH 19/23] add Audit Logging skeleton --- menu.md | 1 + 1 file changed, 1 insertion(+) diff --git a/menu.md b/menu.md index 8e8432d11..7a27061b4 100644 --- a/menu.md +++ b/menu.md @@ -147,6 +147,7 @@ - [TypeScript](node.js/typescript) - [Fiori Support](node.js/fiori) - [Best Practices](node.js/best-practices) +- [Audit Logging](node.js/auditlog) ### [Security](security/) From 4dc908dd82b357aead8bc61457b2d2287730d5c0 Mon Sep 17 00:00:00 2001 From: D050513 Date: Wed, 23 Aug 2023 12:07:03 +0200 Subject: [PATCH 20/23] remove broken link --- guides/data-privacy/sample-app.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/guides/data-privacy/sample-app.md b/guides/data-privacy/sample-app.md index bb807278a..ae2ae0e3e 100644 --- a/guides/data-privacy/sample-app.md +++ b/guides/data-privacy/sample-app.md @@ -64,7 +64,7 @@ should we offer this? ### Deploy your Application -... for example, as documented in the [Deploy to Cloud Foundry](../../deployment/to-cf) guide. +... for example, as documented in the Deploy to Cloud Foundry guide. --> TODO: add link to guide In essence, for a single-tenant app, run these commands in order: From 58c2c7fc5a36a7c53457ecda011aebc803125bbf Mon Sep 17 00:00:00 2001 From: D050513 Date: Wed, 23 Aug 2023 12:45:34 +0200 Subject: [PATCH 21/23] rm Node.js - Audit Logging from menu --- menu.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/menu.md b/menu.md index 7a27061b4..b58f06e92 100644 --- a/menu.md +++ b/menu.md @@ -147,7 +147,7 @@ - [TypeScript](node.js/typescript) - [Fiori Support](node.js/fiori) - [Best Practices](node.js/best-practices) -- [Audit Logging](node.js/auditlog) + ### [Security](security/) From a12c39f4376f61fdb215ffee955e60578bc5294b Mon Sep 17 00:00:00 2001 From: D050513 Date: Wed, 23 Aug 2023 13:11:30 +0200 Subject: [PATCH 22/23] fix more links --- guides/data-privacy/audit-log.md | 182 ---------------------------- guides/data-privacy/drm.md | 2 +- guides/data-privacy/introduction.md | 49 +++++--- guides/data-privacy/sample-app.md | 2 +- 4 files changed, 36 insertions(+), 199 deletions(-) delete mode 100644 guides/data-privacy/audit-log.md diff --git a/guides/data-privacy/audit-log.md b/guides/data-privacy/audit-log.md deleted file mode 100644 index e11dfad9b..000000000 --- a/guides/data-privacy/audit-log.md +++ /dev/null @@ -1,182 +0,0 @@ ---- -# layout: cookbook -shorty: Audit Log -synopsis: > - Enable and use audit-log capabilities with your CAP application. -breadcrumbs: - - Cookbook - - Data Privacy - - Audit Logging -status: released ---- - - -# Audit Logging with CAP - -{{ $frontmatter.synopsis }} - -## Introduction - -This section deals with Audit Logging for reading sensitive data and changes to personal data. As a prerequisite, you have [indicated entities and elements in your domain model, which will contain personal data](introduction#indicate-privacy). - - - -In CAP, audit logging can be handled mostly automatically by adding certain annotations to your business entity definitions and adding some configuration to your project. - -::: warning _❗ Data Subject and Data Object_
-For each audit log on a data object (like a Sales Order) a valid data subject (like a Customer) is needed. -The application has to clarify that this link between data object and data subject - which is typically induced by an annotation like -`Customer @PersonalData.FieldSemantics : 'DataSubjectID';` - is never broken. Thus, semantically correct audit logs can only be written on top of a semantically correct built application. - -Make sure that the data subject is a valid CAP entity, otherwise the metadata-driven automatism will not work. -::: - -## About the Audited Object - -### Data Subject - -In our case `Customers` is the main master data entity with the semantics of 'Data Subject'. - -```cds -using { cuid, Country } from '@sap/cds/common'; - - entity Customers : cuid, managed { - email : String; - firstName : String; - lastName : String; - creditCardNo : String; - dateOfBirth : Date; - postalAddress : Association to CustomerPostalAddress on postalAddress.Customer = $self; -} - -``` - -This entity is annotated in the _db/data-privacy.cds_ file. - -```cds - -annotate bookshop.Customers with @PersonalData : { - DataSubjectRole : 'Customer', - EntitySemantics : 'DataSubject' -} -{ - ID @PersonalData.FieldSemantics : 'DataSubjectID'; - emailAddress @PersonalData.IsPotentiallyPersonal; - firstName @PersonalData.IsPotentiallyPersonal; - lastName @PersonalData.IsPotentiallyPersonal; - creditCardNo @PersonalData.IsPotentiallySensitive; - dateOfBirth @PersonalData.IsPotentiallyPersonal; -} - -annotate bookshop.Customers with @AuditLog.Operation : { - Read : true, - Insert : true, - Update : true, - Delete : true -}; - -``` - -Here we again have the four levels of annotations as already described in the chapter [Indicate Personal Data in Your Domain Model](introduction#indicate-privacy). - -When you've annotated your (business) entity like this, the audit logs for read access and data modifications will be written automatically by the underlying CAP framework. - -In the context of audit logging, the `@PersonalData.IsPotentiallyPersonal` field-level annotation is relevant for inducing audit logs for _Insert_, _Update_, and _Delete_, whereas the `@PersonalData.IsPotentiallySensitive` annotation is relevant for _Read_ access audit logs. - -::: warning _Warning_ -The `@PersonalData.IsPotentiallySensitive` annotation induces an audit log for each and every _Read_ access. ---- Only use this annotation in [relevant cases](https://ec.europa.eu/info/law/law-topic/data-protection/reform/rules-business-and-organisations/legal-grounds-processing-data/sensitive-data/what-personal-data-considered-sensitive_en). ---- Avoid unnecessary logging activities in your application. -::: - - -### Data Subject Details - -In the first example, the audited object was identical to the data subject, but this is not always the case. - -In many cases you have additional master data describing more details of the data subject stored in a separate entity. -In our terminology this has the semantics 'Data Subject Details'. - -In our example we have the additional entity `CustomerPostalAddress` which contains additional master data belonging to a certain 'Customer', but which are stored in a separate entity, for better clarity or better separation of concerns. - -```cds - -entity CustomerPostalAddress : cuid, managed { - Customer : Association to one Customers; - street : String(128); - town : String(128); - country : Country; - someOtherField : String(128); -}; - -``` - -This entity is annotated in the _db/data-privacy.cds_ file. - -```cds -annotate bookshop.CustomerPostalAddress with @PersonalData : { - DataSubjectRole : 'Customer', - EntitySemantics : 'DataSubjectDetails' -} -{ - Customer @PersonalData.FieldSemantics : 'DataSubjectID'; - street @PersonalData.IsPotentiallyPersonal; - town @PersonalData.IsPotentiallyPersonal; - country @PersonalData.IsPotentiallyPersonal; -} - -annotate bookshop.CustomerPostalAddress with @AuditLog.Operation : { - Read : true, - Insert : true, - Update : true, - Delete : true -}; - -``` - -Very similarly to the section on 'Data Subject' this entity is as well annotated in four levels. -More details on these annotations can be found in the chapter [Indicate Personal Data in Your Domain Model](introduction#indicate-privacy). - -### Transactional Data - -In the section on 'Data Subject' and 'Data Subject Details' we have seen, how to annotate the master data entities carrying the semantical information of the 'Data Subject'. - -Now we have a look at classical transactional data. - -In the Personal Data Terminology all transactional data like 'Sales Orders', 'Shipments', 'Payments' are summarizes under the classification 'Other', which means they are relevant for Data Privacy, but they are neither 'Data Subject' nor 'Data Subject Details'. -More details on this Terminology can be found in the chapter [Indicate Personal Data in Your Domain Model](introduction#indicate-privacy). - -In our example we have the entity 'Orders' - -```cds -entity Orders : cuid, managed { - OrderNo : String @title:'Order Number'; //> readable key - Items : Composition of many OrderItems on Items.parent = $self; - currency : Currency; - Customer : Association to Customers; - personalComment : String; -} -``` - -To ensure proper audit logging we annotate using the usual four levels as described in the chapter [Indicate Personal Data in Your Domain Model](introduction#indicate-privacy). - -```cds -annotate bookshop.Orders with @PersonalData.EntitySemantics : 'Other' -{ - ID @PersonalData.FieldSemantics : 'ContractRelatedID'; - Customer @PersonalData.FieldSemantics : 'DataSubjectID'; - personalComment @PersonalData.IsPotentiallyPersonal; -} -annotate bookshop.Orders with @AuditLog.Operation : { - Read : true, - Insert : true, - Update : true, - Delete : true -}; -``` - -Finally, we annotate all standard operations (`Read`, `Insert`, `Update`, `Delete`) as relevant for the audit log - which should be the default case for most of the relevant business entities. - -
- - \ No newline at end of file diff --git a/guides/data-privacy/drm.md b/guides/data-privacy/drm.md index f6f2920e5..177cbba1f 100644 --- a/guides/data-privacy/drm.md +++ b/guides/data-privacy/drm.md @@ -7,7 +7,7 @@ breadcrumbs: - Cookbook - Data Privacy - Data Retention -#status: released +status: released --- diff --git a/guides/data-privacy/introduction.md b/guides/data-privacy/introduction.md index 83e31de26..c529ecd84 100644 --- a/guides/data-privacy/introduction.md +++ b/guides/data-privacy/introduction.md @@ -66,10 +66,10 @@ The most essential requests you have to answer are those in the table below, wit | Question / Request | Discipline | | ------------------------------------------- | -------------------------------------------------------- | -| *When was personal data stored/changed?* | → [Audit Logging](#audit-logging) | -| *What data about me do you have stored?* | → [Personal Data Management](#sap-personal-data-manager) | +| *When was personal data stored/changed?* | → [Audit Logging](audit-logging.md) | +| *What data about me do you have stored?* | → [Personal Data Management](pdm.md) | | → "Right of access to personal data" | | -| *Please delete all personal data about me!* | → [Retention Management](#sap-data-retention-manager) | +| *Please delete all personal data about me!* | → [Retention Management](drm.md) | | → "Right to be forgotten" | |
@@ -80,15 +80,16 @@ While CAP and SAP BTP services greatly facilitate fulfilling the obligations rel ::: - -## Indicate Personal Data in Your Domain Model { #indicate-privacy } +## @PersonalData { #indicate-privacy } See full sample in [cloud-cap-samples](https://github.com/SAP-samples/cloud-cap-samples/tree/gdpr/gdpr). + + ### Base Model In the remainder of this guide, we'll use this domain model as the base to add data privacy and audit logging. @@ -131,12 +132,15 @@ entity Orders : cuid, managed { } ``` + ### Annotating Personal Data Let's annotate our data model to identify personal data. In essence, in all our entities we search for elements which carry personal data, such as person names, birth dates, etc., and tag them accordingly. All found entities are classified as either *Data Subjects*, *Data Subject Details* or *Other*. Use `@PersonalData` annotations to indicate entities and elements in your domain model, which will contain personal data. +This allows you to manage the data privacy-related actions on a fine granular level only using metadata definitions with annotations and without any need of implementation. + For more details on the `@PersonalData` vocabulary, see [this](https://github.com/SAP/odata-vocabularies/blob/main/vocabularies/PersonalData.md). Data Subjects.drawio @@ -197,10 +201,9 @@ You can annotate different CDS artifacts, such as entities or fields. The data p - The **field-level annotations** identify elements containing personal data. -### Entity-Level Annotations -#### EntitySemantics +### .EntitySemantics Entity-level annotations indicate which entities are relevant for data privacy. @@ -225,7 +228,8 @@ Make sure that the data subject is a valid CAP entity, otherwise the metadata-dr ::: -#### DataSubjectRole + +### .DataSubjectRole ```cds @PersonalData.DataSubjectRole: '' @@ -243,27 +247,42 @@ annotate Customers with @PersonalData: { ``` -### Key-Level Annotations + +### .FieldSemantics Key-level annotations indicate the corresponding key information. ```cds -@PersonalData.FieldSemantics: 'DataSubjectID' +annotate Customers with { + ID @PersonalData.FieldSemantics: 'DataSubjectID' +}; ``` This key information consists of the `DataSubject` (= Person) and its identifiers and the corresponding personal documents (such as Order, Consent, ...) and its identifiers. The latter is always captured implicitly, so we mainly have to specify the type and the key of the `DataSubject`. -### Field-Level Annotations -Field-level annotations tag which fields are relevant for data privacy in detail. +### .IsPotentiallyPersonal + +`@PersonalData.IsPotentiallyPersonal` tags which fields are personal and, for example, require audit logs in case of modification. ```cds -@PersonalData.IsPotentiallyPersonal -@PersonalData.IsPotentiallySensitive +annotate Customers with { + firstName @PersonalData.IsPotentiallyPersonal +}; ``` -This allows you to manage the data privacy-related actions on a fine granular level only using metadata definitions with annotations and without any need of implementation. + + +### .IsPotentiallySensitive + +`@PersonalData.IsPotentiallySensitive` tags which fields are sensitive and, for example, require audit logs in case of access. + +```cds +annotate Customers with { + ID @PersonalData.FieldSemantics: 'DataSubjectID' +}; +``` ::: warning _Warning_ Please see [Audit Logging](./audit-logging.md) for implications before marking data as sensitive. diff --git a/guides/data-privacy/sample-app.md b/guides/data-privacy/sample-app.md index ae2ae0e3e..41514574d 100644 --- a/guides/data-privacy/sample-app.md +++ b/guides/data-privacy/sample-app.md @@ -7,7 +7,7 @@ breadcrumbs: - Cookbook - Data Privacy - Sample App -#status: released +status: released --- From c1f0055c8a51e914be4e90423b91981b3cc88692 Mon Sep 17 00:00:00 2001 From: D050513 Date: Wed, 23 Aug 2023 13:27:28 +0200 Subject: [PATCH 23/23] fix broken links --- about/features.md | 2 +- get-started/PARKED-using-mock-servers.md | 2 +- guides/security/data-protection-privacy.md | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/about/features.md b/about/features.md index 68b968aa1..82c58a671 100644 --- a/about/features.md +++ b/about/features.md @@ -124,7 +124,7 @@ Following is an index of the features currently covered by CAP, with status and | [Temporal Data](../guides/temporal-data) | | | | | [Dynamic Extensibility](../guides/extensibility/) | | | | | Monitoring / Logging [[Node.js](../node.js/cds-log)\|[Java](../java/observability#logging)] | | | | -| Audit Logging [[Node.js](../guides/data-privacy/audit-log)\|[Java](../java/auditlog)] | | | | +| Audit Logging [[Node.js](../guides/data-privacy/audit-logging)\|[Java](../java/auditlog)] | | | |
diff --git a/get-started/PARKED-using-mock-servers.md b/get-started/PARKED-using-mock-servers.md index 5e63c3780..7645d13b1 100644 --- a/get-started/PARKED-using-mock-servers.md +++ b/get-started/PARKED-using-mock-servers.md @@ -259,7 +259,7 @@ GET http://localhost:4004/api-business-partner/reset * [Serving OData APIs](../advanced/odata) * [Serving SAP Fiori UIs](../advanced/fiori) * [Deploying to the Cloud](../guides/deployment/) -* [Adding Audit Logging](../guides/data-privacy/audit-log) +* [Adding Audit Logging](../guides/data-privacy/audit-logging) * [Using Monitoring](../advanced/monitoring) & Analytics * Adding Tests * [Using CI/CD](../guides/deployment/cicd) diff --git a/guides/security/data-protection-privacy.md b/guides/security/data-protection-privacy.md index 04b2ca99e..80196cb83 100644 --- a/guides/security/data-protection-privacy.md +++ b/guides/security/data-protection-privacy.md @@ -57,7 +57,7 @@ CAP provides several [features](../data-privacy/) to help applications meet DPP- - The [Personal Data Management (PDM)](../data-privacy/pdm) integration has a configurable **retrieval function**, which can be used to inform data subjects about personal data stored related to them. - CAP also provides a *fully model-driven* approach to track **changes in personal data** or **read access to sensitive personal data** in the audit log. -Having [declared personal data](../data-privacy/introduction#indicate-privacy) in your model, CAP automatically triggers corresponding [audit log events](../data-privacy/audit-log). +Having [declared personal data](../data-privacy/introduction#indicate-privacy) in your model, CAP automatically triggers corresponding [audit log events](../data-privacy/audit-logging). ::: warning ❗ So far, applications have to integrate [SAP Data Retention Manager](https://help.sap.com/docs/DATA_RETENTION_MANAGER) to implement an adequate **erasure function** for personal data out of retention period.