Merge pull request #701 from JoelGauci/feature/firestore-facade

Cloud Firestore facade reference

CODEOWNERS

@@ -13,6 +13,7 @@
 /references/data-api                    @danistrebel
 /references/data-converters-shared-flow @tyayers
 /references/dutch-healthcare            @seymen
+/references/firestore-facade            @joelgauci
 /references/gcp-sa-auth-shared-flow     @danistrebel
 /references/identity-facade             @joelgauci
 /references/java-callout                @omidtahouri

README.md

@@ -59,6 +59,8 @@ further to fit a particular use case.
   popular API auth schemes
 - [reCAPTCHA enterprise](references/recaptcha-enterprise) - A reference for
   API protection against bot leveraging reCAPTCHA enterprise
+- [Firestore Facade](references/firestore-facade) - Reference implementation
+  for a long term caching/storage solution based on Cloud Firestore
 
 ## Tools
 

references/firestore-facade/.gitignore

@@ -0,0 +1 @@
+AM-SetFirestoreMock.xml

references/firestore-facade/README.md

@@ -0,0 +1,163 @@
+# Firestore Facade in Apigee X/hybrid
+
+[Cloud Firestore](https://firebase.google.com/docs/firestore)
+is a flexible, scalable database for mobile, web, and server development
+from Firebase and Google Cloud.
+Apigee can act as a facade in front of Cloud Firestore, to implement the
+following use cases:
+
+- Long term storage: using Cloud Firestore to cache documents on long term
+- Data as a Service (DaaS) pattern: some data of a Cloud Firestore database
+(db) are exposed as an API
+
+The use case that is proposed in the Firestore facade reference is the one
+based on long term storage.
+Indeed, in situations where you need a caching mechanism for data,
+which must be cached for more than 30 days (max TTL for caching data in
+Apigee X) a storage solution is required.
+Cloud Firestore is the perfect solution to consider in case of long
+term storage needs.
+
+## How it works?
+
+Two Apigee **sharedflows** are used as a facade in front of a Cloud Firestore
+db. A **cache key** is used to lookup and populate data into the Cloud
+Firestore db.
+
+By default, the cache key is defined as the following:
+
+```cacheKey = base64Encoding(basePath + '/' + pathSuffix)```
+
+This can be modified depending on the use case you need to implement
+
+The 2 Shared Flows on Apigee, acting as a facade in front of a Cloud
+Firestore db are:
+
+- ```sf-firestore-facade-lookup-v1```: to lookup into a Cloud Firestore db
+- ```sf-firestore-facade-populate-v1```: to populate a Cloud Firestore db
+with backend responses
+
+## Apigee runtime options
+
+The Firestore facade reference can be deployed on both Apigee X and
+hybrid. This reference would also work with Apigee Edge if the Service Account
+token is obtained through the sharedflows, which invoke the Cloud Firestore
+API endpoint (```sf-firestore-facade-lookup-v1``` and
+```sf-firestore-facade-populate-v1```).
+
+## Dependencies
+
+- [Maven](https://maven.apache.org/)
+- [NodeJS](https://nodejs.org/en/) LTS version or above
+- An Apigee organization
+- [Google Cloud Platform](https://cloud.google.com/) (GCP)
+- **plantuml.1.2020.23.jar** (only if you want to execute ```generate-docs.sh```)
+
+This reference leverages Apigee and Cloud Firestore.
+Therefore, it is important to note that:
+
+- A GCP service account is needed by the Apigee configuration
+to securely invoke the Cloud Firestore API endpoint.
+This service account is created during the deployment process on the GCP
+project you are currently using: the ```pipeline.sh``` will attempt to create
+a service account only if it doesn't exist.
+
+In case you want to create this service account manually - or with Terraform,
+please note that the role ```roles/firestore.admin``` must be granted
+to it.
+
+## Quick start
+
+### Apigee X / hybrid
+
+    export APIGEE_X_ORG=xxx
+    export APIGEE_X_ENV=xxx
+    export APIGEE_X_HOSTNAME=xxx
+
+    ./pipeline.sh
+
+## Deployment options
+
+There are 2 options to deploy the Firestore facade reference.
+
+```Option 1``` is the default deployment option.
+
+### Option 1: Cloud Firestore is mocked
+
+    export IS_FIRESTORE_MOCK_ENABLED=true
+
+With this option (*default*) no Cloud Firestore db is
+invoked but a mock response similar to a real one is delivered.
+
+**Functional tests are executed only when this deployment option is
+selected**.
+
+### Option 2: Cloud Firestore is used
+
+    export IS_FIRESTORE_MOCK_ENABLED=false
+
+With this option, it is not possible to execute functional tests.
+Nevertheless, you can request the Firestore facade API using the
+HTTP client of your choice.
+
+Note that this reference implementation assumes the same GCP Project
+is used for Firestore. If that's not the case, please
+update the ```<URL>``` in the target endpoint.
+
+## Script outputs
+
+The pipeline script deploys on Apigee X / hybrid two
+**sharedflows** (```sf-firestore-facade-lookup-v1```
+and ```sf-firestore-facade-populate-v1```)
+containing the full configuration of the Firestore
+facade reference.
+
+An API Proxy, acting as a data proxy, is also part of the reference:
+
+- ```firestore-data-proxy-v1```: a data proxy, which calls the two
+**sharedflows** accordingly.
+
+The target endpoint of this proxy is [mocktarget.apigee.net/echo](https://mocktarget.apigee.net/echo)
+
+## Cloud Firestore & Apigee Sequence Diagram
+
+The following sequence diagram provides all the interactions between:
+
+- End-user
+- Client app
+- Apigee: sharedflows and data proxy
+- Cloud Firestore
+- Backend
+
+This sequence diagram is available as a
+[text file](./diagram/sequence-firestore-facade.txt).
+
+If needed, you can modify this file and re-generate the related picture (png)
+using the following command:
+
+    ./generate_docs.sh
+
+Here is the original sequence diagram:
+
+![Firestore facade](./diagram/sequence-firestore-facade.png "Seq. Diagram")
+
+## Testing the Firestore facade reference
+
+In order to test this reference, you need an HTTP client.
+If you execute your test using a real Cloud Firestore db, you have to create
+it on your Google Cloud platform. You can use the ```(default)``` database.
+
+### cURL command
+
+Using cURL, the request is the following:
+
+    curl https://${APIGEE_X_HOSTNAME}/v1/firestore/users/123
+
+Connect to your Cloud Firestore instance to check that data (response
+of the Apigee Mock API) have been insterted into your Cloud Firestore db (
+cf. **data** on the right side of the picture):
+
+![default database](./images/cloud-firestore.png "default db in Cloud Firestore")
+
+As you can notice, both ```basePath``` (**collectionId** in Cloud Firestore) and
+```pathSuffix``` (**documentId** in Cloud Firestore) are base64 encoded.

references/firestore-facade/diagram/sequence-firestore-facade.txt

@@ -0,0 +1,50 @@
+@startuml
+
+title "Firestore Facade in Apigee X/hybrid"
+
+actor User as u
+entity "Client App" as b
+entity "Cloud\nFirestore database" as gfd
+box "Apigee API Platform" #LightBlue
+entity "API Proxy\nfirestore-data-proxy-v1" as fdp
+entity "SharedFlow\nsf-firestore-facade-lookup-v1" as ffl
+entity "SharedFlow\nsf-firestore-facade-populate-v1" as ffp
+end box
+participant "Backend" as backend
+
+u -> b: User interaction
+b -> b: App activity
+b -> fdp: Access the firestore facade api
+
+note over gfd,ffp: "Apigee API proxy and shared flows acting as a facade in front of Cloud Firestore db" 
+
+fdp -> ffl: Lookup data from the Cloud Firestore db based on base path, path suffix and encoding type (base64 only)
+ffl -> ffl: calculate the document key (cache key)\ncacheKey = encodingType( basePath + pathSuffix)
+ffl -> gfd: Lookup shared flow acting as a facade with Cloud Firestore, using the cache key\ncall is executed using an ID token
+gfd -> gfd: lookup in the Cloud Firestore db using cache key
+
+opt Data retrieved from Cloud Firestore
+    gfd -> ffl: data is retrieved from Cloud Firestore (lookup status)
+    ffl -> ffl: set context variables:\nflow.lookup.hit = true \nflow.lookup.content = "<json content retrieved from cache>" \nflow.lookup.status.code = 200
+    ffl -> fdp: shared flow response
+end
+
+opt Data is NOT retrieved from Cloud Firestore
+    gfd -> ffl: data is not retrieved from Cloud Firestore (lookp status)
+    ffl -> ffl: set context variables:\nflow.lookup.hit = false \nflow.lookup.content = "none" \nflow.lookup.status.code > 399
+    ffl -> fdp: shared flow response
+    fdp -> backend: request is forwarded to the backend API
+    backend -> fdp: backend response
+    fdp -> ffp: Populate data from the Cloud Firestore db based on base path, path suffix and encoding type (base64 only)
+    ffp -> ffp: calculate the cache key\ncacheKey = encodingType( basePath + pathSuffix)
+    ffp -> gfd: Populate shared flow acting as a facade with Cloud Firestore, using the cache key\ncall is executed using an ID token
+    gfd -> gfd: populate backend response in the Firestore db using cache key
+    gfd -> ffp: firestore populate status
+    ffp -> ffp: set context variables:\nflow.populate.content = true \nflow.populate.status.code = 200 \nflow.populate.cachekey = <cacheKey> \nflow.populate.extcache.documentid = <firestore documentId> \nflow.populate.extcache.collectionid = <firestore collectionId>
+    ffp -> fdp: shared flow response
+
+end
+
+fdp -> b: JSON response is sent back to the app (200 OK)
+
+@enduml

references/firestore-facade/diagram/sequence-firestore-kms-facade.txt

@@ -0,0 +1,60 @@
+@startuml
+
+title "Firestore Facade in Apigee X/hybrid"
+
+actor User as u
+entity "Client App" as b
+entity "Cloud\nFirestore database" as gfd
+entity "Cloud\nKMS" as kms
+box "Apigee API Platform" #LightBlue
+entity "API Proxy\nfirestore-data-proxy-v1" as fdp
+entity "SharedFlow\nsf-firestore-facade-lookup-v1" as ffl
+entity "SharedFlow\nsf-firestore-facade-populate-v1" as ffp
+end box
+participant "Backend" as backend
+
+u -> b: User interaction
+b -> b: App activity
+b -> fdp: Access the firestore facade api
+
+note over gfd,ffp: "Apigee API proxy and shared flows acting as a facade in front of Cloud Firestore db" 
+
+fdp -> ffl: Lookup data from the Cloud Firestore db based on base path, path suffix and encoding type (base64 only)
+ffl -> ffl: calculate the cache key\ncacheKey = encodingType( basePath + pathSuffix)
+ffl -> gfd: Lookup shared flow acting as a facade with Cloud Firestore, using the cache key\ncall is executed using an ID token
+gfd -> gfd: lookup in the Cloud Firestore db using cache key
+
+opt Data retrieved from Cloud Firestore
+    gfd -> ffl: (encrypted) data is retrieved from Cloud Firestore (lookup status)
+    ffl -> ffl: extract the encrypted dek from the response (encDek)
+    ffl -> kms: call cloud kms api to decrypt the encrypted dek
+    kms -> ffl: return the decrypted dek
+    ffl -> ffl: decrypt the encrypted content using the dek (decrypted)
+    ffl -> ffl: set context variables:\nflow.lookup.hit = true \nflow.lookup.content = "<json content retrieved from cache>" \nflow.lookup.status.code = 200
+    ffl -> fdp: shared flow response
+end
+
+opt Data is NOT retrieved from Cloud Firestore
+    gfd -> ffl: data is not retrieved from Cloud Firestore (lookp status)
+    ffl -> ffl: set context variables:\nflow.lookup.hit = false \nflow.lookup.content = "none" \nflow.lookup.status.code > 399
+    ffl -> fdp: shared flow response
+    fdp -> backend: request is forwarded to the backend API
+    backend -> fdp: backend response
+    fdp -> fdp: generate a random encryption key
+    fdp -> fdp: encrypt the content of the response
+    fdp -> kms: call cloud kms api to encrypt the encrypted dek
+    kms -> fdp: return the encrypted dek
+    fdp -> fdp: prepare the content to be cached = encrypted content + encrypted dek (envelope encryption pattern)
+    fdp -> ffp: Populate data from the Cloud Firestore db based on base path, path suffix and encoding type (base64 only)
+    ffp -> ffp: calculate the cache key\ncacheKey = encodingType( basePath + pathSuffix)
+    ffp -> gfd: Populate shared flow acting as a facade with Cloud Firestore, using the cache key\ncall is executed using an ID token
+    gfd -> gfd: populate backend response in the Firestore db using cache key
+    gfd -> ffp: firestore populate status
+    ffp -> ffp: set context variables:\nflow.populate.content = true \nflow.populate.status.code = 200 \nflow.populate.cachekey = <cacheKey> \nflow.populate.extcache.documentid = <firestore documentId> \nflow.populate.extcache.collectionid = <firestore collectionId>
+    ffp -> fdp: shared flow response
+
+end
+
+fdp -> b: JSON response is sent back to the app (200 OK)
+
+@enduml

references/firestore-facade/firestore-data-proxy-v1/apiproxy/firestore-data-proxy-v1.xml

@@ -0,0 +1,14 @@
+<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
+<!--
+ Copyright 2023 Google LLC
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+      http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<APIProxy revision="1" name="firestore-data-proxy-v1"/>

references/firestore-facade/firestore-data-proxy-v1/apiproxy/policies/AM-PathSuffixFalse.xml

@@ -0,0 +1,20 @@
+<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
+<!--
+ Copyright 2023 Google LLC
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+      http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<AssignMessage name="AM-PathSuffixFalse">
+    <AssignVariable>
+        <Name>target.copy.pathsuffix</Name>
+        <Value>false</Value>
+    </AssignVariable>
+    <AssignTo createNew="false" transport="http" type="request"/>
+</AssignMessage>

references/firestore-facade/firestore-data-proxy-v1/apiproxy/policies/AM-SetResponse.xml

@@ -0,0 +1,25 @@
+<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
+<!--
+ Copyright 2023 Google LLC
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+      http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<AssignMessage name="AM-SetResponse">
+    <AssignVariable>
+        <Name>message.content</Name>
+        <Ref>flow.lookup.content</Ref>
+    </AssignVariable>
+    <AssignVariable>
+        <Name>message.header.content-type</Name>
+        <Value>application/json</Value>
+    </AssignVariable>
+    <IgnoreUnresolvedVariables>true</IgnoreUnresolvedVariables>
+    <AssignTo createNew="false" transport="http" type="response"/>
+</AssignMessage>

references/firestore-facade/firestore-data-proxy-v1/apiproxy/policies/FC-LookupExternalCache.xml

@@ -0,0 +1,21 @@
+<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
+<!--
+ Copyright 2023 Google LLC
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+      http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<FlowCallout name="FC-LookupExternalCache">
+    <Parameters>
+        <Parameter name="encodingType">base64</Parameter>
+        <Parameter name="basePath">{proxy.basepath}</Parameter>
+        <Parameter name="pathSuffix">{proxy.pathsuffix}</Parameter>
+    </Parameters>
+    <SharedFlowBundle>sf-firestore-facade-lookup-v1</SharedFlowBundle>
+</FlowCallout>