Features for v11.2.0

CHANGELOG.md

@@ -1,5 +1,24 @@
 # GraphDB Helm chart release notes
 
+## Version 11.2.0
+
+### New
+
+- Added CronJob for scheduling GraphDB backups. The CronJob supports both local and cloud backups.
+    - Added new configurations under `backup`: `backup.enabled` for toggling the backup CronJob, `backup.type` for selecting between local and
+      cloud and more.
+    - Local backups support saving the GraphDB backup archives in volume from an existing persistent volume claim, configured
+      with `backup.local`
+    - Cloud backups support uploading the GraphDB backup archives in one of the supported cloud object storage services, configured with `backup.cloud`
+    - Added a new example under [examples/backup-local](examples/backup-local) showing how to use the local backup feature with `backup.local`
+- Added a new property `security.provisioner.passwordHash` to define an initial password for the provisioner user as a bcrypt hash.
+- Configured `graphdb.extra.plugins` to load plugins from `/opt/graphdb/home/extra-plugins` by default
+
+### Fixed
+
+- Updated the GraphDB containers to explicitly use `/tmp` as a working directory to avoid permission errors due to the
+  default security context's `readOnlyRootFilesystem` when the container has a starting folder different from `/tmp`.
+
 ## Version 11.1.5
 
 ### New

Chart.yaml

@@ -5,7 +5,7 @@ apiVersion: v2
 name: graphdb
 description: GraphDB is a highly efficient, scalable and robust graph database with RDF and SPARQL support.
 type: application
-version: 11.1.5
+version: 11.2.0
 appVersion: 10.7.4
 kubeVersion: ^1.26.0-0
 home: https://graphdb.ontotext.com/

README.md

@@ -1,7 +1,7 @@
 # Helm Chart for GraphDB
 
 [![CI](https://github.com/Ontotext-AD/graphdb-helm/actions/workflows/ci.yml/badge.svg)](https://github.com/Ontotext-AD/graphdb-helm/actions/workflows/ci.yml)
-![Version: 11.1.5](https://img.shields.io/badge/Version-11.1.5-informational?style=flat-square)
+![Version: 11.2.0](https://img.shields.io/badge/Version-11.2.0-informational?style=flat-square)
 ![AppVersion: 10.7.4](https://img.shields.io/badge/AppVersion-10.7.4-informational?style=flat-square)
 
 <!--
@@ -310,6 +310,19 @@ IMPORTANT: This is generated by helm-docs, do not attempt modifying it on hand a
 | annotations | object | `{}` |  |
 | args | list | `[]` |  |
 | automountServiceAccountToken | bool | `false` |  |
+| backup.cloud.bucketUri | string | `""` |  |
+| backup.enabled | bool | `false` |  |
+| backup.extraEnv | list | `[]` |  |
+| backup.extraEnvFrom | list | `[]` |  |
+| backup.failedJobsHistoryLimit | int | `3` |  |
+| backup.local.existingPVC | string | `""` |  |
+| backup.local.mountPath | string | `"/opt/graphdb/backups/"` |  |
+| backup.options.backupSystemData | bool | `true` |  |
+| backup.optionsSecret.existingSecret | string | `""` |  |
+| backup.optionsSecret.secretKey | string | `"backup_options.json"` |  |
+| backup.schedule | string | `"@midnight"` |  |
+| backup.successfulJobsHistoryLimit | int | `3` |  |
+| backup.type | string | `"cloud"` |  |
 | cluster.clusterCreationTimeout | int | `60` |  |
 | cluster.config.configmapKey | string | `"cluster-config.json"` |  |
 | cluster.config.existingConfigmap | string | `""` |  |
@@ -384,7 +397,7 @@ IMPORTANT: This is generated by helm-docs, do not attempt modifying it on hand a
 | ingress.path | string | `""` |  |
 | ingress.pathType | string | `"Prefix"` |  |
 | ingress.tls.enabled | bool | `false` |  |
-| ingress.tls.secretName | string | `nil` |  |
+| ingress.tls.secretName | string | `""` |  |
 | initContainerDataPermissions.enabled | bool | `false` |  |
 | initContainerDataPermissions.securityContext.runAsNonRoot | bool | `false` |  |
 | initContainerDataPermissions.securityContext.runAsUser | int | `0` |  |
@@ -588,6 +601,7 @@ IMPORTANT: This is generated by helm-docs, do not attempt modifying it on hand a
 | security.initialUsers.users | object | `{}` |  |
 | security.provisioner.existingSecret | string | `""` |  |
 | security.provisioner.password | string | `"iHaveSuperpowers"` |  |
+| security.provisioner.passwordHash | string | `""` |  |
 | security.provisioner.tokenKey | string | `"GRAPHDB_AUTH_TOKEN"` |  |
 | security.provisioner.username | string | `"provisioner"` |  |
 | securityContext.allowPrivilegeEscalation | bool | `false` |  |
@@ -638,6 +652,16 @@ DNS is broken. This is a common issue with Minikube between system restarts or w
 Minikube driver is used. Please refer to
 https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/.
 
+**Filesystem provisioning errors (in Multi-Node Minikube Cluster)**
+
+When expanding your Minikube cluster from one to two or more nodes to deploy different GraphDB
+instances across multiple nodes to ensure high availability, you may encounter errors when
+setting up persistent storage. These issues are due to implementation problems with the storage
+provisioner included with Minikube. To resolve this, you need to adjust your environment accordingly.
+Follow the steps outlined in the official Minikube documentation under the ["CSI Driver and Volume
+Snapshots"](https://minikube.sigs.k8s.io/docs/tutorials/volume_snapshots_and_csi/) section, specifically
+in the "Multi-Node Clusters" chapter.
+
 ## Maintainers
 
 | Name | Email | Url |

README.md.gotmpl

@@ -323,6 +323,16 @@ DNS is broken. This is a common issue with Minikube between system restarts or w
 Minikube driver is used. Please refer to
 https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/.
 
+**Filesystem provisioning errors (in Multi-Node Minikube Cluster)**
+
+When expanding your Minikube cluster from one to two or more nodes to deploy different GraphDB
+instances across multiple nodes to ensure high availability, you may encounter errors when
+setting up persistent storage. These issues are due to implementation problems with the storage
+provisioner included with Minikube. To resolve this, you need to adjust your environment accordingly.
+Follow the steps outlined in the official Minikube documentation under the ["CSI Driver and Volume
+Snapshots"](https://minikube.sigs.k8s.io/docs/tutorials/volume_snapshots_and_csi/) section, specifically
+in the "Multi-Node Clusters" chapter.
+
 ## Maintainers
 
 {{ template "chart.maintainersTable" . }}

examples/aws/backup/README.md

@@ -0,0 +1,129 @@
+# AWS Cloud Backup Examples
+
+The examples here show different ways of configuring the cloud backup CronJob to send backups in [AWS S3](https://aws.amazon.com/s3/) and
+how to authenticate in AWS:
+
+* Using inline credentials
+* Using credentials from a Secret
+* Using ServiceAccount and [IRSA](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html)
+
+## Inline Credentials
+
+⚠️ **Warning:** This example is not recommended. Instead of using long term credentials, consider using a ServiceAccount.
+
+The easiest and most straightforward way is to configure long-term credentials in the S3 bucket URI property:
+
+```yaml
+backup:
+  enabled: true
+  type: cloud
+  cloud:
+    bucketUri: s3:///<bucket>/<folder>/${BACKUP_NAME}?region=<region>&AWS_ACCESS_KEY_ID=<key>&AWS_SECRET_ACCESS_KEY=<secret>
+```
+
+Where:
+
+- `<bucket>` is the name of your bucket.
+- `<folder>` is an optional path argument, you can skip it for storing the backup at the root.
+- `${BACKUP_NAME}` is replaced with an auto-generated name. Optionally, you can use a fixed name.
+- `<key>` is the key of your static AWS credentials.
+- `<secret>` is the secret of your static AWS credentials.
+
+## Credentials from a Secret
+
+⚠️ **Warning:** This example is not recommended. Instead of using long-term credentials, consider using a ServiceAccount.
+
+In order to avoid providing static credentials in `values.yaml` for obvious security reasons, you can move them to a Kubernetes Secret:
+
+```yaml
+apiVersion: v1
+kind: Secret
+metadata:
+  name: graphdb-backup-s3-credentials
+type: Opaque
+stringData:
+  ACCESS_KEY_ID: "XXXXX"
+  ACCESS_KEY: "XXXXX"
+```
+
+You can then configure your `values.yaml` like so:
+
+```yaml
+backup:
+  enabled: true
+  type: cloud
+  cloud:
+    bucketUri: s3:///<bucket>/${BACKUP_NAME}?region=<region>&AWS_ACCESS_KEY_ID=${ACCESS_KEY_ID}&AWS_SECRET_ACCESS_KEY=${ACCESS_KEY}
+  extraEnvFrom:
+  - secretRef:
+      name: graphdb-backup-s3-credentials
+```
+
+The backup script will pick up the environment variables and interpolate them using the passed environment variables.
+
+## Using ServiceAccount and IRSA
+
+Instead of hardcoding credentials in `values.yaml` or arbitrary Kubernetes Secrets, the best solution is to use temporary credentials
+provided by a ServiceAccount mapped to an IAM role in AWS. This ServiceAccount can be used by GraphDB to obtain temporary credentials for
+interacting with the S3 bucket as long as the role mapped to the ServiceAccount has the necessary permissions for the given bucket.
+See https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html.
+
+In the next example, the chart is configured to create a new ServiceAccount for GraphDB with a special AWS EKS annotation that will map the
+ServiceAccount to an IAM role:
+
+```yaml
+backup:
+  enabled: true
+  type: cloud
+  schedule: "@midnight"
+  cloud:
+    bucketUri: s3:///<bucket>/${BACKUP_NAME}?region=<region>
+serviceAccount:
+  create: true
+  annotations:
+    eks.amazonaws.com/role-arn: arn:aws:iam::<AWS_ACCOUNT_ID>:role/<ROLE_NAME>
+```
+
+Where:
+
+- `AWS_ACCOUNT_ID` is the account in which the IAM role exists
+- `ROLE_NAME` is the name of the IAM role that will be mapped to the ServiceAccount
+
+Effectively, when a new backup request is received by GraphDB, it will use the AWS SDK to obtain temporary credentials via the
+ServiceAccount.
+
+## Customizing
+
+### Reuse Backup Name
+
+You can configure the backups to reuse the same name for every new backup like so:
+
+```yaml
+backup:
+  enabled: true
+  type: cloud
+  cloud:
+    bucketUri: s3:///<bucket>/<folder>/backup.tar?region=<region>
+```
+
+### Backup Options
+
+You can also configure backup specific options by providing them directly to `backup.options`:
+
+```yaml
+backup:
+  # Rendered as JSON in the backups Job
+  options:
+    backupSystemData: false
+    repositories: [ .... ]
+```
+
+Or by passing an existing Secret object with:
+
+```yaml
+backup:
+  optionsSecret:
+    existingSecret: graphdb-custom-backup-options
+```
+
+See https://graphdb.ontotext.com/documentation/10.7/backup-and-restore.html#backup-options for supported options.

examples/aws/ingress/README.md

@@ -17,4 +17,8 @@ This document describes the steps on how to configure the GraphDB Helm chart to
 
 ## Note
 
-After you deploy and the ingress is created please change the externalUrl value to the DNS name of the ALB or Route53.
+After you deploy the GraphDB chart you should either point an DNS name to the ALB and set
+the externalUrl property in the chart and re-apply it, or the other option you can use nip.io and map it's
+public ip address to nip.io. In order to do that use the kubectl get ingress commands and copy the Public IP
+for the GraphDB ingress, then go to the values file and set the externalUrl property to http://ip.address.nip.io/.
+Otherwise Workbench won't be accessible.

examples/aws/ingress/values.yaml

@@ -1,14 +1,18 @@
 # This example shows how to deploy and expose GraphDB with AWS Load Balancer Controller Ingress without SSL.
 
 configuration:
-  externalUrl: http://graphdb-example-dns-name.com/ # Change this to your ALB DNS name or Route53 if you use it.
+  # Change this to your Route53 domain name or use nip.io which you can do after you deploy the chart since you
+  # need to map the public IP address of the AWS Load Balancer to the nip.io.
+  externalUrl: http://ip.address.nip.io/
 
 ingress:
   enabled: true
   className: alb
   annotations:
    alb.ingress.kubernetes.io/load-balancer-name: graphdb-ingress
    alb.ingress.kubernetes.io/target-type: ip
+   # This ensures the load balancer is public
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/inbound-cidrs: "0.0.0.0/0"
+   # Define the ports on which the Ingress should create listener
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}]'

examples/aws/ingress/values_https.yaml

@@ -1,17 +1,23 @@
 # This example shows how to deploy and expose GraphDB with AWS Load Balancer Controller Ingress with SSL enabled.
 
 configuration:
-  externalUrl: https://graphdb-example-dns-name.com/ # Change this to your ALB DNS name or Route53 if you use it.
+  # Change this to your Route53 domain name or use nip.io which you can do after you deploy the chart since you
+  # need to map the public IP address of the AWS Load Balancer to the nip.io.
+  externalUrl: https://ip.address.nip.io
 
 ingress:
   enabled: true
   className: alb
   annotations:
    alb.ingress.kubernetes.io/load-balancer-name: graphdb-ingress
    alb.ingress.kubernetes.io/target-type: ip
+   # This ensures the load balancer is public
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/inbound-cidrs: "0.0.0.0/0"
+   # Define the ports on which the Ingress should create listener
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS": 443}]'
-   alb.ingress.kubernetes.io/certificate-arn: "" # ARN of the ACM SSL Certificate that will be used
+   # ARN of the ACM SSL Certificate that will be used
+   alb.ingress.kubernetes.io/certificate-arn: ""
    alb.ingress.kubernetes.io/ssl-policy: "ELBSecurityPolicy-TLS13-1-2-2021-06"
-   alb.ingress.kubernetes.io/ssl-redirect: '443'
+   # Enable SSL redirect on the listener
+   alb.ingress.kubernetes.io/ssl-redirect: '443'

examples/aws/lb-example/README.md

@@ -11,4 +11,4 @@ This folder contains examples of using GraphDB with the AWS Network Load Balance
 ## Example
 
 * [values.yaml](values.yaml) - Example of how to deploy and expose GraphDB with Network Load Balancer without SSL enabled.
-* [values_https.yaml] - Example of how to deploy and expose GraphDB with Network Load Balancer with SSL enabled.
+* [values_https.yaml](values_https.yaml) - Example of how to deploy and expose GraphDB with Network Load Balancer with SSL enabled.

examples/aws/lb-example/values.yaml

@@ -10,5 +10,6 @@ service:
    service.beta.kubernetes.io/aws-load-balancer-name: "graphdb-lb"
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "ip"
    service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
-   service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing"  # This ensures the load balancer is internet-facing
+   # This ensures the load balancer is public
+   service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing"
    service.beta.kubernetes.io/load-balancer-source-ranges: "0.0.0.0/0"

examples/aws/lb-example/values_https.yaml

@@ -10,8 +10,10 @@ service:
    service.beta.kubernetes.io/aws-load-balancer-name: "graphdb-lb"
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "ip"
    service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
-   service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing"  # This ensures the load balancer is internet-facing
+    # This ensures the load balancer is public
+   service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing"
    service.beta.kubernetes.io/load-balancer-source-ranges: "0.0.0.0/0"
-   service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "" # ARN of the ACM SSL Certificate that will be used
-   service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443" # Ports to be used for the SSL
-   service.beta.kubernetes.io/aws-load-balancer-ssl-negotiation-policy: "ELBSecurityPolicy-TLS13-1-2-2021-06"
+    # ARN of the ACM SSL Certificate that will be used
+   service.beta.kubernetes.io/aws-load-balancer-ssl-cert: ""
+   service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
+   service.beta.kubernetes.io/aws-load-balancer-ssl-negotiation-policy: "ELBSecurityPolicy-TLS13-1-2-2021-06"

examples/aws/service-account/README.md

@@ -2,7 +2,8 @@
 
 This folder contains examples of using GraphDB with the Service account to gain access to the AWS services.
 
-The main reason that we want to use service accounts is that GraphDB relies on S3 for the Cloud Backups. But you can use that service account to use other AWS services from the EKS cluster as well, 
+The main reason that we want to use service accounts is that GraphDB relies on S3 for the Cloud Backups. 
+But you can use that service account to use other AWS services from the EKS cluster as well, 
 when you have configured your IAM policies properly.
 
 ## Pre-requisites
@@ -15,4 +16,3 @@ when you have configured your IAM policies properly.
 ## Example
 
 * [values.yaml](values.yaml) - Example of how to deploy the service account.
-

examples/aws/service-account/values.yaml

@@ -1,5 +1,5 @@
 serviceAccount:
   create: true
-  name: graphdb
   annotations:
-    eks.amazonaws.com/role-arn: arn:aws:iam::<AWS_ACCOUNT_ID>:role/eks-service-account-role # Example ARN Role, replace with your actual IAM Role ARN
+    # Example ARN Role, replace with your actual IAM Role ARN
+    eks.amazonaws.com/role-arn: arn:aws:iam::<AWS_ACCOUNT_ID>:role/eks-service-account-role