Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add option to create indices from existingConfigmap #153

Open
wants to merge 2 commits into
base: feature/11.4
Choose a base branch
from
Open

Add option to create indices from existingConfigmap #153

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
4eaa602
Add option to create indices from existingConfigmap
ck-netlution Dec 30, 2024
7db2811
Remove verbose option for index creation and update docs
ck-netlution Jan 9, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -387,6 +387,7 @@ IMPORTANT: This is generated by helm-docs, do not attempt modifying it on hand a
| import.volumeMount.volumeClaimTemplate.name | string | `"import"` | |
| import.volumeMount.volumeClaimTemplate.spec.accessModes[0] | string | `"ReadWriteOnce"` | |
| import.volumeMount.volumeClaimTemplate.spec.resources.requests.storage | string | `"10Gi"` | |
| indices.existingConfigmap | string | `""` | |
| ingress.annotations | object | `{}` | |
| ingress.className | string | `""` | |
| ingress.enabled | bool | `true` | |
Expand Down
38 changes: 38 additions & 0 deletions files/scripts/graphdb.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -155,6 +155,44 @@ function cloudBackup {
log "Backup ${BACKUP_NAME} completed successfully!"
}

function createIndicesFromFiles() {
local node_count=$1
local indicesConfigsLocation=$2
local success=true
local response
local repository

set -v
ck-netlution marked this conversation as resolved.
Show resolved Hide resolved

waitAllNodes "$node_count"

for filename in "${indicesConfigsLocation}"/*; do
repository="$(basename "$filename")"
response=$(
curl --connect-timeout 60 --retry 3 --retry-all-errors --retry-delay 10 \
-H "Authorization: Basic ${GRAPHDB_AUTH_TOKEN}" \
-H 'Accept: application/json' \
-H 'Content-Type: application/sparql-update' \
--data-binary @"${filename}" \
"http://${GRAPHDB_POD_NAME}-0.${GRAPHDB_SERVICE_NAME}:${GRAPHDB_SERVICE_PORT}/repositories/${repository}/statements"
)

if [ -z "$response" ]; then
log "Successfully created index for ${filename}"
else
log "Could not create index for ${filename}, response:"
log "$response"
if ! grep -q "already exists." <<<$response; then
success=false
fi
fi
done

if [ $success != true ]; then
exit 1
fi
}

function localBackup() {
BACKUP_TIMESTAMP="$(date +'%Y-%m-%d_%H-%M-%S')"
BACKUP_NAME="graphdb-backup-${BACKUP_TIMESTAMP}.tar"
Expand Down
4 changes: 4 additions & 0 deletions templates/jobs/_labels.tpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,10 @@ Helper functions for labels related to Job and provisioning resources
{{- printf "%s-%s" (include "graphdb.fullname" .) "provision-repositories" -}}
{{- end -}}

{{- define "graphdb.fullname.job.provision-indices" -}}
{{- printf "%s-%s" (include "graphdb.fullname" .) "provision-indices" -}}
{{- end -}}

{{- define "graphdb.fullname.job.scale-down-cluster" -}}
{{- printf "%s-%s" (include "graphdb.fullname" .) "scale-down-cluster" -}}
{{- end -}}
Expand Down
99 changes: 99 additions & 0 deletions templates/jobs/job-create-indices.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
{{- if .Values.indices.existingConfigmap }}
apiVersion: batch/v1
kind: Job
metadata:
name: {{ include "graphdb.fullname.job.provision-indices" . }}
namespace: {{ include "graphdb.namespace" . }}
labels:
{{- include "graphdb.labels" . | nindent 4 }}
annotations:
"helm.sh/hook": post-install, post-upgrade, post-rollback
"helm.sh/hook-delete-policy": before-hook-creation, hook-succeeded, hook-failed
"helm.sh/hook-weight": "5"
{{- with .Values.annotations }}
{{- tpl (toYaml .) $ | nindent 4 }}
{{- end }}
spec:
backoffLimit: {{ .Values.jobs.backoffLimit }}
ttlSecondsAfterFinished: {{ .Values.jobs.ttlSecondsAfterFinished }}
template:
spec:
restartPolicy: Never
automountServiceAccountToken: false
{{- if .Values.jobs.schedulerName }}
schedulerName: {{ .Values.jobs.schedulerName }}
{{- end }}
{{- with .Values.jobs.dnsConfig }}
dnsConfig: {{ toYaml . | nindent 8 }}
{{- end }}
{{- if .Values.jobs.dnsPolicy }}
dnsPolicy: {{ .Values.jobs.dnsPolicy }}
{{- end }}
{{- if .Values.jobs.priorityClassName }}
priorityClassName: {{ .Values.jobs.priorityClassName }}
{{- end }}
{{- with .Values.jobs.nodeSelector }}
nodeSelector: {{- tpl (toYaml .) $ | nindent 8 }}
{{- end }}
{{- with .Values.jobs.affinity }}
affinity: {{- tpl (toYaml .) $ | nindent 8 }}
{{- end }}
{{- with .Values.jobs.tolerations }}
tolerations: {{- tpl (toYaml .) $ | nindent 8 }}
{{- end }}
{{- with .Values.jobs.topologySpreadConstraints }}
topologySpreadConstraints: {{- tpl (toYaml .) $ | nindent 8 }}
{{- end }}
{{- if or .Values.global.imagePullSecrets .Values.image.pullSecrets }}
imagePullSecrets:
{{- include "graphdb.combinedImagePullSecrets" . | nindent 8 }}
{{- end }}
{{- with .Values.jobs.podSecurityContext }}
securityContext: {{ toYaml . | nindent 8 }}
{{- end }}
volumes:
- name: job-temp
emptyDir: {{ .Values.jobs.persistence.emptyDir | toYaml | nindent 12 }}
- name: indices-config
configMap:
name: {{ .Values.indices.existingConfigmap }}
- name: graphdb-utils
configMap:
name: {{ include "graphdb.fullname.configmap.utils" . }}
containers:
- name: provision-indices
image: {{ include "graphdb.image" . }}
imagePullPolicy: {{ .Values.image.pullPolicy }}
env:
- name: GRAPHDB_POD_NAME
value: {{ include "graphdb.fullname" . }}
- name: GRAPHDB_SERVICE_NAME
value: {{ include "graphdb.fullname.service.headless" . }}
- name: GRAPHDB_SERVICE_PORT
value: {{ .Values.headlessService.ports.http | quote }}
- name: GRAPHDB_AUTH_TOKEN
valueFrom:
secretKeyRef:
name: {{ (tpl .Values.security.provisioner.existingSecret .) | default (include "graphdb.fullname.secret.provisioning-user" .) }}
key: {{ .Values.security.provisioner.tokenKey }}
{{- with .Values.jobs.securityContext }}
securityContext: {{ toYaml . | nindent 12 }}
{{- end }}
{{- with .Values.jobs.resources }}
resources: {{ toYaml . | nindent 12 }}
{{- end }}
volumeMounts:
- name: job-temp
mountPath: /tmp
- name: graphdb-utils
mountPath: /tmp/utils
- name: indices-config
mountPath: /tmp/indices-config
workingDir: /tmp
command: ["bash"]
args:
- "/tmp/utils/graphdb.sh"
- "createIndicesFromFiles"
- "{{ .Values.replicas }}"
- "/tmp/indices-config"
{{- end }}
10 changes: 10 additions & 0 deletions values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -969,6 +969,16 @@ repositories:
# Each key in the existing configmap will be treated as the config.ttl of a repository.
existingConfigmap: ""

################################
# Custom Lucene Configurations #
################################

indices:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BTW, this could be used for all kinds of SPARQL update queries. I provisioned a repository with some statements using this. However, it should be used with care as each helm upgrade would trigger the job.

We could merge this as is for now and make it a more generic functionality in the next major release.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good! Our use-case was purely indices but I can imagine other queries to be useful. I took inspiration from provision-repositories. Ideally queries should be idempotent to work well with upgrades.

If you make it generic, it might be a good idea to use index creation as an example how a configmap should look like.

# Optional configmap containing indices configuration .ttl file(s).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The description is not accurate, the job does not expect .ttl files or other repo configurations.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I change the description to reflect the fact that the name is the repository name and the contents are the SPARQL query in ttl format.

# GraphDB will automatically create indices with the provided repositories configuration files.
# Each key in the existing configmap will be treated as the config.ttl of a repository.
existingConfigmap: ""

################################################
# Persistent Volume Permissions Configurations #
################################################
Expand Down