diff --git a/docs/dev/docker.md b/docs/dev/docker.md index dc20fca237..5a4a2a49f6 100644 --- a/docs/dev/docker.md +++ b/docs/dev/docker.md @@ -22,4 +22,4 @@ Please note that some of these containers will only be useful if utilised by the ## Requirements -Before you can build and run these containers you will need to install Docker along with the compose plugin. Information on how to do this can be found in the [docker docs](https://docs.docker.com/get-docker/). +Before you can build and run these containers you will need to install Docker or a compatible equivalent (e.g. Podman). diff --git a/docs/dev/kubernetes-guide/add-libraries.md b/docs/dev/kubernetes-guide/add-libraries.md index 5eb49aed65..1f1b4f40dc 100644 --- a/docs/dev/kubernetes-guide/add-libraries.md +++ b/docs/dev/kubernetes-guide/add-libraries.md @@ -9,7 +9,7 @@ By default with the Gaffer deployment you get access to the: If you want more libraries than this (either one of ours of one of your own) you will need to customise the docker images and use them in place of the defaults. -You will need a basic Gaffer instance deployed on Kubernetes, [here](deploy-empty-graph.md) is how you do that. +You will need a [basic Gaffer instance deployed on Kubernetes](deploy-empty-graph.md). ## Overwrite the REST war file @@ -49,9 +49,7 @@ docker build -t custom-gaffer-accumulo:latest . # Switch the images in the deployment -You will need a way of making the custom images visible to the kubernetes cluster. With EKS, you can do this by uploading the images to ECR. There is an example for how to do that in one of our [other guides](aws-eks-deployment.md). With KinD, you just run `kind load docker-image `. - -Once visible you can switch them out. Create a `custom-images.yaml` file with the following contents: +You will need a way of making the custom images visible to the kubernetes cluster. Once visible you can switch them out. Create a `custom-images.yaml` file with the following contents: ```yaml api: diff --git a/docs/dev/kubernetes-guide/aws-eks-deployment.md b/docs/dev/kubernetes-guide/aws-eks-deployment.md deleted file mode 100644 index 3125236f78..0000000000 --- a/docs/dev/kubernetes-guide/aws-eks-deployment.md +++ /dev/null @@ -1,299 +0,0 @@ -# Deploying Gaffer on AWS EKS - -The following instructions will guide you through provisioning and configuring an [AWS EKS](https://aws.amazon.com/eks/) cluster that our Helm Charts can be deployed on. - -## Install CLI Tools - -- [docker compose](https://github.com/docker/compose/releases/latest) -- [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) -- [helm](https://github.com/helm/helm/releases) -- [aws-cli](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html) -- [eksctl](https://github.com/weaveworks/eksctl/releases/latest) - -## Container Images - -If the versions of the container images you would like to deploy are not available in [Docker Hub](https://hub.docker.com/u/gchq) then you will need to host them in a registry yourself. - -The following instructions build all the container images and host them in AWS ECR when run from the ./kubernetes folder: - -```bash -export HADOOP_VERSION=${HADOOP_VERSION:-3.3.3} -export GAFFER_VERSION=${GAFFER_VERSION:-2.0.0} - -docker compose --project-directory ../docker/accumulo/ -f ../docker/accumulo/docker-compose.yaml build -docker compose --project-directory ../docker/gaffer-road-traffic-loader/ -f ../docker/gaffer-road-traffic-loader/docker-compose.yaml build - -HADOOP_IMAGES="hdfs" -GAFFER_IMAGES="gaffer gaffer-rest gaffer-road-traffic-loader" - -ACCOUNT=$(aws sts get-caller-identity --query Account --output text) -[ "${REGION}" = "" ] && REGION=$(aws configure get region) -[ "${REGION}" = "" ] && REGION=$(curl --silent -m 5 http://169.254.169.254/latest/dynamic/instance-identity/document | grep region | cut -d'"' -f 4) -REPO_PREFIX="${ACCOUNT}.dkr.ecr.${REGION}.amazonaws.com/gchq" - -for repo in ${HADOOP_IMAGES} ${GAFFER_IMAGES}; do - aws ecr create-repository --repository-name gchq/${repo} -done - -echo $(aws ecr get-login-password) | docker login -u AWS --password-stdin https://${ACCOUNT}.dkr.ecr.${REGION}.amazonaws.com - -for repo in ${HADOOP_IMAGES}; do - docker image tag gchq/${repo}:${HADOOP_VERSION} ${REPO_PREFIX}/${repo}:${HADOOP_VERSION} - docker image push ${REPO_PREFIX}/${repo}:${HADOOP_VERSION} -done - -for repo in ${GAFFER_IMAGES}; do - docker image tag gchq/${repo}:${GAFFER_VERSION} ${REPO_PREFIX}/${repo}:${GAFFER_VERSION} - docker image push ${REPO_PREFIX}/${repo}:${GAFFER_VERSION} -done -``` - -## EKS Cluster - -There are a number of ways to provision an AWS EKS cluster. This guide uses a cli tool called `eksctl`. [Documentation](https://docs.aws.amazon.com/eks/latest/userguide/getting-started.html) is available for some of the other methods. - -Before issuing any commands, the subnets that will be used by your EKS cluster need to be tagged accordingly: -Subnet Type | Tag Key | Tag Value ------------| ------- | --------- -Public | kubernetes.io/role/elb | 1 -Private | kubernetes.io/role/internal-elb | 1 - -If you want the cluser to spin up a VPC that is not the default, then set `$VPC_ID`. - -```bash -EKS_CLUSTER_NAME=${EKS_CLUSTER_NAME:-gaffer} -KUBERNETES_VERSION=${KUBERNETES_VERSION:-1.15} - -[ "${VPC_ID}" = "" ] && VPC_ID=$(aws ec2 describe-vpcs --filters Name=isDefault,Values=true --query Vpcs[0].VpcId --output text) -[ "${VPC_ID}" = "" ] && echo "Unable to detect default VPC ID, please set \$VPC_ID" && exit 1 - -# Obtain a list of public and private subnets that the cluster will be deployed into by querying for the required 'elb' tags -PUBLIC_SUBNET_IDS=$(aws ec2 describe-subnets --filters Name=vpc-id,Values=${VPC_ID} Name=tag-key,Values=kubernetes.io/role/elb --query Subnets[].SubnetId --output text | tr -s '[:blank:]' ',') -PRIVATE_SUBNET_IDS=$(aws ec2 describe-subnets --filters Name=vpc-id,Values=${VPC_ID} Name=tag-key,Values=kubernetes.io/role/internal-elb --query Subnets[].SubnetId --output text | tr -s '[:blank:]' ',') -[ "${PUBLIC_SUBNET_IDS}" = "" ] && echo "Unable to detect any public subnets. Make sure they are tagged: kubernetes.io/role/elb=1" && exit 1 -[ "${PRIVATE_SUBNET_IDS}" = "" ] && echo "Unable to detect any private subnets. Make sure they are tagged: kubernetes.io/role/internal-elb=1" && exit 1 - -eksctl create cluster \ - -n "${EKS_CLUSTER_NAME}" \ - --version "${KUBERNETES_VERSION}" \ - --managed \ - --nodes 3 \ - --nodes-min 3 \ - --nodes-max 12 \ - --node-volume-size 20 \ - --full-ecr-access \ - --alb-ingress-access \ - --vpc-private-subnets "${PRIVATE_SUBNET_IDS}" \ - --vpc-public-subnets "${PUBLIC_SUBNET_IDS}" - -aws eks update-kubeconfig --name ${EKS_CLUSTER_NAME} -``` - -## Ingress - -Deploy the AWS ALB Ingress Controller, using the [docs](https://docs.aws.amazon.com/eks/latest/userguide/alb-ingress.html). - -At the time of writing, this involves issuing the following commands: - -```bash -EKS_CLUSTER_NAME=${EKS_CLUSTER_NAME:-gaffer} - -[ "${ACCOUNT}" = "" ] && ACCOUNT=$(aws sts get-caller-identity --query Account --output text) -[ "${REGION}" = "" ] && REGION=$(aws configure get region) -[ "${REGION}" = "" ] && REGION=$(curl --silent -m 5 http://169.254.169.254/latest/dynamic/instance-identity/document | grep region | cut -d'"' -f 4) -[ "${REGION}" = "" ] && echo "Unable to detect AWS region, please set \$REGION" && exit 1 - -eksctl utils associate-iam-oidc-provider \ - --region "${REGION}" \ - --cluster "${EKS_CLUSTER_NAME}" \ - --approve - -aws iam create-policy \ - --policy-name ALBIngressControllerIAMPolicy \ - --policy-document https://raw.githubusercontent.com/kubernetes-sigs/aws-alb-ingress-controller/v1.1.4/docs/examples/iam-policy.json - -kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/aws-alb-ingress-controller/v1.1.4/docs/examples/rbac-role.yaml - -eksctl create iamserviceaccount \ - --region "${REGION}" \ - --name alb-ingress-controller \ - --namespace kube-system \ - --cluster "${EKS_CLUSTER_NAME}" \ - --attach-policy-arn arn:aws:iam::${ACCOUNT}:policy/ALBIngressControllerIAMPolicy \ - --override-existing-serviceaccounts \ - --approve - -curl https://raw.githubusercontent.com/kubernetes-sigs/aws-alb-ingress-controller/v1.1.4/docs/examples/alb-ingress-controller.yaml | sed "s/# - --cluster-name=devCluster/- --cluster-name=${EKS_CLUSTER_NAME}/" | kubectl apply -f - - -``` - -## Deploying Helm Charts - -Below are instructions on deploying the different Helm Charts. - -??? example "Gaffer" - - All scripts listed here are intended to be run from the kubernetes/gaffer folder. - - #### Using ECR - If you are hosting the container images in your AWS account, using ECR, then run the following commands to configure the Helm Charts to use them: - - ```bash - ACCOUNT=$(aws sts get-caller-identity --query Account --output text) - [ "${REGION}" = "" ] && REGION=$(aws configure get region) - [ "${REGION}" = "" ] && REGION=$(curl --silent -m 5 http://169.254.169.254/latest/dynamic/instance-identity/document | grep region | cut -d'"' -f 4) - if [ "${REGION}" = "" ]; then - echo "Unable to detect AWS region, please set \$REGION" - else - REPO_PREFIX="${ACCOUNT}.dkr.ecr.${REGION}.amazonaws.com/gchq" - - EXTRA_HELM_ARGS="" - EXTRA_HELM_ARGS+="--set gaffer.hdfs.namenode.repository=${REPO_PREFIX}/hdfs " - EXTRA_HELM_ARGS+="--set gaffer.hdfs.datanode.repository=${REPO_PREFIX}/hdfs " - EXTRA_HELM_ARGS+="--set gaffer.hdfs.shell.repository=${REPO_PREFIX}/hdfs " - EXTRA_HELM_ARGS+="--set gaffer.accumulo.image.repository=${REPO_PREFIX}/gaffer " - EXTRA_HELM_ARGS+="--set gaffer.api.image.repository=${REPO_PREFIX}/gaffer-rest " - EXTRA_HELM_ARGS+="--set loader.image.repository=${REPO_PREFIX}/gaffer-road-traffic-loader " - fi - ``` - - #### Deploy Helm Chart - By default the Gaffer graph uses the in-memory MapStore. If you want to use an alternative store, we have a guide for that [here](deploy-empty-graph.md). - - ```bash - export HADOOP_VERSION=${HADOOP_VERSION:-3.3.3} - export GAFFER_VERSION=${GAFFER_VERSION:-2.0.0} - - helm dependency update ../accumulo/ - helm dependency update ../gaffer/ - helm dependency update - - helm install gaffer . -f ./values-eks-alb.yaml \ - ${EXTRA_HELM_ARGS} \ - --set gaffer.accumulo.hdfs.namenode.tag=${HADOOP_VERSION} \ - --set gaffer.accumulo.hdfs.datanode.tag=${HADOOP_VERSION} \ - --set gaffer.accumulo.hdfs.shell.tag=${HADOOP_VERSION} \ - --set gaffer.accumulo.image.tag=${GAFFER_VERSION} \ - --set gaffer.api.image.tag=${GAFFER_VERSION} \ - --set loader.image.tag=${GAFFER_VERSION} - - helm test road-traffic - ``` - -??? example "Gaffer Road Traffic Dataset" - - All scripts listed here are intended to be run from the kubernetes/gaffer-road-traffic folder. - - #### Using ECR - If you are hosting the container images in your AWS account, using ECR, then run the following commands to configure the Helm Chart to use them: - - ```bash - ACCOUNT=$(aws sts get-caller-identity --query Account --output text) - [ "${REGION}" = "" ] && REGION=$(aws configure get region) - [ "${REGION}" = "" ] && REGION=$(curl --silent -m 5 http://169.254.169.254/latest/dynamic/instance-identity/document | grep region | cut -d'"' -f 4) - if [ "${REGION}" = "" ]; then - echo "Unable to detect AWS region, please set \$REGION" - else - REPO_PREFIX="${ACCOUNT}.dkr.ecr.${REGION}.amazonaws.com/gchq" - - EXTRA_HELM_ARGS="" - EXTRA_HELM_ARGS+="--set gaffer.accumulo.hdfs.namenode.repository=${REPO_PREFIX}/hdfs " - EXTRA_HELM_ARGS+="--set gaffer.accumulo.hdfs.datanode.repository=${REPO_PREFIX}/hdfs " - EXTRA_HELM_ARGS+="--set gaffer.accumulo.hdfs.shell.repository=${REPO_PREFIX}/hdfs " - EXTRA_HELM_ARGS+="--set gaffer.accumulo.image.repository=${REPO_PREFIX}/gaffer " - EXTRA_HELM_ARGS+="--set gaffer.api.image.repository=${REPO_PREFIX}/gaffer-rest " - EXTRA_HELM_ARGS+="--set loader.image.repository=${REPO_PREFIX}/gaffer-road-traffic-loader " - fi - ``` - - #### Deploy Helm Chart - The last thing before deploying is to set the passwords for the various accumulo users in the values.yaml file. These are found under `accumulo.config.userManagement`. - - Finally, deploy the Helm Chart by running this from the kubernetes/gaffer-road-traffic folder: - - ```bash - export HADOOP_VERSION=${HADOOP_VERSION:-3.3.3} - export GAFFER_VERSION=${GAFFER_VERSION:-2.0.0} - - helm dependency update ../accumulo/ - helm dependency update ../gaffer/ - helm dependency update - - helm install road-traffic . -f ./values-eks-alb.yaml \ - ${EXTRA_HELM_ARGS} \ - --set gaffer.hdfs.namenode.tag=${HADOOP_VERSION} \ - --set gaffer.hdfs.datanode.tag=${HADOOP_VERSION} \ - --set gaffer.hdfs.shell.tag=${HADOOP_VERSION} \ - --set gaffer.accumulo.image.tag=${GAFFER_VERSION} \ - --set gaffer.api.image.tag=${GAFFER_VERSION} \ - --set loader.image.tag=${GAFFER_VERSION} - - helm test road-traffic - ``` - -??? example "HDFS" - - All scipts listed here are intended to be run from the kubernetes/hdfs folder. - - #### Using ECR - If you are hosting the container images in your AWS account, using ECR, then run the following commands to configure the Helm Chart to use them: - - ```bash - ACCOUNT=$(aws sts get-caller-identity --query Account --output text) - [ "${REGION}" = "" ] && REGION=$(aws configure get region) - [ "${REGION}" = "" ] && REGION=$(curl --silent -m 5 http://169.254.169.254/latest/dynamic/instance-identity/document | grep region | cut -d'"' -f 4) - if [ "${REGION}" = "" ]; then - echo "Unable to detect AWS region, please set \$REGION" - else - REPO_PREFIX="${ACCOUNT}.dkr.ecr.${REGION}.amazonaws.com/gchq" - - EXTRA_HELM_ARGS="" - EXTRA_HELM_ARGS+="--set namenode.repository=${REPO_PREFIX}/hdfs " - EXTRA_HELM_ARGS+="--set datanode.repository=${REPO_PREFIX}/hdfs " - EXTRA_HELM_ARGS+="--set shell.repository=${REPO_PREFIX}/hdfs " - fi - ``` - - #### Deploy Helm Chart - Finally, deploy the Helm Chart by running this from the kubernetes/hdfs folder: - - ```bash - export HADOOP_VERSION=${HADOOP_VERSION:-3.3.3} - - helm install hdfs . -f ./values-eks-alb.yaml \ - ${EXTRA_HELM_ARGS} \ - --set hdfs.namenode.tag=${HADOOP_VERSION} \ - --set hdfs.datanode.tag=${HADOOP_VERSION} \ - --set hdfs.shell.tag=${HADOOP_VERSION} - - helm test hdf - ``` - -## Access Web UIs - -The AWS ALB Ingress Controller will create an application load balancer (ALB) for each ingress resource deployed into the EKS cluster. - -You can find the URL that you can use to access each ingress with `kubectl get ing`. - -!!! warning - - By default, the security group assigned to the ALBs will allow anyone to access them. We highly recommend attaching a combination of the [other annotations available](https://kubernetes-sigs.github.io/aws-alb-ingress-controller/guide/ingress/annotation/#security-groups) to each of your ingress resources to control access to them. - -## Uninstall - -To uninstall the deployment run: - -```bash -EKS_CLUSTER_NAME=${EKS_CLUSTER_NAME:-gaffer} - -// Use helm to uninstall any deployed charts -for release in $(helm ls --short); do - helm uninstall ${release} -done - -// Ensure EBS volumes are deleted -kubectl get pvc --output name | xargs kubectl delete - -// Delete the EKS cluster -eksctl delete cluster --name "${EKS_CLUSTER_NAME}" -``` diff --git a/docs/dev/kubernetes-guide/change-graph-metadata.md b/docs/dev/kubernetes-guide/change-graph-metadata.md index 158bc97eb1..603db49c99 100644 --- a/docs/dev/kubernetes-guide/change-graph-metadata.md +++ b/docs/dev/kubernetes-guide/change-graph-metadata.md @@ -2,7 +2,7 @@ By default, the default Gaffer deployment ships with the Graph name "simpleGraph" and description "A graph for demo purposes" These are just placeholders and can be overwritten. This guide will show you how. -The first thing you will need to do is [deply an empty graph](deploy-empty-graph.md). +The first thing you will need to do is [deploy an empty graph](deploy-empty-graph.md). ## Changing the description diff --git a/docs/dev/kubernetes-guide/deploy-schema.md b/docs/dev/kubernetes-guide/deploy-schema.md index f951bbb097..fa80719ef2 100644 --- a/docs/dev/kubernetes-guide/deploy-schema.md +++ b/docs/dev/kubernetes-guide/deploy-schema.md @@ -2,7 +2,7 @@ Gaffer uses schema files to describe the data contained in a Graph. This guide will tell you how to deploy your own schemas with a Gaffer Graph. -The first thing you will need to do is deploy a simple graph. We have a guide for how to do that [here](deploy-empty-graph.md). +You will first need [a basic Gaffer instance deployed on Kubernetes] (deploy-empty-graph.md). Once you have that deployed we can change the schema. diff --git a/docs/dev/kubernetes-guide/kind-deployment.md b/docs/dev/kubernetes-guide/kind-deployment.md deleted file mode 100644 index 946890750d..0000000000 --- a/docs/dev/kubernetes-guide/kind-deployment.md +++ /dev/null @@ -1,320 +0,0 @@ -# How to deploy a Kubernetes cluster using Kind - -The following instructions will guide you through provisioning and configuring a local Kubernetes cluster, using [kind](https://kind.sigs.k8s.io/) (Kubernetes IN Docker), that our Helm Charts can be deployed on. - -## Requirements - -You will need to install certain CLI tools: - -- [docker compose](https://github.com/docker/compose/releases/latest) -- [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) -- [helm](https://github.com/helm/helm/releases) -- [kind](https://kind.sigs.k8s.io/docs/user/quick-start/) - -## Kubernetes Cluster - -Simply run the following command to spin up a local Kubernetes cluster, running inside a Docker container: - -```bash -kind create cluster --image kindest/node:v1.24.4 -``` - -## Container Images - -If the versions of the container images you would like to deploy are not available in [Docker Hub](https://hub.docker.com/u/gchq) then you will need to build them yourself and import them into your kind cluster. - -To import the images, run this from the kubernetes directory: - -```bash -export HADOOP_VERSION=${HADOOP_VERSION:-3.3.3} -export GAFFER_VERSION=${GAFFER_VERSION:-2.0.0} - -docker compose --project-directory ../docker/accumulo/ -f ../docker/accumulo/docker-compose.yaml build -docker compose --project-directory ../docker/gaffer-road-traffic-loader/ -f ../docker/gaffer-road-traffic-loader/docker-compose.yaml build - -kind load docker-image gchq/hdfs:${HADOOP_VERSION} -kind load docker-image gchq/gaffer:${GAFFER_VERSION}-accumulo-${ACCUMULO_VERSION} -kind load docker-image gchq/gaffer-rest:${GAFFER_VERSION}-accumulo-${ACCUMULO_VERSION} -kind load docker-image gchq/gaffer-road-traffic-loader:${GAFFER_VERSION} -``` - -From here you should be able to follow the respective kind-deployment files for the services you would like to run. - -## Ingress - -Deploy the Nginx Ingress Controller: - -```bash -INGRESS_NGINX_VERSION="nginx-0.30.0" -kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/${INGRESS_NGINX_VERSION}/deploy/static/mandatory.yaml -kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/${INGRESS_NGINX_VERSION}/deploy/static/provider/baremetal/service-nodeport.yaml -``` - -## Deploy Helm Charts - -Below are the instructions for deploying the different Helm Charts. - -### Gaffer - -This runs through how to deploy a standard Gaffer instance using kind. This will give you an in-memory store, to change this see the [guide]() to change the store type. - -All the scripts found here are designed to be run from the kubernetes/gaffer folder. - -??? example "Deploying Gaffer using kind" - - #### Deploy Helm Charts - ```bash - export HADOOP_VERSION=${HADOOP_VERSION:-3.3.3} - export GAFFER_VERSION=${GAFFER_VERSION:-2.0.0} - - helm dependency update - - helm install gaffer . \ - --set hdfs.namenode.tag=${HADOOP_VERSION} \ - --set hdfs.datanode.tag=${HADOOP_VERSION} \ - --set hdfs.shell.tag=${HADOOP_VERSION} \ - --set accumulo.image.tag=${GAFFER_VERSION} \ - --set api.image.tag=${GAFFER_VERSION} - - helm test gaffer - ``` - #### Accessing Web UIs (via `kubectl port-forward`) - Run the following in the command line: - ```bash - kubectl port-forward svc/gaffer-api 8080:80 - ``` - - Access the following URL in your browser which will take you to the Gaffer REST: [http://localhost:8080/rest/](http://localhost:8080/rest/) - - #### Accessing Web UIs (via [Nginx Ingress Controller](https://github.com/kubernetes/ingress-nginx)) - Register the FQDNs for each component in DNS, e.g. - ```bash - echo "127.0.0.1 gaffer.k8s.local accumulo.k8s.local hdfs.k8s.local" | sudo tee -a /etc/hosts - ``` - - Update the Gaffer deployment to route ingress based on FQDNs: - ```bash - helm upgrade gaffer . -f ./values-host-based-ingress.yaml --reuse-values - ``` - - Set up port forwarding to the nginx ingress controller: - ```bash - sudo KUBECONFIG=$HOME/.kube/config kubectl port-forward -n ingress-nginx svc/ingress-nginx 80:80 - ``` - - Access the web UIs using the following URLs: - - Component | URL - --------- | --- - Gaffer REST | [http://gaffer.k8s.local/rest/](http://gaffer.k8s.local/rest/) - -### Gaffer Road Traffic Dataset - -This runs through how to deploy the example Gaffer Road Traffic Dataset instance using kind. All scripts listed here are intended to be run from the kubernetes/gaffer-road-traffic folder. - -??? example "Deploying Gaffer Road Traffic Dataset using kind" - - After the cluster is provisioned, update the values.yaml with the passwords for the various Accumulo users. - - #### Deploy Helm Charts - ```bash - export HADOOP_VERSION=${HADOOP_VERSION:-3.3.3} - export GAFFER_VERSION=${GAFFER_VERSION:-2.0.0} - - helm dependency update ../accumulo/ - helm dependency update ../gaffer/ - helm dependency update - - helm install road-traffic . \ - --set gaffer.hdfs.namenode.tag=${HADOOP_VERSION} \ - --set gaffer.hdfs.datanode.tag=${HADOOP_VERSION} \ - --set gaffer.hdfs.shell.tag=${HADOOP_VERSION} \ - --set gaffer.accumulo.image.tag=${GAFFER_VERSION} \ - --set gaffer.api.image.tag=${GAFFER_VERSION} \ - --set loader.image.tag=${GAFFER_VERSION} - - helm test road-traffic - ``` - #### Accessing Web UIs (via `kubectl port-forward`) - Component | Command | URL - --------- | ------- | ---- - Gaffer REST | `kubectl port-forward svc/gaffer-api 8080:80` | [http://localhost:8080/rest/](http://localhost:8080/rest/) - Accumulo | `kubectl port-forward svc/road-traffic-gaffer-monitor 9995:80` | [http://localhost:9995/](http://localhost:9995/) - HDFS | `kubectl port-forward svc/road-traffic-hdfs-namenodes 9870:9870` | [http://localhost:9870/](http://localhost:9870/) - - #### Accessing Web UIs (via [Nginx Ingress Controller](https://github.com/kubernetes/ingress-nginx)) - Register the FQDNs for each component in DNS, e.g. - ```bash - echo "127.0.0.1 gaffer.k8s.local accumulo.k8s.local hdfs.k8s.local" | sudo tee -a /etc/hosts - ``` - - Update the Gaffer deployment to route ingress based on FQDNs: - ```bash - helm upgrade road-traffic . -f ./values-host-based-ingress.yaml --reuse-values - ``` - - Set up port forwarding to the nginx ingress controller: - ```bash - sudo KUBECONFIG=$HOME/.kube/config kubectl port-forward -n ingress-nginx svc/ingress-nginx 80:80 - ``` - - Access the web UIs using the following URLs: - - Component | URL - --------- | --- - Gaffer REST | [http://gaffer.k8s.local/rest/](http://gaffer.k8s.local/rest/) - HDFS | [http://hdfs.k8s.local/](http://hdfs.k8s.local/) - Accumulo | [http://accumulo.k8s.local/](http://accumulo.k8s.local/) - -### JupyterHub - -This runs through how to deploy JupyterHub for Gaffer using kind. All the scripts found here are designed to be run from the kubernetes/gaffer-jhub folder. - -??? example "Deploying JupyterHub for Gaffer using kind" - - #### Container Images - Use the following commands to build and deploy the extra containers used by JupyterHub: - ```bash - source ../../docker/gaffer-pyspark-notebook/.env - source ../../docker/gaffer-jhub-options-server/get-version.sh - - # Build Container Images - docker compose --project-directory ../../docker/gaffer-pyspark-notebook/ -f ../../docker/gaffer-pyspark-notebook/docker-compose.yaml build notebook - docker compose --project-directory ../../docker/spark-py/ -f ../../docker/spark-py/docker-compose.yaml build - docker compose --project-directory ../../docker/gaffer-jhub-options-server/ -f ../../docker/gaffer-jhub-options-server/docker-compose.yaml build - - # Deploy Images to Kind - kind load docker-image gchq/gaffer-pyspark-notebook:${GAFFER_VERSION} - kind load docker-image gchq/spark-py:${SPARK_VERSION} - kind load docker-image gchq/gaffer-jhub-options-server:${JHUB_OPTIONS_SERVER_VERSION} - ``` - #### Deploy Helm Chart - Once that is done, use the following commands to deploy a JupyterHub instance with Gaffer extensions: - ```bash - helm dependency update - helm install jhub . -f ./values-insecure.yaml - - helm test jhub - ``` - - #### Accessing JupyterHub Web UIs (via `kubectl port-forward`) - Run the following in the command line: - ```bash - kubectl port-forward svc/proxy-public 8080:80 - ``` - - Access the following URL in your browser: [http://localhost:8080](http://localhost:8080) - - By default, JupyterHub's Dummy Authenticator is used so you can login using any username and password. - - #### Accessing example notebooks - There are some example notebooks that demonstrate how to interact with HDFS, Gaffer and Spark. Copy them into your working directory, to make them easier to view and execute, by starting a Terminal tab and submitting the following command: - ```bash - $ cp -r /examples . - ``` - -### Accumulo - -This runs through how to deploy Accumulo using kind. All the scripts found here are designed to be run from the kubernetes/accumulo folder. - -??? example "Deploying Accumulo using kind" - - #### Deploy Helm Charts - ```bash - export HADOOP_VERSION=${HADOOP_VERSION:-3.3.3} - export GAFFER_VERSION=${GAFFER_VERSION:-2.0.0} - - helm dependency update - - helm install accumulo . \ - --set hdfs.namenode.tag=${HADOOP_VERSION} \ - --set hdfs.datanode.tag=${HADOOP_VERSION} \ - --set hdfs.shell.tag=${HADOOP_VERSION} \ - --set accumulo.image.tag=${GAFFER_VERSION} - - helm test accumulo - ``` - - #### Accessing Web UIs (via `kubectl port-forward`) - Run the following in the command line: - ```bash - kubectl port-forward svc/road-traffic-gaffer-monitor 9995:80 - ``` - - Access the following URL in your browser: [http://localhost:9995/](http://localhost:9995/) - - #### Accessing Web UIs (via [Nginx Ingress Controller](https://github.com/kubernetes/ingress-nginx)) - Register the FQDNs for each component in DNS, e.g. - ```bash - echo "127.0.0.1 gaffer.k8s.local accumulo.k8s.local hdfs.k8s.local" | sudo tee -a /etc/hosts - ``` - - Update the Gaffer deployment to route ingress based on FQDNs: - ```bash - helm upgrade accumulo . -f ./values-host-based-ingress.yaml --reuse-values - ``` - - Set up port forwarding to the nginx ingress controller: - ```bash - sudo KUBECONFIG=$HOME/.kube/config kubectl port-forward -n ingress-nginx svc/ingress-nginx 80:80 - ``` - - Access the web UIs using the following URLs: - - Component | URL - --------- | --- - Accumulo | [http://accumulo.k8s.local/](http://accumulo.k8s.local/) - -### HDFS - -This runs through how to deploy HDFS using kind. All the scripts found here are designed to be run from the kubernetes/hdfs folder. - -??? example "Deploying HDFS using kind" - - #### Deploy Helm Charts - ```bash - export HADOOP_VERSION=${HADOOP_VERSION:-3.3.3} - - helm install hdfs . \ - --set hdfs.namenode.tag=${HADOOP_VERSION} \ - --set hdfs.datanode.tag=${HADOOP_VERSION} \ - --set hdfs.shell.tag=${HADOOP_VERSION} - - helm test hdfs - ``` - - #### Accessing Web UIs (via `kubectl port-forward`) - Run the following in the command line: - ```bash - kubectl port-forward svc/hdfs-namenodes 9870:9870 - ``` - - Access the following URL in your browser: [http://localhost:9870/](http://localhost:9870/) - - #### Accessing Web UIs (via [Nginx Ingress Controller](https://github.com/kubernetes/ingress-nginx)) - Register the FQDNs for each component in DNS, e.g. - ```bash - echo "127.0.0.1 hdfs.k8s.local" | sudo tee -a /etc/hosts - ``` - - Update the Gaffer deployment to route ingress based on FQDNs: - ```bash - helm upgrade hdfs . -f ./values-host-based-ingress.yaml --reuse-values - ``` - - Set up port forwarding to the nginx ingress controller: - ```bash - sudo KUBECONFIG=$HOME/.kube/config kubectl port-forward -n ingress-nginx svc/ingress-nginx 80:80 - ``` - - Access the web UIs using the following URLs: - - Component | URL - --------- | --- - HDFS | [http://hdfs.k8s.local/](http://hdfs.k8s.local/) - -## Uninstall - -```bash -kind delete cluster -``` diff --git a/docs/dev/kubernetes-guide/kubernetes.md b/docs/dev/kubernetes-guide/kubernetes.md index 59c1bda53e..d54ba1e876 100644 --- a/docs/dev/kubernetes-guide/kubernetes.md +++ b/docs/dev/kubernetes-guide/kubernetes.md @@ -2,7 +2,8 @@ The [gaffer-docker](https://github.com/gchq/gaffer-docker) repository contains all code needed to run Gaffer using Docker. -All the files needed to get started using Gaffer in Kubernetes are contained in the ['kubernetes'](https://github.com/gchq/gaffer-docker/tree/develop/kubernetes) sub-folder. In this directory you can find the Helm charts required to deploy various applications onto Kubernetes clusters. +All the files needed to get started using Gaffer in Kubernetes are contained in the ['kubernetes'](https://github.com/gchq/gaffer-docker/tree/develop/kubernetes) sub-folder of the [gaffer-docker](https://github.com/gchq/gaffer-docker) repository. +In this directory you can find the Helm charts required to deploy various applications onto Kubernetes clusters. The Helm charts and associated information for each application can be found in the following places: @@ -16,9 +17,9 @@ These charts can be accessed by cloning our repository or by using our Helm repo ## Requirements -Before you can deploy any of these applications you need to have installed Kubernetes. Information on how to do this can be found in the [Kubernetes docs](https://kubernetes.io/docs/setup/) +Before you can deploy any of these applications you need to have installed Kubernetes. Information on how to do this can be found in the [Kubernetes docs](https://kubernetes.io/docs/setup/). -You will also need to install Docker along with the compose plugin. Information on how to do this can be found in the [Docker docs](https://docs.docker.com/get-docker/). +You will also need to install a container management engine, for example Docker or Podman, to build, run and manage your containers. ## Adding this repo to Helm diff --git a/mkdocs.yml b/mkdocs.yml index 9e5f159d6c..e5ae306b12 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -44,95 +44,93 @@ plugins: - index.md nav: - - Home: "index.md" - - "Gaffer 2.0": - - Changelist: "gaffer2.0/changelist.md" - - Deprecations: "gaffer2.0/deprecations.md" - - Dependencies: "gaffer2.0/dependencies.md" - - "Accumulo Migration": "gaffer2.0/accumulo-migration.md" - - "Federation Changes": "gaffer2.0/federation-changes.md" - - "Accumulo Kerberos": "gaffer2.0/accumulo-kerberos.md" - - "Log4j in Gaffer": "gaffer2.0/log4j.md" - - "Getting Started": - - "Quickstart": "getting-started/quickstart.md" - - "Gaffer Basics": "getting-started/basics.md" - - "Gaffer Guide": - - "Gaffer Guide": "getting-started/guide/guide.md" - - "CSV Import and Export": "getting-started/guide/csv.md" - - "Cardinality": "getting-started/guide/cardinality.md" - - "Deploying Gaffer": "getting-started/deploying.md" - - "Dev Info": - - "Development": "dev/development.md" - - "Managing Dependencies": "dev/managing-dependencies/managing-dependencies.md" - - "Ways of Working": "dev/ways-of-working.md" - - "Sketches custom deserialisation": "dev/rest-api-sketches.md" - - "Docker": "dev/docker.md" - - Components: - - "Cache": "dev/components/cache.md" - - "Data": "dev/components/data.md" - - "Graph": "dev/components/graph.md" - - "Store": "dev/components/store.md" - - "Operation": "dev/components/operation.md" - - "Serialisation": "dev/components/serialisation.md" - - "Integration Tests": "dev/components/integration-test.md" - - "Core REST": "dev/components/core-rest.md" - - "Spring REST": "dev/components/spring-rest.md" - - "Accumulo Store": "dev/components/accumulo-store.md" - - Libraries: - - dev/components/libraries/bitmap.md - - dev/components/libraries/flink.md - - dev/components/libraries/sketches.md - - dev/components/libraries/spark.md - - dev/components/libraries/time.md + - Home: 'index.md' + - 'Gaffer 2.0': + - Changelist: 'gaffer2.0/changelist.md' + - Deprecations: 'gaffer2.0/deprecations.md' + - Dependencies: 'gaffer2.0/dependencies.md' + - 'Accumulo Migration': 'gaffer2.0/accumulo-migration.md' + - 'Federation Changes': 'gaffer2.0/federation-changes.md' + - 'Accumulo Kerberos': 'gaffer2.0/accumulo-kerberos.md' + - 'Log4j in Gaffer': 'gaffer2.0/log4j.md' + - 'Getting Started': + - 'Quickstart': 'getting-started/quickstart.md' + - 'Gaffer Basics': 'getting-started/basics.md' + - 'Gaffer Guide': + - 'Gaffer Guide': 'getting-started/guide/guide.md' + - 'CSV Import and Export': 'getting-started/guide/csv.md' + - 'Cardinality': 'getting-started/guide/cardinality.md' + - 'Deploying Gaffer': 'getting-started/deploying.md' + - 'Dev Info': + - 'Development': 'dev/development.md' + - 'Managing Dependencies': 'dev/managing-dependencies/managing-dependencies.md' + - 'Ways of Working': 'dev/ways-of-working.md' + - 'Sketches custom deserialisation': 'dev/rest-api-sketches.md' + - 'Docker': 'dev/docker.md' - Kubernetes: - - "Kubernetes Guide": "dev/kubernetes-guide/kubernetes.md" - - "Kind Deployment": "dev/kubernetes-guide/kind-deployment.md" - - "AWS EKS Deployment": "dev/kubernetes-guide/aws-eks-deployment.md" - - "Deploy an Empty Graph": "dev/kubernetes-guide/deploy-empty-graph.md" - - "Add your schema": "dev/kubernetes-guide/deploy-schema.md" - - "Change the Graph Id and Description": "dev/kubernetes-guide/change-graph-metadata.md" - - "Adding your own libraries and functions": "dev/kubernetes-guide/add-libraries.md" - - "Changing Accumulo Passwords": "dev/kubernetes-guide/change-accumulo-passwords.md" - - "Reference": - - "Reference Guide": "reference/intro.md" + - 'Kubernetes Guide': 'dev/kubernetes-guide/kubernetes.md' + - 'Deploy an Empty Graph': 'dev/kubernetes-guide/deploy-empty-graph.md' + - 'Add your schema': 'dev/kubernetes-guide/deploy-schema.md' + - 'Change the Graph Id and Description': 'dev/kubernetes-guide/change-graph-metadata.md' + - 'Adding your own libraries and functions': 'dev/kubernetes-guide/add-libraries.md' + - 'Changing Accumulo Passwords': 'dev/kubernetes-guide/change-accumulo-passwords.md' + - Components: + - 'Cache': 'dev/components/cache.md' + - 'Data': 'dev/components/data.md' + - 'Graph': 'dev/components/graph.md' + - 'Store': 'dev/components/store.md' + - 'Operation': 'dev/components/operation.md' + - 'Serialisation': 'dev/components/serialisation.md' + - 'Integration Tests': 'dev/components/integration-test.md' + - 'Core REST': 'dev/components/core-rest.md' + - 'Spring REST': 'dev/components/spring-rest.md' + - 'Accumulo Store': 'dev/components/accumulo-store.md' + - Libraries: + - dev/components/libraries/bitmap.md + - dev/components/libraries/flink.md + - dev/components/libraries/sketches.md + - dev/components/libraries/spark.md + - dev/components/libraries/time.md + - 'Reference': + - 'Reference Guide': 'reference/intro.md' - Stores: - - "Store Guide": "reference/stores-guide/stores.md" - - "Map Store": "reference/stores-guide/map.md" - - "Accumulo Store": "reference/stores-guide/accumulo.md" - - "Proxy Store": "reference/stores-guide/proxy.md" - - "Federated Store": "reference/stores-guide/federated.md" + - 'Store Guide': 'reference/stores-guide/stores.md' + - 'Map Store': 'reference/stores-guide/map.md' + - 'Accumulo Store': 'reference/stores-guide/accumulo.md' + - 'Proxy Store': 'reference/stores-guide/proxy.md' + - 'Federated Store': 'reference/stores-guide/federated.md' - Properties: - - "Properties Guide": "reference/properties-guide/properties.md" - - "Basic Properties": "reference/properties-guide/basic.md" - - "Advanced Properties": "reference/properties-guide/advanced.md" - - "Type Properties": "reference/properties-guide/type.md" - - "Map & Set Properties": "reference/properties-guide/map-set.md" + - 'Properties Guide': 'reference/properties-guide/properties.md' + - 'Basic Properties': 'reference/properties-guide/basic.md' + - 'Advanced Properties': 'reference/properties-guide/advanced.md' + - 'Type Properties': 'reference/properties-guide/type.md' + - 'Map & Set Properties': 'reference/properties-guide/map-set.md' - Predicates: - - "Predicates Guide": "reference/predicates-guide/predicates.md" - - "Gaffer Predicates": "reference/predicates-guide/gaffer-predicates.md" - - "Koryphe Predicates": "reference/predicates-guide/koryphe-predicates.md" - - "Binary Operators": - - "Binary Operators Guide": "reference/binary-operators-guide/binary-operators.md" - - "Koryphe Operators Guide": "reference/binary-operators-guide/koryphe-operators.md" + - 'Predicates Guide': 'reference/predicates-guide/predicates.md' + - 'Gaffer Predicates': 'reference/predicates-guide/gaffer-predicates.md' + - 'Koryphe Predicates': 'reference/predicates-guide/koryphe-predicates.md' + - 'Binary Operators': + - 'Binary Operators Guide': 'reference/binary-operators-guide/binary-operators.md' + - 'Koryphe Operators Guide': 'reference/binary-operators-guide/koryphe-operators.md' - Functions: - - "Functions Guide": "reference/functions-guide/functions.md" - - "Gaffer Functions": "reference/functions-guide/gaffer-functions.md" - - "Koryphe Functions": "reference/functions-guide/koryphe-functions.md" + - 'Functions Guide': 'reference/functions-guide/functions.md' + - 'Gaffer Functions': 'reference/functions-guide/gaffer-functions.md' + - 'Koryphe Functions': 'reference/functions-guide/koryphe-functions.md' - Operations: - - "Operations Guide": "reference/operations-guide/operations.md" - - "Core Operations": "reference/operations-guide/core.md" - - "Join Operation": "reference/operations-guide/join.md" - - "Get Operations": "reference/operations-guide/get.md" - - "Control Flow Operations": "reference/operations-guide/flow.md" - - "Job Operations": "reference/operations-guide/job.md" - - "Named Operations": "reference/operations-guide/named.md" - - "Export Operations": "reference/operations-guide/export.md" - - "Generate Operations": "reference/operations-guide/generate.md" - - "Flink Operations": "reference/operations-guide/flink.md" - - "HDFS Operations": "reference/operations-guide/hdfs.md" - - "Accumulo Operations": "reference/operations-guide/accumulo.md" - - "Spark Operations": "reference/operations-guide/spark.md" - - "Misc Operations": "reference/operations-guide/misc.md" + - 'Operations Guide': 'reference/operations-guide/operations.md' + - 'Core Operations': 'reference/operations-guide/core.md' + - 'Join Operation': 'reference/operations-guide/join.md' + - 'Get Operations': 'reference/operations-guide/get.md' + - 'Control Flow Operations': 'reference/operations-guide/flow.md' + - 'Job Operations': 'reference/operations-guide/job.md' + - 'Named Operations': 'reference/operations-guide/named.md' + - 'Export Operations': 'reference/operations-guide/export.md' + - 'Generate Operations': 'reference/operations-guide/generate.md' + - 'Flink Operations': 'reference/operations-guide/flink.md' + - 'HDFS Operations': 'reference/operations-guide/hdfs.md' + - 'Accumulo Operations': 'reference/operations-guide/accumulo.md' + - 'Spark Operations': 'reference/operations-guide/spark.md' + - 'Misc Operations': 'reference/operations-guide/misc.md' theme: name: material @@ -158,4 +156,4 @@ theme: primary: green toggle: &tolight icon: material/weather-sunny - name: Switch to light mode + name: Switch to light mode \ No newline at end of file