Changes responding to PR comments:

* Allow service and container ports to be configurable (in yaml at least)
* Minor fixes to documentation
* Integrate running of the deploy_hyperion.py script into deploy_hyperion_to_k8s.sh
* Sanity check for checked out vs image version in deployment
* Minor change to deploy_hyperion.py to be able to get the install folder
* By default the deploy_hyperion_to_k8s.sh will now log into the k8s cluster
* Fix ghcr login case issue

Dockerfile.release

@@ -24,6 +24,9 @@ pip install --no-cache-dir --no-compile --no-deps -e ../dodal
 # Everything above this line should be in the image cache unless pyproject.toml changes
 #
 ADD .git /app/hyperion/.git
+# Restore the repository at the current commit instead of copying, to exclude uncommitted changes
+# This is so that if you build a developer image from this dockerfile then _version.py will not
+# append the dirty workdir hash, which causes complications during deployments that mount from a clean folder.
 RUN git restore .
 
 # Regenerate _version.py with the correct version - this should run quickly since we already have our dependencies

docs/developer/hyperion/deploying-hyperion.rst

@@ -11,6 +11,15 @@ you will first need to create a GH personal access token and then log in with po
 Pushing the docker image
 ------------------------
 
+Obtaining a GitHub access token
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+You will need to obtain a GitHub personal access token (classic) - not the new fine-grained token.
+It will need the specific permission scopes as detailed in the `ghcr documentation <https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry#authenticating-with-a-personal-access-token-classic>`_
+
+Building the image
+~~~~~~~~~~~~~~~~~~
+
 If building a test image, the image should be pushed to your personal GH account:
 
 ::
@@ -19,10 +28,6 @@ If building a test image, the image should be pushed to your personal GH account
 
 where ``mysecretfile`` contains your personal access token
 
-::
-
- podman push ghcr.io/<your gh login>/
-
 Then run the ``build_docker_image.sh`` script.
 
 Troubleshooting
@@ -68,23 +73,16 @@ Once the docker image is built, the image can be deployed to kubernetes using th
 Production deployment
 ~~~~~~~~~~~~~~~~~~~~~
 
-* From a development hyperion workspace, first deploy the source folders
+Then create and deploy the helm release
 
 ::
 
- python utility_scripts/deploy/deploy_hyperion.py --kubernetes <beamline>
+ ./utility_scripts/deploy/deploy_hyperion_to_k8s.sh --beamline=<beamline> --checkout-to-prod hyperion
 
-* Then create and deploy the helm release
-
-::
-
- module load helm
- cd <path to deployed hyperion folder in /dls_sw>
- ./utility_scripts/deploy/deploy_hyperion_to_k8s.sh --beamline=<beamline> hyperion
-
-This will create a helm release "hyperion". The source folders will be mounted as 
-bind mounts to allow the pod to pick up changes in production. For production these are expected to be in the normal 
-place defined in ``values.yaml``.
+This will run the ``deploy_hyperion.py`` script to deploy the latest hyperion to ``/dls_sw``.
+You will be prompted to log into the beamline cluster, then it will create a helm release "hyperion".
+The source folders will be mounted as bind mounts to allow the pod to pick up changes in production. 
+For production these are expected to be in the normal place defined in ``values.yaml``.
 
 Development deployment
 ~~~~~~~~~~~~~~~~~~~~~~
@@ -102,6 +100,10 @@ The dev deployment bind-mounts the current ``hyperion`` workspace and ``../dodal
 run against your own development code. **Clusters do not allow bind mounts from arbitrary directories so 
 your workspace will have to be in a permitted directory such as your home directory.**
 
+By default the script will log into the ``argus`` cluster, if you want to deploy to an alternate cluster, 
+log in with ``kubectl set-context --current --namespace=<NAMESPACE>`` and then specify ``--no-login`` when running the 
+script
+
 Please note, the deployment script is intended to be run from a checked-out matching version of the git repository.
 
-``helm list`` should then show details of the installed release 
+``helm list`` should then show details of the installed release on a successful install

docs/user/how-to/run-container.rst

@@ -10,6 +10,6 @@ Starting the container
 
 To pull the container from github container registry and run::
 
- $ docker run ghcr.io/DiamondLightSource/mx-bluesky:main --version
+ $ docker run ghcr.io/diamondlightsource/mx-bluesky:main --version
 
 To get a released version, use a numbered release instead of ``main``.

helmchart/templates/deployment.yaml

@@ -62,7 +62,7 @@ spec:
  memory: "1Gi"
  ports:
  - name: hyperion-api
- containerPort: 5005
+ containerPort: {{ .Values.hyperion.containerPort }}
  protocol: TCP
  env:
  - name: HYPERION_LOG_DIR

helmchart/templates/ingress.yaml

@@ -18,6 +18,6 @@ spec:
  service:
  name: hyperion-svc # this must match the name of the service you want to target
  port:
- number: 5005
+ number: {{ .Values.hyperion.containerPort }}
  {{- end }}
 

helmchart/templates/service.yaml

@@ -10,8 +10,8 @@ spec:
  {{- end }}
  ports:
  - name: hyperion-api
- port: 5005
+ port: {{ .Values.hyperion.servicePort }}
  protocol: TCP
- targetPort: 5005
+ targetPort: {{ .Values.hyperion.containerPort }}
  selector:
  app: hyperion

helmchart/values.yaml

@@ -1,5 +1,7 @@
 hyperion:
- imageRepository: ghcr.io/DiamondLightSource
+ containerPort: 5005
+ servicePort: 80
+ imageRepository: ghcr.io/diamondlightsource
  # i03-hyperion user and group
  runAsUser: 36101
  runAsGroup: 36101

utility_scripts/build_docker_image.sh

@@ -59,7 +59,7 @@ if [[ $BUILD == 1 ]]; then
 fi
 
 if [[ $PUSH == 1 ]]; then
- NAMESPACE=`podman login --get-login ghcr.io`
+ NAMESPACE=$(podman login --get-login ghcr.io | tr '[:upper:]' '[:lower:]')
  if [[ $? != 0 ]]; then
  echo "Not logged in to ghcr.io"
  exit 1

utility_scripts/deploy/deploy_hyperion.py

@@ -18,9 +18,16 @@
 DEV_DEPLOY_LOCATION = "/scratch/30day_tmp/hyperion_release_test/bluesky"
 
 
+class Options(NamedTuple):
+ release_dir: str
+ kubernetes: bool = False
+ print_release_dir: bool = False
+ quiet: bool = False
+
+
 class Deployment:
  # Set name, setup remote origin, get the latest version"""
- def __init__(self, name: str, repo_args):
+ def __init__(self, name: str, repo_args, options: Options):
  self.name = name
  self.repo = Repo(repo_args)
 
@@ -32,7 +39,10 @@ def __init__(self, name: str, repo_args):
  t.name for t in self.repo.tags if VERSION_PATTERN_COMPILED.match(t.name)
  ]
  self.versions.sort(key=Version, reverse=True)
- print(f"Found {self.name}_versions:\n{os.linesep.join(self.versions)}")
+
+ if not options.quiet:
+ print(f"Found {self.name}_versions:\n{os.linesep.join(self.versions)}")
+
  self.latest_version_str = self.versions[0]
 
  def deploy(self):
@@ -64,13 +74,8 @@ def set_deploy_location(self, release_area):
  )
 
 
-class Options(NamedTuple):
- release_dir: str
- kubernetes: bool = False
-
-
 # Get the release directory based off the beamline and the latest hyperion version
-def _get_hyperion_release_dir_from_args() -> Options:
+def _parse_options() -> Options:
  parser = argparse.ArgumentParser(
  prog="deploy_hyperion", description="Deploy hyperion to a beamline"
  )
@@ -85,15 +90,24 @@ def _get_hyperion_release_dir_from_args() -> Options:
  choices=recognised_beamlines,
  help=f"The beamline to deploy hyperion to. 'dev' installs to {DEV_DEPLOY_LOCATION}",
  )
-
+ parser.add_argument(
+ "--print-release-dir",
+ action=argparse.BooleanOptionalAction,
+ help="Print the path to the release folder and then exit",
+ )
  args = parser.parse_args()
  if args.beamline == "dev":
  print("Running as dev")
  release_dir = DEV_DEPLOY_LOCATION
  else:
  release_dir = f"/dls_sw/{args.beamline}/software/bluesky"
 
- return Options(release_dir=release_dir, kubernetes=args.kubernetes)
+ return Options(
+ release_dir=release_dir,
+ kubernetes=args.kubernetes,
+ print_release_dir=args.print_release_dir,
+ quiet=args.print_release_dir,
+ )
 
 
 def _create_environment_from_control_machine(
@@ -129,25 +143,30 @@ def main(options: Options):
  release_area = options.release_dir
  this_repo_top = os.path.abspath(os.path.join(os.path.dirname(__file__), "../.."))
 
- print(f"Repo top is {this_repo_top}")
+ if not options.quiet:
+ print(f"Repo top is {this_repo_top}")
 
  hyperion_repo = Deployment(
- name="hyperion",
- repo_args=os.path.join(this_repo_top, ".git"),
+ name="hyperion", repo_args=os.path.join(this_repo_top, ".git"), options=options
  )
 
  if hyperion_repo.name != "hyperion":
  raise ValueError("This function should only be used with the hyperion repo")
 
  release_area_version = os.path.join(
- release_area, f"hyperion_{hyperion_repo.latest_version_str}"
+ release_area, f"mx_bluesky_{hyperion_repo.latest_version_str}"
  )
 
+ if options.print_release_dir:
+ print(release_area_version)
+ return
+
  print(f"Putting releases into {release_area_version}")
 
  dodal_repo = Deployment(
  name="dodal",
  repo_args=os.path.join(this_repo_top, "../dodal/.git"),
+ options=options,
  )
 
  dodal_repo.set_deploy_location(release_area_version)
@@ -218,5 +237,5 @@ def create_symlink_by_tmp_and_rename(dirname, target, linkname):
 
 if __name__ == "__main__":
  # Gives path to /bluesky
- options = _get_hyperion_release_dir_from_args()
+ options = _parse_options()
  main(options)

utility_scripts/deploy/deploy_hyperion_to_k8s.sh

@@ -1,5 +1,7 @@
 #!/bin/bash
 # Installs helm package to kubernetes
+LOGIN=true
+
 for option in "$@"; do
  case $option in
  -b=*|--beamline=*)
@@ -10,6 +12,10 @@ for option in "$@"; do
  DEV=true
  shift
  ;;
+ --checkout-to-prod)
+ CHECKOUT=true
+ shift
+ ;;
  --repository=*)
  REPOSITORY="${option#*=}"
  shift
@@ -18,16 +24,30 @@ for option in "$@"; do
  APP_VERSION="${option#*=}"
  shift
  ;;
+ --no-login)
+ LOGIN=false
+ shift
+ ;;
  --help|--info|--h)
  CMD=`basename $0`
  echo "$CMD [options] <release>"
- echo "Deploys hyperion to kubernetes"
- echo " --help This help"
- echo " --dev Install to a development kubernetes cluster (assumes project checked out under /home)"
- echo " -b, --beamline=BEAMLINE Overrides the BEAMLINE environment variable with the given beamline"
- echo " --repository=REPOSITORY Override the repository to fetch the image from"
- echo " --appVersion=version Version of the image to fetch from the repository otherwise it is deduced
- from the setuptools_scm"
+ cat <<EOM
+Deploys hyperion to kubernetes
+
+Important!
+If you do not specify --checkout-to-prod YOU MUST run this from the hyperion directory that will be bind-mounted to
+the container, NOT the directory that you built the container image from. 
+
+ --help This help
+ --appVersion=version Version of the image to fetch from the repository otherwise it is deduced
+ from the setuptools_scm
+ -b, --beamline=BEAMLINE Overrides the BEAMLINE environment variable with the given beamline
+ --checkout-to-prod Checkout source folders to the production folder using deploy_hyperion.py
+ --dev Install to a development kubernetes cluster (assumes project checked out under /home)
+ (default cluster is argus in user namespace)
+ --no-login Do not attempt to log in to kubernetes instead use the current namespace and cluster
+ --repository=REPOSITORY Override the repository to fetch the image from
+EOM
  exit 0
  ;;
  -*|--*)
@@ -49,9 +69,45 @@ if [[ -z $RELEASE ]]; then
  exit 1
 fi
 
+if [[ -n $DEV ]]; then
+ if [[ -n $CHECKOUT ]]; then
+ echo "Cannot specify both --dev and --checkout-to-prod"
+ exit 1
+ fi
+ # First extract the version and location that will be deployed
+ DEPLOY_HYPERION="python $PROJECTDIR/utility_scripts/deploy/deploy_hyperion.py"
+ HYPERION_BASE=$($DEPLOY_HYPERION --print-release-dir)
+
+ echo "Running deploy_hyperion.py to deploy to production folder..."
+ $DEPLOY_HYPERION --kubernetes $BEAMLINE || (echo "Deployment failed, aborting."; exit 1)
+
+ NEW_PROJECTDIR=$HYPERION_BASE/hyperion
+ echo "Changing directory to $NEW_PROJECTDIR..."
+ cd $NEW_PROJECTDIR
+ PROJECTDIR=$NEW_PROJECTDIR
+ HYPERION_BASENAME=$(basename $HYPERION_BASE)
+ CHECKED_OUT_VERSION=${HYPERION_BASENAME#hyperion_v}
+else
+ CHECKED_OUT_VERSION=$(git describe --tag)
+fi
+
 HELM_OPTIONS=""
 PROJECTDIR=$(readlink -e $(dirname $0)/../..)
 
+
+if [[ $LOGIN = true ]]; then
+ if [[ -n $DEV ]]; then
+ CLUSTER=argus
+ NAMESPACE=$(whoami)
+ else
+ CLUSTER=k8s-$BEAMLINE
+ NAMESPACE=$BEAMLINE-beamline
+ fi
+
+ module load $CLUSTER
+ kubectl config set-context --current --namespace=$NAMESPACE
+fi
+
 ensure_version_py() {
  # We require the _version.py to be created, this needs a minimal virtual environment
  if [[ ! -d $PROJECTDIR/.venv ]]; then
@@ -80,6 +136,13 @@ if [[ -z $APP_VERSION ]]; then
  APP_VERSION=$(app_version)
 fi
 
+echo "Checked out version that will be bind-mounted in $PROJECTDIR is $CHECKED_OUT_VERSION"
+echo "Container image version that will be pulled is $APP_VERSION" 
+
+if [[ $APP_VERSION != $CHECKED_OUT_VERSION ]]; then
+ echo "WARNING: Checked out version and container image versions differ!"
+fi
+
 if [[ -n $DEV ]]; then
  GID=`id -g`
  SUPPLEMENTAL_GIDS=37904
@@ -93,13 +156,15 @@ hyperion.externalHostname=test-hyperion.diamond.ac.uk "
  mkdir -p $PROJECTDIR/tmp
  DEPLOYMENT_DIR=$PROJECTDIR
 else
- DEPLOYMENT_DIR=/dls_sw/i03/software/bluesky/hyperion_v${APP_VERSION}/hyperion
+ DEPLOYMENT_DIR=/dls_sw/i03/software/bluesky/mx_bluesky_v${APP_VERSION}/hyperion
 fi
 
 HELM_OPTIONS+="--set hyperion.appVersion=$APP_VERSION,\
 hyperion.projectDir=$DEPLOYMENT_DIR,\
 dodal.projectDir=$DEPLOYMENT_DIR/../dodal " 
 
+module load helm
+
 helm package $PROJECTDIR/helmchart --app-version $APP_VERSION
 # Helm package generates a file suffixed with the chart version
 helm upgrade --install $HELM_OPTIONS $RELEASE hyperion-0.0.1.tgz