Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bump persistenceagent rock to ckf-1.8 (kfp 2.0.3) #62

Merged
merged 4 commits into from
Jan 16, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 5 additions & 4 deletions persistenceagent/rockcraft.yaml
Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
# Dockerfile: https://github.com/kubeflow/pipelines/blob/2.0.3/backend/Dockerfile.persistenceagent
name: kfp-persistence
summary: Reusable end-to-end ML workflows built using the Kubeflow Pipelines SDK
description: |
This component serves as the backend persistence agent of Kubeflow pipelines.
version: 2.0.0-alpha.7_22.04_1 # version format: <KF-upstream-version>_<Charmed-KF-version>
version: 2.0.3
license: Apache-2.0
base: ubuntu:22.04
run-user: _daemon_
Expand All @@ -24,9 +25,9 @@ parts:
plugin: go
source: https://github.com/kubeflow/pipelines
source-type: git
source-tag: 2.0.0-alpha.7 # upstream branch
source-tag: 2.0.3
build-snaps:
- go
- go/1.20/stable
build-packages:
- git
- openssl
Expand All @@ -40,7 +41,7 @@ parts:
./hack/install-go-licenses.sh
$GOBIN/go-licenses check ./backend/src/agent/persistence
$GOBIN/go-licenses csv ./backend/src/agent/persistence > $CRAFT_PART_INSTALL/third_party/licenses.csv && \
diff $CRAFT_PART_INSTALL/third_party/licenses.csv backend/third_party_licenses/swf.csv && \
diff $CRAFT_PART_INSTALL/third_party/licenses.csv backend/third_party_licenses/persistence_agent.csv && \
$GOBIN/go-licenses save ./backend/src/agent/persistence --save_path $CRAFT_PART_INSTALL/third_party/NOTICES

# security requirement
Expand Down
75 changes: 41 additions & 34 deletions persistenceagent/tox.ini
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
[tox]
skipsdist = True
skip_missing_interpreters = True
envlist = pack, export-to-docker, sanity, integration

[testenv]
setenv =
Expand All @@ -12,57 +13,63 @@ setenv =
CHARM_BRANCH=main
LOCAL_CHARM_DIR=charm_repo

[testenv:unit]
[testenv:pack]
passenv = *
allowlist_externals =
bash
tox
rockcraft
deps =
juju~=2.9.0
pytest
pytest-operator
ops
commands =
# build and pack rock
rockcraft pack

[testenv:export-to-docker]
passenv = *
allowlist_externals =
bash
skopeo
yq
commands =
# pack rock and export to docker
bash -c 'NAME=$(yq eval .name rockcraft.yaml) && \
VERSION=$(yq eval .version rockcraft.yaml) && \
ARCH=$(yq eval ".platforms | keys" rockcraft.yaml | awk -F " " '\''{ print $2 }'\'') && \
ROCK="$\{NAME\}_$\{VERSION\}_$\{ARCH\}" && \
sudo skopeo --insecure-policy copy oci-archive:$ROCK.rock docker-daemon:$ROCK:$VERSION && \
docker save $ROCK > $ROCK.tar'
# run rock tests
pytest -v --tb native --show-capture=all --log-cli-level=INFO {posargs} {toxinidir}/tests
ARCH=$(yq eval ".platforms | keys | .[0]" rockcraft.yaml) && \
ROCK="$\{NAME\}_$\{VERSION\}_$\{ARCH\}.rock" && \
DOCKER_IMAGE=$NAME:$VERSION && \\
echo "Exporting $ROCK to docker as $DOCKER_IMAGE" && \
skopeo --insecure-policy copy oci-archive:$ROCK docker-daemon:$DOCKER_IMAGE'

[testenv:sanity]
passenv = *
allowlist_externals =
echo
commands =
# TODO: Implement sanity tests
echo "WARNING: This is a placeholder test - no test is implemented here."

[testenv:integration]
passenv = *
allowlist_externals =
echo
bash
git
rm
tox
rockcraft
deps =
juju~=2.9.0
juju<4.0
pytest
pytest-operator
ops
commands =
# build and pack rock
rockcraft pack
# clone related charm
rm -rf {env:LOCAL_CHARM_DIR}
git clone --branch {env:CHARM_BRANCH} {env:CHARM_REPO} {env:LOCAL_CHARM_DIR}
# upload rock to docker and microk8s cache, replace charm's container with local rock reference
bash -c 'NAME=$(yq eval .name rockcraft.yaml) && \
VERSION=$(yq eval .version rockcraft.yaml) && \
ARCH=$(yq eval ".platforms | keys" rockcraft.yaml | awk -F " " '\''{ print $2 }'\'') && \
ROCK="$\{NAME\}_$\{VERSION\}_$\{ARCH\}" && \
sudo skopeo --insecure-policy copy oci-archive:$ROCK.rock docker-daemon:$ROCK:$VERSION && \
docker save $ROCK > $ROCK.tar && \
microk8s ctr image import $ROCK.tar && \
yq e -i ".resources.oci-image.upstream-source=\"$ROCK:$VERSION\"" {env:LOCAL_CHARM_DIR}/charms/kfp-persistence/metadata.yaml'
# run charm integration test with rock
tox -c {env:LOCAL_CHARM_DIR} -e integration

echo "WARNING: This is a placeholder test - no test is implemented here."
; Remove warning and uncomment below once https://github.com/canonical/kfp-operators/issues/377
; is resolved
; # clone related charm
; rm -rf {env:LOCAL_CHARM_DIR}
; git clone --branch {env:CHARM_BRANCH} {env:CHARM_REPO} {env:LOCAL_CHARM_DIR}
; # upload rock to docker and microk8s cache, replace charm's container with local rock reference
; bash -c 'NAME=$(yq eval .name rockcraft.yaml) && \
; VERSION=$(yq eval .version rockcraft.yaml) && \
; DOCKER_IMAGE=$NAME:$VERSION && \
; docker save $DOCKER_IMAGE > $DOCKER_IMAGE.tar && \
; sudo microk8s ctr image import $DOCKER_IMAGE.tar --digests=true && \
; yq e -i ".resources.oci-image.upstream-source=\"$DOCKER_IMAGE\"" {env:LOCAL_CHARM_DIR}/charms/kfp-persistence/metadata.yaml'
; # run bundle integration tests with rock
; tox -c {env:LOCAL_CHARM_DIR} -e bundle-integration-v2 -- --model kubeflow
Loading