Added Ml metadata server rock image #160

BON4 · 2024-12-02T13:17:56Z

Description

Add rockcraft.yaml
Add tests
Add tox.ini

This is re-open request for #115.

Using mlmd-operator. Modified metadata.yaml. Replaced "gcr.io/tfx-oss-public/ml_metadata_store_server:1.14.0" with "vyurchenko581/ml-metadata-store-server:1.14.0".

Juju status after all tests passed:

Model     Controller          Cloud/Region        Version  SLA          Timestamp
kubeflow  microk8s-localhost  microk8s/localhost  3.4.6    unsupported  22:16:25+02:00

App                Version  Status   Scale  Charm              Channel        Rev  Address         Exposed  Message
grafana-agent-k8s  0.32.1   waiting      1  grafana-agent-k8s  latest/stable   45  10.152.183.246  no       installing agent
mlmd                        active       1  mlmd                                0  10.152.183.183  no       

Unit                  Workload  Agent  Address      Ports  Message
grafana-agent-k8s/0*  blocked   idle   10.1.34.132         logging-consumer: off, grafana-cloud-config: off
mlmd/0*               active    idle   10.1.34.180

Logs of passed integration tests tox -vve integration -- --model kubeflow --keep-models:

============================================================================================= test session starts ==============================================================================================
platform linux -- Python 3.10.12, pytest-8.1.1, pluggy-1.4.0 -- /home/bon/repos/mlmd-operator/.tox/integration/bin/python
cachedir: .tox/integration/.pytest_cache
rootdir: /home/bon/repos/mlmd-operator
configfile: pyproject.toml
plugins: operator-0.34.0, asyncio-0.21.1, anyio-4.4.0
asyncio: mode=auto
collected 3 items                                                                                                                                                                                              

tests/integration/test_charm.py::test_build_and_deploy 
------------------------------------------------------------------------------------------------ live log setup ------------------------------------------------------------------------------------------------
INFO     pytest_operator.plugin:plugin.py:708 Connecting to existing model microk8s-localhost:kubeflow on unspecified cloud
------------------------------------------------------------------------------------------------ live log call -------------------------------------------------------------------------------------------------
INFO     pytest_operator.plugin:plugin.py:561 Using tmp_path: /home/bon/repos/mlmd-operator/.tox/integration/tmp/pytest/kubeflow0
INFO     pytest_operator.plugin:plugin.py:1031 Building charm mlmd
INFO     pytest_operator.plugin:plugin.py:1036 Built charm mlmd in 29.63s
INFO     test_charm:test_charm.py:28 Built charm /home/bon/repos/mlmd-operator/.tox/integration/tmp/pytest/kubeflow0/charms/mlmd_ubuntu-20.04-amd64.charm
INFO     juju.model:model.py:2097 Deploying local:mlmd-0
INFO     juju.model:model.py:2957 Waiting for model:
  mlmd/0 [allocating] waiting: installing agent
INFO     juju.model:model.py:2957 Waiting for model:
  mlmd/0 [allocating] waiting: agent initialising
INFO     juju.model:model.py:2957 Waiting for model:
  mlmd/0 [idle] active: 
INFO     charmed_kubeflow_chisme.testing.cos_integration:cos_integration.py:65 deploying grafana-agent-k8s from latest/stable channel
INFO     juju.model:model.py:2097 Deploying ch:amd64/jammy/grafana-agent-k8s-45
INFO     charmed_kubeflow_chisme.testing.cos_integration:cos_integration.py:95 Adding relation: mlmd:logging and grafana-agent-k8s:logging-provider
INFO     juju.model:model.py:2957 Waiting for model:
  grafana-agent-k8s/0 [allocating] waiting: installing agent
INFO     juju.model:model.py:2957 Waiting for model:
  grafana-agent-k8s/0 [allocating] waiting: agent initialising
INFO     juju.model:model.py:2957 Waiting for model:
  grafana-agent-k8s/0 [idle] blocked: logging-consumer: off, grafana-cloud-config: off
PASSED
tests/integration/test_charm.py::test_logging 
------------------------------------------------------------------------------------------------ live log call -------------------------------------------------------------------------------------------------
INFO     charmed_kubeflow_chisme.testing.cos_integration:cos_integration.py:138 found relations [<Relation id=1 mlmd:logging grafana-agent-k8s:logging-provider>] for grafana-agent-k8s:logging
INFO     charmed_kubeflow_chisme.testing.cos_integration:cos_integration.py:238 running cmd `relation-get --format=yaml -r 1 - grafana-agent-k8s/0` on unit grafana-agent-k8s/0
PASSED
tests/integration/test_charm.py::test_using_charm 
------------------------------------------------------------------------------------------------ live log call -------------------------------------------------------------------------------------------------
INFO     root:test_charm.py:63 Using temporary directory /home/bon/repos/mlmd-operator/.tox/integration/tmp/pytest/test_using_charm0
INFO     root:test_charm.py:64 cwd = /home/bon/repos/mlmd-operator
INFO     root:test_charm.py:65 script_abs_path = /home/bon/repos/mlmd-operator/tests/integration/data/interact_with_mlmd.sh
+ MODEL=kubeflow
MODEL=kubeflow
+ echo MODEL=kubeflow
+ rm -rf ml_metadata/proto
+ wget https://raw.githubusercontent.com/google/ml-metadata/master/ml_metadata/proto/metadata_store.proto
--2024-12-01 22:15:37--  https://raw.githubusercontent.com/google/ml-metadata/master/ml_metadata/proto/metadata_store.proto
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.111.133, 185.199.109.133, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 52201 (51K) [text/plain]
Saving to: ‘metadata_store.proto’

metadata_store.proto                                100%[===================================================================================================================>]  50.98K  --.-KB/s    in 0.05s   

2024-12-01 22:15:38 (1.08 MB/s) - ‘metadata_store.proto’ saved [52201/52201]

+ wget https://raw.githubusercontent.com/google/ml-metadata/master/ml_metadata/proto/metadata_store_service.proto
--2024-12-01 22:15:38--  https://raw.githubusercontent.com/google/ml-metadata/master/ml_metadata/proto/metadata_store_service.proto
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.110.133, 185.199.109.133, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 55595 (54K) [text/plain]
Saving to: ‘metadata_store_service.proto’

metadata_store_service.proto                        100%[===================================================================================================================>]  54.29K  --.-KB/s    in 0.04s   

2024-12-01 22:15:38 (1.19 MB/s) - ‘metadata_store_service.proto’ saved [55595/55595]

+ wget -O- https://github.com/fullstorydev/grpcurl/releases/download/v1.8.0/grpcurl_1.8.0_linux_x86_64.tar.gz
+ tar -xzv
--2024-12-01 22:15:38--  https://github.com/fullstorydev/grpcurl/releases/download/v1.8.0/grpcurl_1.8.0_linux_x86_64.tar.gz
Resolving github.com (github.com)... 140.82.121.3
Connecting to github.com (github.com)|140.82.121.3|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/111431261/88be8800-4f53-11eb-94fd-6a7ef1143069?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20241201%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20241201T201538Z&X-Amz-Expires=300&X-Amz-Signature=15678bc0f7e071b32801fa8904c801a9995fb0f2e73e26d2a32b2b342d5502f4&X-Amz-SignedHeaders=host&response-content-disposition=attachment%3B%20filename%3Dgrpcurl_1.8.0_linux_x86_64.tar.gz&response-content-type=application%2Foctet-stream [following]
--2024-12-01 22:15:38--  https://objects.githubusercontent.com/github-production-release-asset-2e65be/111431261/88be8800-4f53-11eb-94fd-6a7ef1143069?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20241201%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20241201T201538Z&X-Amz-Expires=300&X-Amz-Signature=15678bc0f7e071b32801fa8904c801a9995fb0f2e73e26d2a32b2b342d5502f4&X-Amz-SignedHeaders=host&response-content-disposition=attachment%3B%20filename%3Dgrpcurl_1.8.0_linux_x86_64.tar.gz&response-content-type=application%2Foctet-stream
Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.108.133, 185.199.111.133, 185.199.109.133, ...
Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.108.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 4986341 (4.8M) [application/octet-stream]
Saving to: ‘STDOUT’

-                                                     0%[                                                                                                                    ]       0  --.-KB/s               LICENSE
grpcurl
-                                                   100%[===================================================================================================================>]   4.75M  4.98MB/s    in 1.0s    

2024-12-01 22:15:40 (4.98 MB/s) - written to stdout [4986341/4986341]

+ mkdir -p ml_metadata/proto/
+ mv metadata_store.proto ml_metadata/proto/
++ microk8s kubectl get services/metadata-grpc-service -n kubeflow -oyaml
++ yq e .spec.clusterIP -
+ SERVICE=10.152.183.194
+ ./grpcurl -v --proto=metadata_store_service.proto --plaintext 10.152.183.194:8080 ml_metadata.MetadataStoreService/GetArtifacts

Resolved method descriptor:
// Gets all the artifacts.
rpc GetArtifacts ( .ml_metadata.GetArtifactsRequest ) returns ( .ml_metadata.GetArtifactsResponse );

Request metadata to send:
(empty)

Response headers received:
content-type: application/grpc
grpc-accept-encoding: identity, deflate, gzip

Response contents:
{
  
}

Response trailers received:
(empty)
Sent 0 requests and received 1 response
PASSED
---------------------------------------------------------------------------------------------- live log teardown -----------------------------------------------------------------------------------------------
INFO     pytest_operator.plugin:plugin.py:834 Model status:

Model     Controller          Cloud/Region        Version  SLA          Timestamp
kubeflow  microk8s-localhost  microk8s/localhost  3.4.6    unsupported  22:15:41+02:00

App                Version  Status   Scale  Charm              Channel        Rev  Address         Exposed  Message
grafana-agent-k8s  0.32.1   waiting      1  grafana-agent-k8s  latest/stable   45  10.152.183.246  no       installing agent
mlmd                        active       1  mlmd                                0  10.152.183.183  no       

Unit                  Workload  Agent  Address      Ports  Message
grafana-agent-k8s/0*  blocked   idle   10.1.34.132         logging-consumer: off, grafana-cloud-config: off
mlmd/0*               active    idle   10.1.34.180         

INFO     pytest_operator.plugin:plugin.py:840 Juju error logs:


INFO     pytest_operator.plugin:plugin.py:904 Forgetting main...


======================================================================================== 3 passed in 200.52s (0:03:20) =========================================================================================
integration: 219884 I exit 0 (201.22 seconds) /home/bon/repos/mlmd-operator> pytest -v --tb native --asyncio-mode=auto /home/bon/repos/mlmd-operator/tests/integration --log-cli-level=INFO -s --model kubeflow --keep-models pid=740143 [tox/execute/api.py:280]
  integration: OK (219.78=setup[18.56]+cmd[201.22] seconds)
  congratulations :) (219.82 seconds)

NohaIhab

thanks @BON4, I left a few comments kindly address them

ml-metadata-store-server/rockcraft.yaml

NohaIhab

thx @BON4, almost there

ml-metadata/rockcraft.yaml

Co-authored-by: Noha Ihab <49988746+NohaIhab@users.noreply.github.com>

misohu

Run local build of rock + tests and looks like everything is working. Good job :)

BON4 added 2 commits December 2, 2024 09:56

ml-metadata-store-server rock

36d84d4

ml-metadata-store-server rock and test

839d2bd

BON4 requested a review from a team as a code owner December 2, 2024 13:17

ml-metadata-store-server rock and test

8603bfb

NohaIhab mentioned this pull request Dec 6, 2024

Review ml-metadata-store rock #134

Closed

NohaIhab reviewed Dec 6, 2024

View reviewed changes

BON4 added 4 commits December 12, 2024 21:57

ml-metadata-store-server rock and test

08d0828

ml-metadata-store-server rock and test

bf00545

ml-metadata-store-server rock and test

35182f6

ml-metadata-store-server rock and test

5e77b3d

BON4 requested a review from NohaIhab December 18, 2024 10:12

NohaIhab mentioned this pull request Dec 18, 2024

Integrate the rock into mlmd-operator charm canonical/mlmd-operator#111

Closed

NohaIhab reviewed Dec 20, 2024

View reviewed changes

ml-metadata/rockcraft.yaml Outdated Show resolved Hide resolved

Update ml-metadata/rockcraft.yaml

d0d4eb6

Co-authored-by: Noha Ihab <49988746+NohaIhab@users.noreply.github.com>

BON4 requested a review from NohaIhab January 8, 2025 08:15

misohu approved these changes Jan 10, 2025

View reviewed changes

NohaIhab merged commit 2a80c7b into canonical:main Jan 13, 2025
7 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added Ml metadata server rock image #160

Added Ml metadata server rock image #160

BON4 commented Dec 2, 2024

NohaIhab left a comment

NohaIhab left a comment

misohu left a comment

Added Ml metadata server rock image #160

Added Ml metadata server rock image #160

Conversation

BON4 commented Dec 2, 2024

Description

NohaIhab left a comment

Choose a reason for hiding this comment

NohaIhab left a comment

Choose a reason for hiding this comment

misohu left a comment

Choose a reason for hiding this comment