[ODSC-48942]opctl create/publish error on M1 and M2 #381

z7ye · 2023-10-20T23:39:26Z

https://jira.oci.oraclecorp.com/browse/ODSC-48942

Testing Step

build the image

ads opctl build-image job-local

conda create

ads opctl conda create -n testing -f ~/Downloads/environment.yaml

conda publish

ads opctl conda publish -s /Users/z7ye/conda/testing_v1 -a security_token

Evidence

github-actions · 2023-10-21T00:41:06Z

📌 Cov diff with main:

No lines with coverage information in this diff.

📌 Overall coverage:

github-actions · 2023-10-24T05:09:16Z

⚠️ This PR changed pyproject.toml file. ⚠️

PR Creator must update 📃 THIRD_PARTY_LICENSES.txt, if any 📚 library added/removed in pyproject.toml.
PR Approver must confirm 📃 THIRD_PARTY_LICENSES.txt updated, if any 📚 library added/removed in pyproject.toml.

github-actions · 2023-10-24T06:08:03Z

📌 Cov diff with main:

📌 Overall coverage:

ads/opctl/conda/cmds.py

mrDzurb · 2023-10-24T22:37:50Z

ads/opctl/conda/pack.py

    with tempfile.TemporaryDirectory() as td:
        process = subprocess.Popen(
            ["conda", "env", "export", "--prefix", pack_folder_path],
            stdout=subprocess.PIPE,
            stderr=subprocess.PIPE,
        )
        stdout, stderr = process.communicate()
-        if stderr:
+
+        if process.returncode and stderr:
            print(stderr)


Let's use logger for debug purpose?

I remember we could only use print() to show the msgs when running this code in container?

Yep yep, you are right.

mrDzurb · 2023-10-24T22:42:04Z

ads/opctl/docker/Dockerfile_arm.job

NIT: Maybe Dockerfile.arm.job?

good suggestion! will change.

mrDzurb · 2023-10-24T22:46:23Z

ads/opctl/docker/Dockerfile_arm.job

Just curious, the ARM docker file looks much bigger than x86 one? Was it copied from the jupyter repo?
This file I guess needs to be cleaned up. The Jobs file was much lighter in comparison with the jupyter one. We don't need to install jlab related libraries for example, and YAM i think needs to be installed differently.

updated with Liuda's PR

mrDzurb · 2023-10-24T22:58:23Z

ads/opctl/docker/base-env-arm.yaml

Is this base yaml different from the one that we already have?

updated with Liuda's PR

mrDzurb · 2023-10-24T23:02:32Z

ads/opctl/docker/Symantec_Private_SSL_SHA256_CA.cer

We probably don't need this certificates if we get rid of the libraries that relay on it.

mrDzurb · 2023-10-24T23:02:56Z

ads/opctl/docker/common/install-runtime-deps.sh

Do we really need this file?

updated with Liuda's PR

mrDzurb · 2023-10-24T23:03:32Z

ads/opctl/docker/etc/supervisord.conf

We probably don't need this file, if we don't want to run the JLab server?

updated with Liuda's PR

mrDzurb · 2023-10-24T23:04:00Z

ads/opctl/docker/etc/supervisord.d/jupyterlab.ini

We probably can get rid of this file?

updated with Liuda's PR

github-actions · 2023-10-25T18:23:02Z

⚠️ This PR changed pyproject.toml file. ⚠️

PR Creator must update 📃 THIRD_PARTY_LICENSES.txt, if any 📚 library added/removed in pyproject.toml.
PR Approver must confirm 📃 THIRD_PARTY_LICENSES.txt updated, if any 📚 library added/removed in pyproject.toml.

github-actions · 2023-10-25T19:26:16Z

📌 Cov diff with main:

📌 Overall coverage:

…into bug_fix

github-actions · 2023-10-27T21:34:03Z

⚠️ This PR changed pyproject.toml file. ⚠️

PR Creator must update 📃 THIRD_PARTY_LICENSES.txt, if any 📚 library added/removed in pyproject.toml.
PR Approver must confirm 📃 THIRD_PARTY_LICENSES.txt updated, if any 📚 library added/removed in pyproject.toml.

github-actions · 2023-10-27T22:16:41Z

⚠️ This PR changed pyproject.toml file. ⚠️

PR Creator must update 📃 THIRD_PARTY_LICENSES.txt, if any 📚 library added/removed in pyproject.toml.
PR Approver must confirm 📃 THIRD_PARTY_LICENSES.txt updated, if any 📚 library added/removed in pyproject.toml.

github-actions · 2023-10-27T22:17:28Z

📌 Cov diff with main:

📌 Overall coverage:

github-actions · 2023-10-27T23:16:52Z

📌 Cov diff with main:

📌 Overall coverage:

ads/opctl/conda/cmds.py

mrDzurb · 2023-10-30T16:19:58Z

ads/opctl/conda/pack.py

    with tempfile.TemporaryDirectory() as td:
        process = subprocess.Popen(
            ["conda", "env", "export", "--prefix", pack_folder_path],
            stdout=subprocess.PIPE,
            stderr=subprocess.PIPE,
        )
        stdout, stderr = process.communicate()
-        if stderr:
+
+        if process.returncode and stderr:
            print(stderr)


Yep yep, you are right.

mrDzurb · 2023-10-30T16:27:52Z

ads/opctl/utils.py

@@ -161,7 +162,14 @@ def build_image(
            )
        proc = _build_custom_operator_image(gpu, source_folder, dst_image)
    else:
-        image, dockerfile, target = _get_image_name_dockerfile_target(image_type, gpu)
+        # https://stackoverflow.com/questions/66842004/get-the-processor-type-using-python-for-apple-m1-processor-gives-me-an-intel-pro
+        import cpuinfo


It might be better to create a separate function - _get_architecture()? We probably will reuse this method in different other places.

github-actions · 2023-11-02T21:18:56Z

⚠️ This PR changed pyproject.toml file. ⚠️

PR Creator must update 📃 THIRD_PARTY_LICENSES.txt, if any 📚 library added/removed in pyproject.toml.
PR Approver must confirm 📃 THIRD_PARTY_LICENSES.txt updated, if any 📚 library added/removed in pyproject.toml.

github-actions · 2023-11-02T22:12:28Z

📌 Cov diff with main:

📌 Overall coverage:

github-actions · 2023-11-13T18:03:43Z

⚠️ This PR changed pyproject.toml file. ⚠️

PR Creator must update 📃 THIRD_PARTY_LICENSES.txt, if any 📚 library added/removed in pyproject.toml.
PR Approver must confirm 📃 THIRD_PARTY_LICENSES.txt updated, if any 📚 library added/removed in pyproject.toml.

github-actions · 2023-11-13T18:04:31Z

📌 Cov diff with main:

No success to gather report. 😿

📌 Overall coverage:

No success to gather report. 😿

github-actions · 2023-11-13T18:16:49Z

📌 Cov diff with main:

No success to gather report. 😿

📌 Overall coverage:

No success to gather report. 😿

liudmylaru · 2023-10-24T20:28:34Z

ads/opctl/docker/Dockerfile_arm.job

+WORKDIR /home/datascience
+
+COPY base-env-arm.yaml /opt/base-env.yaml
+COPY jupyter-arm.yaml /opt/jupyter.yaml


I think we need only 1 yaml here. In Dockerfile.job - we have one. And in jobs Dockerfile.arm (https://bitbucket.oci.oraclecorp.com/projects/ODSC/repos/odsc_jobs/browse/jobs/Dockerfile.arm) also 1 yaml used - https://bitbucket.oci.oraclecorp.com/projects/ODSC/repos/odsc_jobs/browse/jobs/Dockerfile.arm#89-90

liudmylaru · 2023-10-24T20:31:52Z

ads/opctl/docker/Dockerfile_arm.job

+ARG PIP_INDEX_URL
+
+############# Setup Conda environment tools ###########################
+ARG RAND=1


We creating special directory for operators, setting RUN_WORKING_DIR etc., not sue if this a requirement for the image, but Dockerfile.job has this lines - https://github.com/oracle/accelerated-data-science/blob/main/ads/opctl/docker/Dockerfile.job#L108C1-L108C17
And we need to finish file with USER datascience. I would suggest to copy lines from link above to this Dockerfile also.

liudmylaru · 2023-10-24T20:36:49Z

ads/opctl/docker/etc/yum.repos.d/ol7x-base-default.repo

@@ -0,0 +1,8 @@
+[ol7-yum]
+name=ol7-yum
+baseurl=https://artifactory.oci.oraclecorp.com/ol7-yum/


Is Artifactory in reach for users? This is not a public repository

liudmylaru · 2023-10-24T20:37:53Z

pyproject.toml

@@ -123,6 +123,7 @@ opctl = [
  "nbconvert",
  "nbformat",
  "oci-cli",
+  "py-cpuinfo",


New library will need an input into https://github.com/oracle/accelerated-data-science/blob/main/THIRD_PARTY_LICENSES.txt

liudmylaru · 2023-10-24T20:41:26Z

ads/opctl/docker/Dockerfile_arm.job

+WORKDIR /home/datascience
+
+# Note in order to run sudo commands as a non root user, you must specify --credential yes if using qemu static to build the image
+RUN wget -nv https://artifactory.oci.oraclecorp.com/odsc-dev-generic-local/datascience/Miniconda3-${MINICONDA_VER}-$(uname)-$(uname -m).sh \


We can't use here link to artifactory, this is internal resource.
Use repo.anaconda link for miniconda - https://bitbucket.oci.oraclecorp.com/projects/ODSC/repos/odsc_jobs/browse/jobs/Dockerfile.arm#73
And gihub for miniforge, like described here - https://github.com/conda-forge/miniforge#mambaforge.

…o bug_fix

github-actions · 2023-11-14T17:51:09Z

⚠️ This PR changed pyproject.toml file. ⚠️

PR Creator must update 📃 THIRD_PARTY_LICENSES.txt, if any 📚 library added/removed in pyproject.toml.
PR Approver must confirm 📃 THIRD_PARTY_LICENSES.txt updated, if any 📚 library added/removed in pyproject.toml.

github-actions · 2023-11-14T18:22:04Z

📌 Cov diff with main:

📌 Overall coverage:

github-actions · 2023-11-14T19:49:56Z

⚠️ This PR changed pyproject.toml file. ⚠️

PR Creator must update 📃 THIRD_PARTY_LICENSES.txt, if any 📚 library added/removed in pyproject.toml.
PR Approver must confirm 📃 THIRD_PARTY_LICENSES.txt updated, if any 📚 library added/removed in pyproject.toml.

github-actions · 2023-11-14T20:24:13Z

📌 Cov diff with main:

📌 Overall coverage:

first working version

1f18adc

oracle-contributor-agreement bot added the OCA Verified All contributors have signed the Oracle Contributor Agreement. label Oct 20, 2023

z7ye changed the title ~~first working version~~ opctl create/publish error on M1 and M2 Oct 20, 2023

z7ye added 2 commits October 23, 2023 22:07

use arm base image when local machine is detected as m# machine

f3f189d

use arm base image when local machine is detected as m# machine

c29bd5d

z7ye changed the title ~~opctl create/publish error on M1 and M2~~ [ODSC-48942]opctl create/publish error on M1 and M2 Oct 24, 2023

mrDzurb reviewed Oct 24, 2023

View reviewed changes

mayoor force-pushed the bug_fix branch from c29bd5d to f3f189d Compare October 25, 2023 18:22

z7ye requested a review from mrDzurb October 27, 2023 17:31

z7ye added 2 commits October 27, 2023 10:32

Merge branch 'bug_fix' of github.com:oracle/accelerated-data-science …

509a052

…into bug_fix

rebase to main

32761ee

revert back the path for docker build

8535ef1

z7ye requested a review from liudmylaru October 27, 2023 22:17

mrDzurb previously approved these changes Oct 30, 2023

View reviewed changes

update the code based on comments

e16021d

z7ye dismissed mrDzurb’s stale review via e16021d November 2, 2023 21:18

z7ye requested a review from mrDzurb November 2, 2023 21:18

mrDzurb previously approved these changes Nov 3, 2023

View reviewed changes

fix the unit test

5bc1dc9

z7ye dismissed mrDzurb’s stale review via 5bc1dc9 November 13, 2023 18:03

z7ye requested a review from mrDzurb November 13, 2023 18:03

z7ye requested a review from qiuosier November 13, 2023 18:03

liudmylaru previously approved these changes Nov 13, 2023

View reviewed changes

Merge branch 'main' of github.com:oracle/accelerated-data-science int…

65cec58

…o bug_fix

mrDzurb previously approved these changes Nov 14, 2023

View reviewed changes

adding license info for py-cpuinfo

8a57a5d

z7ye dismissed stale reviews from mrDzurb and liudmylaru via 8a57a5d November 14, 2023 19:49

z7ye requested review from mrDzurb and liudmylaru November 14, 2023 19:49

liudmylaru approved these changes Nov 14, 2023

View reviewed changes

mrDzurb approved these changes Nov 15, 2023

View reviewed changes

mrDzurb merged commit 34105b5 into main Nov 15, 2023
13 of 14 checks passed

[ODSC-48942]opctl create/publish error on M1 and M2 #381

[ODSC-48942]opctl create/publish error on M1 and M2 #381

Conversation

z7ye commented Oct 20, 2023 • edited Loading

Testing Step

build the image

conda create

conda publish

Evidence

github-actions bot commented Oct 21, 2023

github-actions bot commented Oct 24, 2023

github-actions bot commented Oct 24, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Oct 25, 2023

github-actions bot commented Oct 25, 2023

github-actions bot commented Oct 27, 2023

github-actions bot commented Oct 27, 2023

github-actions bot commented Oct 27, 2023

github-actions bot commented Oct 27, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Nov 2, 2023

github-actions bot commented Nov 2, 2023

github-actions bot commented Nov 13, 2023

github-actions bot commented Nov 13, 2023

github-actions bot commented Nov 13, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Nov 14, 2023

github-actions bot commented Nov 14, 2023

github-actions bot commented Nov 14, 2023

github-actions bot commented Nov 14, 2023

z7ye commented Oct 20, 2023 •

edited

Loading