BGD-6054 bigdata-operator update for spark-watcher 0.6.3 archive

[abunghez]

Jira ticket

https://spotinst.atlassian.net/browse/BGD-6054

Description

Update bigdata-operator chart after spark-watcher embedded chart update.

Demo

See integration tests below.

Checklist

• I have added a Jira ticket link
• I have filled in the test plan
• I have executed the tests and filled in the test results
• I have updated/created relevant documentation

How to test

Integration tests on prod cluster

Test plan and results

Test	Description	Result	Notes
1	Integration tests on prod cluster	Pass

[abunghez]

bandrei@bandrei-mac-0 ~/code/bigdata-charts %  devctl cp test run-with-params --token=$DEVCTL_PAT --accountId=$DEVCTL_ACCOUNT --clusterId=$DEVCTL_CLUSTER --optimizer-url=$OPTIMIZER_URL --concurrency-limit=5
2:22PM INF Checking cluster availability clusterId=osc-940e76ef
2:22PM INF Cluster is available, tests are ready to run clusterId=osc-940e76ef
2:22PM INF Start running tests concurrency-limit=5 now=2024-11-25T14:22:58+01:00 version=v1.0.48
2:22PM INF Optimizer Url: https://api.spotinst.io
2:22PM INF accountId: act-12f6b1b9
2:22PM INF clusterId: osc-940e76ef
2:22PM INF sparkProps: [{Version:3.5.1 Image:}]
2:22PM INF 20 tests selected: [list_applications, kill_application_pending_3.5.1, kill_application_running_3.5.1, kill_application_completed_3.5.1, list_jobs, get_job, put_job, list_clusters, get_config_template, create_valid_config_template, create_invalid_config_template, list_config_templates, put_config_templates, delete_config_template, run_workspace, run_spark_connect_application-3.5.1, run_spark_connect_application_code-3.5.1, run_notebook_application-python-3.5.1, run_notebook_application-scala-3.5.1, run_application-3.5.1]
2:22PM INF Starting suite=get_job
2:22PM INF Starting suite=list_jobs
2:22PM INF Starting suite=list_applications
2:22PM INF Starting suite=kill_application_completed_3.5.1
2:22PM INF Starting suite=kill_application_running_3.5.1
2:22PM INF Submitted app integration-tests-job-get-lthjic-662d4-stuff suite=get_job
2:22PM INF Submitted app integration-tests-09cee-tuner suite=kill_application_completed_3.5.1
2:22PM INF Submitted app integration-tests-14b04-input suite=kill_application_running_3.5.1
2:22PM INF Success suite=list_jobs
2:22PM INF Starting suite=put_job
2:22PM INF Success suite=list_applications
2:22PM INF Starting suite=list_clusters
2:22PM INF Job fetched successfully suite=get_job
2:22PM INF Got a correct job suite=get_job
2:22PM INF App is processed suite=kill_application_completed_3.5.1
2:22PM INF App is processed suite=kill_application_running_3.5.1
2:22PM INF Success suite=list_clusters
2:22PM INF Starting suite=get_config_template
2:23PM INF Config template created successfully suite=get_config_template
2:23PM INF Submitted app integration-tests-job-put-jugygv-6711c-dimly suite=put_job
2:23PM INF Job config has updated suite=put_job
2:23PM INF Updated job labels are correct suite=put_job
2:23PM INF Config template fetched successfully suite=get_config_template
2:23PM INF Got a correct config template suite=get_config_template
2:23PM INF Success suite=get_config_template
2:23PM INF Starting suite=create_valid_config_template
2:23PM INF Trying to create config template integration-tests-config-template-create-uilrxr suite=create_valid_config_template
2:23PM INF Created config template integration-tests-config-template-create-uilrxr suite=create_valid_config_template
2:23PM INF Trying to create config template with same name integration-tests-config-template-create-uilrxr suite=create_valid_config_template
2:23PM INF Successfully received bad request response for integration-tests-config-template-create-uilrxr suite=create_valid_config_template
2:23PM INF Success suite=create_valid_config_template
2:23PM INF Starting suite=create_invalid_config_template
2:23PM INF Trying to create invalid config template integration-tests-config-template-create-ftcptg suite=create_invalid_config_template
2:23PM INF Successfully received unprocessable entity response for integration-tests-config-template-create-ftcptg suite=create_invalid_config_template
2:23PM INF Success suite=create_invalid_config_template
2:23PM INF Starting suite=list_config_templates
2:23PM INF Config template created successfully suite=list_config_templates
2:23PM INF Trying to list the config templates suite=list_config_templates
2:23PM INF Config template fetched successfully suite=list_config_templates
2:23PM INF Got a correct config template list suite=list_config_templates
2:23PM INF Success suite=list_config_templates
2:23PM INF Starting suite=put_config_templates
2:23PM INF Config template created successfully suite=put_config_templates
2:23PM INF Config template successfully updated suite=put_config_templates
2:23PM INF Config template fetched successfully suite=put_config_templates
2:23PM INF Got a correct config template suite=put_config_templates
2:23PM INF Success suite=put_config_templates
2:23PM INF Starting suite=delete_config_template
2:23PM INF Config template created successfully suite=delete_config_template
2:23PM INF Config template successfully deleted suite=delete_config_template
2:23PM INF Success suite=delete_config_template
2:23PM INF Starting suite=run_workspace
2:23PM INF Workspace creation test suite=run_workspace
2:23PM INF Workspace created: wksp-324159-5b989eb3 suite=run_workspace
2:23PM INF Workspace is running: wksp-324159-5b989eb3 suite=run_workspace
2:23PM INF Workspace killed: wksp-324159-5b989eb3 suite=run_workspace
2:23PM INF Success suite=run_workspace
2:23PM INF Starting suite=run_spark_connect_application-3.5.1
2:23PM INF Testing spark connect app submit with properties {3.5.1 } suite=run_spark_connect_application-3.5.1
2:23PM INF Submitted spark connect app integration-tests-7955e-doles suite=run_spark_connect_application-3.5.1
2:23PM INF Tests status failure=[] in_progress=["run_spark_connect_application-3.5.1","kill_application_running_3.5.1","kill_application_completed_3.5.1","get_job","put_job"] success=10 to_do=4
2:24PM INF Tests status failure=[] in_progress=["put_job","run_spark_connect_application-3.5.1","kill_application_running_3.5.1","kill_application_completed_3.5.1","get_job"] success=10 to_do=4
2:25PM INF Tests status failure=[] in_progress=["kill_application_running_3.5.1","kill_application_completed_3.5.1","get_job","run_spark_connect_application-3.5.1","put_job"] success=10 to_do=4
2:26PM INF App is now running. suite=kill_application_running_3.5.1
2:26PM INF App killed suite=kill_application_running_3.5.1
2:26PM INF Success suite=kill_application_running_3.5.1
2:26PM INF Starting suite=run_spark_connect_application_code-3.5.1
2:26PM INF Testing spark connect app submit with properties {3.5.1 } suite=run_spark_connect_application_code-3.5.1
2:26PM INF Submitted spark connect app integration-tests-cd434-major suite=run_spark_connect_application_code-3.5.1
2:26PM INF Spark connect app is now running suite=run_spark_connect_application_code-3.5.1
2:26PM INF Success suite=get_job
2:26PM INF Starting suite=run_notebook_application-python-3.5.1
2:26PM INF Creating config template it-notebook-python-mbaejx. suite=run_notebook_application-python-3.5.1
2:26PM INF Testing Spark submit with properties Python suite=run_notebook_application-python-3.5.1
2:26PM INF Success suite=put_job
2:26PM INF Starting suite=run_notebook_application-scala-3.5.1
2:26PM INF Creating config template it-notebook-scala-sopmlv. suite=run_notebook_application-scala-3.5.1
2:26PM INF Testing Spark submit with properties Scala suite=run_notebook_application-scala-3.5.1
2:26PM INF Spark connect app is now running suite=run_spark_connect_application-3.5.1
2:26PM INF Spark connect app killed suite=run_spark_connect_application-3.5.1
2:26PM INF Success suite=run_spark_connect_application-3.5.1
2:26PM INF Starting suite=run_application-3.5.1
2:26PM INF Testing Spark submit with properties {3.5.1 } suite=run_application-3.5.1
2:26PM INF Tests status failure=[] in_progress=["run_spark_connect_application_code-3.5.1","run_application-3.5.1","kill_application_completed_3.5.1","run_notebook_application-python-3.5.1","run_notebook_application-scala-3.5.1"] success=14 to_do=0
2:26PM INF Submitted app integration-tests-b887b-crawl suite=run_application-3.5.1
2:26PM INF App is processed suite=run_application-3.5.1
2:27PM INF Live kubernetes logs are available suite=run_application-3.5.1
2:27PM INF App is now running. suite=run_application-3.5.1
2:27PM INF Live driver logs are available suite=run_application-3.5.1
2:27PM INF Listing cluster notebook applications for type: Python suite=run_notebook_application-python-3.5.1
2:27PM INF App is processed suite=run_notebook_application-python-3.5.1
2:27PM INF App is now running. suite=run_notebook_application-python-3.5.1
2:27PM INF Get App status. suite=run_notebook_application-python-3.5.1
2:27PM INF Cleaning up config template it-notebook-python-mbaejx. suite=run_notebook_application-python-3.5.1
2:27PM INF Success suite=run_notebook_application-python-3.5.1
2:27PM INF Listing cluster notebook applications for type: Scala suite=run_notebook_application-scala-3.5.1
2:27PM INF App is processed suite=run_notebook_application-scala-3.5.1
2:27PM INF App is now running. suite=run_notebook_application-scala-3.5.1
2:27PM INF Get App status. suite=run_notebook_application-scala-3.5.1
2:27PM INF Cleaning up config template it-notebook-scala-sopmlv. suite=run_notebook_application-scala-3.5.1
2:27PM INF Success suite=run_notebook_application-scala-3.5.1
2:27PM INF Tests status failure=[] in_progress=["run_application-3.5.1","kill_application_completed_3.5.1","run_spark_connect_application_code-3.5.1"] success=16 to_do=0
2:28PM INF Tests status failure=[] in_progress=["run_application-3.5.1","kill_application_completed_3.5.1","run_spark_connect_application_code-3.5.1"] success=16 to_do=0
2:29PM INF App is completed suite=kill_application_completed_3.5.1
2:29PM INF Success suite=kill_application_completed_3.5.1
2:29PM INF Spark connect app executed sql suite=run_spark_connect_application_code-3.5.1
2:29PM INF Spark connect app killed suite=run_spark_connect_application_code-3.5.1
2:29PM INF Success suite=run_spark_connect_application_code-3.5.1
2:29PM INF Tests status failure=[] in_progress=["run_application-3.5.1"] success=18 to_do=0
2:30PM INF App is completed suite=run_application-3.5.1
2:30PM INF Tests status failure=[] in_progress=["run_application-3.5.1"] success=18 to_do=0
2:31PM INF Tests status failure=[] in_progress=["run_application-3.5.1"] success=18 to_do=0
2:32PM INF Tests status failure=[] in_progress=["run_application-3.5.1"] success=18 to_do=0
2024/11/25 14:33:02 c.ReadMessage: websocket: RSV1 set, FIN not set on control
2:33PM INF Tests status failure=[] in_progress=["run_application-3.5.1"] success=18 to_do=0
2:34PM INF Tests status failure=[] in_progress=["run_application-3.5.1"] success=18 to_do=0
2:35PM INF Tests status failure=[] in_progress=["run_application-3.5.1"] success=18 to_do=0
2:36PM INF Tests status failure=[] in_progress=["run_application-3.5.1"] success=18 to_do=0
2:37PM INF Tests status failure=[] in_progress=["run_application-3.5.1"] success=18 to_do=0
2:38PM INF Tests status failure=[] in_progress=["run_application-3.5.1"] success=18 to_do=0
2:39PM INF Logs are processed suite=run_application-3.5.1
2:39PM INF Driver log archive is available suite=run_application-3.5.1
2:39PM INF Success suite=run_application-3.5.1
2:39PM INF Starting sequential tests
2:39PM INF Suites run final status failure=[] in_progress=[] success=19 to_do=0
2:39PM INF Starting suite=kill_application_pending_3.5.1
2:39PM INF Submitted app integration-tests-c645d-slimy suite=kill_application_pending_3.5.1
2:39PM INF App is processed suite=kill_application_pending_3.5.1
2:39PM INF App killed suite=kill_application_pending_3.5.1
2:39PM INF Success suite=kill_application_pending_3.5.1
2:39PM INF All tests are successful