chore(airflow): Update container images ahead of Stackable Release 24.11

[NickLarsenNZ]

Part of #831.

• Add: 2.10.2 (2.9.3 remains the latest/LTS, and was already added in chore: Bump airflow to 2.9.3 #809).
• Remove: 2.6.x, 2.8.x (already done in chore: Bump airflow to 2.9.3 #809).

Tip

Please add the scheduled-for/20XX-XX label.

Update tasks

Preview Give feedback

1. Update versions.py to reflect the agreed upon versions in the spreadsheet (including the removal of old versions).
2. Download new constraints file (see airflow/download_constraints.sh).
3. Update other dependencies if applicable (eg: python, statsd_exporter, etc).
4. Check other operators (getting_started / kuttl) for usage of the versions. Add the PR(s) to the list below.
5. Update the version in demos. Add the PR(s) to the list below.
6. Research memory OOM problems in demo when using Airflow 2.10.2 -> Maybe increase default memory for webservers

Related Pull Requests

Preview Give feedback

1. feat: Add airflow 2.10.2 #877
   +0
   
2. feat: Add Airflow 2.10.2 to tests airflow-operator#512
   5 of 5
   +0

Acceptance

Preview Give feedback

1. Can build image locally
2. Kuttl smoke tests passes locally
3. Release notes written in a comment below
4. Applicable release-note label added to this issue

Testing instructions

# See the latest version at https://pypi.org/project/image-tools-stackabletech/
pip install image-tools-stackabletech==0.0.12

bake --product airflow=x.y.z # where x.y.z is the new version added in this PR

kind load docker-image docker.stackable.tech/stackable/airflow:x.y.z-stackable0.0.0-dev

# Change directory into the airflow-operator repository and update the
# product version in tests/test-definition.yaml
./scripts/run-tests --test-suite smoke-latest # or similar

Please consider updating this template if these instructions are wrong, or
could be made clearer.

[NickLarsenNZ]

I see a new statsd_exporter version available. Will update that in a separate PR.

[NickLarsenNZ]

Smoke tests pass:

--- PASS: kuttl (423.52s)
    --- PASS: kuttl/harness (0.00s)
        --- PASS: kuttl/harness/smoke_airflow-2.10.2_openshift-false_executor-kubernetes (423.49s)
PASS

[NickLarsenNZ]

airflow-operator getting_started passes, but will leave it at the current stable version.

diff

diff --git a/docs/modules/airflow/examples/getting_started/code/airflow.yaml b/docs/modules/airflow/examples/getting_started/code/airflow.yaml
index 9389942..9165718 100644
--- a/docs/modules/airflow/examples/getting_started/code/airflow.yaml
+++ b/docs/modules/airflow/examples/getting_started/code/airflow.yaml
@@ -5,7 +5,8 @@ metadata:
   name: airflow
 spec:
   image:
-    productVersion: 2.9.3
+    productVersion: 2.10.2
+    pullPolicy: Never
   clusterConfig:
     loadExamples: true
     exposeConfig: false

[NickLarsenNZ]

The demo does not work with 2.10.2, that will need some work.

In case it helps, @adwk67:

1. Launching the cluster via stackablectl demo install airflow-scheduled-job, but then there is a failed job in the UI (with very little detail in the logs).
2. There appears to be a new button in the UI next to each DAG, which lets you re-parse the DAGs. After hitting that, the Airflow UI is unreachable, and stackablectl stacklet list shows the cluster as Reconciling.

diff

diff --git a/stacks/airflow/airflow.yaml b/stacks/airflow/airflow.yaml
index 5e8b96c..7b12ce0 100644
--- a/stacks/airflow/airflow.yaml
+++ b/stacks/airflow/airflow.yaml
@@ -6,7 +6,7 @@ metadata:
   name: airflow
 spec:
   image:
-    productVersion: 2.9.3
+    productVersion: 2.10.2
   clusterConfig:
     listenerClass: external-unstable
     loadExamples: false
diff --git a/stacks/stacks-v2.yaml b/stacks/stacks-v2.yaml
index 12a55e9..b51ea27 100644
--- a/stacks/stacks-v2.yaml
+++ b/stacks/stacks-v2.yaml
@@ -85,7 +85,7 @@ stacks:
         default: adminadmin
   airflow:
     description: Stack containing Airflow scheduling platform
-    stackableRelease: 24.7
+    stackableRelease: dev
     stackableOperators:
       - commons
       - listener
@@ -95,9 +95,9 @@ stacks:
     labels:
       - airflow
     manifests:
-      - helmChart: https://raw.githubusercontent.com/stackabletech/demos/main/stacks/_templates/postgresql-airflow.yaml
-      - helmChart: https://raw.githubusercontent.com/stackabletech/demos/main/stacks/_templates/redis-airflow.yaml
-      - plainYaml: https://raw.githubusercontent.com/stackabletech/demos/main/stacks/airflow/airflow.yaml
+      - helmChart: ./stacks/_templates/postgresql-airflow.yaml
+      - helmChart: ./stacks/_templates/redis-airflow.yaml
+      - plainYaml: ./stacks/airflow/airflow.yaml
     supportedNamespaces: []
     resourceRequests:
       cpu: 3400m

[sbernauer]

I would prefer to keep the Issue open until the problem is solved to not forget about it. We don't need to ship Airflow 2.10 with SDP 24.11, although I would prefer investing the time (at a min. 1-2 days) to properly research the problem

[sbernauer]

Gave it a very quick try (installed the airflow-scheduled-job demo) and updated to 2.10.2. Everything worked perfectly fine, even the DAG re-parsing. At some point of time the webserver OOMed. Did yours too?
Maybe we only need to give Airflow a bit more memory and everything works

[sbernauer]

Update: Seems to work perfectly fine after I increased the webserver memory 1 -> 2 Gi 🥳

[NickLarsenNZ]

Moving this back to Development Done, because @adwk67 solved the problem with the failing test task.
@sbernauer, as far as I understand, there was no memory issue.

[sbernauer]

as far as I understand, there was no memory issue.

I mean the Pod in my demo OOMed, so something is going on. But when you say everything is working now I'm happy :)