How to monitor many online spark jobs?

[tolstykh-da]

Hi, In yarn, we can monitor spark jobs online and view them through spark ui, what about spark-operator?
We are thinking of using a separate spark master and going through --conf spark.kubernetes.driverEnv.SPARK_MASTER_URL
Will Stackable Operator for Apache Spark support this?

[lfrancke]

Hi,
depends on what exactly you mean.
Each Spark job still has the UI that you can view but there is no UI that displays all running spark jobs.
For that you'd need to look at Kubernetes and check all SparkApplications.

But before I go into detail you can maybe clarify what it is exactly you're looking for?

[tolstykh-da]

yes, we want the UI to display all running Spark jobs. To solve this problem, we are thinking of raising the spark master separately and placing drivers in it in order to see all tasks in one ui

[lfrancke]

Understood.
I'm afraid you can only get this now using kubectl for example: kubectl get -A sparkapplication or a tool like k9s on the command line as well.
There are also Kubernetes Web dashboards which show the objects.

On top of that we have just started a project to build our own UI that will display all Stackable "things": https://stackable.tech/en/building-a-kubernetes-hosted-web-ui-for-our-data-platform/

But right now, I'm afraid you'll have to resort to the Kubernetes native things.

We do not support the Spark Standalone mode.

[tolstykh-da]

that is, the operator does not support now? --conf spark.kubernetes.driverEnv.SPARK_MASTER_URL
master of the spark we can deploy ourselves

[lfrancke]

I'm afraid there is no "master" in the Spark K8s mode so to speak of.
In Standalone mode you have the standalone master and in YARN mode you have the YARN RM, in K8s mode it uses the current K8s context instead.

Long story short: No, we don't support that.

If I understand you correctly you're trying to run Spark in Standalone mode on Kubernetes. That can be done but we don't support it.
I would recommend using the native K8s mode instead and one of the methods mentioned above to look at your jobs.

I'd be interested in feedback on what exactly you need. Is it just a list of all running applications?

[tolstykh-da]

we want to see how in yarn (id, name, status, cpu, ram, pod job) and the ability to jump in spark UI. In principle, yunikorn suits us. especially if it is replaced instead of the default scheduler. But there is no transition in ui spark and we don't know yet how it intersects with your operator