How to use seatunnel-web with seatunnel server cluster in k8s?

[psychiche]

I deployed a separate cluster(master-worker) of seatunnel services in k8s of our production environment according to the documentation.

However, I was unsuccessful in deploying seatunnel-web and there is no documentation available.

Some of the mentioned documents are about how to use it in a standalone environment, and I don't know how to solve the most critical network configuration issues (such as ServiceCount, Ingress, etc.) and Docker image problems in a cluster environment.

Please help, thank you.

[liunaijie]

Hi @psychiche , which version you deployed on k8s, can you share your hazelcast.yaml config, in current version if we use kubernetes joiner strategy. it has some issue.
related discussion #7750 and pr #7757

[psychiche]

Thanks for your reply. The version of my service is as follows:

• seatunnel release2.3.7
• seatunnel-web: release1.0.1

My hazelcast.yaml config is here:

apiVersion: v1
kind: Service
metadata:
  name: seatunnel
spec:
  selector:
    app: seatunnel
  ports:
  - port: 5801
    name: seatunnel-master
  - port: 5802
    name: seatunnel-worker
  clusterIP: None
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: hazelcast-client
  namespace: seatunnel
data:
  hazelcast-client.yaml: |
    hazelcast-client:
      cluster-name: seatunnel
      properties:
        hazelcast.logging.type: log4j2
      connection-strategy:
        connection-retry:
          cluster-connect-timeout-millis: 3000
      network:
        cluster-members:
              - seatunnel-master-0.seatunnel:5801
              - seatunnel-master-1.seatunnel:5801
              - seatunnel-master-2.seatunnel:5801
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: hazelcast-master
  namespace: seatunnel
data:
  hazelcast-master.yaml: |
    hazelcast:
      cluster-name: seatunnel
      network:
        rest-api:
          enabled: true
          endpoint-groups:
            CLUSTER_WRITE:
              enabled: true
            DATA:
              enabled: true
        join:
          tcp-ip:
            enabled: true
            member-list:
              - seatunnel-master-0.seatunnel:5801
              - seatunnel-master-1.seatunnel:5801
              - seatunnel-master-2.seatunnel:5801
              - seatunnel-worker-0.seatunnel:5802
              - seatunnel-worker-1.seatunnel:5802
              - seatunnel-worker-2.seatunnel:5802
        port:
          auto-increment: false
          port: 5801
      properties:
        hazelcast.invocation.max.retry.count: 20
        hazelcast.tcp.join.port.try.count: 30
        hazelcast.logging.type: log4j2
        hazelcast.operation.generic.thread.count: 50
        hazelcast.heartbeat.failuredetector.type: phi-accrual
        hazelcast.heartbeat.interval.seconds: 2
        hazelcast.max.no.heartbeat.seconds: 180
        hazelcast.heartbeat.phiaccrual.failuredetector.threshold: 10
        hazelcast.heartbeat.phiaccrual.failuredetector.sample.size: 200
        hazelcast.heartbeat.phiaccrual.failuredetector.min.std.dev.millis: 100
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: hazelcast-worker
  namespace: seatunnel
data:
  hazelcast-worker.yaml: |+
    hazelcast:
      cluster-name: seatunnel
      network:
        join:
          tcp-ip:
            enabled: true
            member-list:
              - seatunnel-master-0.seatunnel:5801
              - seatunnel-master-1.seatunnel:5801
              - seatunnel-master-2.seatunnel:5801
              - seatunnel-worker-0.seatunnel:5802
              - seatunnel-worker-1.seatunnel:5802
              - seatunnel-worker-2.seatunnel:5802
        port:
          auto-increment: false
          port: 5802
      properties:
        hazelcast.invocation.max.retry.count: 20
        hazelcast.tcp.join.port.try.count: 30
        hazelcast.logging.type: log4j2
        hazelcast.operation.generic.thread.count: 50
        hazelcast.heartbeat.failuredetector.type: phi-accrual
        hazelcast.heartbeat.interval.seconds: 2
        hazelcast.max.no.heartbeat.seconds: 180
        hazelcast.heartbeat.phiaccrual.failuredetector.threshold: 10
        hazelcast.heartbeat.phiaccrual.failuredetector.sample.size: 200
        hazelcast.heartbeat.phiaccrual.failuredetector.min.std.dev.millis: 100
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: seatunnel
  namespace: seatunnel
data:
  seatunnel.yaml: |-
    seatunnel:
      engine:
        history-job-expire-minutes: 1440
        backup-count: 1
        queue-type: blockingqueue
        print-execution-info-interval: 60
        print-job-metrics-info-interval: 60
        slot-service:
          dynamic-slot: true
        checkpoint:
          interval: 10000
          timeout: 60000
          storage:
            type: hdfs
            max-retained: 3
            plugin-config:
              namespace: /tmp/seatunnel/checkpoint_snapshot
              storage.type: hdfs
              fs.defaultFS: file:///tmp/ # Ensure that the directory has written permission

And my seatunnel-web.yaml as follows:

kind: ConfigMap
metadata:
  name: hazelcast-client
  namespace: seatunnel
data:
  hazelcast-client.yaml: |
    hazelcast-client:
      cluster-name: seatunnel
      properties:
        hazelcast.logging.type: log4j2
      connection-strategy:
        connection-retry:
          cluster-connect-timeout-millis: 3000
      network:
        cluster-members:
              - seatunnel-master-0.seatunnel:5801
              - seatunnel-master-1.seatunnel:5801
              - seatunnel-master-2.seatunnel:5801
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: seatunnel-web
  labels:
    app: seatunnel
  namespace: seatunnel
spec:
  replicas: 1
  selector:
    matchLabels:
      app: seatunnel
  template:
    metadata:
      labels:
        app: seatunnel
    spec:
      containers:
      - name: seatunnel-web
        image: xxx/seatunnel-web:1.0.1
        command: [ "/bin/sh", "/opt/seatunnel-web/bin/seatunnel-backend-daemon.sh", "start" ]
        imagePullPolicy: IfNotPresent
        resources:
          limits:
            cpu: "1"
            memory: 4G
          requests:
            cpu: "1"
            memory: 2G
---
apiVersion: v1
kind: Service
metadata:
  name: seatunnel-web
  labels:
    app: seatunnel
  namespace: seatunnel
spec:
  selector:
    app: seatunnel
  ports:
  - name: seatunnel-web-ports
    protocol: TCP
    port: 8801
    nodePort: 31075
    targetPort: 8801
  type: NodePort

Thanks again.