Frame size exceeds frame max

[shayktrust]

Describe the bug

Connection to RabbitMQ fails with Error: Frame size exceeds frame max.

I've run into some issues with RabbitMQ connections. When I direct the application to the RabbitMQ LB connection string, such as this one: amqps://test:xxx-debugg-password@mq.debugg.xxx.io:5671, I receive an error message stating: "Failed: Error: Frame size exceeds frame max," with a timestamp of 1714901442519.

However, when I use the connection string locally (amqp://test:xxx-debugg-password@rabbitmq.rabbitmq.svc.cluster.local:5672), the application establishes a connection successfully.

Reproduction steps

1. Use the following RabbitMQ configuration:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: rabbitmq
  namespace: argocd
spec:
  destination:
    namespace: rabbitmq
    server: https://kubernetes.default.svc
  project: default
  source:
    chart: rabbitmq
    repoURL: https://charts.bitnami.com/bitnami
    targetRevision: 14.0.2
    helm:
      releaseName: rabbitmq
      values: |
        service:
          type: ClusterIP
          portName: amqp
          port: 5672
          tlsPortName: amqp-ssl
          tlsPort: 5671
          distPortName: dist
          distPort: 25672
          managerPortEnabled: true
          managerPort: 15672
          managerPortName: stats
          metricsPort: 9419
          metricsPortName: metrics
        replicaCount: 3
        extraConfiguration: |-
          default_vhost = default-vhost
          default_permissions.configure = .*
          default_permissions.read = .*
          default_permissions.write = .*
          max_message_size = 536870912
        memoryHighWatermark:
          enabled: true
          type: "relative"
          value: 0.4
        auth:
          username: "test"
          password: "xxx-debugg-password"
          erlangCookie: "test"      
          tls:
            enabled: true
            existingSecret: "rabbitmq-tls"
            existingSecretFullChain: true
            sslOptionsVerify: "verify_peer"
            failIfNoPeerCert: false
            sslOptionsPassword:
              enabled: false
              existingSecret: "rabbitmq-tls"
        resources:
          requests:
            cpu: "500m"
            memory: "512Mi"
          limits:
            cpu: "1000m"
            memory: "2048Mi"
        extraPlugins: rabbitmq_auth_backend_http rabbitmq_delayed_message_exchange rabbitmq_shovel rabbitmq_shovel_management rabbitmq_amqp1_0 rabbitmq_federation rabbitmq_consistent_hash_exchange
        communityPlugins: []
        clustering:
          enabled: true
          name: "rabbitmq-cluster"
          addressType: hostname
          rebalance: true
          forceBoot: false
        pdb:
          create: true
          minAvailable: 2
        dnsPolicy: ""
        dnsConfig: {}
        hostAliases: []
        containerPorts:
          amqp: 5672
          amqpTls: 5671
          dist: 25672
          manager: 15672
          epmd: 4369
          metrics: 15692
        persistence:
          enabled: true
          storageClass: "efs-rabbitmq"
          accessModes: ["ReadWriteMany"]
          size: 5Gi
        metrics:
          enabled: true
          serviceMonitor:
            enabled: true
            namespace: "monitoring"
            interval: "30s"
            scrapeTimeout: "30s"
            additionalLabels:
              release: prometheus-operator
          prometheusRule:
            enabled: true
            additionalLabels:
              role: alert-rules
            rules:
              - alert: RabbitMQHighMemoryUsage
                annotations:
                  description: "RabbitMQ instance {{`{{ $labels.instance }}`}} is using more than 80% of its memory limit."
                  summary: "RabbitMQ instance {{`{{ $labels.instance }}`}} high memory usage detected"
                expr: rabbitmq_memory_used_bytes / rabbitmq_memory_limit_bytes > 0.8
                for: 10m
                labels:
                  severity: critical
        extraSecrets:
          load-definition:
            load_definition.json: |
              {
                "users": [
                  {
                    "name": "test",
                    "password": "xxx-debugg-password",
                    "tags": "administrator"
                  }
                ],
                "permissions": [{
                  "user": "test",
                  "vhost": "/",
                  "configure": ".*",
                  "write": ".*",
                  "read": ".*"
                }],
                "topic_permissions": [{
                  "user": "test",
                  "vhost": "/",
                  "configure": ".*",
                  "write": ".*",
                  "read": ".*"
                }], 
                "policies": [
                  {
                    "name": "ha-all",
                    "pattern": ".*",
                    "vhost": "/",
                    "definition": {
                      "ha-mode": "all",
                      "ha-sync-mode": "automatic",
                      "ha-sync-batch-size": 1
                    }
                  },
                  {
                    "name": "AWS-DEFAULT-POLICY-SINGLE-INSTANCE",
                    "pattern": ".*",
                    "apply-to":"all",
                    "vhost": "/",
                    "definition": {
                      "queue-mode": "lazy"
                    }
                    "priority": 0
                  }
                ],
                "exchanges": [],
                "bindings": [],
                "vhosts": [
                  {
                    "name": "/",
                  }
                ]
              }
  syncPolicy:
    automated: {}
    syncOptions:
      - CreateNamespace=true

Expected behavior

2. Verify that the connection is successfully established using the local connection string: amqp://test:[xxx-debugg-password@rabbitmq.rabbitmq.svc.cluster.local](mailto:xxx-debugg-password@rabbitmq.rabbitmq.svc.cluster.local):5672.

3. Expect to encounter an error while using the LB connection string: amqps://test:[xxx-debugg-password@mq.debugg.xxx.io](mailto:xxx-debugg-password@mq.debugg.xxx.io):5671.

Let me know if you need anything else.

Additional context

No response

[michaelklishin]

@shayktrust per our community support policy, you need to provide an executable way to reproduce and be using the latest release series, or our team will disengage.

Without a way to reproduce, all we can say is that it is very likely a concurrency hazard in your application. Take a traffic capture for clues.

I'm talking about this kind of limitations. Although concurrent writes to the socket on a shared connection are usually synchronized by clients (unlikely publish operations on a shared channel), you haven't
provided any code or even the name of the client, so I cannot say whether the client used here does that.

It's unlikely that the issue has anything to do with the load balancer. A TCP load balancer should not cause any behavior differences except for these ones with certain settings. That case is not applicable here.