Connections are blocked

[ashitmazumdar]

Describe the bug

Hi,

Can someone please help me on the issue as described below. I have applied persistent storage to be 10GB and the resources as below. But the connections always goes into a blocked state even the value is set tp vm_memory_high_watermark= 0.8.

There are amq connections and stomp conenction enabled for the cluster to connect.

Also I have 4 replica set and none of the node accets the connection. Screen shot attached

resources:
limits:
cpu: '1'
memory: 512Mi
requests:
cpu: 100m
memory: 256Mi

persistence:
storage: 10Gi

Reproduction steps

Apply the rabbitmq cluster with the below yaml file

apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
name: rabbit-cluster
namespace: {{changenamespace}}
spec:
delayStartSeconds: 30
image: rabbitmq:3.12.2-management
override:
service:
spec:
ports:
- name: stomp-wss
port: 15671
protocol: TCP
targetPort: 15674
persistence:
storage: 10Gi
rabbitmq:
additionalConfig: |
cluster_partition_handling = pause_minority
vm_memory_high_watermark_paging_ratio = 0.99
disk_free_limit.relative = 1.0
collect_statistics_interval = 10000
default_vhost = sm600
default_user = default_user_7zU0N4KH09240i1kMkj
default_pass = Go2XX080LlqodUmLQoyVM1VvoyP0rkYm
default_user_tags.administrator = true
web_stomp.tcp.port = 15674
auth_backends.1 = internal
auth_backends.2 = http
auth_http.http_method = get
auth_http.user_path = http://rabbitmqauth.{{changenamespace}}.svc.cluster.local:80/rabbitmqauth/user
auth_http.vhost_path = http://rabbitmqauth.{{changenamespace}}.svc.cluster.local:80/rabbitmqauth/vhost
auth_http.resource_path = http://rabbitmqauth.{{changenamespace}}.svc.cluster.local:80/rabbitmqauth/resource
auth_http.topic_path = http://rabbitmqauth.{{changenamespace}}.svc.cluster.local:80/rabbitmqauth/vhost
additionalPlugins:
- rabbitmq_auth_backend_http
- rabbitmq_management
- rabbitmq_management_agent
- rabbitmq_web_dispatch
- rabbitmq_web_stomp
replicas: 3
resources:
limits:
cpu: '1'
memory: 512Mi
requests:
cpu: 100m
memory: 256Mi
secretBackend:
externalSecret:
name: rabbitmq-secret
service:
type: ClusterIP
terminationGracePeriodSeconds: 604800

Expected behavior

The connection should not be in blocked state. The cluster should manage itself to handle the connections and recover by themselves.

Additional context

No response

[michaelklishin]

You are looking at a resource alarm.

They will "manage themselves" when node's memory usage drops below the watermark, which is set to 164 MiB for these nodes.

164 MiB won't be enough for many environments, not if you plan on having 100s of connections and channels, and any queues that keep at least some data in memory (streams are very efficient at moving data straight to disk).

[ashitmazumdar]

Thank you @michaelklishin , will update to the memory as recommended and keep you posted on the issue.