Allow Setting Both Memory Requests and Limits for HdfsCluster Pods

[fjammes]

Current Behavior

The HdfsCluster Custom Resource Definition (CRD) currently only allows specifying memory.limit under HdfsCluster.spec.nameNodes.config.resources.memory. This configuration automatically sets memory.request equal to memory.limit for pods.

Problem

This behavior is problematic for CI platforms with limited memory resources. Forcing the memory request to match the limit can lead to resource scheduling issues, making it difficult to run HdfsCluster pods in resource-constrained environments.

Desired Behavior

Allow users to independently set both memory.request and memory.limit under HdfsCluster.spec.nameNodes.config.resources. For example:

spec:
  nameNodes:
    config:
      resources:
        memory:
          request: "512Mi"
          limit: "1Gi"

This flexibility ensures that pods can be scheduled even on CI platforms with tight resource constraints.

Proposed Solution

• Update the HdfsCluster CRD to support separate request and limit fields for memory under resources.
• Ensure backward compatibility by defaulting request to the same value as limit if request is not explicitly set.

Impact

This change would make the HdfsCluster CRD more usable in diverse environments, including CI platforms with restricted memory resources.

Additional Context

This enhancement would enable more efficient use of resources while maintaining compatibility with existing configurations.

[xeniape]

Hey @fjammes, thanks for getting in touch!

While the hdfs-operator does not support specifying the memory requests using a dedicated CRD field, you could try using podOverrides to achieve the same result:

spec:
  nameNodes:
    config:
      ...
    podOverrides: 
      spec:
        containers:
          - name: namenode
            resources:
              requests:
                memory: "512Mi"

I hope this helps your case at least for now 😄