Prevent node removal for at least x seconds (old ttlSecondsAfterEmpty)

[alekc]

Hi all,

We are currently facing a following issue: the cluster is optimized for its size, so no consolidation is happening. We launch a new CI job which triggers a spawining of a new node, as soon as the job is finished karpenter removes the node which means that we will need to wait for a new node to come up, pull the image, etc increasing the overall wait time.

Previously this could have been fixed by using ttlSecondsAfterEmpty but as far as I can see, there is no such option anymore. Not sure if #735 covers the issue (it less about the consolidation and more about the removal of an empty node).

My nodepool config:

apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  annotations:
    karpenter.sh/nodepool-hash: '5345755618547637684'
  creationTimestamp: '2024-01-18T10:08:26Z'
  generation: 2
  name: default
  resourceVersion: '132113143'
  uid: 3d869c5a-cfe8-4369-ac06-0a2d44719dab
  selfLink: /apis/karpenter.sh/v1beta1/nodepools/default
status:
  resources:
    cpu: '10'
    ephemeral-storage: 83827632Ki
    memory: 32025644Ki
    pods: '440'
    vpc.amazonaws.com/pod-eni: '9'
spec:
  disruption:
    consolidationPolicy: WhenUnderutilized
    expireAfter: 168h0m0s
  limits:
    cpu: 500
  template:
    metadata: {}
    spec:
      kubelet:
        maxPods: 110
      nodeClassRef:
        name: default
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values:
            - spot
            - on-demand
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values:
            - c
            - m
            - t
        - key: karpenter.k8s.aws/instance-hypervisor
          operator: In
          values:
            - nitro
        - key: karpenter.k8s.aws/instance-generation
          operator: Gt
          values:
            - '2'
        - key: karpenter.k8s.aws/instance-size
          operator: In
          values:
            - medium
            - large
            - xlarge
        - key: kubernetes.io/arch
          operator: In
          values:
            - arm64
            - amd64
        - key: kubernetes.io/os
          operator: In
          values:
            - linux

Sadly consolidateAfter doesnt work with consolidationPolicy: WhenUnderutilized. Any suggestions?