Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding volumes to StatefulSet after the initial creation #367

Open
2 tasks
leochr opened this issue Apr 6, 2022 · 3 comments · May be fixed by #412
Open
2 tasks

Adding volumes to StatefulSet after the initial creation #367

leochr opened this issue Apr 6, 2022 · 3 comments · May be fixed by #412
Assignees
Labels
enhancement New feature or request future

Comments

@leochr
Copy link
Member

leochr commented Apr 6, 2022

Kubernetes StatefulSet itself doesn't allow volumes to be added to StatefulSets after their initial creation.

Trying to add a volume, results in this error:
Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden

The same limitation is encountered via operator.

  • create StatefulSet by setting:
spec:
  statefulSet: {}
  • try to add a volume to it
spec.
  statefulSet: 
    storage:
      size: 100Mi
      mountPath: /logs

Tasks:

  • understand the reasons behind StatefulSet not supporting
  • if it's not an anti-pattern to add volume afterwards, support adding it via operator. StatefulSet probably have to be deleted and recreated (meaning there will be some downtime)
@leochr leochr added enhancement New feature or request future labels Apr 6, 2022
@halim-lee halim-lee self-assigned this Jun 24, 2022
@halim-lee halim-lee assigned kabicin and unassigned halim-lee Jul 6, 2022
@kabicin
Copy link
Collaborator

kabicin commented Jul 11, 2022

The reason for StatefulSet not supporting modification of other fields (in general):

  • Deleting the StatefulSet and pod(s) will not delete bound PVs and it's PVCs. This is to preserve data safety, which is put at a higher precedence than deleting all StatefulSet resources.
  • Difficulty determining an update strategy for pods
    • Currently, a non-cascading delete and then re-creation of the StatefulSet will try and reuse it's X orphaned pods still using old claims to PVs, despite any new modifications made to the volume template.
  • No guarantee on deletion of all pods when a StatefulSet is deleted

For this specific use case outlined above, PVs and PVCs are not an issue but ensuring deletion of all pods is necessary to ensure recreation with the correct volume from StatefulSetSpec.

Edit: I am reconsidering to not ensure deletion of all pods since this specific use case will never cause a forced rollback state and so we can depend on StatefulSet controller to gracefully delete a pods with statefulSet: {}

@kabicin kabicin linked a pull request Jul 13, 2022 that will close this issue
2 tasks
@kabicin
Copy link
Collaborator

kabicin commented Jul 13, 2022

The PR above only implements adding volumes to instances without storage specified, i.e. statefulSet: {}.
Given the above findings, do we want to support modifying volumes to RCO instances which already have StatefulSet + volume provisioned? For example, changing the mountPath PV size (which currently updates RC spec but does not actually update the StatefulSet spec until it is manually restarted).

@halim-lee
Copy link
Collaborator

I would recommend adding support only for adding volumes for now. Since making changes to StatefulSet volumes also requires deletion of pods, I am not sure if it is safe to make that too available. If we decide to support modifying the existing volumes, we can implement it later.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request future
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants