Add the manifests overlay for Kubeflow Training V2 #2382

Doris-xm · 2025-01-09T14:54:28Z

What this PR does / why we need it:

This PR adds the manifests overlay for Kubeflow Training V2, allowing to install it within Kubeflow Platform.

Which issue(s) this PR fixes :
Fixes #2381

Checklist:

Docs included if any changes are user facing

Signed-off-by: Xinmin Du <10803082+doris-xm@user.noreply.gitee.com>

Electronic-Waste · 2025-01-09T15:25:45Z

Will review it later.

/cc @kubeflow/wg-training-leads @kubeflow/release-team @saileshd1402
/ok-to-test
/rerun-all

google-oss-prow · 2025-01-09T15:25:50Z

@Electronic-Waste: GitHub didn't allow me to request PR reviews from the following users: kubeflow/release-team, saileshd1402.

Note that only kubeflow members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

Will review it later.

/cc @kubeflow/wg-training-leads @kubeflow/release-team @saileshd1402
/ok-to-test
/rerun-all

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Electronic-Waste

Thanks for creating this @Doris-xm! I left some initial comments for you.

Btw, I recommend that you could learn more about the concept of Training V2. This will help you update the manifests overlay in training-operator and kubeflow/manifests:)

FYI: KubeCon 2024 NA Talk by Andrey and Yuki

Electronic-Waste · 2025-01-10T01:49:25Z

manifests/v2/overlays/kubeflow/kubeflow-training-roles.yaml

+    resources:
+      - trainjobs/status
+    verbs:
+      - get


Suggested change

- get

- get

Please add a blank line in the end of file

Thanks for your careful review！

Electronic-Waste · 2025-01-10T01:49:43Z

manifests/v2/overlays/kubeflow/kustomization.yaml

+  - name: training-operator-v2-webhook-cert
+    namespace: kubeflow-system
+    options:
+      disableNameSuffixHash: true


Suggested change

disableNameSuffixHash: true

disableNameSuffixHash: true

Same as above

Electronic-Waste · 2025-01-10T01:54:30Z

manifests/v2/overlays/kubeflow/kustomization.yaml

+kind: Kustomization
+namespace: kubeflow
+resources:
+  - ../../base


There does not exist a kustomization.yaml file under ../../base dir. So we may need to reference those resources in subdir explicitly.

Like:

training-operator/manifests/v2/overlays/only-manager/kustomization.yaml

Lines 5 to 8 in be2e29e

- ../../base/crds

- ../../base/manager

- ../../base/rbac

- ../../base/webhook

Electronic-Waste · 2025-01-10T02:00:54Z

manifests/v2/overlays/kubeflow/kustomization.yaml

+resources:
+  - ../../base
+  - kubeflow-training-roles.yaml
+  - https://github.com/kubernetes-sigs/jobset/releases/download/v0.6.0/manifests.yaml


Please add comments to make this item consistent with:

training-operator/manifests/v2/overlays/standalone/kustomization.yaml

Lines 10 to 11 in be2e29e

# TODO (andreyvelich): JobSet should support kubeflow-system namespace.

- https://github.com/kubernetes-sigs/jobset/releases/download/v0.6.0/manifests.yaml

I am wondering for Kubeflow Platform installation, should we actually deploy JobSet controller in the kubeflow namespace ?
@kannon92 also made it possible: kubernetes-sigs/jobset#719
cc @kubeflow/wg-manifests-leads

Electronic-Waste · 2025-01-10T02:04:23Z

manifests/v2/overlays/kubeflow/kubeflow-training-roles.yaml

+metadata:
+  name: kubeflow-training-admin-v2
+  labels:
+    rbac.authorization.kubeflow.org/aggregate-to-kubeflow-admin: "true"


This label will conflict with:

training-operator/manifests/overlays/kubeflow/kubeflow-training-roles.yaml

Line 6 in be2e29e

rbac.authorization.kubeflow.org/aggregate-to-kubeflow-admin: "true"

Please rename it to rbac.authorization.kubeflow.org/aggregate-to-kubeflow-admin-v2 to avoid conflicts.

Electronic-Waste · 2025-01-10T02:04:36Z

manifests/v2/overlays/kubeflow/kubeflow-training-roles.yaml

+aggregationRule:
+  clusterRoleSelectors:
+    - matchLabels:
+        rbac.authorization.kubeflow.org/aggregate-to-kubeflow-training-admin: "true"


Same as above.

Electronic-Waste · 2025-01-10T02:04:57Z

manifests/v2/overlays/kubeflow/kubeflow-training-roles.yaml

+    rbac.authorization.kubeflow.org/aggregate-to-kubeflow-edit: "true"
+    rbac.authorization.kubeflow.org/aggregate-to-kubeflow-training-admin: "true"


Same as above.

Electronic-Waste · 2025-01-10T02:05:11Z

manifests/v2/overlays/kubeflow/kubeflow-training-roles.yaml

+metadata:
+  name: kubeflow-training-view-v2
+  labels:
+    rbac.authorization.kubeflow.org/aggregate-to-kubeflow-view: "true"


Same as above.

Electronic-Waste · 2025-01-10T02:18:05Z

manifests/v2/overlays/kubeflow/kubeflow-training-roles.yaml

+  - apiGroups:
+      - ""
+    resources:
+      - events
+    verbs:
+      - get
+      - list
+      - watch


I haven't seen any event resources in Training V2. Maybe we should remove them

training-operator/pkg/controller.v2/trainjob_controller.go

Lines 69 to 72 in be2e29e

// +kubebuilder:rbac:groups=kubeflow.org,resources=trainjobs,verbs=get;list;watch;create;update;patch;delete

// +kubebuilder:rbac:groups=kubeflow.org,resources=trainjobs/status,verbs=get;update;patch

// +kubebuilder:rbac:groups=kubeflow.org,resources=trainjobs/finalizers,verbs=get;update;patch

training-operator/pkg/controller.v1/pytorch/pytorchjob_controller.go

Lines 106 to 114 in be2e29e

// +kubebuilder:rbac:groups=kubeflow.org,resources=pytorchjobs,verbs=get;list;watch;create;update;patch;delete

// +kubebuilder:rbac:groups=kubeflow.org,resources=pytorchjobs/status,verbs=get;update;patch

// +kubebuilder:rbac:groups=kubeflow.org,resources=pytorchjobs/finalizers,verbs=update

// +kubebuilder:rbac:groups="",resources=pods,verbs=get;list;watch;create;update;patch;delete

// +kubebuilder:rbac:groups="",resources=services,verbs=get;list;watch;create;delete

// +kubebuilder:rbac:groups=autoscaling,resources=horizontalpodautoscalers,verbs=get;list;watch;create;update;patch;delete

// +kubebuilder:rbac:groups=scheduling.volcano.sh,resources=podgroups,verbs=get;list;watch;create;update;patch;delete

// +kubebuilder:rbac:groups=scheduling.x-k8s.io,resources=podgroups,verbs=get;list;watch;create;update;patch;delete

// +kubebuilder:rbac:groups="",resources=events,verbs=get;list;watch;create;update;patch;delete

PTAL👀 @kubeflow/wg-training-leads. Do you think we should reserve events resource in the manifests?

Thanks for your review! It seems event resources are required for EventRecorder used in TrainJobReconciler to log events related to training jobs.

training-operator/pkg/controller.v2/trainjob_controller.go

Lines 53 to 58 in be2e29e

type TrainJobReconciler struct {

log logr.Logger

client client.Client

recorder record.EventRecorder

runtimes map[string]jobruntimes.Runtime

}

However, the event resource permissions are not included in the kubebuilder annotations in the trainjob_controller.go

training-operator/pkg/controller.v2/trainjob_controller.go

Lines 69 to 71 in be2e29e

// +kubebuilder:rbac:groups=kubeflow.org,resources=trainjobs,verbs=get;list;watch;create;update;patch;delete

// +kubebuilder:rbac:groups=kubeflow.org,resources=trainjobs/status,verbs=get;update;patch

// +kubebuilder:rbac:groups=kubeflow.org,resources=trainjobs/finalizers,verbs=get;update;patch

I'm not sure I understand it correctly. Appreciate your clarification.

SGTM! However, in controller v2, we haven't called r.recorder.Eventf() yet. We just initialize it in:

training-operator/pkg/controller.v2/trainjob_controller.go

Lines 53 to 67 in be2e29e

type TrainJobReconciler struct {

log logr.Logger

client client.Client

recorder record.EventRecorder

runtimes map[string]jobruntimes.Runtime

}

func NewTrainJobReconciler(client client.Client, recorder record.EventRecorder, runtimes map[string]jobruntimes.Runtime) *TrainJobReconciler {

return &TrainJobReconciler{

log: ctrl.Log.WithName("trainjob-controller"),

client: client,

recorder: recorder,

runtimes: runtimes,

}

}

Maybe we could add events resource's permission to the Kubeflow Manifests when we call the function.

Actually, events will be required in the SDK, like for V1: https://github.com/kubeflow/training-operator/blob/master/sdk/python/kubeflow/training/api/training_client.py#L1309
We haven't implemented it yet in V2, but we will do it in the future.

Electronic-Waste · 2025-01-10T02:21:12Z

manifests/v2/overlays/kubeflow/kubeflow-training-roles.yaml

+      - clustertrainingruntimes
+      - trainingruntimes


We could also edit trainingruntimes and clustertrainingruntimes in the Kubeflow platform. I think, their perssmission granularity should be the same with trainjobs:)

Electronic-Waste · 2025-01-10T02:30:03Z

/rerun-all

Signed-off-by: Xinmin Du <10803082+doris-xm@user.noreply.gitee.com>

Electronic-Waste

Basically LGTM. Thank you for your great contributions!

/lgtm
/assign @kubeflow/wg-training-leads @kubeflow/wg-manifests-leads

google-oss-prow · 2025-01-10T11:28:24Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Electronic-Waste
Once this PR has been reviewed and has the lgtm label, please assign jeffwan for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

andreyvelich

Thank you for this great contribution @willb!
/assign @kubeflow/wg-manifests-leads @kubeflow/release-team

andreyvelich · 2025-01-10T16:27:05Z

manifests/v2/overlays/kubeflow/kustomization.yaml

@@ -0,0 +1,20 @@
+apiVersion: kustomize.config.k8s.io/v1beta1


Can we name this overlay: kubeflow-platform to make it consistent with the naming: https://www.kubeflow.org/docs/started/introduction/#what-is-kubeflow-platform.

andreyvelich · 2025-01-10T16:28:31Z

manifests/v2/overlays/kubeflow/kustomization.yaml

+resources:
+  - ../../base
+  - kubeflow-training-roles.yaml
+  - https://github.com/kubernetes-sigs/jobset/releases/download/v0.6.0/manifests.yaml


I am wondering for Kubeflow Platform installation, should we actually deploy JobSet controller in the kubeflow namespace ?
@kannon92 also made it possible: kubernetes-sigs/jobset#719
cc @kubeflow/wg-manifests-leads

andreyvelich · 2025-01-10T16:30:40Z

manifests/v2/overlays/kubeflow/kubeflow-training-roles.yaml

+  - apiGroups:
+      - ""
+    resources:
+      - events
+    verbs:
+      - get
+      - list
+      - watch


Actually, events will be required in the SDK, like for V1: https://github.com/kubeflow/training-operator/blob/master/sdk/python/kubeflow/training/api/training_client.py#L1309
We haven't implemented it yet in V2, but we will do it in the future.

Add the manifests overlay for Kubeflow Training V2

0140805

Signed-off-by: Xinmin Du <10803082+doris-xm@user.noreply.gitee.com>

google-oss-prow bot requested review from jinchihe and kuizhiqing January 9, 2025 14:54

google-oss-prow bot added the size/L label Jan 9, 2025

google-oss-prow bot requested a review from a team January 9, 2025 15:25

google-oss-prow bot added the ok-to-test label Jan 9, 2025

Electronic-Waste reviewed Jan 10, 2025

View reviewed changes

Update manifest: adjust permissions, and format changes

f908380

Signed-off-by: Xinmin Du <10803082+doris-xm@user.noreply.gitee.com>

google-oss-prow bot added size/M and removed size/L labels Jan 10, 2025

Electronic-Waste approved these changes Jan 10, 2025

View reviewed changes

google-oss-prow bot assigned Electronic-Waste Jan 10, 2025

google-oss-prow bot added the lgtm label Jan 10, 2025

andreyvelich reviewed Jan 10, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add the manifests overlay for Kubeflow Training V2 #2382

Add the manifests overlay for Kubeflow Training V2 #2382

Doris-xm commented Jan 9, 2025

Electronic-Waste commented Jan 9, 2025

google-oss-prow bot commented Jan 9, 2025

Electronic-Waste left a comment

Electronic-Waste Jan 10, 2025

Doris-xm Jan 10, 2025

Electronic-Waste Jan 10, 2025

Electronic-Waste Jan 10, 2025

Electronic-Waste Jan 10, 2025

andreyvelich Jan 10, 2025

Electronic-Waste Jan 10, 2025

Electronic-Waste Jan 10, 2025

Electronic-Waste Jan 10, 2025

Electronic-Waste Jan 10, 2025

Electronic-Waste Jan 10, 2025

Electronic-Waste Jan 10, 2025

Doris-xm Jan 10, 2025

Electronic-Waste Jan 10, 2025 •

edited

Loading

andreyvelich Jan 10, 2025

Electronic-Waste Jan 10, 2025

Electronic-Waste commented Jan 10, 2025

Electronic-Waste left a comment

google-oss-prow bot commented Jan 10, 2025

andreyvelich left a comment

andreyvelich Jan 10, 2025

andreyvelich Jan 10, 2025

andreyvelich Jan 10, 2025

	- ../../base/crds
	- ../../base/manager
	- ../../base/rbac
	- ../../base/webhook

	# TODO (andreyvelich): JobSet should support kubeflow-system namespace.
	- https://github.com/kubernetes-sigs/jobset/releases/download/v0.6.0/manifests.yaml

		rbac.authorization.kubeflow.org/aggregate-to-kubeflow-edit: "true"
		rbac.authorization.kubeflow.org/aggregate-to-kubeflow-training-admin: "true"

	// +kubebuilder:rbac:groups=kubeflow.org,resources=trainjobs,verbs=get;list;watch;create;update;patch;delete
	// +kubebuilder:rbac:groups=kubeflow.org,resources=trainjobs/status,verbs=get;update;patch
	// +kubebuilder:rbac:groups=kubeflow.org,resources=trainjobs/finalizers,verbs=get;update;patch

	type TrainJobReconciler struct {
	log logr.Logger
	client client.Client
	recorder record.EventRecorder
	runtimes map[string]jobruntimes.Runtime
	}

Add the manifests overlay for Kubeflow Training V2 #2382

Are you sure you want to change the base?

Add the manifests overlay for Kubeflow Training V2 #2382

Conversation

Doris-xm commented Jan 9, 2025

Electronic-Waste commented Jan 9, 2025

google-oss-prow bot commented Jan 9, 2025

Electronic-Waste left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Electronic-Waste Jan 10, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Electronic-Waste commented Jan 10, 2025

Electronic-Waste left a comment

Choose a reason for hiding this comment

google-oss-prow bot commented Jan 10, 2025

andreyvelich left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Electronic-Waste Jan 10, 2025 •

edited

Loading