Support overriding task pod_template via with_overrides #6118
Conversation
Signed-off-by: Nelson Chen <asd3431090@gmail.com>
Signed-off-by: Nelson Chen <asd3431090@gmail.com>
|
#take |
Code Review Agent Run #9af1d5Actionable Suggestions - 7
Review Details
AI Code Review powered by |
Changelist by BitoThis pull request implements the following key changes.
|
| t.Run("Test_enable-distributed-error-aggregation", func(t *testing.T) { | ||
|
|
||
| t.Run("Override", func(t *testing.T) { | ||
| testValue := "1" | ||
|
|
||
| cmdFlags.Set("enable-distributed-error-aggregation", testValue) | ||
| if vBool, err := cmdFlags.GetBool("enable-distributed-error-aggregation"); err == nil { | ||
| testDecodeJson_K8sPluginConfig(t, fmt.Sprintf("%v", vBool), &actual.EnableDistributedErrorAggregation) | ||
|
|
||
| } else { | ||
| assert.FailNow(t, err.Error()) | ||
| } | ||
| }) | ||
| }) |
There was a problem hiding this comment.
Consider adding test cases for both true and false values for the enable-distributed-error-aggregation flag to ensure proper boolean parsing behavior.
Code suggestion
Check the AI-generated fix before applying
@@ -382,14 +382,40 @@ func TestK8sPluginConfig_SetFlags(t *testing.T) {
t.Run("Test_enable-distributed-error-aggregation", func(t *testing.T) {
t.Run("Override with 1", func(t *testing.T) {
testValue := "1"
cmdFlags.Set("enable-distributed-error-aggregation", testValue)
if vBool, err := cmdFlags.GetBool("enable-distributed-error-aggregation"); err == nil {
testDecodeJson_K8sPluginConfig(t, fmt.Sprintf("%v", vBool), &actual.EnableDistributedErrorAggregation)
} else {
assert.FailNow(t, err.Error())
}
})
t.Run("Override with true", func(t *testing.T) {
testValue := "true"
cmdFlags.Set("enable-distributed-error-aggregation", testValue)
if vBool, err := cmdFlags.GetBool("enable-distributed-error-aggregation"); err == nil {
testDecodeJson_K8sPluginConfig(t, fmt.Sprintf("%v", vBool), &actual.EnableDistributedErrorAggregation)
} else {
assert.FailNow(t, err.Error())
}
})
t.Run("Override with false", func(t *testing.T) {
testValue := "false"
cmdFlags.Set("enable-distributed-error-aggregation", testValue)
if vBool, err := cmdFlags.GetBool("enable-distributed-error-aggregation"); err == nil {
testDecodeJson_K8sPluginConfig(t, fmt.Sprintf("%v", vBool), &actual.EnableDistributedErrorAggregation)
} else {
assert.FailNow(t, err.Error())
}
})
})
Code Review Run #9af1d5
Is this a valid issue, or was it incorrectly flagged by the Agent?
- it was incorrectly flagged
| @@ -6953,6 +6959,9 @@ export namespace flyteidl { | |||
|
|
|||
| /** K8sPod dataConfig */ | |||
| dataConfig?: (flyteidl.core.IDataLoadingConfig|null); | |||
|
|
|||
| /** K8sPod primaryContainerName */ | |||
| primaryContainerName?: (string|null); | |||
There was a problem hiding this comment.
Consider making primaryContainerName required instead of optional since it appears to be a critical field for K8sPod configuration. The interface shows it as optional with ? but the class implementation has it as required. A similar issue was also found in flyteidl/gen/pb-js/flyteidl.d.ts (line 6985-6986).
Code suggestion
Check the AI-generated fix before applying
| primaryContainerName?: (string|null); | |
| primaryContainerName: string; |
Code Review Run #9af1d5
Is this a valid issue, or was it incorrectly flagged by the Agent?
- it was incorrectly flagged
| @@ -31,6 +31,7 @@ func buildNodeSpec(n *core.Node, tasks []*core.CompiledTask, errs errors.Compile | |||
| var resources *core.Resources | |||
| var extendedResources *v1alpha1.ExtendedResources | |||
| var containerImage string | |||
| var podtemplate *core.K8SPod | |||
There was a problem hiding this comment.
Consider initializing the podtemplate variable with a default value to avoid potential nil pointer dereference issues.
Code suggestion
Check the AI-generated fix before applying
| var podtemplate *core.K8SPod | |
| var podtemplate *core.K8SPod = &core.K8SPod{} |
Code Review Run #9af1d5
Is this a valid issue, or was it incorrectly flagged by the Agent?
- it was incorrectly flagged
| if overrides.GetPodTemplate() != nil { | ||
| podtemplate = overrides.GetPodTemplate() | ||
| } |
There was a problem hiding this comment.
Consider adding validation for podtemplate before assignment. The current code directly assigns the pod template without any validation which could lead to issues if an invalid template is provided.
Code suggestion
Check the AI-generated fix before applying
| if overrides.GetPodTemplate() != nil { | |
| podtemplate = overrides.GetPodTemplate() | |
| } | |
| if template := overrides.GetPodTemplate(); template != nil { | |
| if err := validatePodTemplate(template); err != nil { | |
| return nil, err | |
| } | |
| podtemplate = template | |
| } |
Code Review Run #9af1d5
Is this a valid issue, or was it incorrectly flagged by the Agent?
- it was incorrectly flagged
| var podspec_override *v1.PodSpec | ||
| err := utils.UnmarshalStructToObj(podtemplate.PodSpec, &podspec_override) |
There was a problem hiding this comment.
Consider using podspec_override instead of podspec_override for better Go naming conventions. Also, error handling could be improved by returning a more descriptive error.
Code suggestion
Check the AI-generated fix before applying
| var podspec_override *v1.PodSpec | |
| err := utils.UnmarshalStructToObj(podtemplate.PodSpec, &podspec_override) | |
| var podSpecOverride *v1.PodSpec | |
| if err := utils.UnmarshalStructToObj(podtemplate.PodSpec, &podSpecOverride); err != nil { | |
| return nil, objectMeta, fmt.Errorf("failed to unmarshal pod spec: %w", err) | |
| } |
Code Review Run #9af1d5
Is this a valid issue, or was it incorrectly flagged by the Agent?
- it was incorrectly flagged
| @@ -1,3 +1,3 @@ | |||
| from setuptools import setup, find_packages | |||
|
|
|||
| setup() | |||
| setup() | |||
There was a problem hiding this comment.
The empty setup() call may be missing required parameters. Consider adding necessary configuration like name, version, packages, etc.
Code suggestion
Check the AI-generated fix before applying
@@ -1,3 +1,9 @@
from setuptools import setup, find_packages
-setup()
+setup(
+ name='flyteidl',
+ version='0.1.0',
+ packages=find_packages(),
+ install_requires=[],
+ python_requires='>=3.7'
+)
Code Review Run #9af1d5
Is this a valid issue, or was it incorrectly flagged by the Agent?
- it was incorrectly flagged
| @@ -148,6 +148,7 @@ func RunPluginEndToEndTest(t *testing.T, executor pluginCore.Plugin, template *i | |||
| }) | |||
| overrides.OnGetExtendedResources().Return(&idlCore.ExtendedResources{}) | |||
| overrides.OnGetContainerImage().Return("") | |||
| overrides.OnGetPodTemplate().Return(&idlCore.K8SPod{}) | |||
There was a problem hiding this comment.
Consider adding error handling for the mock return value setup. The OnGetPodTemplate() call may need error handling to properly simulate failure scenarios.
Code suggestion
Check the AI-generated fix before applying
| overrides.OnGetPodTemplate().Return(&idlCore.K8SPod{}) | |
| overrides.OnGetPodTemplate().Return(&idlCore.K8SPod{}, nil) |
Code Review Run #9af1d5
Is this a valid issue, or was it incorrectly flagged by the Agent?
- it was incorrectly flagged
Tracking issue
Related to #5683
Why are the changes needed?
If we can support pod_template in with_overrides, this would reduce a lot of toil since we can supply pod templates in a central location and override downstream tasks, similar to how we can do so for resources.
What changes were proposed in this pull request?
We can use with_override() to override podtemplate, just like resources.
How was this patch tested?
Excute a workflow and using with_override(pod_template=PodTemplate(xxx)) to override the default podtemplate
Setup process
I ran flyte on my local machine and tested my code with this workflow and task:
Screenshots
Check all the applicable boxes
Related PRs
Docs link
Summary by Bito
This PR implements pod template override functionality through with_overrides() method, adding primary_container_name field to K8sPod and pod_template to TaskNodeOverrides. The implementation spans multiple languages (TypeScript, Go, JavaScript) with protobuf definitions. The changes enable centralized pod template specification with downstream task override capability, similar to resource overrides.Unit tests added: True
Estimated effort to review (1-5, lower is better): 5