[flytepropeller][flyteadmin] Streaming Decks V2 #6053

Future-Outlier · 2024-11-27T15:38:59Z

Tracking issue

Why are the changes needed?

To enhance user visibility into Flyte Decks at different stages of workflow execution (running, failing, and succeeding), enabling better debugging and analysis.

What changes were proposed in this pull request?

Concept:

propeller will turn node info to NodeExecutionEvent, and send it to admin.

flyte/flytepropeller/pkg/controller/nodes/executor.go

Lines 1251 to 1261 in b3330ba

    
           nev, err := ToNodeExecutionEvent( 
        
           	nCtx.NodeExecutionMetadata().GetNodeExecutionID(), 
        
           	p, 
        
           	nCtx.InputReader().GetInputPath().String(), 
        
           	nCtx.NodeStatus(), 
        
           	nCtx.ExecutionContext().GetEventVersion(), 
        
           	nCtx.ExecutionContext().GetParentInfo(), nCtx.Node(), 
        
           	c.clusterID, 
        
           	nCtx.NodeStateReader().GetDynamicNodeState().Phase, 
        
           	c.eventConfig, 
        
           	targetEntity)

Life Cycle:

use new flytekit > 1.14.0

summary:

NO HEAD request to be called. (save resource)
use config from task template to know whether enable deck or not

details:

propeller keep adding DeckURI when the task is running if FLYTE_ENABLE_DECK=true in the task template.
propeller will put DeckURI to node info, and turn it to NodeExecutionEvent to flyte admin.
flyte admin will add DeckURI to Closure

flyte console will get DeckURI by sending request to admin.

flyte/flyteadmin/dataproxy/service.go

Lines 175 to 189 in b3330ba

    
           		nativeURL = node.GetClosure().GetDeckUri() 
        
           	} 
        
           } else { 
        
           	return nil, errors.NewFlyteAdminErrorf(codes.InvalidArgument, "unsupported source [%v]", reflect.TypeOf(req.GetSource())) 
        
           } 
        
           if len(nativeURL) == 0 { 
        
           	return nil, errors.NewFlyteAdminErrorf(codes.Internal, "no deckUrl found for request [%+v]", req) 
        
           } 
        
           ref := storage.DataReference(nativeURL) 
        
           meta, err := s.dataStore.Head(ctx, ref) 
        
           if err != nil { 
        
           	return nil, errors.NewFlyteAdminErrorf(codes.Internal, "failed to head object before signing url. Error: %v", err) 
        
           }

if flyte console can't get the DeckURI from the node Closure, it will not show the Flyte Deck button.

old flytekit <= 1.14.0

summary:

we keep the backward compatible (show deck when succeed)

details:

In the terminal state, use a HEAD request to know if the Deck URI exists or not.
if exist, then put it to the node info.

How was this patch tested?

unit test and remote execution.

python code:

from flytekit import ImageSpec, task, workflow
from flytekit.deck import Deck

flytekit_hash = "6b55930d0a77efc3594ebaac056f2c75024e61b5"
flytekit = f"git+https://github.com/flyteorg/flytekit.git@{flytekit_hash}"

# Define custom image for the task
custom_image = ImageSpec(packages=[flytekit],
                            apt_packages=["git"],
                            registry="localhost:30000",
                            env={"FLYTE_SDK_LOGGING_LEVEL": 10},
                         )

@task(enable_deck=False, container_image=custom_image)
def t_no_deck():
    # Deck.publish()
    print("No Deck")

@task(enable_deck=True, container_image=custom_image)
def t_deck():
    import time
    """
    1st deck only show timeline deck
    2nd will show
    """
    for i in range(3):
        Deck.publish()
        time.sleep(1)

@task(enable_deck=True, container_image=custom_image)
def t_fail_deck():
    import time

    for i in range(3):
        Deck.publish()
        time.sleep(3)
    time.sleep(10)
    raise ValueError("Failed Deck")

@workflow
def wf():
    t_no_deck()
    t_deck()
    t_fail_deck()

if __name__ == "__main__":
    from flytekit.clis.sdk_in_container import pyflyte
    from click.testing import CliRunner
    import os

    runner = CliRunner()
    path = os.path.realpath(__file__)

    result = runner.invoke(pyflyte.main,
                           ["run", path, "t_no_deck"])
    print("Local Execution: ", result.output)

    result = runner.invoke(pyflyte.main,
                           ["run", "--remote", path,"wf"])
    print("Remote Execution: ", result.output)

Setup process

single binary.

flyte: this branch
flytekit: flyteorg/flytekit#2779
flyteconsole: flyteorg/flyteconsole#890

Screenshots

flytekit branch:
flyteorg/flytekit#2779

NEW FLYTEKIT, NO DECK, RUNNING With Deck, SUCCEED, and FAILED

OSS-STREAMING-DECK-small.mov

OLD FLYTEKIT, NO DECK, RUNNING With Deck, SUCCEED, and FAILED

OSS-STREAMING-DECK-OLD-FLYTEKIT-small.mov

Check all the applicable boxes

I updated the documentation accordingly.
All new and existing tests passed.
All commits are signed-off.

Related PRs

follow up questions

should we support Abort phase for the streaming deck?

should we support EPhaseAbort in this file?

https://github.com/flyteorg/flyte/blob/b3330ba4430538f91ae9fc7d868a29a2e96db8bd/flytepropeller/pkg/controller/nodes/handler/transition_info.go

how can we support the auto-refresh UX?

Summary by Bito

Implementation of enhanced Flyte Deck streaming functionality for improved workflow execution visibility. Introduces real-time deck URI handling in Flytepropeller and Flyteadmin, supporting deck information display across various execution states. Maintains backward compatibility with Flytekit <=1.14.0 while optimizing for newer versions through FLYTE_ENABLE_DECK configuration.

Unit tests added: False

Estimated effort to review (1-5, lower is better): 2

Signed-off-by: Future-Outlier <eric901201@gmail.com>

Signed-off-by: Future-Outlier <eric901201@gmail.com> Co-authored-by: Yi Cheng <luyc58576@gmail.com> Co-authored-by: pingsutw <pingsutw@apache.org>

codecov · 2024-11-27T15:45:04Z

Codecov Report

Attention: Patch coverage is 39.75904% with 50 lines in your changes missing coverage. Please review.

Project coverage is 37.02%. Comparing base (27c9edd) to head (04f7fbc).

Files with missing lines	Patch %	Lines
...lytepropeller/pkg/controller/nodes/task/handler.go	39.02%	40 Missing and 10 partials ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #6053   +/-   ##
=======================================
  Coverage   37.01%   37.02%           
=======================================
  Files        1317     1317           
  Lines      132523   132580   +57     
=======================================
+ Hits        49060    49082   +22     
- Misses      79217    79244   +27     
- Partials     4246     4254    +8

Flag	Coverage Δ
unittests-datacatalog	`51.58% <ø> (ø)`
unittests-flyteadmin	`54.25% <100.00%> (+0.02%)`	⬆️
unittests-flytecopilot	`30.99% <ø> (ø)`
unittests-flytectl	`62.29% <ø> (ø)`
unittests-flyteidl	`7.23% <ø> (ø)`
unittests-flyteplugins	`53.85% <ø> (ø)`
unittests-flytepropeller	`42.61% <39.02%> (-0.03%)`	⬇️
unittests-flytestdlib	`55.18% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Signed-off-by: Future-Outlier <eric901201@gmail.com>

hamersaw · 2024-11-27T16:52:38Z

flytepropeller/pkg/controller/nodes/task/handler.go

 	switch pluginTrns.pInfo.Phase() {
 	case pluginCore.PhaseSuccess:
+		// This is to prevent the console from potentially checking the deck URI that does not exist if in final phase(PhaseSuccess).
+		err = pluginTrns.RemoveNonexistentDeckURI(ctx, tCtx)


Does this do a head call on the deck URI for every task that succeeds? Two thoughts here:
(1) does the flyteadmin merge algorithm then remove the deckURI from the execution metadata?
(2) this is incurring a 20-30ms performance degredation to every task execution

will take a look tmr, thank you!!!

Does this do a head call on the deck URI for every task that succeeds?

yes it will do a head call by RemoteFileOutputReader

flyte/flyteplugins/go/tasks/pluginmachinery/ioutils/remote_file_output_reader.go

Lines 306 to 313 in b3330ba

func (r RemoteFileOutputReader) DeckExists(ctx context.Context) (bool, error) {

md, err := r.store.Head(ctx, r.outPath.GetDeckPath())

if err != nil {

return false, err

}

return md.Exists(), nil

}

how do you know the performance degradation?
did you use grafana or other performance tools?

does the flyteadmin merge algorithm then remove the deckURI from the execution metadata?

flyteadmin will set the deckURI in the execution metadata to nil if the propeller removes it.

cc @hamersaw

Future-Outlier · 2024-11-27T17:02:52Z

How to test it?

start a new sandbox

flytectl demo start --image futureoutlier/sandbox:deck-1205-1138 --force

checkout streaming deck flytekit branch

cd flytekit
gh pr checkout 2779

run a failure task (show deck after it failed)

from flytekit import ImageSpec, task, workflow
from flytekit.deck import Deck

flytekit_hash = "473ae1119af6f86c26c0790dee0affa3eb29be64"
flytekit = f"git+https://github.com/flyteorg/flytekit.git@{flytekit_hash}"

# Define custom image for the task
custom_image = ImageSpec(packages=[flytekit],
                            apt_packages=["git"],
                            registry="localhost:30000",
                            env={"FLYTE_SDK_LOGGING_LEVEL": 10},
                         )

@task(enable_deck=True, container_image=custom_image)
def t_deck():
    import time
    """
    1st deck only show timeline deck
    2nd will show
    """
    for i in range(5):
        Deck.publish()
        # # raise Exception("This is an exception")
        time.sleep(3)

@workflow
def wf():
    t_deck()

if __name__ == "__main__":
    from flytekit.clis.sdk_in_container import pyflyte
    from click.testing import CliRunner
    import os

    runner = CliRunner()
    path = os.path.realpath(__file__)

    # result = runner.invoke(pyflyte.main,
    #                        ["run", path, "wf"])
    # print("Local Execution: ", result.output)

    result = runner.invoke(pyflyte.main,
                           ["run", "--remote", path,"wf"])
    # "--remote"
    print("Remote Execution: ", result.output)

EngHabu · 2024-11-28T00:38:55Z

Mind adding screenshots for the rendered deck and refresh to the PR description?

Future-Outlier · 2024-11-28T01:52:29Z

Mind adding screenshots for the rendered deck and refresh to the PR description?

Yes no problem

Future-Outlier · 2024-11-28T02:10:23Z

Mind adding screenshots for the rendered deck and refresh to the PR description?

its provided!
#6053 (comment)

Signed-off-by: Future-Outlier <eric901201@gmail.com>

flyte-bot · 2025-01-02T08:18:59Z

Code Review Agent Run #f3ef5e

Actionable Suggestions - 1

flytepropeller/pkg/controller/nodes/task/handler.go - 1
- Consider impact of removing deckPath parameter · Line 120-120

Additional Suggestions - 1

flytepropeller/pkg/controller/nodes/task/handler.go - 1
- Consider removing unnecessary string cast · Line 43-43

Review Details

Files reviewed - 3 · Commit Range: 54aa165..65b6efe
- flyteadmin/pkg/repositories/transformers/node_execution.go
- flyteadmin/pkg/repositories/transformers/node_execution_test.go
- flytepropeller/pkg/controller/nodes/task/handler.go
Files skipped - 0
Tools
- Golangci-lint (Linter) - ✖︎ Failed
- Whispers (Secret Scanner) - ✔︎ Successful
- Detect-secrets (Secret Scanner) - ✔︎ Successful
- OWASP (Security Vulnerability) - ✔︎ Successful
- GOVULNCHECK (Security Vulnerability) - ✖︎ Failed
- SNYK (Security Vulnerability) - ✔︎ Successful

AI Code Review powered by

flyte-bot · 2025-01-02T08:22:18Z

Changelist by Bito

This pull request implements the following key changes.

Key Change	Files Impacted
Feature Improvement - Enhanced Flyte Deck Streaming Support	- `node_execution.go` - Added DeckUri support in node execution transformers - `node_execution_test.go` - Added tests for DeckUri functionality - `handler.go` - Implemented real-time deck streaming with backward compatibility

flyte-bot · 2025-01-02T08:22:20Z

flytepropeller/pkg/controller/nodes/task/handler.go

+	return nil
+}
+
+func (p *pluginRequestedTransition) CacheHit(outputPath storage.DataReference, entry catalog.Entry) {


Consider impact of removing deckPath parameter

The method signature for CacheHit has been modified to remove the deckPath parameter, but it seems this parameter may still be needed based on the usage in the code. Consider if this change could cause issues with deck path handling.

Code suggestion

Check the AI-generated fix before applying

Suggested change

func (p *pluginRequestedTransition) CacheHit(outputPath storage.DataReference, entry catalog.Entry) {

func (p *pluginRequestedTransition) CacheHit(outputPath storage.DataReference, deckPath *storage.DataReference, entry catalog.Entry) {

Code Review Run #f3ef5e

Is this a valid issue, or was it incorrectly flagged by the Agent?

it was incorrectly flagged

wild-endeavor · 2025-01-06T20:34:16Z

flytepropeller/pkg/controller/nodes/task/handler.go

+	//   - We relied on a HEAD request to check if the deck file exists, then added the URI to the event.
+	//
+	// After (new behavior):
+	//   - If `FLYTE_ENABLE_DECK = true` is set in the task template config (requires Flytekit > 1.14.0),


this comment is no longer correct right?

yes super nice catch

wild-endeavor · 2025-01-06T20:34:25Z

flytepropeller/pkg/controller/nodes/task/handler.go

@@ -380,6 +430,27 @@ func (t Handler) fetchPluginTaskMetrics(pluginID, taskType string) (*taskMetrics
 	return t.taskMetricsMap[metricNameKey], nil
 }

+func GetDeckStatus(ctx context.Context, tCtx *taskExecutionContext) (DeckStatus, error) {
+	// FLYTE_ENABLE_DECK is used when flytekit > 1.14.0


update this comment

wild-endeavor · 2025-01-06T20:39:08Z

flytepropeller/pkg/controller/nodes/task/handler.go

+
+	metadata := template.GetMetadata()
+	if metadata == nil {
+		return DeckUnknown, nil


is this correct in older versions of flytekit? didn't tasks in the past also have this field? this means that this function will always return Disabled right for older versions of flytekit. meaning the condition on line 567 won't get triggered cuz it'll be disabled instead of unknown.

oh yes, you are right, thinking solution.

Signed-off-by: Future-Outlier <eric901201@gmail.com>

flyte-bot · 2025-01-09T05:32:55Z

Code Review Agent Run Status

Limitations and other issues: ❌ Failure - Bito Code Review Agent didn't review this pull request automatically because it exceeded the size limit. No action is needed if you didn't intend for the agent to review it. Otherwise, you can initiate the review by typing /review in a comment below.

Future-Outlier · 2025-01-09T06:01:05Z

flytepropeller/pkg/controller/nodes/task/handler.go

+	// GetDeckStatus determines whether a task generates a deck based on its execution context.
+	//
+	// This function ensures backward compatibility with older Flytekit versions using the following logic:
+	// 1. For Flytekit > 1.14.3, the task template's metadata includes the `generates_deck` flag:
+	//    - If `generates_deck` is set to true, it indicates that the task generates a deck, and DeckEnabled is returned.
+	// 2. If `generates_deck` is set to false or is not set (likely from older Flytekit versions):
+	//    - DeckUnknown is returned as a placeholder status.
+	//    - In terminal states, a HEAD request can be made to check if the deck file exists.
+	//
+	// In future implementations, a `DeckDisabled` status could be introduced for better performance optimization:
+	//    - This would eliminate the need for a HEAD request in the final phase.
+	//    - However, the tradeoff is that a new field would need to be added to FlyteIDL to support this behavior.
+
+	template, err := tCtx.tr.Read(ctx)


Better comments!
cc @wild-endeavor

Future-Outlier and others added 2 commits November 27, 2024 23:36

add tests from Yi-Cheng

54aa165

Signed-off-by: Future-Outlier <eric901201@gmail.com>

helped by Kevin and Yi-Cheng

9ed6b6e

Signed-off-by: Future-Outlier <eric901201@gmail.com> Co-authored-by: Yi Cheng <luyc58576@gmail.com> Co-authored-by: pingsutw <pingsutw@apache.org>

lint

4b4f6bd

Signed-off-by: Future-Outlier <eric901201@gmail.com>

hamersaw reviewed Nov 27, 2024

View reviewed changes

nit

dd774cb

Signed-off-by: Future-Outlier <eric901201@gmail.com>

Future-Outlier requested a review from hamersaw December 12, 2024 13:37

Future-Outlier added 3 commits December 13, 2024 23:18

add comments

0bb8e91

Signed-off-by: Future-Outlier <eric901201@gmail.com>

add comments and better solution for backward compativle

25fea29

Signed-off-by: Future-Outlier <eric901201@gmail.com>

better comments

4e24e91

Signed-off-by: Future-Outlier <eric901201@gmail.com>

Future-Outlier mentioned this pull request Dec 17, 2024

[flyte deck] Streaming Decks flyteorg/flytekit#2779

Open

3 tasks

Future-Outlier added 4 commits December 18, 2024 23:39

DeckStatus

8d1d0e4

Signed-off-by: Future-Outlier <eric901201@gmail.com>

rename GetDeckStatus

31853bb

Signed-off-by: Future-Outlier <eric901201@gmail.com>

comments

4068043

Signed-off-by: Future-Outlier <eric901201@gmail.com>

lint

65b6efe

Signed-off-by: Future-Outlier <eric901201@gmail.com>

flyte-bot reviewed Jan 2, 2025

View reviewed changes

wild-endeavor reviewed Jan 6, 2025

View reviewed changes

Future-Outlier added 2 commits January 9, 2025 12:27

fix

137579f

Signed-off-by: Future-Outlier <eric901201@gmail.com>

Merge branch 'master' into streaming-deck-v2

04f7fbc

Future-Outlier commented Jan 9, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[flytepropeller][flyteadmin] Streaming Decks V2 #6053

[flytepropeller][flyteadmin] Streaming Decks V2 #6053

Future-Outlier commented Nov 27, 2024 •

edited by flyte-bot

Loading

codecov bot commented Nov 27, 2024 •

edited

Loading

hamersaw Nov 27, 2024

Future-Outlier Nov 27, 2024

Future-Outlier Nov 28, 2024 •

edited

Loading

Future-Outlier Nov 28, 2024

Future-Outlier Nov 28, 2024

Future-Outlier Nov 28, 2024

Future-Outlier commented Nov 27, 2024 •

edited

Loading

EngHabu commented Nov 28, 2024

Future-Outlier commented Nov 28, 2024

Future-Outlier commented Nov 28, 2024

flyte-bot commented Jan 2, 2025 •

edited

Loading

Code Review Agent Run #f3ef5e

flyte-bot commented Jan 2, 2025

Changelist by Bito

flyte-bot Jan 2, 2025

wild-endeavor Jan 6, 2025

Future-Outlier Jan 8, 2025

wild-endeavor Jan 6, 2025

wild-endeavor Jan 6, 2025

Future-Outlier Jan 9, 2025

flyte-bot commented Jan 9, 2025 •

edited

Loading

Future-Outlier Jan 9, 2025

	nev, err := ToNodeExecutionEvent(
	nCtx.NodeExecutionMetadata().GetNodeExecutionID(),
	p,
	nCtx.InputReader().GetInputPath().String(),
	nCtx.NodeStatus(),
	nCtx.ExecutionContext().GetEventVersion(),
	nCtx.ExecutionContext().GetParentInfo(), nCtx.Node(),
	c.clusterID,
	nCtx.NodeStateReader().GetDynamicNodeState().Phase,
	c.eventConfig,
	targetEntity)

	nativeURL = node.GetClosure().GetDeckUri()
	}
	} else {
	return nil, errors.NewFlyteAdminErrorf(codes.InvalidArgument, "unsupported source [%v]", reflect.TypeOf(req.GetSource()))
	}

	if len(nativeURL) == 0 {
	return nil, errors.NewFlyteAdminErrorf(codes.Internal, "no deckUrl found for request [%+v]", req)
	}

	ref := storage.DataReference(nativeURL)
	meta, err := s.dataStore.Head(ctx, ref)
	if err != nil {
	return nil, errors.NewFlyteAdminErrorf(codes.Internal, "failed to head object before signing url. Error: %v", err)
	}

	func (r RemoteFileOutputReader) DeckExists(ctx context.Context) (bool, error) {
	md, err := r.store.Head(ctx, r.outPath.GetDeckPath())
	if err != nil {
	return false, err
	}

	return md.Exists(), nil
	}

	func (p *pluginRequestedTransition) CacheHit(outputPath storage.DataReference, entry catalog.Entry) {
	func (p pluginRequestedTransition) CacheHit(outputPath storage.DataReference, deckPath storage.DataReference, entry catalog.Entry) {

[flytepropeller][flyteadmin] Streaming Decks V2 #6053

Are you sure you want to change the base?

[flytepropeller][flyteadmin] Streaming Decks V2 #6053

Conversation

Future-Outlier commented Nov 27, 2024 • edited by flyte-bot Loading

Tracking issue

Why are the changes needed?

What changes were proposed in this pull request?

Concept:

Life Cycle:

use new flytekit > 1.14.0

old flytekit <= 1.14.0

How was this patch tested?

Setup process

Screenshots

NEW FLYTEKIT, NO DECK, RUNNING With Deck, SUCCEED, and FAILED

OLD FLYTEKIT, NO DECK, RUNNING With Deck, SUCCEED, and FAILED

Check all the applicable boxes

Related PRs

follow up questions

Summary by Bito

codecov bot commented Nov 27, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Future-Outlier Nov 28, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Future-Outlier commented Nov 27, 2024 • edited Loading

How to test it?

EngHabu commented Nov 28, 2024

Future-Outlier commented Nov 28, 2024

Future-Outlier commented Nov 28, 2024

flyte-bot commented Jan 2, 2025 • edited Loading

Code Review Agent Run #f3ef5e

flyte-bot commented Jan 2, 2025

Changelist by Bito

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

flyte-bot commented Jan 9, 2025 • edited Loading

Code Review Agent Run Status

Choose a reason for hiding this comment

Future-Outlier commented Nov 27, 2024 •

edited by flyte-bot

Loading

codecov bot commented Nov 27, 2024 •

edited

Loading

Future-Outlier Nov 28, 2024 •

edited

Loading

Future-Outlier commented Nov 27, 2024 •

edited

Loading

flyte-bot commented Jan 2, 2025 •

edited

Loading

flyte-bot commented Jan 9, 2025 •

edited

Loading