Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Make FlyteFile and FlyteDirectory pickleable #6144

Open
2 tasks done
cosmicBboy opened this issue Jan 6, 2025 · 0 comments
Open
2 tasks done

[BUG] Make FlyteFile and FlyteDirectory pickleable #6144

cosmicBboy opened this issue Jan 6, 2025 · 0 comments
Labels
bug Something isn't working untriaged This issues has not yet been looked at by the Maintainers

Comments

@cosmicBboy
Copy link
Contributor

Describe the bug

Currently, FlyteFile and FlyteDirectory objects with a non-default _downloader function (i.e. a _downloader that isn't the default noop function) cannot be pickled.

      File "/opt/micromamba/envs/runtime/lib/python3.11/multiprocessing/reduction.py", line 60, in dump
        ForkingPickler(file, protocol).dump(obj)
    AttributeError: Can't pickle local object 'FlyteDirToMultipartBlobTransformer.async_to_python_value.<locals>._downloader'

Message:

    AttributeError: Can't pickle local object 'FlyteDirToMultipartBlobTransformer.async_to_python_value.<locals>._downloader'

This error is raised when, e.g. users specify FlyteDirectory/File task inputs together with the Elastic plugin, which relies on pickling the inputs when running the task function code in parallel.

Expected behavior

This should just work.

Additional context to reproduce

No response

Screenshots

No response

Are you sure this issue hasn't been raised already?

  • Yes

Have you read the Code of Conduct?

  • Yes
@cosmicBboy cosmicBboy added bug Something isn't working untriaged This issues has not yet been looked at by the Maintainers labels Jan 6, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working untriaged This issues has not yet been looked at by the Maintainers
Projects
Status: Backlog
Development

No branches or pull requests

1 participant