Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(filemanager): add FileStateChange event schema #607

Closed
wants to merge 6 commits into from

Conversation

mmalenic
Copy link
Member

Related to #605, addresses point 1 of #605 (comment).

Changes

  • Adds the FileStateChange schema which will support tagging objects to transition them to another storage class, or delete them.
    • I don't think I've missed anything, as both expiring and transitioning objects is supported.
    • I haven't added any support for NoncurrentVersionTransition, NoncurrentVersionExpiration, AbortIncompleteMultipartUpload, or ExpiredObjectDeleteMarker, because I feel like these would be covered by bucket-wide rules that apply to all objects? It doesn't seem like it would be useful to have these apply differently based on the object prefix. I'm happy to add it in if we think it's useful though?
  • The implementation uses a build command to generate the schema from Rust code. This is more convenient because the same code can be used to parse incoming events.

@mmalenic mmalenic self-assigned this Oct 17, 2024
@mmalenic mmalenic added filemanager an issue relating to the filemanager feature New feature labels Oct 17, 2024
@reisingerf
Copy link
Member

Hm, not sure I understood the intention correctly...

The schema defines a FILE state change event, but it does not really link it to any particular file. It rather feels like a policy or lifecycle transition rule change.

It also does not seem to have provisioning for other "state" changes, like "archived" or "deleted" or "moved" ....

The "source" in the examples is orcabus.filemanager indicating an announcement from the FM?

Also, not sure all the S3 lifecycle details are needed/useful. We can't create rules for each and every file/event...

@mmalenic
Copy link
Member Author

mmalenic commented Oct 17, 2024

Yes, I think this probably isn't a StateChange kind of event, it more a transition event. The source isn't intended to be orcabus.filemanager, but rather a workflow or another service.

It also does not seem to have provisioning for other "state" changes, like "archived" or "deleted" or "moved" ....

On this note, one thing the filemanager could do that isn't possible using s3 lifecycles is to move objects between locations. E.g. a workflow run could say to move the objects to an archive bucket after n-days. Not sure if something like this would be useful.

@victorskl
Copy link
Member

victorskl commented Oct 18, 2024

@reisingerf @mmalenic Let us quick catch up call next week on this, pls. I'll follow up ping you on next Monday.

@mmalenic
Copy link
Member Author

Putting this feature on hold for now, as we need to figure out if using events/tags is the best approach for this.

@mmalenic mmalenic closed this Oct 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature New feature filemanager an issue relating to the filemanager
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants