Reactive S3 adapter - Project Reactor support

[roger751]

Type: Feature

Is your feature request related to a problem? Please describe.
As specified in spring-attic/spring-cloud-aws#802 and in spring-projects/spring-integration-aws#198, there should be support for Reactive Streams in the new AWS SDK v2 rewrite. I would be glad if this would be added to the milestone https://github.com/awspring/spring-cloud-aws/milestone/4.

Describe the solution you'd like
Enable the usage of Project Reactor with spring cloud aws.

Describe alternatives you've considered
As previously mentioned, this is a great example of self implementation https://github.com/eugenp/tutorials/tree/master/aws-reactive.

[markusherbert]

@maciejwalkowiak any updates on this?

[markusherbert]

I think it would also be nice if there would be support for Spring Integration with Project Reactor. Something like ReactiveS3MessageHandler

[maciejwalkowiak]

Not yet, but once #275 is done, this issue will become open for contributions.

[jifeiiii]

Anticipated feature. Also wanted with spring-projects/spring-integration-aws#198 - a Reactive Spring Integration channel adapter.

[maciejwalkowiak]

In regards to reactive - thinking how to do it right. We can easily auto-configure S3AsyncClient but likely we can do more. One is integrating code from Baeldung, another one is wrapping S3AsyncClient CompletableFutures with Mono in S3ReactiveClient. Wrapping could be done actually for all async clients.

If you have more ideas please share, perhaps we can come up here with conclusions and open this issue for contributions.

[markusherbert]

Yeah wrapping S3AsyncClient with a reactive class that will still give access to the underlying client sounds like the ideal way to go

[markusherbert]

I'd put S3AsyncClient as an argument in the constructor of S3ReactiveClient

[jifeiiii]

Any progress and a tracking issue so people can help?

[maciejwalkowiak]

Design work is pending. We are open for suggestion or even PRs but can't guarantee that anything will get merged until we do more research.

[sgarfinkel]

The latest versions of the SDK allows you to return an SdkPublisher<ByteBuffer> (which extends Publisher<ByteBuffer>) which you can use to map to the underlying byte array: https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/core/internal/async/PublisherAsyncResponseTransformer.html.

The main issue with doing this is backpressure management is a bit of a pain here , from what I understand the upstream Publisher will basically buffer endlessly until finished, which for large files with slow consumers isn't necessarily ideal.

[rolkhas2]

@sgarfinkel I see that the Publisher you've mentioned is on an internal directory. I'm not sure if that's targeted for public use.

[lynch19]

@maciejwalkowiak Can you publish the timeline for this issue? Users like me depend on this and cannot use Spring-Cloud-AWS at all without this issue being resolved.

[maciejwalkowiak]

@lynch19 it's almost certain this will not end up in 3.0. We have still too much work with other integrations to do before we can start working on reactive support. But, we're open for suggestions/contributions - if you have an idea how reactive support could look like, please let us know.

[lynch19]

@maciejwalkowiak I can try to take a basic start for this, but I'm not sure what have been already planned, or shall I just open some PR and let the discussion flow

[maciejwalkowiak]

Nothing has been planned so far so feel free to open a sketch PR but please keep in mind that it may not get merged.

[zaredea]

Would there be any progress on this subjectsoon?

[maciejwalkowiak]

@zaredea No unless someone from the community comes up with a sketch PR.

[maciejwalkowiak]

Considering the efforts needed to support reactive APIs we took the decision to not develop it in near future. We may embrace in the future async clients more than we do right now, which would make it easier for users to adapt to reactive apis themselves.