AWS SDK 2.X migration for source connector [KCON-84] #374
Conversation
aindriu-aiven
force-pushed
the
aindriu-aiven/aws-sdk-2X
branch
from
6da63c0 to
7c4aea9
Compare
| @@ -18,10 +18,13 @@ plugins { id("aiven-apache-kafka-connectors-all.java-conventions") } | |||
|
|
|||
| val amazonS3Version by extra("1.12.777") | |||
| val amazonSTSVersion by extra("1.12.777") | |||
| val amazonV2Version by extra("2.29.34") | |||
There was a problem hiding this comment.
This can be updated to 2.29.36 😆 Please try to keep up!!
(Just kidding, but we should keep in mind to bump it to the most recent patch version before release!)
| if (Objects.isNull(config.getAwsS3EndPoint())) { | ||
| return null; | ||
| return S3Client.builder() |
There was a problem hiding this comment.
Nice clean-up for readability here!
| new PredefinedBackoffStrategies.FullJitterBackoffStrategy( | ||
| Math.toIntExact(config.getS3RetryBackoffDelayMs()), | ||
| Math.toIntExact(config.getS3RetryBackoffMaxDelayMs())), | ||
| config.getS3RetryBackoffMaxRetries(), false)); |
There was a problem hiding this comment.
I think you dropped this maxRetries! You can add it later to the r -> r .backoffStrategy(backoffStrategy).maxAttempts(config.getS3RetryBackoffMaxRetries()
There was a problem hiding this comment.
updated and also updated the new method names conventions.
| final ListObjectsV2Request request = ListObjectsV2Request.builder() | ||
| .bucket(bucketName) | ||
| .maxKeys(s3SourceConfig.getS3ConfigFragment().getFetchPageSize() * PAGE_SIZE_FACTOR) | ||
| .prefix(optionalKey(s3SourceConfig.getAwsS3Prefix())) |
There was a problem hiding this comment.
I checked, this looks like the correct way to set/unset a configuration (null/not-null)
| return s3Client.listObjectsV2( | ||
| new ListObjectsV2Request().withContinuationToken(response.getNextContinuationToken())); | ||
| return s3Client.listObjectsV2(ListObjectsV2Request.builder() | ||
| .maxKeys(s3SourceConfig.getS3ConfigFragment().getFetchPageSize() * PAGE_SIZE_FACTOR) |
There was a problem hiding this comment.
Do we need to have maxKeys here, just for my understanding, as we don't have prefix check.?
There was a problem hiding this comment.
I think the maxkeys is still required here but I would probably need to test it to see if it remembers the original max keys setting as I could not find that info in the documentation.
…long term support. Signed-off-by: Aindriu Lavelle <aindriu.lavelle@aiven.io>
Signed-off-by: Aindriu Lavelle <aindriu.lavelle@aiven.io>
Signed-off-by: Aindriu Lavelle <aindriu.lavelle@aiven.io>
…ved from the source connector Signed-off-by: Aindriu Lavelle <aindriu.lavelle@aiven.io>
…retry Signed-off-by: Aindriu Lavelle <aindriu.lavelle@aiven.io>
RyanSkraba
force-pushed
the
aindriu-aiven/aws-sdk-2X
branch
from
ea261ae to
ec1ccdf
Compare
RyanSkraba
changed the title
The AWS 1.X sdk is in maintenance mode and will be out of support by December 2025.
Key differences are
SDK 1.X still in use by sink connector but that will be required to be updated as well in the future, but this means the s3-commons code has both the 1.x and 2.x jars.