ES 1.3.1 testing against a Minio (S3-compatible ) storage backend - please help!

[hrstoyanov]

Discussed in #188

^{Originally posted by hrstoyanov March 28, 2024}
Hello,
I am trying to test ES 1.3.1 (with AWS Java SDK 2.25.18) against a local Minio server - a highly compatible AWS S3 storage server, widely for local development/testing of S3.

This setup code:

var awsBasicCredentials = AwsBasicCredentials.create(s3configuration.accessKey, s3configuration.secretKey);

            var s3Client = S3Client.builder()
                    .region(isEmptyOrNull(s3configuration.region)? Region.AWS_ISO_GLOBAL: Region.of(s3configuration.region))
                    .endpointOverride(s3configuration.endpoint)
                    .credentialsProvider(StaticCredentialsProvider.create(awsBasicCredentials))
                    .build();

            var buckets = s3Client.listBuckets().buckets();//Validate the S3 connection!
            logger.log(INFO, () -> STR."S3 buckets: \{buckets} ");

            var fileSystem = BlobStoreFileSystem.New(S3Connector.Caching(s3Client));

            return EmbeddedStorageFoundation.New()
                    .setConfiguration(StorageConfiguration.Builder()
                                        .setStorageFileProvider(StorageLiveFileProvider.Builder(fileSystem)
                                                                    .setDirectory(fileSystem.ensureDirectoryPath("storage"))
                                                                    .createFileProvider())
                                        .setChannelCountProvider(StorageChannelCountProvider.New(4))
                     .createConfiguration())
                     .onConnectionFoundation(BinaryHandlersJDK8::registerJDK8TypeHandlers)
                     .onConnectionFoundation(BinaryHandlersJDK17::registerJDK17TypeHandlers)
                     .createEmbeddedStorageManager();

generates the following exeception:

software.amazon.awssdk.services.s3.model.S3Exception: null (Service: S3, Status Code: 400, Request ID: 17C0DACB248DA3B8, Extended Request ID: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8)
	at software.amazon.awssdk.protocols.xml@2.25.18/software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleErrorResponse(AwsXmlPredicatedResponseHandler.java:156)
	at software.amazon.awssdk.protocols.xml@2.25.18/software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleResponse(AwsXmlPredicatedResponseHandler.java:108)
	at software.amazon.awssdk.protocols.xml@2.25.18/software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:85)
	at software.amazon.awssdk.protocols.xml@2.25.18/software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:43)

Am I setting up the embedded store manager correctly? In my code, you will see that I list the available buckets from the Minio server, so I know the S3Client connection is working fine. What am I missing?

In the Minio WEB UI admin console I see these traces:

s3.ListBuckets    200 OK
s3.HeadObject     400 Bad Request

[hrstoyanov]

See this:
minio/minio#19381 (comment)

[hrstoyanov]

See this for an alternative approach:
#188 (comment)