Cross-origin copy fails for Linode Object Storage

[AlexanderPodorov]

Hello,

We're using boto3 with Linode Object Storage, which is compatible with AWS S3 according to their documentation.
Everything seems to work well, except cross-origin copy operation.
When I download an object from source origin/bucket and then upload it to destination origin/bucket, everything works well. Although, I'd like to avoid that unnecessary upload/download step.

I have the bucket named test-bucket on both origins. And I'd like to copy the object named test-object from us-east-1 to us-southeast-1 cluster.

Here is the example code I'm using:

from boto3 import client
from boto3.session import Session

sess = Session(
    aws_access_key_id='***',
    aws_secret_access_key='***'
)

s3_client_src = sess.client(
    service_name='s3',  
    region_name='us-east-1',
    endpoint_url='https://us-east-1.linodeobjects.com'
)
# test-bucket and test-object are already exists.

s3_client_trg = sess.client(
    service_name='s3',
    region_name='us-southeast-1',
    endpoint_url='https://us-southeast-1.linodeobjects.com'
)

copy_source = {
    'Bucket': 'test-bucket',
    'Key': 'test-object'
}
s3_client_trg.copy(CopySource=copy_source, Bucket='test-bucket', Key='test-object', SourceClient=s3_client_src)

When I call:

s3_client_src.list_objects(Bucket='test-bucket')['Contents']

It shows me that the test-object exists, But when I run copy, then it throws following message:

An error occurred (NoSuchKey) when calling the CopyObject operation: Unknown

Any help is appreciated!
Thanks!

[tim-finnigan]

Hi @AlexanderPodorov thanks for reaching out. I found an issue that mentioned the same error here: #2759

The premium support article mentioned there may help: https://aws.amazon.com/premiumsupport/knowledge-center/404-error-nosuchkey-s3/

But since you’re using Linode, I think the problem may have to do with the endpoint URL as ths customer mentioned in this comment:

A storage server that serves by my endpoint url created different system of folders' storage. Bucket name must be added to the "Folder1/vniz.txt", so that I had to send "base/Folder1/vniz.txt" where "base" is a bucket name. Thanks I resolved my problem!

[AlexanderPodorov]

Hi @tim-finnigan thanks for the answer! Unfortunately the issue you mentioned is not relevant to my issue. Copy operation within the origin works fine to me. I have the issue with cross-origin copy request. The article you mentioned have not helped as well.
For experiment sake, I intentionally changed Key name to something wrong:

copy_source = {
    'Bucket': 'test-bucket',
    'Key': 'test-object-wrong'
}

I get An error occurred (404) when calling the HeadObject operation: Not Found. While on a correct Key I have: An error occurred (NoSuchKey) when calling the CopyObject operation: Unknown.
If I use SourceClient argument to copy method, should it use s3_client_src and the origin that was set up for it?

[AlexanderPodorov]

After some debugging, I've found out that copy method actually does that:

PUT https://us-southeast-1.linodeobjects.com/test-bucket/test-object

And one of headers is that:

x-amz-copy-source: test-bucket/test-object

So it thinks that I want to copy test-bucket/test-object in us-southeast-1 which is clearly does not exist yet.
So I need to specify us-east-1 region for that somehow.
I've tried that:

x-amz-copy-source: us-east-1.linodeobjects.com/test-bucket/test-object

and that:

x-amz-copy-source: test-bucket.us-east-1.linodeobjects.com/test-object

Is it possible to specify the source region for bucket/object?

[AlexanderPodorov]

Well, how do you specify you want to copy from us-west-2 to us-east-1? How client "knows" you want to copy to us-east-1?

[tim-finnigan]

If you add boto3.set_stream_logger('') to your script then the debug logs will explain the behavior. A warning is given regarding this that says:

botocore.utils [DEBUG] S3 client configured for region us-west-2 but the bucket bucket-2 is in region us-east-1; Please configure the proper region to avoid multiple unnecessary redirects and signing attempts.
botocore.utils [DEBUG] Updating URI from https://s3.us-west-2.amazonaws.com/bucket-2/key-name to https://s3.us-east-1.amazonaws.com/bucket-2/key-name

The S3RegionRedirector.redirect_from_error in botocore will automatically redirect the request to wherever the bucket’s location is.

[AlexanderPodorov]

@tim-finnigan , thanks for info! In my case I have the same name test-bucket in both regions. We might have the same bucket name in 3 or 4 regions. How boto3 decides which bucket from which region to choose? It's ambiguous in this case.

[tim-finnigan]

Hi @AlexanderPodorov you can refer to here in botocore utils.py for more information on how the S3RegionRedirector works.

[AlexanderPodorov]

Hi @tim-finnigan , thank you! I tried to debug some more with S3RegionRedirector.
This condition is always truthy for me line, so the rest of code is not executed
If you take a look here
get_bucket_region method always fails for me, it performs head_bucket operation and extracts x-amz-bucket-region header, which is never returned in my case. Do you have any other ideas?

[AlexanderPodorov]

Closing the discussion.