Impossible to create a bucket in us-east-1 if underlying botocore session sets another default region

[ChristianWeymannTNG]

Describe the bug

When creating an S3 client with region_name=us-east-1 from a boto3.Session, if the underlying botocore.session.Session is setup with a botocore.config.Config that defines a different region_name, creating buckets will always fail with an IllegalLocationConstraintException

Expected Behavior

Being able to create buckets in the us-east-1 region

Current Behavior

Traceback (most recent call last):
  File "scratch_20.py", line 14, in <module>
    client.create_bucket(Bucket="test-bucket-creation-20221111")
  File "site-packages/botocore/client.py", line 495, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "site-packages/botocore/client.py", line 914, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (IllegalLocationConstraintException) when calling the CreateBucket operation: The unspecified location constraint is incompatible for the region specific endpoint this request was sent to.

Traceback (most recent call last):
  File "scratch_20.py", line 14, in <module>
    client.create_bucket(Bucket="test-bucket-creation-20221111", CreateBucketConfiguration={"LocationConstraint": "us-east-1"})
  File "site-packages/botocore/client.py", line 495, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "site-packages/botocore/client.py", line 914, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (IllegalLocationConstraintException) when calling the CreateBucket operation: The us-east-1 location constraint is incompatible for the region specific endpoint this request was sent to.

Reproduction Steps

import boto3
from botocore.config import Config
from botocore.session import Session

config = Config(region_name="eu-west-1")
session = Session()
session.set_default_client_config(config)
boto_session = boto3.Session(botocore_session=session)

client = boto_session.client("s3", region_name="us-east-1")
client.create_bucket(Bucket="test-bucket-creation-20221111") #raises
client.create_bucket(Bucket="test-bucket-creation-20221111", CreateBucketConfiguration={"LocationConstraint": "us-east-1"}) #raises

client = boto_session.client("s3", region_name="ap-southeast-1")
client.create_bucket(Bucket="test-bucket-creation-20221111", CreateBucketConfiguration={"LocationConstraint": "ap-southeast-1"}) #works as expected

Possible Solution

In the example above, it seems like like for the specific case of "us-east-1", the S3 endpoint for "eu-west-1" is still called.

Additional Information/Context

No response

SDK version used

boto3 1.26.1, botocore 1.29.1

Environment details (OS name and version, etc.)

Tested on Ubuntu 20.04 and lambda runtime python3.9

[ChristianWeymannTNG]

As a workaround, we currently reset the config around the calls

config = session._session.get_default_client_config()
config_no_region = config.merge(Config(region_name=None))
session._session.set_default_client_config(config_no_region)
try:
    ...
finally:
    session._session.set_default_client_config(config)

[tim-finnigan]

Hi @ChristianWeymannTNG thanks for reaching out. A similar issue was raised here before: #125. The create-bucket documentation notes that regions outside of us-east-1 require the appropriate LocationConstraint to be specified in order to create the bucket in the desired region. Otherwise the default is us-east-1. Please let us know if you had any further questions on this. Since this issue involves a guidance on expected behavior rather than a bug I am going to convert it to a discussion as per our triage process.

[ChristianWeymannTNG]

Hi @tim-finnigan
Would you consider reopening #3489 ? I really think this is a bug, and it is definitely different from #125 , see my message below.

[ChristianWeymannTNG]

Hi @tim-finnigan

I am aware of the behavior described in #125, this is different, apologies for not making that clear.

Here, I am trying to create a bucket in us-east-1. This fails independently of the LocationConstraint, seemingly because the default endpoint of the session is called, and not the endpoint corresponding to the region set in the client. This is only a problem when creating the client for the us-east-1 region, as can be seen in the reproduction steps.

I do not believe this is expected behavior.

[ChristianWeymannTNG]

Hi @tim-finnigan , thanks for your response.
They should not be conflicting, and they are not for other services and other regions, see the example with ap-southeast-1 in my reproduction steps. The default from the session should be overwritten by the setting passed when creating the client, which is the way it works for all services and regions, EXCEPT for s3 clients on us-east-1. This is why I believe this to be a bug.

[tim-finnigan]

Hi @ChristianWeymannTNG thanks for replying here. After some discussion with the team on this we decided to reach to S3 for more information on this behavior. A related issue was found here in the JS SDK repository: aws/aws-sdk-js#3647. I can share any updates here when we hear back.

[ChristianWeymannTNG]

Hi @tim-finnigan , thanks for following up. I'd be glad to hear back from you when you have updates from them. The issue you linked seems to be describing the same problem as in #125, that you can't set a LocationConstraint for us-east-1. I want to stress that the issue here is different, and making it IMPOSSIBLE to create a bucket in us-east-1, independently of whether the LocationConstraint is set or not.

[jonemo]

Hi @ChristianWeymannTNG, what you are describing can be separated into three components:

First, there are two ways to specify the region when creating a client. One is the region_name argument of the boto3_session.client() or botocore_session.create_client() methods. I will refer to this as "client constructor argument" below. The other is the region_name field in the Config object. The set_default_client_config() method sets the default for the latter. Therefore, for the sake of reproducing the problem, the code sample you provided can be simplified to:

from botocore.config import Config
from botocore.session import Session

session = Session()
client = session.client(
    "s3", region_name="us-east-1", config=Config(region_name="eu-west-1")
)

Second, when the client receives conflicting regions, as in the example above, then the client constructor argument usually takes precedence over the config object entry. However, when the function argument is us-east-1, then the config object takes precedence. Put differently: When either source of region name is not us-east-1, then the non-us-east-1 value will be used.

Finally, us-east-1 is not a valid value for the LocationConstraint. This is a behavior of the S3 API as documented here. Note that "us-east-1" is not listed as a valid value for "LocationConstraint". This is also what is covered in #125 and aws/aws-sdk-js#3647 which are mentioned above.

Putting all three together to resolve your issue of being unable to create a bucket in us-east-1: Please ensure that you do not provide conflicting region names for the client constructor argument and in the config object. In particular, use either the config object's region_name (for example, by using set_default_client_config) or provide a region_name argument to session.client(). If you want to keep the overall structure of your code with a default config region and custom regions for each clients, you could write it like this:

import boto3
from botocore.config import Config
from botocore.session import Session

config = Config(region_name="eu-west-1")
session = Session()
session.set_default_client_config(config)
boto_session = boto3.Session(botocore_session=session)

client = boto_session.client("s3", config=Config(region_name="us-east-1"))

[ChristianWeymannTNG]

Hi @jonemo , thank you for your in depth response.
I think my issue is with what you describe as the second component: that fact that in this seemingly quite arbitrary case, the behavior changes significantly. You say that "the client constructor argument usually takes precedence over the config object entry". Are there other exceptions? Are they documented somewhere?
It seems useful and expected to me to have a way of setting a default region for clients created without specifying it in the constructor arguments, and to be able to overwrite it without affecting other default parameters, like timeouts and retries, which are also set in the config. I thought this way was to set the region_name in the Config, and to overwrite it as needed when instantiating the clients, but I gather from your response that this is not the expected usage. Is there a way to pass a set of "default" parameters for all clients generated from a given session (including a "default" region), and to selectively overwrite them when instantiating the clients?