Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

source-kinesis: bump maximum retry attempts #2167

Merged
merged 1 commit into from
Nov 25, 2024

Conversation

williamhbaker
Copy link
Member

@williamhbaker williamhbaker commented Nov 25, 2024

Description:

The AWS v2 Go SDK has a built in retry mechanism, but the default only retries 3 times before returning a fatal error.

This bumps that limit up so that the connector will continue retrying for ~5 minutes before failing with an error. This is the same configuration as we use for source-dynamodb - see

func (c *config) toClient(ctx context.Context) (*client, error) {
opts := []func(*awsConfig.LoadOptions) error{
awsConfig.WithCredentialsProvider(
credentials.NewStaticCredentialsProvider(c.AWSAccessKeyID, c.AWSSecretAccessKey, ""),
),
awsConfig.WithRegion(c.Region),
awsConfig.WithRetryer(func() aws.Retryer {
// Bump up the number of retry maximum attempts from the default of 3. The maximum retry
// duration is 20 seconds, so this gives us around 5 minutes of retrying retryable
// errors before giving up and crashing the connector.
//
// Ref: https://aws.github.io/aws-sdk-go-v2/docs/configuring-sdk/retries-timeouts/
return retry.AddWithMaxAttempts(retry.NewStandard(), 20)
}),
}

(Describe the high level scope of new or changed features)

Workflow steps:

(How does one use this feature, and how has it changed)

Documentation links affected:

(list any documentation links that you created, or existing ones that you've identified as needing updates, along with a brief description)

Notes for reviewers:

(anything that might help someone review this PR)


This change is Reviewable

The AWS v2 Go SDK has a built in retry mechanism, but the default only retries 3
times before returning a fatal error.

This bumps that limit up so that the connector will continue retrying for ~5
minutes before failing with an error. This is the same configuration as we use
for source-dynamodb.
Copy link
Contributor

@Alex-Bair Alex-Bair left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@williamhbaker williamhbaker merged commit 3fdcc30 into main Nov 25, 2024
52 of 53 checks passed
@williamhbaker williamhbaker deleted the wb/kinesis-more-retries branch November 25, 2024 15:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants