Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fixInfiniteLoopIssueWhenSourceTaskGotCancelled #545

Merged
merged 5 commits into from
Jan 25, 2024

Conversation

xinlian12
Copy link
Contributor

@xinlian12 xinlian12 commented Jan 24, 2024

Issue:
Post this change in kafka connect framework the connector uses a single thread for both
poll() as well as for stop(). Before that change,
the SourceTask::stop method would be invoked on a separate thread from
the one that did the actual data processing for the task (polling the task
for records, transforming and converting those records, then sending them
to Kafka). After that change, SourceTask::stop() is being invoked on the
same thread that handled data processing for the task. This had the effect
that tasks which blocked indefinitely in the SourceTask::poll method with the expectation
that they could stop blocking when SourceTask::stop
was invoked would no longer be capable of graceful shutdown, and may even
hang forever and the reverse is also true. Which is if stop() sleeps on the thread forever
then poll() would never be able to continue.

This issue also related to #543

Changes:

  1. Remove the empty check from the stop() method
  2. Adding .block when calling changeFeedProcessor.stop(), else the changeFeedProcessor will not be stopped and all internal tasks are still running
  3. using dedicated Kafka-cfp-bounded-elastic schedulers

@xinlian12 xinlian12 requested a review from a team as a code owner January 24, 2024 16:42
Copy link
Collaborator

@kushagraThapar kushagraThapar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks @xinlian12

CHANGELOG.md Show resolved Hide resolved
@xinlian12 xinlian12 merged commit 64709c1 into microsoft:dev Jan 25, 2024
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants