Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement graceful shutdown #4050

Open
jjbayer opened this issue Sep 20, 2024 · 0 comments
Open

Implement graceful shutdown #4050

jjbayer opened this issue Sep 20, 2024 · 0 comments

Comments

@jjbayer
Copy link
Member

jjbayer commented Sep 20, 2024

Follow-up to https://github.com/getsentry/team-ingest/issues/345: We want to give Relay plenty of time to burn down an envelope buffer backlog after it has received the SIGTERM shutdown signal, so we should set a long grace period.

Unfortunately Relay currently never shuts down before the end of the grace period, which prevents us from setting a longer grace period. To allow Relay to terminate when it is "done",

  1. Stop accepting envelopes after the shutdown signal has been received,
  2. ensure that all services subscribe to the shutdown signal,
  3. break any communication cycles between services. Cycles might be fine during normal operations but could extend shutdown indefinitely).
  4. Ensure that all services break out of their main loops once they do not receive any new data.
  5. In relay_server::run, do not await Controller::shutdown_handle().finished(). Instead, await a join handle for each service.

Relates to #4037, #4026 (join handles).

See summary of discussion here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant