Enable zero downtime/graceful shutdown

[tmeijn]

Hey @kayman-mk I've been thinking a bit on how to implement graceful shutdown and therefore having the ability to do zero downtime deployments. This is just high level based on some research I did, nothing actually tested yet, want to get your thoughts:

1. We run a script on the instance in the background that polls whether the instance is in the terminating state, ref: https://stackoverflow.com/questions/35914004/how-to-detect-state-of-aws-instance-from-inside-of-itself
   • scratch that, we can actually use the instance metadata api to monitor the target state:

2. Mark the service as non-restartable: https://unix.stackexchange.com/questions/689011/how-to-prevent-restarting-of-systemd-services
3. Once this state is detected, send a SIGQUIT signal to the Runner process. According to the docs this should do a graceful shutdown.
4. Monitor the existence of the process..., if it's still there, send heartbeat for the lifecycle
5. Once Runner process is no longer there, send complete-lifecycle-action
6. Profit?

The "Cleanup Lambda" would have to be triggered on actual termination instead of the terminate-wait hook.

Please let me know what you think! 😄

[kayman-mk]

Just linking #1029 and #631. Now on my list. We should push this forward.

[kayman-mk]

Since #1117 (May 2024) this has been solved. The old Runners are killed as soon as they finished the current jobs. The new Runners are spun up in parallel.