Goroutines may be prioritized incorrectly

[eytan-avisror]

It was noticed when a huge spike of terminating instances happen (150 instances + another 150 instances after few minutes), we may not have enough goroutines or do not prioritize some goroutines efficiently.

The result was that and event that was received started processing 5-6 minutes later and by that time the heartbeat had already expired.

It's possible that newWorker goroutine is trying to spawn but there are so many other goroutines that it cannot.

We should load test and evaulate:

• Giving more priority to newWorker and sendHeartbeat goroutines.
• Overall reduce number of goroutines we spin up

The goal is to make sure we can handle large spikes of scale-downs without starvation

[eytan-avisror]

One improvement that may improve this is refactoring server.go
This block should not be called if message already exist, we should check

lifecycle-manager/pkg/service/server.go

Lines 90 to 93 in 48c3bc7

	// process messags from channel
	for message := range mgr.eventStream {
	go mgr.newWorker(message)
	}

We should move these checks up

lifecycle-manager/pkg/service/server.go

Lines 164 to 182 in 48c3bc7

	event, err := readMessage(message)
	if err != nil {
	err = errors.Wrap(err, "failed to read message")
	mgr.RejectEvent(err, event)
	return
	}
	event.SetQueueURL(url)
	if !event.IsValid() {
	err = errors.Wrap(err, "received invalid event")
	mgr.RejectEvent(err, event)
	return
	}
	if event.IsAlreadyExist(mgr.workQueue) {
	err := errors.New("event already exists")
	mgr.RejectEvent(err, event)
	return
	}

[eytan-avisror]

Message validation has been refactored to avoid spinning up worker goroutines for invalid/duplicate messages, this might help a bit.
But it seems the best improvement we can make here is around the deregistration waiters which spin up N * Cluster TargetGroups * Terminating Instances goroutines, which make up 99% of the goroutines we spin up.
We should come up with some approach to do this differently if possible.