Need a configurable process to auto re-attempt the DLQ

[trevoryager-callaway]

I would not want to always have to ask client to go and re-attempt everything in the DLQ.
We need a configurable way to re-attempt the subscriptions stuck in the DLQ that could be holding up future subscriptions of the same group.

Perhaps a timed process every now and again that pulls from custom metadata that indicates we do want to attempt them every so often.

[ChuckJonas]

When it comes to retries, I think there are two different mechanisms working together...

#7 would help reduce these items from ending up in the dead letter in the first place. However, the granularity of the retry interval can only really be guarantee to execute as often as the Scheduled Cleanup Job runs.

EG if the next attempt is set for 5m from now there isn't actually anyway to trigger the attempt 5m from now. Worst case (no other events are inserted) it won't actually run until the cleanup runs.

Once an item is in the dead letter, it can be reprocessed by simply incrementing the Relay_Attempts__c and Making sure its not Manually` dead lettered:

Subscription_Outbox__c[] deadlettered = [SELECT Id, Relay_Attempts__c FROM Subscription_Outbox__c WHERE Is_Deadletter__c = true];
for(Subscription_Outbox__c outbox :deadlettered){
   outbox.Relay_Attempts__c = outbox.Relay_Attempts__c + 1;
   outbox.Is_Manual_Deadletter__c = false;
}
update deadlettered;

Running the above code in a scheduled job could give you a cadence like:
ATTEMPT, ATTEMPT, ATTEMPT, PAUSE.... ATTEMPT... PAUSE... ATTEMPT

However, at some point you probably do just want the record to fall into the dead letter and someone should take a look at it.