queue: Allow stuck entries (same node) to be reaped. See #169

[sbrossie]

This is a fix to address existing stuck entries to be reaped even in a single node deployment. See #169 (comment)

I tried this on an existing deployment showing the issue by pushing this jar:

 select * from notifications where effective_date < NOW();
+-----------+--------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------+---------------------+---------------------------------------------+---------------------------------------------+---------------------------+------------------+-------------+-------------+-------------+-----------------------------------------+---------------------+--------------------------------------+
| record_id | class_name                                                               | event_json                                                                                                                                                                | user_token                           | created_date        | creating_owner                              | processing_owner                            | processing_available_date | processing_state | error_count | search_key1 | search_key2 | queue_name                              | effective_date      | future_user_token                    |
+-----------+--------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------+---------------------+---------------------------------------------+---------------------------------------------+---------------------------+------------------+-------------+-------------+-------------+-----------------------------------------+---------------------+--------------------------------------+
|     71560 | org.killbill.billing.invoice.notification.NextBillingDateNotificationKey | {"uuidKey":null,"uuidKeys":["abfe39d2-1f98-41fc-92f9-66891dd66835"],"targetDate":"2023-11-10T09:08:03.000Z","isDryRunForInvoiceNotification":false,"isRescheduled":false} | 066bcb36-3f3e-4964-b7fc-2cb7fe1a7632 | 2023-11-09 09:08:05 | ip-172-31-64-209.us-west-2.compute.internal | ip-172-31-64-209.us-west-2.compute.internal | 2023-11-10 09:13:13       | IN_PROCESSING    |           0 |       16224 |        5859 | invoice-service:next-billing-date-queue | 2023-11-10 09:08:03 | 84b8497e-44cc-49cf-bff5-051c4fc066e5 |
|     76528 | org.killbill.billing.invoice.notification.NextBillingDateNotificationKey | {"uuidKey":null,"uuidKeys":["fd908f5d-7bdf-4e20-902b-a0fd9fedca2a"],"targetDate":"2024-01-16T03:42:02.000Z","isDryRunForInvoiceNotification":false,"isRescheduled":false} | 555b6b99-7f59-4894-8f49-f70c40b3549a | 2024-01-15 03:42:03 | ip-172-31-64-209.us-west-2.compute.internal | ip-172-31-64-209.us-west-2.compute.internal | 2024-01-16 03:47:08       | IN_PROCESSING    |           0 |        3014 |        1261 | invoice-service:next-billing-date-queue | 2024-01-16 03:42:02 | 09415c00-165c-4612-97a3-b1a44f21993e |
+---

Then we see:

2024-04-12T02:01:20,080+0000 lvl='WARN', log='DBBackedQueue', th='NotificationReaper-1', xff='', rId='', tok='', aRId='', tRId='', DBBackedQueue-notifications reapEntries: 2 entries were reaped by ip-172-31-64-209.us-west-2.compute.internal java.util.stream.ReferencePipeline$3@5d78cdf4

And then we see:

select * from notifications where effective_date < NOW();
Empty set (0.00 sec)

[pierre]

Just need to also change testWithStuckEntryProcessedByThisNode.

[pierre]

Not used anymore.

[sbrossie]

Fixed in 158f874