New Queue Not Being Picked Up

[jtglass1]

Hi guys. I've a strange issue which is intermittent and I wanted to see if you had come across this before or might know what is causing it. I've two apps - [a] Admin - which runs in one JVM in ECS which allows the user to create queues dynamically via your framework [b] Reception - server which runs in a many JVM cluster also in ECS - it takes messages inbound from a REST API uses headers to derive the SQS queue the message will be stored on, and writes to that queue. 99% of the time this all works - and when a new queue is created by Admin, reception is able to use it immediately. However, sometimes, when reception tries to write to a new queue, it cannot find it. All other queues continue to work and we've never seen the look-up of a queue start to fail when it worked previously - the issue only happens when trying to write to a new queue for the first time. Only by restarting the reception server does the look-up start to work. It feels like there is some sort of cache here which is - for some reason - not getting refreshed reliably. I don't know if the 'can't find queue' is coming from the Spring layers or the AWS layers. It is also possible that there is an attribute I can use when creating the queue which will solve this issue - something like 'force cache reset'. I'm using v 3.1.1. I've put this in discussion as I honestly don't know where to start - AWS issue or Spring Cloud issue. Hoping you might have a view, thanks!

[jtglass1]

One other odd thing. We see this issue only when writing to the new queue. Servers which start listeners to subscribe to the new queue have no issues doing that. Obviously they never read a message - as no message arrives due to the reception issue above - but it is interesting that their subscription to the new queue works while the write (via sqsTemplate) fails.

[tomazfernandes]

Hi @jtglass1, thanks for bringing this up.

What's probably happening is that 1% of the time by the time your app gets the request with the queue name, the queue creation hasn't been propagated for the AWS SQS client to reach. It's not uncommon to have to wait a short time before we can access the queue.

And you're right we have a cache so we don't need to keep fetching queue information for every request, but it's a fairly naive cache implementation without a TTL, that'll replay whatever happened when we first tried to fetch the queue attributes.

spring-cloud-aws/spring-cloud-aws-sqs/src/main/java/io/awspring/cloud/sqs/operations/SqsTemplate.java

Lines 467 to 470 in c6a99ad

	private CompletableFuture<QueueAttributes> getQueueAttributes(String endpointName) {
	return this.queueAttributesCache.computeIfAbsent(endpointName,
	newName -> doGetQueueAttributes(endpointName, newName));
	}

The first idea that comes to mind would be removing the Future from the cache if it fails, but then in case of an unrecoverable error it might end up failing again and again. Which might be ok and an expected behavior TBH.

Let's make this a bug and discuss it further - let me know if you have any other ideas for a solution and feel free to open a PR when we reach a conclusion.

[jtglass1]

Thanks so much Tomaz - huge help. Your response has also helped me realise why we are hitting this. In our lower environments (early integration test) it is not uncommon for the message to come into reception before we have configured the target queue - just due to a lack in comms within the virtual team - and we then fix (by creating the queue) in reaction to the message reception failing. I now realise this means even when the queue is created, reception will never work as it has cached the first failure. We'd never have queue planning issues in higher environments (e.g. prod) but I now know why we see this in the lower ones. I'd be perfectly happy with your suggestion above - no cache on failure feels right to me. TTL for success - you are far better qualified to know if a cache without TTL could lead to issues on success. Your response also helps me workaround the issue - as we will know if we get this 'message before queue' pattern we can bounce reception - but we don't need to bounce reception otherwise. We were worried we would have to bounce reception in all cases. Again - we really appreciate your response - super helpful, John