Add support for Micrometer's Observation API for the SQS pipeline

[sondemar]

📢 Type of change

• Bugfix
• New feature
• Enhancement
• Refactoring

📜 Description

Added Micrometer's Observability support instruments SQS processing out of the box, while additionally propagating the inbound tracing information by looking it up in MessageHeaders.
Closes #646

💡 Motivation and Context

💚 How did you test it?

📝 Checklist

• I reviewed submitted code
• I added tests to verify changes
• I updated reference documentation to reflect the change
• All tests passing
• No breaking changes

🔮 Next steps

[tomazfernandes]

Hi @sondemar, on a first glance this looks great!

I wonder if it'd also make sense to add Observability support to SqsTemplate? Or perhaps leave it to a separate issue / PR?

Let me know your thoughts, thanks!

[sondemar]

Hi @sondemar, on a first glance this looks great!

I wonder if it'd also make sense to add Observability support to SqsTemplate? Or perhaps leave it to a separate issue / PR?

Let me know your thoughts, thanks!

Hi @tomazfernandes , I have also thought about templates (SqsTemplate and SnsTemplate as well). It makes sense to include them because we might want to propagate the tracing context while sending messages to SQS/SNS or receiving messages from SQS (SqsTemplate). Initially, I considered including support for SQS/SNS templates in a separate ticket, but including SqsTemplate with this ticket is also fine.

[tomazfernandes]

Hi @tomazfernandes , I have also thought about templates (SqsTemplate and SnsTemplate as well). It makes sense to include them because we might want to propagate the tracing context while sending messages to SQS/SNS or receiving messages from SQS (SqsTemplate). Initially, I considered including support for SQS/SNS templates in a separate ticket, but including SqsTemplate with this ticket is also fine.

Makes sense, always a good idea to break into smaller PRs. In this case, TBH I'm not super familiar with Micrometer 2, so it'd be helpful to have a PR open for SqsTemplate instrumentation as well so I can run them locally and see the propagation happening, if that's ok with you.

[tomazfernandes]

Another point here is - the most challenging part of instrumenting observation for this project is that it has many points where thread hopping may occur.

The main reason is it supports any combination of Blocking and Non-blocking components such as Interceptors, Message Listener and Error Handlers.

So I believe ideally we would have an Integration Test such as e.g.:

• An AsyncMessageInterceptor that performs an async action using AsyncSqsClient (that should switch to an AWS SDK thread)
• A blocking Messageinterceptor that doesn't really have to do anything (that should switch to a thread from the standard / provided Task executor)
• An async message listener that performs an action using AsyncSqsClient and throws an error (should switch to a AWS SDK Thread)
• A blocking ErrorHandler that doesn't have to do anything (should switch back to a TaskExecutor thread)

As a user I would expect to have Observability context in all these points, regardless of async / sync components.

Of course, we could start supporting only let's say the message listener, but then we'd need to make it clear in the docs what are the limitations.

Makes sense? Please let me know your thoughts. Thanks!

[sondemar]

I have already started the implementation of observability support for SqsTemplate (in the same PR), but now I am waiting for feedback on the following questions: micrometer-metrics/context-propagation#262 and micrometer-metrics/tracing#765.

Regarding the scenarios for the integration test, it makes sense, and I will apply them.

[tomazfernandes]

Hello @sondemar, just to catch up on this

but now I am waiting for feedback on the following questions: micrometer-metrics/context-propagation#262 and micrometer-metrics/tracing#765.

I see you're still looking into this, anything else we could do on this end?

Thanks!

[sondemar]

Hello @sondemar, just to catch up on this

but now I am waiting for feedback on the following questions: micrometer-metrics/context-propagation#262 and micrometer-metrics/tracing#765.

I see you're still looking into this, anything else we could do on this end?

Thanks!

Hi @tomazfernandes !

I have found a solution for the issues I encountered. I am currently completing the integration tests proposed in this comment. I expect to have the full functionality ready by next week.

Thank you for your patience!

[sondemar]

#646

Hi @tomazfernandes ,

I have just pushed the following changes:

1. Added observability for the SQS listener pipeline.
2. Introduced observability for SqsTemplate.
3. Created a test for your proposed scenario in this comment.

[sondemar]

Hello @tomazfernandes, can I do anything to help make the review easier?

[AnakinPt]

I have just one question. This will send the trace when we send sns messages and we have SQS subscribing the SNS?

[sondemar]

I have just one question. This will send the trace when we send sns messages and we have SQS subscribing the SNS?

Hi @AnakinPt , According to this comment, the SnsTemplate is not included in this PR. As I mentioned previously in this earlier comment:

"Initially, I considered including support for SQS/SNS templates in a separate ticket, but including SqsTemplate with this ticket is also fine."