Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix local OTEL. Change OTEL to use tracer and ctx instead of span #2447

Merged
merged 21 commits into from
May 20, 2024

Conversation

mojotalantikite
Copy link
Contributor

@mojotalantikite mojotalantikite commented May 15, 2024

Summary

While investigating how OTEL and the error boundary in React interacted I noticed that our local OTEL setup (i.e. in local dev) wasn't working. It turns out that Jaegar changed how it deals with OTEL traces coming into the system, so this updates the local setup so we can use the most recent Jaegar and collect traces in local dev.

Once that was fixed, I was having issues seeing spans tied properly between app-web and app-api. I've never felt good about how we passed around a span instead of a tracer to our resolvers, so this updates our usage. Basically, it adds the tracer to our Context that is passed around, along with the extracted otel context that we get from the incoming app-web http headers, so in each resolver we can properly create a span tied to it's invocation.

Finally, there were some changes and cleanups that I wanted to do to app-web. We never used the ZoneContextManager, as is recommended by OTEL for frontends. We also had our TracerContext below our routing in react, so we never got navigation events (these are coming in now).

I also needed to add a new env variable for local dev, as we need to distinguish between where app-web and app-api send their traces with the Jaeger changes.

Related issues

https://jiraent.cms.gov/browse/MCR-4159
https://jiraent.cms.gov/browse/MCR-4119
https://jiraent.cms.gov/browse/MCR-4160

@mojotalantikite
Copy link
Contributor Author

mojotalantikite commented May 17, 2024

Also, I verified that we indeed are now recording React app crashes in OTEL, using Hana's settings page crash idea from #2433:
Screenshot 2024-05-17 at 3 09 26 PM

Copy link
Contributor

@haworku haworku left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good - can you demo next RSP what the traces look like too once we get this in production ? Also please add to my docs once my PR lands - lets pull some of the stuff in this PR description into the monitoring docs

@mojotalantikite
Copy link
Contributor Author

I threw my name on a topic to talk about OTEL in RSP. I'll also pull in your changes that just merged and add to the docs!

@mojotalantikite mojotalantikite merged commit 6e6ff98 into main May 20, 2024
28 checks passed
@mojotalantikite mojotalantikite deleted the mt-otel-experiments branch May 20, 2024 19:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants