API for Baggage

[cortopy]

Feature Request

Opentelemetry Rust provides APIs for Baggage. However, there is no integration with the tracing framework, probably because of its lack of support for distributed tracing.

Baggage has two main uses:

1. For propagation across systems. This is currently possible with the Opentelemetry rust libraries. One may use the BaggagePropagator from opentelemetry_sdk to add Baggage to an Extractor.
2. However, Baggage, as defined in the docs, "is contextual information that’s passed between spans". I have tried several options and I just can't find any mechanism to set Baggage for the trace, so that any span within it can add to it or propagate context using BaggagePropagator.

The limitation probably comes from the immutability aspect of Context. It's easy to create a new Context using something like:

let ctx_with_baggage = Context::map_current(|cx| cx.with_baggage([KeyValue::new("uid", "value".to_string())]));

But then there is no way to attach it anywhere, resulting in the span getting lost and never accessible for any span created within the same trace.

Motivation

Being able to share values for things like customer ids, session id, etc. across a trace and, ultimately, propagate it.

Proposal

There is an API provided by tracing-opentelemetry which allows replacing the current trace's context with a new one, so that Baggage can be added to it.

Alternatives

Opentelemetry Rust supports this, but not within the tracing framework.

[mladedav]

It should be possible to add the context with baggage to a new span by calling OpenTelemetrySpanExt::set_parent.

Arguably the API should be improved as context should be set at the creation of the span, but the attached context should be immutable according to OpenTelemetry spec as far as I know.

[cortopy]

thanks @mladedav ! But I don't have access to the next span, as the place where I want to set baggage is an axum extension.

I've tried the following:

let ctx_with_baggage = opentelemetry::Context::map_current(|cx| {
    cx.with_baggage([
        KeyValue::new("uid", web_session.uid().to_string()),
        KeyValue::new("session_id", web_session.id().to_string()),
    ])
});
tracing::Span::current().set_parent(ctx_with_baggage);

but it doesn't work

[mladedav]

When you say "Axum extension" do you mean an extractor? Or just a middleware?

The latter case should be simple enough as you should be able to just create a new span to your liking, the former would be a bit more problematic but I don't see why setting the context wouldn't work.

[cortopy]

I'm using it as a middleware, it's loaded like this:

.layer(middleware::from_fn_with_state(state.clone(), authenticate))

[mladedav]

In that case you should be able to just create a new span with the changed context. You can take a look at TraceLayer but pretty much any tower service that just creates the span with the context with baggage and then calls the inner service, instrumenting it with the new span, should work.

I'll still try and see why changing the context does not work.

[mladedav]

So, quite frankly, I'm not entirely sure how the contexts work, but the baggage is set in the context of the span. You just need to use tracing-opentelemetry to get the current span's context, not opentelemetry itself, that seems to not have been updated.

See the asserts in the following example.

Example

        let root = tracing::info_span!("app_start");
        let _enter = root.enter();

        let context = opentelemetry::Context::current();
        assert!(context.baggage().get("uid").is_none());
        let context = tracing::Span::current().context();
        assert!(context.baggage().get("uid").is_none());

        let ctx_with_baggage = opentelemetry::Context::map_current(|cx| {
            assert!(cx.baggage().get("uid").is_none());
            cx.with_baggage([
                KeyValue::new("uid", "seventeen maybe?"),
            ])
        });
        tracing::Span::current().set_parent(ctx_with_baggage);

        let context = tracing::Span::current().context();
        assert_eq!(context.baggage().get("uid").unwrap().as_str(), "seventeen maybe?");
        let context = opentelemetry::Context::current();
        assert!(context.baggage().get("uid").is_none());

        let child = tracing::info_span!("child");
        let _enter = child.enter();

        let context = tracing::Span::current().context();
        assert_eq!(context.baggage().get("uid").unwrap().as_str(), "seventeen maybe?");
        let context = opentelemetry::Context::current();
        assert!(context.baggage().get("uid").is_none());

So with this you should be able to get the baggage in child spans and use the propagator as you originally intended?

[cortopy]

I've run your code and I can see what the issue is. My middleware doesn't have any child spans. It makes sense because the tree looks something like this:

• request span (uid is not known here)
  • middleware span
  • execute span
    • child span 1
    • child span 2

The solution seems to be to set the baggage somewhere else, but unfortunately this messed up the whole trace. If I set the baggage in the execute span, then this span becomes root and the tree of the trace changes to something like this:

• request span
  • middleware span
• execute span (baggage is set here with set_parent accessing the user id from the request extension. It appears as root in tracing gui. I'm using openobserve)
  • child span 1
  • child span 2

[mladedav]

And you're not creating the execute span? When you create the span you want to have baggage, you just need to take the context of the parent span (which should be current at the time if you're not setting the parent explicitly), add baggage to that context and then create the new span and set its otel parent.

If you cannot do that, you can either create one more level in the execute span or you should be able to set the baggage inside your middleware and then instrument the Service::call inside with your span.