How to enrich context in one place and use the added value elsewhere

[gustaff-weldon]

I'm working on instrumenting several services in the request chain with a distributed traces. I got those working and can see nice waterfalls in zipkin.

Next step is to enrich traces with user data. I can obtain it in one service, would like to add it to all spans in that service (and later in further services, but that's not in the scope of this discussion).

My code works roughly like this:

1. setup Node opentelemetry at the start of the app (includes custom BatchSpanProcessor that adds baggage entries to span and exports to OTLPExporter)

  const provider: NodeTracerProvider = new NodeTracerProvider({
    resource: Resource.default().merge(
      new Resource({
        [SemanticResourceAttributes.SERVICE_NAME]: serviceName,
        [SemanticResourceAttributes.DEPLOYMENT_ENVIRONMENT]:
          deploymentEnvironment,
      })
    ),
  });

const collectorTraceExporter = new OTLPTraceExporter({
  url: tracesUrl,
});

provider.addSpanProcessor(
  new BaggageSpanProcessor(collectorTraceExporter)
);

2. The BaggageSpanProcessor is a simple processor that is supposed to add all Baggage entries to every span:

class BaggageSpanProcessor extends BatchSpanProcessor {
  onStart(span: Span): void {
    const tracingContext = api.context.active();
    const baggage = api.propagation.getBaggage(tracingContext);
    baggage?.getAllEntries().forEach((item: [string, BaggageEntry]) => {
      span.setAttribute(item[0], item[1].value);
    });
  }
}

3. When my app is running, I can only extract user id per request and store it in the baggage:

this.server = new ApolloServer({
  ...
  context: ({ req: { headers } }) => {
      const userData: UserData = extractUserData(headers);
      const baggage = api.propagation.createBaggage({
        "enduser.id": { value: userData.user.id },
      });

      // this creates new context...
      const newCtx = api.propagation.setBaggage(
        api.context.active(),
        baggage
      );

      // the context above is not active is it? 

      return graphqlContext;
  }
  ...
});

How do I add the baggage to current context that the processor would see it?
I cannot make the context active with

api.context.with(ctx.setValue(key, "context 2"), async () => {
  // "context 2" is active
  console.log(api.context.active().getValue(key)) // "context 2"
});

as the code which add spans executes somewhere else when handling graphql request.

[dyladan]

Not sure what you mean by "the code which add spans executes somewhere else when handling graphql request." Is this referring to some sort of thread pool situation? I know there have been several issues with the graphql instrumentation context propagation because of some advanced handling within their module.

[gustaff-weldon]

I mean, when the actual request comes in, it is handled by graphql-gateway, which I believe under the hood uses express.

so the request gets handled roughly like this:

The place where I can add user id to baggage is in the onRequest / context path (that's a callback).
The actual execution path that gets instrumented and generates spans is in the http / graphql handler path.

So whatever I add to baggage in callback, will not get to the main path as I believe that main path would need to wrapped with context.with with that baggage?

[vmarchaud]

yeah you are right but looking at your code (and looking into the apollo documentation) i don't think you can wrap the execution with the context.with method because you don't control the execution (like you can do with express by calling next middleware). I think the best way forward would be to discuss this on the apollo repo

[gustaff-weldon]

@vmarchaud I know, that's why I was asking if there's a way to add baggage from a callback and use it in the root path of the execution.

There does not seem to be, which is a shame, since this problem is common to all frameworks/libraries which use hooks pre/post but do not give you the ability to wrap the actual main execution path.

Thanks for chiming in.

[vmarchaud]

Not currently, dyladan started an effort to implement this with #123 but there was some blocker if i remember correctly. But i think the people from apollo can maybe offer something in the mean time since they already use otel

[dyladan]

Yeah I started work. The blocker is that being able to simply attach/detach isn't enough for many use cases. Imagine the scenario where the pre hook is called using context.with and you attach within that hook. When the hook execution ends, the with block ends and the previous context is reinstated which means you lose the context you attach'd. I'm really not sure how that can be handled in the general case. Making the baggage mutable would be one option (create it when the request comes in, then all execution paths share the same mutable baggage) but then we violate spec which specifically states baggage is immutable.