Cross service traces with TraceIdRatioBasedSampler

[Henrik1979]

Hi,
I'm trying to get started with Tracing using .Net 6 core Web APIs, my infrastructure looks like this:

Web API ----> Database API ----> SQL Server

In the Web API I would like to sample a configurable number of the incoming requests, I'm using this configuration:

  .WithTracing(tracerProviderBuilder => tracerProviderBuilder
    .AddOtlpExporter(options =>
    {
      options.Endpoint = new Uri("http://localhost:9045");
    })
    .AddConsoleExporter()
    .AddSource(OtelTracing.SourceName)
    .SetSampler(new TraceIdRatioBasedSampler(1.00))
    .AddAspNetCoreInstrumentation()
    .AddHttpClientInstrumentation()
    .SetResourceBuilder(TracingResourceBuilder.GetResourceBuilder())
  );

And calling Database API with this dummy code:

[HttpGet]
public async Task<JsonResult> GetDevices()
{
  using var activity = OtelTracing.MyActivitySource.StartActivity();

  try
  {
    Activity? dbActivity = null;

    if (activity != null)
    {
      dbActivity = OtelTracing.MyActivitySource.StartActivity("Loading data from DB", ActivityKind.Internal, activity!.Context);
    }
   
    using (dbActivity?.AddEvent(new ActivityEvent("Executing query")))
    {
      _ = await client.GetAsync("http://localhost:7070/api/database");
    }

    var data = new List<object>
    {
      new { id = 1, Name = "E-Meter" },
      new { id = 2, Name = "MCOM" }
    };

    dbActivity?.Stop();
    activity?.AddEvent(new ActivityEvent("Parsing data"));
    return new JsonResult(data);
  }
  catch (Exception ex)
  {
    activity?.SetStatus(ActivityStatusCode.Error, "Something bad happened!");
    activity.RecordException(ex);
    return new JsonResult(Enumerable.Empty<object>());
  }
}

At the Database API I only want traces if the caller provides a trace ( I want a full trace through the system, when TraceIdRatioBasedSampler chooses to sample)

Configuration:

builder.Services.AddOpenTelemetry()
  .WithTracing(tracerProviderBuilder => tracerProviderBuilder
    .AddOtlpExporter(options =>
    {
      options.Endpoint = new Uri("http://localhost:9045");
    })
   .AddSource(OtelTracing.SourceName)
    .AddAspNetCoreInstrumentation()
    .SetResourceBuilder(TracingResourceBuilder.GetResourceBuilder())
    .AddConsoleExporter()
  );

if .AddAspNetCoreInstrumentation() is in the configuration everything is traced , if it's removed nothing is traced

    [HttpGet]
    public async Task<string> Get()
    {
      var headers = Request.Headers.ToDictionary(h => h.Key, h => h.Value.ToString());

      const string traceparent = "traceparent";
     
      // Activity.Current.Parent //always null
      //Activity.Current.Context.IsRemote // always false

      if (headers.ContainsKey(traceparent))
      {
        var dbActivity = OtelTracing.MyActivitySource.StartActivity("Loading data from DB", ActivityKind.Client, headers[traceparent]);
        dbActivity.AddEvent(new("do stuff "));
      }
     
      await Task.Delay(1500);
      return string.Empty;
    }

Can anyone please tell me if what I'm trying to do is possible and if so point me in the right direction