Datadog ingestion problem with .net and OTL collector

[rubencc]

Hi everyone

We have implemented tracing with opentelemetry on .net and integrated with datadog. I detect a lot of trace spans are not ingested by Datadog. My doubt is about this lack of ingestion is product of the configuration on our sampling on .net apps and the OTL collector or is a Datadog decision.

May I configure a particular sampler?
Its a missconfiguration on the OTL collector?
Is Datadog take the decision?
Is tailsamplingprocessor the right approach for a 100% trace ingestion?

Thanks!

[zacharycmontoya]

Hi Ruben, could you please describe your setup in more detail? How are you configuring the OpenTelemetry SDK in code and how are you configuring the OTEL collector? And can you clarify, are you missing endpoints entirely or are you getting fewer than expected traces being ingested?

[rubencc]

Hi Zachary,

We are not missing enpoints. We are missing traces but we need a 100% of these.

We use this config for our services (some are asp .net host, some have nservicebus, this code is one that have all):

return services.AddOpenTelemetryTracing(builder =>
        {
            builder.SetResourceBuilder(ResourceBuilder.CreateDefault()
                .AddService(serviceName: assemblyName.Name,
                    serviceVersion: assemblyName.Version!.ToString()));

            builder
                .AddAspNetCoreInstrumentation(options =>
                {
                    options.RecordException = true;
                    options.Filter = (req) =>
                        req.Request.Path.ToUriComponent() != "/" &&
                        !req.Request.Path.ToUriComponent().Contains("index.html", StringComparison.OrdinalIgnoreCase) &&
                        !req.Request.Path.ToUriComponent().Contains("favicon", StringComparison.OrdinalIgnoreCase) &&
                        !req.Request.Path.ToUriComponent().Contains("swagger", StringComparison.OrdinalIgnoreCase) &&
                        !req.Request.Path.ToString().Contains("/healthz", StringComparison.OrdinalIgnoreCase) &&
                        !req.Request.Path.ToString().Contains("/status", StringComparison.OrdinalIgnoreCase) &&
                        !(req.Request.Method == HttpMethod.Options.Method && req.Request.Headers.AccessControlRequestMethod.Any()); 
                })
                .AddMongoDBInstrumentation()
                .AddNServiceBusInstrumentation()
                .AddHttpClientInstrumentation(options =>
                {
                    options.SetHttpFlavor = true;
                    options.RecordException = true;
                    options.Filter = message =>
                    {
                        if (message.RequestUri?.ToString()
                                .Contains("livediagnostics.monitor.azure.com") ?? false)
                        {
                            return false;
                        }

                        if (message.RequestUri?.ToString()
                                .StartsWith("https://dc.services.visualstudio.com") ?? false)
                        {
                            return false;
                        }

                        if (message.RequestUri?.ToString()
                                .StartsWith("https://rt.services.visualstudio.com") ?? false)
                        {
                            return false;
                        }

                        if (message.RequestUri?.ToString()
                                .Contains("https://http-intake.logs.datadoghq") ?? false)
                        {
                            return false;
                        }

                        return true;
                    };
                })
                .AddConsoleExporter()
                .AddExporters(configuration, c =>
                {
                    c.UseOtelCollector();
                    c.UseAzureMonitor();
                });
        });

And the OTL collector config is:

  chart:
    spec:
      chart: opentelemetry-collector
      version: "0.35.0"
      sourceRef:
        kind: HelmRepository
        name: helm-registry-git-repository
        namespace: flux-system
      interval: 1m
  values:
    mode: deployment
    image:
      tag: 0.61.0
    config:
      receivers:
        jaeger: null
        prometheus: null
        zipkin: null
      exporters:
        datadog:
          api:
            site: datadoghq.com
            key: <APIKEY>
      processors:
        attributes:
          actions:
            - key: env
              value: <ENVName>
              action: insert
      service:
        pipelines:
          traces:
            receivers:
              - otlp
            processors:
              - batch
              - attributes
            exporters:
              - datadog
          metrics:
            receivers:
              - otlp
            processors:
              - batch
              - attributes
            exporters:
              - datadog
          logs: null
    ports:
      jaeger-compact:
        enabled: false
      jaeger-thrift:
        enabled: false
      jaeger-grpc:
        enabled: false
      zipkin:
        enabled: false

[dineshg13]

@rubencc which org is this ? Can you please create support ticket, so that we can investigate this in our backend.

[rubencc]

Despite the breakdown information shown by datadog by adding a retention filter for all traces that do not contain error these are available for the selected period.

[robvanpamel]

Hi @rubencc

I'm not sure if this question is still relevant, but I just came across this one.
If you look at the datadog exporter, in the example see (https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/datadogexporter/examples/collector.yaml) you notice there is a batch processor, which specifies a batch size to avoid the problem you describe.

Not sure if you have seen this one. I see you are using a batch processor, but not the configuration.

[rubencc]

Hi Rob,

We try these config with the default parameters and the recomended parameters by Datadog. It's no works because the problem is not in the collector side.

The solution were create a retention filter for the 100% traces in datadog.