OpenTelemetry log exporter and ElasticSearch APM 8+

[p10tyr]

Hi

I am just trying to see if I can get logs into Elastic Stack. It seems that since ES version 8 it is supported with the unstable data model for now. (I am on ES 8.1)

Tracking this is seems like there should be some very basic stuff coming though by now
#2482

I am looking at this example https://github.com/open-telemetry/opentelemetry-dotnet/blob/main/examples/Console/TestLogs.cs and it seems to use a local collector but I do not see where the URI is being configured? Is the default just localhost:4318 or something as per the docker instructions? Just curios because I am setting my own endpoint and not sure what the default value would be?? if any

So for my logging configuration I am using as below and I don't think I need to explicitly specify GRPC (but that is the only protocol Elastics supports for now) neither the simple exporter?

Sadly I do not see anything show up in Elastic streams and by the looks of #2482 there should be some basic data coming through

Using OpenTelemetry.Exporter.OpenTelemetryProtocol.Logs 1.0.0-rc9.2 (lowest to match)
To match stable OpenTelemetry 1.2.0

Unless I need to bump up to Otel 1.3.0 and logs-rc9.3? (I tried but because some of our stuff still used 4.7.2 I cant bump up other dependencies without a massive fallout)

 .AddLogging(configure => configure.AddOpenTelemetry(options =>
                {
                    options.IncludeScopes = true;

                    options.AddOtlpExporter(otlpExporter =>
                    {
                        otlpExporter.Protocol = OpenTelemetry.Exporter.OtlpExportProtocol.Grpc;
                        otlpExporter.ExportProcessorType = ExportProcessorType.Simple;
                        otlpExporter.Endpoint = new System.Uri("https://xyz.apm.aws.cloud.es.io");
                        otlpExporter.Headers = "Authorization=Bearer supersecret";
                    });
                }));

[cijothomas]

Please take the newest version from myget - https://github.com/open-telemetry/opentelemetry-dotnet#release-schedule
(as some of the OTLPLogExporter fixes are very new and hasn't released to nuget yet).

Or please collect sdk logs https://github.com/open-telemetry/opentelemetry-dotnet/tree/main/src/OpenTelemetry#self-diagnostics that can tell if something is wrong within the sdk/exporter.

[p10tyr]

Well here is a revelation! (using TestLogs)

It now decided to show up in log stream (maybe there is a long delay in processing)
Problem is I changed stuff so many times I don't know what the combination was

[cijothomas]

It looks like https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter has a elasticsearchexporter and elasticexporter, the latter being not recommended due to native support added...

[p10tyr]

Yea. I have no idea. After spending the whole day today no logs were showing up from my real application.

Now, all of a sudden. No code changes .. its streaming.

So to confirm I am using OpenTelemetry 1.2.0 with OpenTelemetry.Exporter.OpenTelemetryProtocol.Logs 1.0.0-rc9.2

I know some things are missing but for now just the logs and correlation ID's are super helpful.

I just wish I knew what it was that started the Logs to stream in Elastics.. I must have gone somewhere or clicked something while messing around in the Management settings that kicked it off ? Very confused.

Thank you for your help!

[cijothomas]

Please use the newest versions as they release (or from myget feed..), and share feedbacks. OTel Logging SIG is about to declare Logs Protocol as stable, so we are unblocked from that perspective to release a Stable version of OTLP LogExporter.

[p10tyr]

My real world application I cant just use your package because it is trying to upgrade a lot of Net packages.. that I cant just do with out a massive headache :( I am dealing with a hybrid / half migrated application.. So I will do that when I have 1 week to burn.

But this is good new regardless. I cant wait for it to go stable!

I would like ES Exporter to be thing.. Wonder if I should take a look at writing something because I am adding extra attributes to enrich ES logs. I will provide them here for the use of other people.

var otelAttributes = new List<KeyValuePair<string, object>>();
              otelAttributes.Add(new KeyValuePair<string, object>("deployment.environment", environmentName));

options.SetResourceBuilder(ResourceBuilder.CreateDefault()
   .AddService("ServiceName")  <-- Populates the service name if you need it
   .AddAttributes(otelAttributes));  <-- Enables filtering by environment in ES (also add any extra attributes if you need them)

[ejsmith]

I just got everything being sent to Elasticsearch 8.2 using otel collector and the ES apm server. It was a huge PITA to get it all working. You can view my setup in this branch: https://github.com/exceptionless/Exceptionless/tree/feature/elastic8x

Docker compose to run ES 8.x, Kibana and APM server all talking to each other with SSL turned on because it's required for APM server.
https://github.com/exceptionless/Exceptionless/blob/feature/elastic8x/docker-compose.yml

Otel collector config for collecting and forwarding to APM server:
https://github.com/exceptionless/Exceptionless/blob/feature/elastic8x/otel-config.yml

Ideally, we should be able to send directly to the APM Server and not have to run an otel collector instance, but I wasn't able to get it working.

My .NET configuration code to collect traces, metrics and logs from my app:
https://github.com/exceptionless/Exceptionless/blob/feature/elastic8x/src/Exceptionless.Web/ApmExtensions.cs

[p10tyr]

I am sending directly to Elastic Cloud APM Agent that is running in Fleet. Version 8.1

 otlpExporter.Endpoint = new Uri("https://redacted.apm.redacted.aws.cloud.es.io");
 otlpExporter.Headers = "Authorization=Bearer redacted";

To be fair. I am not 100% certain how it is all configured in the background. Maybe they spin up a collector on our behalf and that is why I was not seeing any thing before.

Maybe I triggered something when I ran this repo test code and all of a sudden logs started spitting out even for my older versions

[p10tyr]

Really cool stuff in the ApmExtensions! I am going to be going over that tomorrow morning 💯 ❤️

[ejsmith]

Yeah, it's a LOT easier that way. 😆

[p10tyr]

I'm definitely going to give your service a test drive with another project I'm working on when I sort this out 😄
Can't say that I have seen or heard of this before. Looks really awesome.

[ejsmith]

Awesome! Let me know if you need anything or have any questions.