Interoperability with existing applications and different metric libraries

[jonatan-ivanov]

What is the plan for interoperability with existing applications that are using different metric libraries?
On the Metric Workshop, we talked a little about this topic, it seems OpenTelemetry is compatible with OpenTracing and OpenCensus that are non-GA and most probably are not used widely in production. What about interoperability with other libraries that are widely used in prod, e.g.: Micrometer, Dropwizard Metrics, Prometheus?

Slightly connected discussion: #1368

cc: @shakuzen @brian-brazil @RichiH @bogdandrutu @jmacd @ebullient @matthiasr @jsuereth

[aavileli]

Definitely agree that other libraries that are widely used in prod, e.g.: Micrometer, Dropwizard Metrics, Prometheus should be included into the open-telemetry as this allows people using these existing libraries to work with a standard OpenTelemetry API directly.

[ebullient]

Stated differently as a series of questions:

1. Will OpenTelemetry define a metrics API?
2. Who is the target audience/consumer of that API (instrumentation agent developers, library developers, application developers, ..?)
3. What is the expectation for applications or libraries already instrumented with Micrometer or Dropwizard, etc.

Direct usage of an OpenTelemetry Metrics API is the question, as that requires code change and migration, where interop with existing instrumentation libraries (like Micrometer, Dropwizard, & Prometheus) could constrain changes to dependencies.

I would answer the above questions this way (if I had my way ;) ):
OpenTelemetry will provide a metrics API, but its use (by applications or libraries) is optional (rather than expected)
Micrometer, e.g., can use the OTel Metrics API internally, to allow existing micrometer-based instrumentation to work with OTel w/o requiring changes to those applications or libraries that use Micrometer.

That has some ramifications to the API surface, some of which I think will be resolved as the data model questions settle, in my opinion.

[tedsuo]

Hi @ebullient @jonatan-ivanov @aavileli, thank you for the questions, I can provide a high level answer.

So, we have two goals which drive us to create a metrics API.

One is to have an API which is integrated with Tracing. Both APIs are built on top of the same context propagation primitives, and can share state with each other under the hood. This expands the labeling options for metrics, and allows the collective data (tracing, metrics, logging) to all be exported in a single stream (OTLP protocol) which saves on overhead.

The other goal is to have APIs which are decoupled from any implementation. In OTel, the implementation is separate from the API, so taking on an API dependency does not pull in a dependency chain. This allows OSS libraries to instrument themselves natively while minimizing the risk of that instrumentation creating dependency creating conflicts.

(A more prosaic goal is that we need an API for adding metrics to the instrumentation we provide, but the above points apply there as well.)

So those are the features that we see as important added value to the metrics community. Most current metrics implementations do not have a distributed tracing tool to help them manage context to improve their labeling, and most metrics interfaces are tied to a specific implementation. Actually, MicroMeter does a good job of managing this abstraction, but it is only available in Java, with no plans to offer a similar abstracted API in other languages. Otherwise we would be very interested – we talked to them about this and they are advising us on our own metrics work.

Ok, now to answer your questions about interoperability, which are good questions and an important subject to us.

Most important, If someone has already instrumented their application with another metrics tool and is fine with their current setup, then they don't need to use the OpenTelemetry metrics API at all. If they just want to use tracing, they can disable metrics entirely by not installing a MetricsProvider (the implementation). There is no need to haul everything in if they just want one part of OpenTelemetry.

If they want to use the OTel metrics API intermixed in with another API, a MetricsProvider that is implemented with Prometheus (or their client) of choice could be written. That would allow for a progressive re-instrumentation, or allow them to make use of our metrics instrumentation (or an OSS library's instrumentation) without first having to rewrite their own instrumentation to do so. This could also work in the other direction: implementing MicroMeter with the OpenTelemetry client.

If they want to send their existing metrics data to the OpenTelemetry Collector, and make use of the processing features we provide there, they can use a Collector Receiver plugin for to receive data in that format.

Likewise, if they want to instrument everything with the OpenTelemetry API and then send the data to Prometheus, they can use the Prometheus Exporter – either in the Collector, or directly in the client if they don't want to run a Collector.

Ensuring that Prometheus/OpenMetrics data can be processed in this manner is a priority. I expect the above scenarios to be pretty common – many users will be interested in tracing and the collector, but will have a more nuanced story about metrics and logs. So we are keeping that in mind when we design the project.