Receiver enforces label order

[obeleh]

I'm using open telemetry python

Code:

from typing import Iterable

from opentelemetry import metrics
from opentelemetry.metrics import CallbackOptions, Observation
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.exporter.prometheus_remote_write import (
    PrometheusRemoteWriteMetricsExporter
)
from opentelemetry.sdk.metrics._internal.export import PeriodicExportingMetricReader

exporter = PrometheusRemoteWriteMetricsExporter(
    endpoint="http://localhost:19291/api/v1/receive"
)

labels = {"some": "label"}
reader = PeriodicExportingMetricReader(exporter)
provider = MeterProvider(metric_readers=[reader])
metrics.set_meter_provider(provider)
meter = metrics.get_meter("test-meter")


def observe(options: CallbackOptions) -> Iterable[Observation]:
    return [Observation(
        123.0,
        {"a": 1}
    )]


meter.create_observable_gauge(
    name="my_gauge",
    callbacks=[observe],
)


provider.force_flush(100_000)

I get the following error

caller=writer.go:164 component=receive component=receive-writer tenant=default-tenant msg="Error on series with out-of-order labels" numDropped=1

I can see this code checks if the labels are in order.

The python code doesn't order the labels. In fact when I patch the code to sort labels before sending the samples it works. I can out of the blue raise an issue and a PR to solve this. But I wonder. Is there somewhere described that OTel should send these labels in order? Is it a bug on the python side? Why does receiver expect the ordering?

[matej-g]

Hey @obeleh,

I'm not sure about the OTel, but Prometheus states this, i.e. labels being ordered alphabetically, as a requirement. I have recently opened an issue to get this fully clarified (prometheus/prometheus#11505) and there you can find few more information and related issues (also look at the remote write spec draft that is linked in that issue, it's also one of the requirements).