Storing future/forecasted metric data in thanos

[ghost]

Is it possible to store the future/forecasted metric data in thanos?

Using thanos-receive API we were able to store the future data (e.g 4 days in future) in thanos. The future timestamps were used while doing the prometheus remote write. Thanos-receive did not complain.

By using the @ modifier we were able to query the data also. For example

apm_request_duration_sum{server_k8s_namespace_name="opentelemetry-demo"}[300s] @ 1690707237 (i.e. 2023/7/30 17:12:00 GMT)

But after this storing any real time generated metrics using prometheus remote write was throwing following error.

2023-07-26T22:05:41.353+0530 error exporterhelper/queued_retry.go:391 Exporting failed. The error is not retryable. Dropping data. {"kind": "exporter", "data_type": "metrics", "name": "prometheusremotewrite", "error": "Permanent error: Permanent error: remote write returned HTTP status 409 Conflict; err = %!w(<nil>): store locally for endpoint 127.0.0.1:10901: add 26 samples: out of bounds\n", "dropped_items": 12} go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send

[matej-g]

Hi @draviraj,

Normally the data that is being ingested should be sent in order, i.e. you should be always sending newer samples than the current latest sample, and not older samples. If you started first with ingesting future data, the 'real-time' data will be refused with the error you are getting (out of bounds).

There is a feature to allow out-of-order ingestion (see hidden flag tsdb.out-of-order.time-window - https://github.com/thanos-io/thanos/blob/main/cmd/thanos/receive.go#L907). However, this is usually intended for smaller discrepancies between samples (perhaps in terms of minutes or max hours) that can be caused by infrastructure issues such as networking problems.

If you want to ingest data out of order with such big time differences (4 days), you might want to consider having separate receiver instances for your future data and current data.

I hope that helps.