Skip to content

Commit

Permalink
feat: add sampling to open telemetry, exclude ingestion routes. (#2096)
Browse files Browse the repository at this point in the history 
* feat: add sampling to open telemetry, exclude ingestion routes.

* feat: exclude endpoint query routes
  • Loading branch information
@Ziinc
Ziinc authored Jun 14, 2024
1 parent 47654c5 commit 3ff14b7
Show file tree
Hide file tree
Showing 5 changed files with 113 additions and 21 deletions.
16 changes: 15 additions & 1 deletion config/runtime.exs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -121,6 +121,7 @@ log_level =
"warn" -> :warning
"warning" -> :warning
"info" -> :info
"debug" -> :debug
"error" -> :error
_ -> nil
end
Expand Down Expand Up @@ -288,8 +289,21 @@ config :libcluster,
if(System.get_env("LIBCLUSTER_TOPOLOGY") == "postgres", do: postgres_topology, else: [])

if System.get_env("LOGFLARE_OTEL_ENDPOINT") do
config :logflare, opentelemetry_enabled?: true

config :opentelemetry,
traces_exporter: :otlp
traces_exporter: :otlp,
sampler:
{:parent_based,
%{
root:
{LogflareWeb.OpenTelemetrySampler,
%{
probability:
System.get_env("LOGFLARE_OPEN_TELEMETRY_SAMPLE_RATIO", "0.001")
|> String.to_float()
}}
}}

config :opentelemetry_exporter,
otlp_protocol: :grpc,
Expand Down
38 changes: 20 additions & 18 deletions docs/docs.logflare.com/docs/self-hosting/index.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -19,24 +19,26 @@ All browser authentication will be disabled when in single-tenant mode.

### Common Configuration

| Env Var | Type | Description |
| ---------------------------------- | ------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `LOGFLARE_SINGLE_TENANT` | Boolean, defaults to `false` | If enabled, a singular user will be seeded. All browser usage will default to the user. |
| `LOGFLARE_API_KEY` | string, defaults to `nil` | If set, this API Key can be used for interacting with the Logflare API. API key will be automatically generated if not set. |
| `LOGFLARE_SUPABASE_MODE` | Boolean, defaults to `false` | A special mode for Logflare, where Supabase-specific resources will be seeded. Intended for Suapbase self-hosted usage. |
| `PHX_HTTP_PORT` | Integer, defaults to `4000` | Allows configuration of the HTTP server port. |
| `DB_SCHEMA` | String, defaults to `nil` | Allows configuration of the database schema to scope Logflare operations. |
| `LOGFLARE_LOG_LEVEL` | String, defaults to `info`. <br/>Options: `error`,`warning`, `info` | Allows runtime configuration of log level. |
| `LOGFLARE_NODE_HOST` | string, defaults to `127.0.0.1` | Sets node host on startup, which affects the node name `logflare@<host>` |
| `LOGFLARE_LOGGER_METADATA_CLUSTER` | string, defaults to `nil` | Sets global logging metadata for the cluster name. Useful for filtering logs by cluster name. |
| `LOGFLARE_PUBSUB_POOL_SIZE` | Integer, defaults to `10` | Sets the number of `Phoenix.PubSub.PG2` partitions to be created. Should be configured to the number of cores of your server for optimal multi-node performance. |
| `LOGFLARE_ALERTS_ENABLED` | Boolean, defaults to `true` | Flag for enabling and disabling query alerts. |
| `LOGFLARE_ALERTS_MIN_CLUSTER_SIZE` | Integer, defaults to `1` | Sets the required cluster size for Query Alerts to be run. If cluster size is below the provided value, query alerts will not run. |
| `LOGFLARE_MIN_CLUSTER_SIZE` | Integer, defaults to `1` | Sets the target cluster size, and emits a warning log periodically if the cluster is below the set number of nodes.. |
| `LOGFLARE_OTEL_ENDPOINT` | String, defaults to `nil` | Sets the OpenTelemetry Endpoint to send traces to via gRPC. Port number can be included, such as `https://logflare.app:443` |
| `LOGFLARE_OTEL_SOURCE_UUID` | String, defaults to `nil`, optionally required for OpenTelemetry. | Sets the appropriate header for ingesting OpenTelemetry events into a Logflare source. |
| `LOGFLARE_OTEL_ACCESS_TOKEN` | String, defaults to `nil`, optionally required for OpenTelemetry. | Sets the appropriate authentication header for ingesting OpenTelemetry events into a Logflare source. |

| Env Var | Type | Description |
| -------------------------------------- | ------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `LOGFLARE_SINGLE_TENANT` | Boolean, defaults to `false` | If enabled, a singular user will be seeded. All browser usage will default to the user. |
| `LOGFLARE_API_KEY` | string, defaults to `nil` | If set, this API Key can be used for interacting with the Logflare API. API key will be automatically generated if not set. |
| `LOGFLARE_SUPABASE_MODE` | Boolean, defaults to `false` | A special mode for Logflare, where Supabase-specific resources will be seeded. Intended for Suapbase self-hosted usage. |
| `PHX_HTTP_PORT` | Integer, defaults to `4000` | Allows configuration of the HTTP server port. |
| `DB_SCHEMA` | String, defaults to `nil` | Allows configuration of the database schema to scope Logflare operations. |
| `LOGFLARE_LOG_LEVEL` | String, defaults to `info`. <br/>Options: `error`,`warning`, `info` | Allows runtime configuration of log level. |
| `LOGFLARE_NODE_HOST` | string, defaults to `127.0.0.1` | Sets node host on startup, which affects the node name `logflare@<host>` |
| `LOGFLARE_LOGGER_METADATA_CLUSTER` | string, defaults to `nil` | Sets global logging metadata for the cluster name. Useful for filtering logs by cluster name. |
| `LOGFLARE_PUBSUB_POOL_SIZE` | Integer, defaults to `10` | Sets the number of `Phoenix.PubSub.PG2` partitions to be created. Should be configured to the number of cores of your server for optimal multi-node performance. |
| `LOGFLARE_ALERTS_ENABLED` | Boolean, defaults to `true` | Flag for enabling and disabling query alerts. |
| `LOGFLARE_ALERTS_MIN_CLUSTER_SIZE` | Integer, defaults to `1` | Sets the required cluster size for Query Alerts to be run. If cluster size is below the provided value, query alerts will not run. |
| `LOGFLARE_MIN_CLUSTER_SIZE` | Integer, defaults to `1` | Sets the target cluster size, and emits a warning log periodically if the cluster is below the set number of nodes.. |
| `LOGFLARE_OTEL_ENDPOINT` | String, defaults to `nil` | Sets the OpenTelemetry Endpoint to send traces to via gRPC. Port number can be included, such as `https://logflare.app:443` |
| `LOGFLARE_OTEL_SOURCE_UUID` | String, defaults to `nil`, optionally required for OpenTelemetry. | Sets the appropriate header for ingesting OpenTelemetry events into a Logflare source. |
| `LOGFLARE_OTEL_ACCESS_TOKEN` | String, defaults to `nil`, optionally required for OpenTelemetry. | Sets the appropriate authentication header for ingesting OpenTelemetry events into a Logflare source. |
| `LOGFLARE_OPEN_TELEMETRY_SAMPLE_RATIO` | Float, defaults to `0.001`, optionally required for OpenTelemetry. | Sets the sample ratio for server traces. Ingestion and Endpoint routes are dropped and are not included in tracing. |

LOGFLARE_OPEN_TELEMETRY_SAMPLE_RATIO
Additional environment variable configurations for the OpenTelemetry libraries used can be found [here](https://hexdocs.pm/opentelemetry_exporter/readme.html).perf/bq-pipeline-sharding

### BigQuery Backend Configuration
Expand Down
6 changes: 4 additions & 2 deletions lib/logflare/application.ex
View file
Original file line number Diff line number Diff line change
Expand Up @@ -31,8 +31,10 @@ defmodule Logflare.Application do

children = get_children(env)

:opentelemetry_cowboy.setup()
OpentelemetryPhoenix.setup(adapter: :cowboy2)
if Application.get_env(:logflare, :opentelemetry_enabled?) do
:opentelemetry_cowboy.setup()
OpentelemetryPhoenix.setup(adapter: :cowboy2)
end

# See https://hexdocs.pm/elixir/Supervisor.html
# for other strategies and supported options
Expand Down
67 changes: 67 additions & 0 deletions lib/logflare_web/open_telemetry_sampler.ex
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
defmodule LogflareWeb.OpenTelemetrySampler do
require OpenTelemetry.Tracer, as: Tracer
@behaviour :otel_sampler

@impl :otel_sampler
def setup(opts) do
:otel_sampler_trace_id_ratio_based.setup(opts.probability)
end

@impl :otel_sampler
def description(_sampler_config) do
"LogflareWeb.OpenTelemetrySampler"
end

@doc """
Drops traces for ingest and endpoints related routes.
Delegates to trace id sampler.
iex> ctx = %{}
iex> trace_id = 75141356756228984281078696925651880580
iex> links = {:links, 128, 128, :infinity, 0, []}
iex> span_name = "HTTP POST"
iex> span_kind = :server
iex> attributes = %{ "http.method": "POST", "http.target": "/logs/json"}
iex> sampler_config = %{probability: 0.001, id_upper_bound: 9223372036854776.0}
iex> {decision, _, _state} = should_sample(ctx, trace_id, links, span_name, span_kind, attributes, sampler_config)
iex> decision in [:drop, :record_and_sample]
true
"""
@impl :otel_sampler
def should_sample(
ctx,
trace_id,
links,
span_name,
span_kind,
attributes,
sampler_config
) do
tracestate = Tracer.current_span_ctx(ctx) |> OpenTelemetry.Span.tracestate()

exclude_route? =
case Map.get(attributes, "http.target") do
"/logs" <> _ -> true
"/api/logs" <> _ -> true
"/api/events" <> _ -> true
"/endpoints/query" <> _ -> true
"/api/endpoints/query" <> _ -> true
_ -> false
end

if exclude_route? do
{:drop, [], tracestate}
else
:otel_sampler_trace_id_ratio_based.should_sample(
ctx,
trace_id,
links,
span_name,
span_kind,
attributes,
sampler_config
)
end
end
end
7 changes: 7 additions & 0 deletions test/logflare_web/open_telemetry_sampler_test.exs
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
defmodule LogflareWeb.OpenTelemetrySamplerTest do
@moduledoc false
use LogflareWeb.ConnCase
import LogflareWeb.OpenTelemetrySampler

doctest LogflareWeb.OpenTelemetrySampler
end

0 comments on commit 3ff14b7

Please sign in to comment.