feat: tweak schema sampling (#2116)

* feat: tweak schema sampling * perf: tweak sample rate further
Logflare · Jun 24, 2024 · f0b06f3 · f0b06f3
1 parent 11dcb69
commit f0b06f3
Show file tree

Hide file tree

Showing 2 changed files with 5 additions and 3 deletions.
diff --git a/docs/docs.logflare.com/docs/concepts/ingestion/index.mdx b/docs/docs.logflare.com/docs/concepts/ingestion/index.mdx
@@ -104,7 +104,7 @@ metadata: {
 :::note
 On high ingestion volume, Logflare will sample incoming events instead of checking each event. The sample rate decreases as the ingestion rate increases. Ingestion rates are compared only on an individual local server that is performing the ingestion.
 
-From 1000-5000 events per second, sample rate is 0.1. Above 5000 events per second, sample rate is 0.01.
+From 10-100 events per second, sample rate is 0.2. From 100-500 events per second, sample rate is 0.1. From 500-1000 events per second, sample rate is 0.05. Above 1000 events per second, sample rate is 0.01.
 :::
 
 ### Key Transformation

diff --git a/lib/logflare/source/bigquery/pipeline.ex b/lib/logflare/source/bigquery/pipeline.ex
@@ -194,8 +194,10 @@ defmodule Logflare.Source.BigQuery.Pipeline do
     # random sample if local ingest rate is above a certain level
     probability =
       case PubSubRates.Cache.get_local_rates(source.token) do
-        %{average_rate: avg} when avg > 5000 -> 0.01
-        %{average_rate: avg} when avg > 1000 -> 0.1
+        %{average_rate: avg} when avg > 1000 -> 0.01
+        %{average_rate: avg} when avg > 500 -> 0.05
+        %{average_rate: avg} when avg > 100 -> 0.1
+        %{average_rate: avg} when avg > 10 -> 0.2
         _ -> 1
       end