Treat aggregation of profiles different per profile type.

[simonswine]

When aggregating a profile we have two dimension we aggregate over:

• vertical: Profiles part another series (another process, pod)
• temporal: Profiles from the same series taken at a different time.

We need to treat those different depending of the profile samples are counters or gauges.:

• Gauges: memory_in_use_*, goroutines
• Counters: profiles_cpu, memory_allocated

I think aggregating gauges by sum creates non meaningful values.

For gauges we sum vertically, but need to use either an (avg,min,max, quantile) over the temporal aggregation. Any thoughts?

[aocenas]

Would it make sense to bake this somehow into the query language? In a similar way to Prometheus?

In general I still feel like the actual absolute values in the flame graph don't really matter that much, you are probably not going to use a flamegraph to sort of alert on something like memory going over some limit.