CPU Pressure Percent Verifcation

[rimbesat]

Hi team!

Just wanted to reach out to see if you have any ideas. Currently, we run experiments on our application and the application metrics go to our Datadog Dashboard.
With cpu pressure we can now specify how much cpu it will use with a percentage. We really want to validate the cpu usage with metrics, to show other users that various cpu percentages has a different effect. We currently use the following metric in our Datadog dashboard: kubernetes.cpu.usage.total/kubernetes.cpu.limits*100 to get an idea of the overall cpu % limits, but this metric does not vary according to different cpu percents given in the experiment definition.

In this screenshot i ran a 33 percent experiment and then a 60 percent experiment though i could not see a difference.
This process metric also does not show us what we are looking for

The requests and limits are stated as follows
requests:
memory: "1536Mi"
cpu: "3"
limits:
memory: "1536Mi"
cpu: "3"

I was able to use the sysbench cpu test you provided in your docs to validate on the container that the cpu pressure does correctly inject with various levels of pressure and it correctly shows the change in number of events per second. I was just wondering whether there are metrics you may know of that provide similar information on cpu performance, or some calculation that I should look into to prove the various percentages being injected.
I also wanted to ask a few more questions, if i am running a 40 percent cpu pressure experiment, am i running 40% of the limit or 40% of the request.
The target pods are autoscaling so how does the HPA know to do that, based on what exactly.

Thanks, in advance.

[rimbesat]

Hi @ptnapoleon, not sure who to tag for this, but could someone confirm that this has been seen? Thanks

[aymericDD]

Have you tried on a single node only with your pod scheduled on it and verify if the CPU usage is correct? Don't hesitate to increase the CPU request/limit to remove totally the throttling.

[rimbesat]

thanks @aymericDD we will give this a go

[rimbesat]

Hi @aymericDD, we tried out your suggestion. We scheduled our application pod onto one node, and we increased our request and limit to 15 cores in our deployment object. This was scehduled on a node that had 32 cores capacity. There are two application pods, both set to this configuration:

resources:
limits:
cpu: '15'
memory: 1Gi
requests:
cpu: '15'
memory: 680M

However, we could not reduce throttling with this approach, our application still throttles (at 60 percent cpu and 50 percent cpu) , and we found it difficult to get around this issue. Though, we still want to share our findings. We used three methods to give as much visibility into the cpu usage as possible. I used sysbench cpu run , kubectl top pod and Datadog metrics to see if they correspond well

SYSBENCH

No CPU pressure

CPU speed:
    events per second:  4076.32
 
General statistics:
    total time:                          10.0001s
    total number of events:              40766
 
Latency (ms):
         min:                                    0.24
         avg:                                    0.25
         max:                                    0.57
         95th percentile:                        0.25
         sum:                                 9995.14
 
Threads fairness:
    events (avg/stddev):           40766.0000/0.00
    execution time (avg/stddev):   9.9951/0.00

60 percent cpu

 
CPU speed:
    events per second:  1312.14
 
General statistics:
    total time:                          10.0213s
    total number of events:              13150
 
Latency (ms):
         min:                                    0.24
         avg:                                    0.76
         max:                                   63.67
         95th percentile:                        0.69
         sum:                                 9987.89
 
Threads fairness:
    events (avg/stddev):           13150.0000/0.00
    execution time (avg/stddev):   9.9879/0.00

50 percent cpu

 
Threads started!
 
CPU speed:
    events per second:  1760.19
 
General statistics:
    total time:                          10.0001s
    total number of events:              17603
 
Latency (ms):
         min:                                    0.24
         avg:                                    0.57
         max:                                   37.58
         95th percentile:                        0.28
         sum:                                 9970.26
 
Threads fairness:
    events (avg/stddev):           17603.0000/0.00
    execution time (avg/stddev):   9.9703/0.00

33 percent cpu

 
CPU speed:
    events per second:  2536.56
 
General statistics:
    total time:                          10.0001s
    total number of events:              25367
 
Latency (ms):
         min:                                    0.24
         avg:                                    0.39
         max:                                   38.89
         95th percentile:                        0.27
         sum:                                 9975.80
 
Threads fairness:
    events (avg/stddev):           25367.0000/0.00
    execution time (avg/stddev):   9.9758/0.00

10 percent cpu

CPU speed:
    events per second:  3568.12
 
General statistics:
    total time:                          10.0002s
    total number of events:              35684
 
Latency (ms):
         min:                                    0.24
         avg:                                    0.28
         max:                                   10.44
         95th percentile:                        0.26
         sum:                                 9991.59
 
Threads fairness:
    events (avg/stddev):           35684.0000/0.00
    execution time (avg/stddev):   9.9916/0.00

5 percent

CPU speed:
    events per second:  3631.65
 
General statistics:
    total time:                          10.0001s
    total number of events:              36319
 
Latency (ms):
         min:                                    0.24
         avg:                                    0.28
         max:                                   20.13
         95th percentile:                        0.26
         sum:                                 9992.05
 
Threads fairness:
    events (avg/stddev):           36319.0000/0.00
    execution time (avg/stddev):   9.9920/0.00

KUBECTL TOP

No CPU pressure

POD                                                               NAME                       CPU(cores)   MEMORY(bytes)
fault-injection-showcase-plugin-master-template-79dd4895c97pptp   fault-injection-showcase   23m          652Mi

60 percent

POD                                                               NAME                       CPU(cores)   MEMORY(bytes)
fault-injection-showcase-plugin-master-template-79dd4895c97pptp   fault-injection-showcase   12754m       669Mi

50 percent

POD                                                               NAME                       CPU(cores)   MEMORY(bytes)
fault-injection-showcase-plugin-master-template-79dd4895c97pptp   fault-injection-showcase   9713m        673Mi

33 percent

POD                                                               NAME                       CPU(cores)   MEMORY(bytes)
fault-injection-showcase-plugin-master-template-79dd4895c97pptp   fault-injection-showcase   8046m        679Mi

10 percent

POD                                                               NAME                       CPU(cores)   MEMORY(bytes)
fault-injection-showcase-plugin-master-template-79dd4895c97pptp   fault-injection-showcase   4318m        681Mi

5 percent

POD                                                               NAME                       CPU(cores)   MEMORY(bytes)
fault-injection-showcase-plugin-master-template-79dd4895c97pptp   fault-injection-showcase   1544m        686Mi

DATADOG METRICS

Below shows various different metric dashboard to represent the data, and it is annotated with the corresponding percentage cpu experiement that was run. We are seeing that 12 cores are being used during a 60 percent cpu experiment however we have set the cores to 15, and we expect 9 cores to be used. And the other dashboard does say 80 percent used. But disregarding the ones that throttle, 33 percent cpu experiment show usage of 8 cores when it should be around 5. The same mismatch is happening with 10 percent cpu experiment that shows 3 core usage but we would expect 1.5 cores.
We face similar issues with kubectl top command.

One thing to note is that the first graph showing container.cpu.usage stays below the limit of 15 but still throttles, when 60 pe3rcent cpu is applied. Another aspect is that we also have a graph with kubernetes.cpu.usage.total which reaches even 20 cores which could explain the throttling however these is quite a difference between these two graphs that are meant to measure the same aspects.

If theres anything else I could do to explore further, let me know. If you have seen this behaviour elsewhere it would be good to know too.

[aymericDD]

Thanks for your reply and sorry for the delay 🙏

The percentage defined in the CPU pressure selects a number of cores, but when an application uses the CPU, the kernel allocates an amount of CPU time during a period. For example, the kernel can allocate to your application the usage of 20ms CPU time during a period of 100ms. If all the CPU time is consumed during this period, the application is throttled. Many factors can change the way the CPU time is allocated (QOS, specific kernel config, number of application running, etc, etc). By default, Kubernetes does not enforce exclusive access to a CPU, hence an app can be allocated all CPUs of the server, and several container can hence share the same CPU https://kubernetes.io/docs/tasks/administer-cluster/cpu-management-policies/#none-policy. If we take your experiment, it looks the application used 20 cores on a 15 app CPU limit. This means, we see on cpu-stress at least 20 cores, and we try to stress those cores at 80%, hence we are throttled at some point (by the linux scheduler).

If we look at the graph the impact seems greater than expected, still impact follow a valid trend. So I proposed to detect a behaviour change during the application of a CPU pressure by defining a range. If you apply CPU pressure to test HPA, I will redirect you to the datadog support and they will find an interlocutor to help you.

[rimbesat]

HI @aymericDD, I appreciate your explanation. I want to round this discussion off with some of our findings. We really wanted to find some form of metric that can prove the amount cpu pressure applied. We found the user cpu on our container, correlated very well. For example Datadog uses this process.stat.container.cpu.user_pct{*}, to give me this user cpu graph at container level when running a 33% cpu pressure experiment on one container

However we only found this is the case for a specific ec2 instance type m5a.4xlarge. On other instance types such as c6a.4xlarge, m5.2xlarge the cpu is much higher when running a 33% cpu pressure on one container. (c6a.4xlarge, m5.2xlarge results shown below)
.

I initally thought it was to do with the number of cores on the instance. However m5a.4xlarge shares the same number of cores as c6a.4xlarge.

This message is just to give you some insight into what we are finding, and we don't intend to spend more time into investigating as we are aware of the numerous variables that can affect cpu behaviour. We were just wondering whether this is repeatable on your end, or if you think the user cpu isn't meant to correlate in any way.

Anyway, just want to say again, thank you for the continuous support while we have been investigating this.

[rimbesat]

Hi, if anyone has a chance to look at this again , our main question is,
how do you measure cpu metrics on Datadog dashboards?