Profile/pressure analisys best practices

[bhack]

Can you add a clear list of best practices to test the sidecar resource allocation? From the current documentation it seems to be a little bit "black magic".
It is hard to understand on real workload if the sidecar is under pressure and it requires additional memory and CPU limits.

[bhack]

See also #35

[songjiaxun]

Thanks for the question. This is a great suggestion. Our team will discuss how to provide best practices guidance and update you here.

[bhack]

@songjiaxun Do you think that this could be solved by your comment at #61 (comment) ?

[bhack]

@songjiaxun As an alternative to unlimited resources do you think that we could occasionally log the the CPU occupancy so that we could fine-tune resource reservation?

[bhack]

As on autopilot we need to limit the CPU resources assigned to the sidecar how we could be sure that the sidecar is not a bottleneck?
On deep learning jobs we have dataloaders/dataworkers that require assigned CPU Resource on the main container. These need to cooperated with the sidecar resources to transfer file, preprocessing and feed GPUs.

We need to have a reliable way to understand when the sidecar is the bottleneck or instead it is on the dataloaders/datowrkers assigned resources.