[misc] Layerwise profile updates

[varun-sundar-rabindranath]

Layerwise profile - changes and updates:

1. Add ability to profile the scenario where the engine is processing requests of different output-lengths.
   • Remove --output-length from offline_profile.py CLI args.
   • Add sub-commands run_num_steps and run_to_completion in its place.
   • run_num_steps captures the user-intent more clearly than --output-length. i.e. Profile n engine steps.
   • run_to_completion lets the user specify the number of requests the engine should complete every decode step. This provides layer-wise profile information for a range of batch-sizes.
2. Split gemm_ops into cutlass-gemm-ops and gemm-ops
3. Fix ProfileContext read in visualization code.
4. Bug fix : Incorrect use of rstrip()
5. Update layerwise_profile and visualization code to store and display metadata.

Examples:
run_num_steps sub-command:

model=neuralmagic/Meta-Llama-3-8B-Instruct-FP8
python3 examples/offline_profile.py --model ${model} --batch-size 128 --prompt-len 64 --json llama-8b-fp8-num-steps --csv llama-8b-fp8-num-ste    ps --enforce-eager run_num_steps -n 5

Graph:
command :

python3 tools/profiler/visualize_layerwise_profile.py --json-trace llama-8b-fp8-num-steps.json --output-directory profile-breakdown-num-steps     --level kernel --plot-metric cuda_time_ms --step-plot-interval 1

run_to_completion subcommand:

model=neuralmagic/Meta-Llama-3-8B-Instruct-FP8
python3 examples/offline_profile.py --model ${model} --batch-size 128 --prompt-len 64 --json llama-8b-fp8-to-completion --csv llama-8b-fp8-to-    completion --enforce-eager run_to_completion -n 16 

Graphs

python3 tools/profiler/visualize_layerwise_profile.py --json-trace llama-8b-fp8-to-completion.json --output-directory profile-breakdown-to-com    pletion --level kernel --plot-metric cuda_time_ms --step-plot-interval 1

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

• Add ready label to the PR
• Enable auto-merge.

🚀

[varun-sundar-rabindranath]

@LucasWilkinson i made some updates to layerwise-profile . Can you please take a look. Thanks !

[LucasWilkinson]

could we maybe keep a --output-len option? thats mutually exclusive with --max-output-len and --min-output-len?, seems a bit cumbersome if I want say all to have output-len of 8 to do --max-output-len 8 --min-output-len 8

[varun-sundar-rabindranath]

Hey. I have updated to CLI args to pass in num_steps directly. I believe it is better in capturing the intent. PTAL.

[LucasWilkinson]

maybe we should check for gemm in the name too? not that we use cutlass for anything else right now, but might prevent future confusion if we use cutlass for convolution or something

[varun-sundar-rabindranath]

Yeah, you are right. Ill add it.

[LucasWilkinson]

Left a few comments otherwise LGTM. Thanks, these seem like nice improvements!

[varun-sundar-rabindranath]

Hi @LucasWilkinson - I've made some non trivial changes. PTAL, thanks !

[LucasWilkinson]

I'm confused by the 'batch size = 32', when step_request[0] = 128, this doesn't seem to align with the '--complete-num-requests-per-step' comment where it is batch size 128

[varun-sundar-rabindranath]

My mistake. It should be "batch_size = 128" ... fixed now 👍