Profile-Guided Optimization (PGO) benchmark report

[zamazan4ik]

Hi!

I was interested in optimizing the library's performance even further. I evaluated Profile-Guided Optimization (PGO) on many projects - all the results are available at https://github.com/zamazan4ik/awesome-pgo . Since this compiler optimization works well in many places, especially different parsers, I decided to apply it to the project - here are my benchmark results.

Test environment

• Fedora 40
• Linux kernel 6.9.4
• AMD Ryzen 9 5900x
• 48 Gib RAM
• SSD Samsung 980 Pro 2 Tib
• Compiler - Rustc 1.79
• musli version: main branch on commit af9cd871af456e1b56eea5ca0d7d5112bb1c9c27
• Disabled Turbo boost

Benchmark

For benchmark purposes, I use built-in into the project benchmarks. For PGO optimization I use cargo-pgo tool. Release bench result I got with taskset -c 0 cargo bench --workspace --all-features command. The PGO training phase is done with taskset -c 0 cargo pgo bench -- --workspace --all-features, PGO optimization phase - with taskset -c 0 cargo pgo optimize bench -- --workspace --all-features. taskset -c 0 is used for reducing the OS scheduler influence on the results. All measurements are done on the same machine, with the same background "noise" (as much as I can guarantee).

Results

I got the following results:

• Release: https://gist.github.com/zamazan4ik/17740e1a02a081dbbb853708c8328086
• PGO optimized compared to Release: https://gist.github.com/zamazan4ik/77ec01c7a015f99670e7dbcfc62b8b74
• (just for reference) PGO instrumented compared to Release: https://gist.github.com/zamazan4ik/a0ebab1b383a9096b616a8a1e450f86d

According to the results, PGO measurably improves the library's performance in many cases.

Further steps

I can suggest the following action points:

• Perform more PGO benchmarks with other datasets (if you are interested enough in it). If it shows improvements - add a note to the documentation (the README file, I guess) about possible improvements in the library's performance with PGO.
• Probably, you can try to get some insights about how the code can be optimized further based on the changes that the compiler performed with PGO. It can be done via analyzing flamegraphs before and after applying PGO to understand the difference or checking some assembly/LLVM IR differences before and after PGO.

I would be happy to answer your questions about PGO.

[udoprog]

Note that if you're benchmarking musli, you should not be using --all-features since that both enables third party serialization frameworks and enables features which disables things not supported across all frameworks. Instead, please use use the default features.

As for PGO, it's unclear to me how to translate benchmarking outcomes into concrete improvements without significant effort. If you do have more concrete guidance on this, for example what the full workflow of implementing an improvement looks like it would be appreciated.

[zamazan4ik]

Instead, please use the default features.

Got it, thanks! Here are the updated results for cargo bench --workspace:

• Release: https://gist.github.com/zamazan4ik/224b8fffe9b695603f40a73c32aeadb4
• PGO optimized compared to Release: https://gist.github.com/zamazan4ik/e98c70d49bc57165a535ed7bdf7ca6d3
• (just for reference) PGO instrumented compared to Release: https://gist.github.com/zamazan4ik/eb4f97b997fd0bf62ffa64f5b39203f8

Results remain the same - from PGO I see measurable improvements in musli's performance.

As for PGO, it's unclear to me how to translate benchmarking outcomes into concrete improvements without significant effort. If you do have more concrete guidance on this, for example what the full workflow of implementing an improvement looks like it would be appreciated.

Fair point. I also don't know an easy way to "extract" knowledge from PGO build directly to the library - digging into the resulting assembly and/or LLVM IR is a time-consuming process.

Instead, I suggest you simply write a note in the README file about PGO's effects on the library performance. Since the README says "excellent performance", I think such a note will be valuable for users.