Benchmarking Set for HEIR Optimizations and Hardware Accelerators

[code-perspective]

Let's start building a benchmarking set to understand how well HEIR optimizations and hardware speedups work across different levels in the FHE stack.

Application Level

• Focus: End-to-end FHE applications
• Metrics: Runtime, memory usage, hardware specification, precision-loss(where applicable) at a specific security level
• Goals: Evaluate impact of optimizations, end-to-end hardware scheduling, precision vs performance tradeoff, etc across schemes, etc.
• Examples: Encrypted machine learning (training/inference), SQL like query processing in DB, any other application you can think of.

Scheme Level

• Focus: Cryptographic primitives
• Metrics: Runtime of single op, amortized runtime, memory usage for a specific set of parameters
• Goals: Identify performance bottlenecks within schemes and how math kernels translate to scheme level op speed up.
• Examples: Bootstrapping, Homomorphic operations (addition, multiplication, etc.), Relinearizations, KeySwitch, etc

Math Level

• Focus: Polynomial-level operations, hardware kernels for FHE
• Metrics: Runtime, throughput, resource utilization (memory, compute) for a specific set of parameters
• Goals: Assess the efficiency of low-level mathematical operations and evaluate the performance of hardware-specific kernels
• Examples: Number Theoretic Transform (NTT), inverse NTT, Toeplitz matrix multiplication, RNS multiplication

Let’s start collecting some of the benchmarks used in papers and keep updating this thread. The benchmarks can be at various levels. Please add your comments on what kind of benchmarks you think would be useful to have. Please also add benchmarks that will showcase the bounds of an optimization, hardware, etc.

Once we have enough, we can start designing a HEIR benchmarking set and put together some starter input programs or MLIR-files to run through passes and pipelines.