-
Notifications
You must be signed in to change notification settings - Fork 116
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add sycl-bench benchmarks #2047
Closed
Closed
Changes from 6 commits
Commits
Show all changes
28 commits
Select commit
Hold shift + click to select a range
939b4d0
add syclbench.py
mateuszpn 335faaa
unisa-hpc/sycl-bench benchmarks added
mateuszpn 56cf9cf
Merge branch 'main' into add-sycl-bench
mateuszpn d5b553a
Merge branch 'oneapi-src:main' into add-sycl-bench
mateuszpn 5f374f6
Benchamrk names simplified
mateuszpn c4854bf
Merge branch 'add-sycl-bench' of github.com:mateuszpn/unified-runtime…
mateuszpn 0ab5039
Merge branch 'oneapi-src:main' into add-sycl-bench
mateuszpn 535c723
update
mateuszpn 7d751ba
update
mateuszpn 6895f21
update
mateuszpn ff3a0f1
WIP
mateuszpn 20d6859
Performance chart and geometrical mean
mateuszpn 162681b
Merge branch 'oneapi-src:main' into add-sycl-bench
mateuszpn 2af01f7
push benchmarks results
mateuszpn 98f338b
Merge branch 'add-sycl-bench' of github.com:mateuszpn/unified-runtime…
mateuszpn b1acce4
WIP [skip ci]
mateuszpn 1499fdd
WIP [skip ci]
mateuszpn 8479e59
Merge branch 'oneapi-src:main' into add-sycl-bench
mateuszpn 3b2f85e
png added
mateuszpn 8df45d8
Merge branch 'add-sycl-bench' of github.com:mateuszpn/unified-runtime…
mateuszpn 716f103
action added [skip ci]
mateuszpn 50309a8
ascii bar chart added to benchmarks results [skip ci]
mateuszpn a2affaa
ascii bar fix [skip ci]
mateuszpn 0358db2
bar chart update [skip ci]
mateuszpn 90608c8
update [skip ci]
mateuszpn ebe4160
update [skip ci]
mateuszpn 379e1bd
update [skip ci]
mateuszpn 7af46ef
Merge branch 'oneapi-src:main' into add-sycl-bench
mateuszpn File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,369 @@ | ||
# Copyright (C) 2024 Intel Corporation | ||
# Part of the Unified-Runtime Project, under the Apache License v2.0 with LLVM Exceptions. | ||
# See LICENSE.TXT | ||
# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception | ||
|
||
import os | ||
import csv | ||
import io | ||
from utils.utils import run, git_clone, create_build_path | ||
from .base import Benchmark | ||
from .result import Result | ||
from .options import options | ||
|
||
class SyclBench: | ||
def __init__(self, directory): | ||
self.directory = directory | ||
self.built = False | ||
self.setup() | ||
return | ||
|
||
def setup(self): | ||
if self.built: | ||
return | ||
|
||
repo_path = git_clone(self.directory, "sycl-bench-repo", "https://github.com/mateuszpn/sycl-bench.git", "1e6ab2cfd004a72c5336c26945965017e06eab71") | ||
build_path = create_build_path(self.directory, 'sycl-bench-build') | ||
|
||
configure_command = [ | ||
"cmake", | ||
f"-B {build_path}", | ||
f"-S {repo_path}", | ||
f"-DCMAKE_BUILD_TYPE=Release", | ||
f"-DCMAKE_CXX_COMPILER={options.sycl}/bin/clang++", | ||
f"-DCMAKE_C_COMPILER={options.sycl}/bin/clang", | ||
f"-DSYCL_IMPL=dpcpp" | ||
] | ||
run(configure_command, add_sycl=True) | ||
|
||
run(f"cmake --build {build_path} -j", add_sycl=True) | ||
|
||
self.built = True | ||
self.bins = build_path | ||
return | ||
|
||
class SyclBenchmark(Benchmark): | ||
def __init__(self, bench, name, test): | ||
self.bench = bench | ||
self.bench_name = name | ||
self.test = test | ||
super().__init__(bench.directory) | ||
|
||
def bin_args(self) -> list[str]: | ||
return [] | ||
|
||
def extra_env_vars(self) -> dict: | ||
return {} | ||
|
||
def unit(self): | ||
return "ms" | ||
|
||
def setup(self): | ||
self.bench.setup() | ||
self.benchmark_bin = os.path.join(self.bench.bins, self.bench_name) | ||
|
||
def run(self, env_vars) -> Result: | ||
outputfile = f"{self.bench.directory}/{self.test}.csv" | ||
command = [ | ||
f"{self.benchmark_bin}", | ||
f"--warmup-run", | ||
f"--num-runs=3", | ||
f"--output={outputfile}" | ||
] | ||
|
||
command += self.bin_args() | ||
env_vars.update(self.extra_env_vars()) | ||
|
||
result = self.run_bench(command, env_vars) | ||
|
||
with open(outputfile, 'r') as f: | ||
reader = csv.reader(f) | ||
res_list = [] | ||
for row in reader: | ||
if not row[0].startswith('#'): | ||
res_list.append( | ||
Result(label=row[0], | ||
value=float(row[12]) * 1000, # convert to ms | ||
command=command, | ||
env=env_vars, | ||
stdout=result)) | ||
|
||
median_list = [] | ||
for label in set(result.label for result in res_list): | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. this is already done for all the benchmarks in main.py. why repeat it? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Removed |
||
values = [result.value for result in res_list if result.label == label] | ||
median_value = sorted(values)[len(values) // 2] | ||
median_list.append(Result(label=label, value=median_value, command=command, env=env_vars, stdout=result)) | ||
|
||
return median_list | ||
|
||
def teardown(self): | ||
return | ||
|
||
def name(self): | ||
return self.test | ||
|
||
class Arith(SyclBenchmark): | ||
def __init__(self, bench): | ||
super().__init__(bench, "arith", "Arith_int32_512") | ||
|
||
def bin_args(self) -> list[str]: | ||
return [ | ||
f"--size=16384", | ||
] | ||
|
||
class TwoDConvolution(SyclBenchmark): | ||
def __init__(self, bench): | ||
super().__init__(bench, "2DConvolution", "2DConvolution") | ||
|
||
class Two_mm(SyclBenchmark): | ||
def __init__(self, bench): | ||
super().__init__(bench, "2mm", "2mm") | ||
|
||
def bin_args(self) -> list[str]: | ||
return [ | ||
f"--size=512", | ||
] | ||
|
||
class Three_mm(SyclBenchmark): | ||
def __init__(self, bench): | ||
super().__init__(bench, "3mm", "3mm") | ||
|
||
def bin_args(self) -> list[str]: | ||
return [ | ||
f"--size=512", | ||
] | ||
|
||
class Atax(SyclBenchmark): | ||
def __init__(self, bench): | ||
super().__init__(bench, "atax", "Atax") | ||
|
||
def bin_args(self) -> list[str]: | ||
return [ | ||
f"--size=8192", | ||
] | ||
|
||
class Atomic_reduction(SyclBenchmark): | ||
def __init__(self, bench): | ||
super().__init__(bench, "atomic_reduction", "ReductionAtomic_fp64") | ||
|
||
class Bicg(SyclBenchmark): | ||
def __init__(self, bench): | ||
super().__init__(bench, "bicg", "Bicg") | ||
|
||
def bin_args(self) -> list[str]: | ||
return [ | ||
f"--size=8192", | ||
] | ||
|
||
class Correlation(SyclBenchmark): | ||
def __init__(self, bench): | ||
super().__init__(bench, "correlation", "Correlation") | ||
|
||
def bin_args(self) -> list[str]: | ||
return [ | ||
f"--size=512", | ||
] | ||
|
||
class Covariance(SyclBenchmark): | ||
def __init__(self, bench): | ||
super().__init__(bench, "covariance", "Covariance") | ||
|
||
def bin_args(self) -> list[str]: | ||
return [ | ||
f"--size=512", | ||
] | ||
|
||
class Gemm(SyclBenchmark): | ||
def __init__(self, bench): | ||
super().__init__(bench, "gemm", "Gemm") | ||
|
||
def bin_args(self) -> list[str]: | ||
return [ | ||
f"--size=1024", | ||
] | ||
|
||
class Gesumv(SyclBenchmark): | ||
def __init__(self, bench): | ||
super().__init__(bench, "gesummv", "Gesummv") | ||
|
||
def bin_args(self) -> list[str]: | ||
return [ | ||
f"--size=8192", | ||
] | ||
|
||
class Gramschmidt(SyclBenchmark): | ||
def __init__(self, bench): | ||
super().__init__(bench, "gramschmidt", "Gramschmidt") | ||
|
||
def bin_args(self) -> list[str]: | ||
return [ | ||
f"--size=512", | ||
] | ||
|
||
class KMeans(SyclBenchmark): | ||
def __init__(self, bench): | ||
super().__init__(bench, "kmeans", "Kmeans") | ||
|
||
def bin_args(self) -> list[str]: | ||
return [ | ||
f"--size=67108864", | ||
] | ||
|
||
class LinRegCoeff(SyclBenchmark): | ||
def __init__(self, bench): | ||
super().__init__(bench, "lin_reg_coeff", "LinearRegressionCoeff") | ||
|
||
class LinRegError(SyclBenchmark): | ||
def __init__(self, bench): | ||
super().__init__(bench, "lin_reg_error", "LinearRegression") | ||
|
||
class MatmulChain(SyclBenchmark): | ||
def __init__(self, bench): | ||
super().__init__(bench, "matmulchain", "MatmulChain") | ||
|
||
def bin_args(self) -> list[str]: | ||
return [ | ||
f"--size=1024", | ||
] | ||
|
||
# ** bad input file path ** | ||
# | ||
# class Median(SyclBenchmark): | ||
# def __init__(self, bench): | ||
# super().__init__(bench, "median", "MedianFilter") | ||
# | ||
# def bin_args(self) -> list[str]: | ||
# return [ | ||
# f"--size=512", | ||
# ] | ||
|
||
class MolDyn(SyclBenchmark): | ||
def __init__(self, bench): | ||
super().__init__(bench, "mol_dyn", "MolecularDynamics") | ||
|
||
|
||
class Mvt(SyclBenchmark): | ||
def __init__(self, bench): | ||
super().__init__(bench, "mvt", "Mvt") | ||
|
||
def bin_args(self) -> list[str]: | ||
return [ | ||
f"--size=16384", | ||
] | ||
|
||
# ** verification fail ** | ||
# | ||
# class NBody(SyclBenchmark): | ||
# def __init__(self, bench): | ||
# super().__init__(bench, "nbody", "NBody_") | ||
|
||
class Sf(SyclBenchmark): | ||
def __init__(self, bench): | ||
super().__init__(bench, "sf", "sf_16") | ||
|
||
def bin_args(self) -> list[str]: | ||
return [ | ||
f"--size=--size=100000000", | ||
] | ||
|
||
# bad input file path | ||
# | ||
# class SobelX(SyclBenchmark): | ||
# def __init__(self, bench): | ||
# super().__init__(bench, "sobel", "SobelFilter") | ||
|
||
class Syr2k(SyclBenchmark): | ||
def __init__(self, bench): | ||
super().__init__(bench, "syr2k", "Syr2k") | ||
|
||
def bin_args(self) -> list[str]: | ||
return [ | ||
f"--size=1024", | ||
] | ||
|
||
class Syrk(SyclBenchmark): | ||
def __init__(self, bench): | ||
super().__init__(bench, "syrk", "Syrk") | ||
|
||
def bin_args(self) -> list[str]: | ||
return [ | ||
f"--size=1024", | ||
] | ||
|
||
# multi benchmarks | ||
class Blocked_transform(SyclBenchmark): | ||
def __init__(self, bench): | ||
super().__init__(bench, "blocked_transform", "BlockedTransform_multi") | ||
|
||
def bin_args(self) -> list[str]: | ||
return [ | ||
f"--size=512", | ||
] | ||
|
||
class DagTaskI(SyclBenchmark): | ||
def __init__(self, bench): | ||
super().__init__(bench, "dag_task_throughput_independent", "IndependentDAGTaskThroughput_multi") | ||
|
||
def bin_args(self) -> list[str]: | ||
return [ | ||
f"--size=512", | ||
] | ||
|
||
class DagTaskS(SyclBenchmark): | ||
def __init__(self, bench): | ||
super().__init__(bench, "dag_task_throughput_sequential", "DAGTaskThroughput_multi") | ||
|
||
def bin_args(self) -> list[str]: | ||
return [ | ||
f"--size=512", | ||
] | ||
|
||
class HostDevBandwidth(SyclBenchmark): | ||
def __init__(self, bench): | ||
super().__init__(bench, "host_device_bandwidth", "HostDeviceBandwidth_multi") | ||
|
||
class LocalMem(SyclBenchmark): | ||
def __init__(self, bench): | ||
super().__init__(bench, "local_mem", f"LocalMem_multi") | ||
|
||
def bin_args(self) -> list[str]: | ||
return [ | ||
f"--size=512", | ||
] | ||
|
||
class Pattern_L2(SyclBenchmark): | ||
def __init__(self, bench): | ||
super().__init__(bench, "pattern_L2", "L2_multi") | ||
|
||
class Reduction(SyclBenchmark): | ||
def __init__(self, bench): | ||
super().__init__(bench, "reduction", "Pattern_Reduction_multi") | ||
|
||
class ScalarProd(SyclBenchmark): | ||
def __init__(self, bench): | ||
super().__init__(bench, "scalar_prod", "ScalarProduct_multi") | ||
|
||
class SegmentReduction(SyclBenchmark): | ||
def __init__(self, bench): | ||
super().__init__(bench, "segmentedreduction", "Pattern_SegmentedReduction_multi") | ||
|
||
class UsmAccLatency(SyclBenchmark): | ||
def __init__(self, bench): | ||
super().__init__(bench, "usm_accessors_latency", "USM_Latency_multi") | ||
|
||
class UsmAllocLatency(SyclBenchmark): | ||
def __init__(self, bench): | ||
super().__init__(bench, "usm_allocation_latency", "USM_Allocation_latency_multi") | ||
|
||
class UsmInstrMix(SyclBenchmark): | ||
def __init__(self, bench): | ||
super().__init__(bench, "usm_instr_mix", "USM_Instr_Mix_multi") | ||
|
||
class UsmPinnedOverhead(SyclBenchmark): | ||
def __init__(self, bench): | ||
super().__init__(bench, "usm_pinned_overhead", "USM_Pinned_Overhead_multi") | ||
|
||
class VecAdd(SyclBenchmark): | ||
def __init__(self, bench): | ||
super().__init__(bench, "vec_add", "VectorAddition_multi") | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not sure if I follow. We open an "output" file that contains some benchmark definitions? Shouldn't that be input file then?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is an output file containing results - when put in file, they are structured as csv and easy to analyze. I must analyze the output, whether it is in stdout or file, to return results.