Skip to content

Latest commit

 

History

History
76 lines (69 loc) · 6.76 KB

build-guide.md

File metadata and controls

76 lines (69 loc) · 6.76 KB
layout title nav_order parent
page
Build Parameters for Velox Backend
4
Getting-Started

Build Parameters

Native build parameters for buildbundle-veloxbe.sh or builddeps-veloxbe.sh

Please set them via --, e.g. --build_type=Release.

Parameters Description Default
build_type Build type for Velox & gluten cpp, CMAKE_BUILD_TYPE. Release
build_tests Build gluten cpp tests. OFF
build_examples Build udf example. OFF
build_benchmarks Build gluten cpp benchmarks. OFF
build_jemalloc Build with jemalloc. ON
build_protobuf Build protobuf lib. ON
enable_qat Enable QAT for shuffle data de/compression. OFF
enable_iaa Enable IAA for shuffle data de/compression. OFF
enable_hbm Enable HBM allocator. OFF
enable_s3 Build with S3 support. OFF
enable_gcs Build with GCs support. OFF
enable_hdfs Build with HDFS support. OFF
enable_abfs Build with ABFS support. OFF
enable_ep_cache Enable caching for external project build (Velox). OFF
enable_vcpkg Enable vcpkg for static build. OFF
run_setup_script Run setup script to install Velox dependencies. ON
velox_repo Specify your own Velox repo to build. ""
velox_branch Specify your own Velox branch to build. ""
velox_home Specify your own Velox source path to build. ""
build_velox_tests Build Velox tests. OFF
build_velox_benchmarks Build Velox benchmarks (velox_tests and connectors will be disabled if ON) OFF
compile_arrow_java Compile arrow java for gluten build to use to fix invalid pointer issues. OFF

Velox build parameters for build_velox.sh

Please set them via --, e.g., --velox_home=/YOUR/PATH.

Parameters Description Default
velox_home Specify Velox source path to build. GLUTEN_SRC/ep/build-velox/build/velox_ep
build_type Velox build type, i.e., CMAKE_BUILD_TYPE. Release
enable_s3 Build Velox with S3 support. OFF
enable_gcs Build Velox with GCS support. OFF
enable_hdfs Build Velox with HDFS support. OFF
enable_abfs Build Velox with ABFS support. OFF
run_setup_script Run setup script to install Velox dependencies before build. ON
enable_ep_cache Enable and reuse cache of Velox build. OFF
build_test_utils Build Velox with cmake arg -DVELOX_BUILD_TEST_UTILS=ON if ON. OFF
build_tests Build Velox test. OFF
build_benchmarks Build Velox benchmarks. OFF
compile_arrow_java Build arrow java for gluten build to use to fix invalid pointer issues. OFF

Maven build parameters

The below parameters can be set via -P for mvn.

Parameters Description Default state
backends-velox Build Gluten Velox backend. disabled
backends-clickhouse Build Gluten ClickHouse backend. disabled
rss Build Gluten with Remote Shuffle Service, only applicable for Velox backend. disabled
delta Build Gluten with Delta Lake support. disabled
iceberg Build Gluten with Iceberg support. disabled
spark-3.2 Build Gluten for Spark 3.2. enabled
spark-3.3 Build Gluten for Spark 3.3. disabled
spark-3.4 Build Gluten for Spark 3.4. disabled

Gluten Jar for Deployment

The gluten jar built out is under GLUTEN_SRC/package/target/. It's name pattern is gluten-<backend_type>-bundle-spark<spark.bundle.version>_<scala.binary.version>-<os.detected.release>_<os.detected.release.version>-<project.version>.jar.

Spark Version spark.bundle.version scala.binary.version
3.2.2 3.2 2.12
3.3.1 3.3 2.12
3.4.2 3.4 2.12