diff --git a/build-scripts/README.md b/build-scripts/README.md index 46a281a2d2cd8..b249938e4f3e9 100644 --- a/build-scripts/README.md +++ b/build-scripts/README.md @@ -1,277 +1,44 @@ -# How to Build `wazuh-indexer` DEB and RPM Packages +# Wazuh Indexer packages generation guide -> [!CAUTION] -> -> Be aware that there might be some problems while following the steps in this guide due to outdated information. -> This document is pending a review. Let us know if you find any issues. +This guide includes instructions to generate distribution packages locally using Docker. -The packages' generation process consists on 2 steps: -[build.yml](../.github/workflows/build.yml) -- **Build**: compiles the Java application and bundles it into a package. -- **Assembly**: uses the package from the previous step and inflates it with plugins and - configuration files, ready for production deployment. +Wazuh Indexer supports any of these combinations: -We usually generate the packages using GitHub Actions, however, the process is designed to -be independent enough for maximum portability. GitHub Actions provides infrastructure, while -the building process is self-contained in the application code. +- distributions: `['tar', 'deb', 'rpm']` +- architectures: `['x64', 'arm64']` -This guide includes instructions to generate packages locally, using Act or Docker. +Windows is currently not supported. -The names of the packages are managed by the `baptizer.sh` script. - -## Local environment - -### Build - -1. Build a wazuh-indexer minimal package - ```bash - bash build-scripts/build.sh -a [ARCHITECTURE] -d [PACKAGE_TYPE] -n $(bash build-scripts/baptizer.sh -a [ARCHITECTURE] -d [PACKAGE_TYPE] -m) - ``` - > A basic package including only the Wazuh Indexer engine without extra plugin is generated first using the `build.sh` script. - > Take a look at the `build.yml` workflow file for an example of usage. -2. Build the wazuh-indexer plugins. (Repeat for each plugin `command-manager`, `setup`, and `reporting`) - ```bash - cd path/to/wazuh/indexer/plugin/ - ./gradlew build -Dversion=[VERSION] -Drevision=[REVISION] - ``` - > Follow the [DEVELOPER_GUIDE.md](https://github.com/wazuh/wazuh-indexer-plugins/blob/master/DEVELOPER_GUIDE.md) instructions to build the plugins. The build scripts expect the plugins in the Maven local repository or under the `artifacts/plugins` folder. -3. Publish the plugins to the local Maven repository. (Repeat for each plugin `command-manager`, `setup`, and `reporting`) - ```bash - ./gradlew publishToMavenLocal - ``` - > Alternatively, copy the generated zip files to the `artifacts/plugins` folder. - > ```bash - > cp build/distributions/wazuh-indexer-[PLUGIN_NAME]-[VERSION]-[REVISION].zip /wazuh-indexer/artifacts/plugins/ - > ``` - -### Assemble - -> **Note:** set the environment variable `TEST=true` to assemble a package with the required plugins only, -speeding up the assembly process. - -1. Assemble the `wazuh-indexer` complete package - ```bash - bash build-scripts/assemble.sh -a [ARCHITECTURE] -d [PACKAGE_TYPE] - ``` - -## Docker environment +The process to build packages requires Docker and Docker Compose. - [Install Docker](https://docs.docker.com/engine/install/) +- [Install Docker Compose](https://docs.docker.com/compose/install/linux/) -The `builder` image automates the build and assemble process for the Wazuh Indexer and its plugins, making it easy to create packages on any system. - -1. Build the image: - ```bash - cd docker/builder && docker build -t wazuh-indexer-builder . && cd ../.. - ``` -2. Execute the package build process - ```bash - docker run --rm -v ./artifacts/dist:/home/indexer/artifacts wazuh-indexer-builder - ``` -The resulting package will be stored at `./artifacts/dist`. - -#### Environment Variables -You can customize the build process by setting the following environment variables: - -- `INDEXER_BRANCH`: The branch to use for the Wazuh Indexer (default: `master`). -- `INDEXER_PLUGINS_BRANCH`: The branch to use for the Wazuh Indexer plugins (default: `master`). -- `INDEXER_REPORTING_BRANCH`: The branch to use for the Wazuh Indexer reporting (default: `master`). -- `REVISION`: The revision number for the build (default: `0`). -- `IS_STAGE`: Whether the build is a staging build (default: `false`). -- `DISTRIBUTION`: The distribution format for the package (default: `tar`). -- `ARCHITECTURE`: The architecture for the package (default: `x64`). - -Example usage with custom environment variables: -```bash -docker run --rm -e INDEXER_BRANCH="5.0.0" -e INDEXER_PLUGINS_BRANCH="5.0.0" -e INDEXER_REPORTING_BRANCH="5.0.0" -v ./artifacts/dist:/home/indexer/artifacts wazuh-indexer-builder -``` - -## Build and Assemble in Act - -- [Install Act](https://github.com/nektos/act) - -Use Act to run the `build.yml` workflow locally. The `act.input.env` file contains the inputs -for the workflow. As the workflow clones the `wazuh-indexer-plugins` repository, the `GITHUB_TOKEN` -is required. You can use the `gh` CLI to authenticate, as seen in the example below. - -```bash -act -j build -W .github/workflows/build.yml --artifact-server-path ./artifacts --input-file build-scripts/act.input.env -s GITHUB_TOKEN="$(gh auth token)" -``` - -## Bash scripts reference - -The packages' generation process is guided through bash scripts. This section list and describes -them, as well as their inputs and outputs. - -```yml -scripts: - - file: build.sh - description: | - generates a distribution package by running the appropiate Gradle task - depending on the parameters. - inputs: - architecture: [x64, arm64] # Note: we only build x86_64 packages - distribution: [tar, deb, rpm] - name: the name of the package to be generated. - outputs: - package: minimal wazuh-indexer package for the required distribution. - - - file: assemble.sh - description: | - bundles the wazuh-indexer package generated in by build.sh with plugins, - configuration files and demo certificates (certificates yet to come). - inputs: - architecture: [x64, arm64] # Note: we only build x86_64 packages - distribution: [tar, deb, rpm] - revision: revision number. 0 by default. - outputs: - package: wazuh-indexer package. - - - file: provision.sh - description: Provision script for the assembly of DEB packages. - - - file: baptizer.sh - description: generate the wazuh-indexer package name depending on the parameters. - inputs: - architecture: [x64, arm64] # Note: we only build x86_64 packages - distribution: [tar, deb, rpm] - revision: revision number. 0 by default. - plugins_hash: Commit hash of the `wazuh-indexer-plugins` repository. - reporting_hash: Commit hash of the `wazuh-indexer-reporting` repository. - is_release: if set, uses release naming convention. - is_min: if set, the package name will start by `wazuh-indexer-min`. Used on the build stage. - outputs: - package: the name of the wazuh-indexer package. -``` - -### Assemble flow reference - -#### TAR +Before you get started, make sure to clean your environment by running `./gradlew clean`. -The assembly process for tarballs consists on: +## Building wazuh-indexer packages -1. Extract. -2. Install plugins. -3. Add Wazuh's configuration files and tools. -4. Compress. +Use the script under `wazuh-indexer/docker/builder/builder.sh` to build a package. ```bash -bash build-scripts/assemble.sh -a x64 -d tar -r 1 +./builder.sh -h +Usage: ./builder.sh [args] + +Arguments: +-p INDEXER_PLUGINS_BRANCH [Optional] wazuh-indexer-plugins repo branch, default is 'master'. +-r INDEXER_REPORTING_BRANCH [Optional] wazuh-indexer-reporting repo branch, default is 'master'. +-R REVISION [Optional] Package revision, default is '0'. +-s STAGE [Optional] Staging build, default is 'false'. +-d DISTRIBUTION [Optional] Distribution, default is 'rpm'. +-a ARCHITECTURE [Optional] Architecture, default is 'x64'. +-D Destroy the docker environment +-h Print help ``` -#### DEB - -For DEB packages, the `assemble.sh` script will perform the following operations: - -1. Extract the deb package using `ar` and `tar` tools. - - > By default, `ar` and `tar` tools expect the package to be in `wazuh-indexer/artifacts/tmp/deb`. - > The script takes care of creating the required folder structure, copying also the min package - > and the Makefile. - - Current folder loadout at this stage: - - ``` - artifacts/ - |-- dist - | |-- wazuh-indexer-min_5.0.0_amd64.deb - `-- tmp - `-- deb - |-- Makefile - |-- data.tar.gz - |-- debmake_install.sh - |-- etc - |-- usr - |-- var - `-- wazuh-indexer-min_5.0.0_amd64.deb - ``` - - `usr`, `etc` and `var` folders contain `wazuh-indexer` files, extracted from `wazuh-indexer-min-*.deb`. - `Makefile` and the `debmake_install` are copied over from `wazuh-indexer/distribution/packages/src/deb`. - The `wazuh-indexer-performance-analyzer.service` file is also copied from the same folder. It is a dependency of the SPEC file. - -2. Install the plugins using the `opensearch-plugin` CLI tool. -3. Set up configuration files. - - > Included in `min-package`. Default files are overwritten. - -4. Bundle a DEB file with `debmake` and the `Makefile`. - - > `debmake` and other dependencies can be installed using the `provision.sh` script. - > The script is invoked by the GitHub Workflow. - - Current folder loadout at this stage: - - ``` - artifacts/ - |-- artifact_name.txt - |-- dist - | |-- wazuh-indexer-min_5.0.0_amd64.deb - | `-- wazuh-indexer_5.0.0_amd64.deb - `-- tmp - `-- deb - |-- Makefile - |-- data.tar.gz - |-- debmake_install.sh - |-- etc - |-- usr - |-- var - |-- wazuh-indexer-min_5.0.0_amd64.deb - `-- debian/ - | -- control - | -- copyright - | -- rules - | -- preinst - | -- prerm - | -- postinst - ``` - -### RPM - -The `assemble.sh` script will use the output from the `build.sh` script and use it as a -base to bundle together a final package containing the plugins, the production configuration -and the service files. - -The script will: - -1. Extract the RPM package using `rpm2cpio` and `cpio` tools. - - > By default, `rpm2cpio` and `cpio` tools expect the package to be in `wazuh-indexer/artifacts/tmp/rpm`. The script takes care of creating the required folder structure, copying also the min package and the SPEC file. - - Current folder loadout at this stage: - - ``` - /rpm/$ARCH - /etc - /usr - /var - wazuh-indexer-min-*.rpm - wazuh-indexer.rpm.spec - ``` - - `usr`, `etc` and `var` folders contain `wazuh-indexer` files, extracted from `wazuh-indexer-min-*.rpm`. - `wazuh-indexer.rpm.spec` is copied over from `wazuh-indexer/distribution/packages/src/rpm/wazuh-indexer.rpm.spec`. - The `wazuh-indexer-performance-analyzer.service` file is also copied from the same folder. It is a dependency of the SPEC file. - -2. Install the plugins using the `opensearch-plugin` CLI tool. -3. Set up configuration files. - - > Included in `min-package`. Default files are overwritten. - -4. Bundle an RPM file with `rpmbuild` and the SPEC file `wazuh-indexer.rpm.spec`. - - > `rpmbuild` is part of the `rpm` OS package. +The resulting package will be stored at `wazuh-indexer/artifacts/dist`. - > `rpmbuild` is invoked from `wazuh-indexer/artifacts/tmp/rpm`. It creates the {BUILD,RPMS,SOURCES,SRPMS,SPECS,TMP} folders and applies the rules in the SPEC file. If successful, `rpmbuild` will generate the package in the `RPMS/` folder. The script will copy it to `wazuh-indexer/artifacts/dist` and clean: remove the `tmp\` folder and its contents. +> The `STAGE` option defines the naming of the package. When set to `false`, the package will be unequivocally named with the commits' SHA of the `wazuh-indexer`, `wazuh-indexer-plugins` and `wazuh-indexer-reporting` repositories, in that order. For example: `wazuh-indexer_5.0.0-0_x86_64_aff30960363-846f143-494d125.rpm`. - Current folder loadout at this stage: +## Building wazuh-indexer docker images - ``` - /rpm/$ARCH - /{BUILD,RPMS,SOURCES,SRPMS,SPECS,TMP} - /etc - /usr - /var - wazuh-indexer-min-*.rpm - wazuh-indexer.rpm.spec - ``` +> TBD. For manual building refer to [our Docker containers guide](../docker/README.md). diff --git a/build-scripts/REFERENCE.md b/build-scripts/REFERENCE.md new file mode 100644 index 0000000000000..62cb4c41cffa7 --- /dev/null +++ b/build-scripts/REFERENCE.md @@ -0,0 +1,224 @@ +# Wazuh Indexer build scripts and flow reference + +## Introduction + +The packages' generation process consists on 2 steps: + +- **Build**: compiles the Java application and bundles it into a package. +- **Assembly**: uses the package from the previous step and inflates it with plugins and configuration files, ready for production deployment. + +We usually generate the packages using GitHub Actions, however, the process is designed to be independent enough for maximum portability. GitHub Actions provides infrastructure, while the building process is self-contained in the application code. + +This guide includes development worthy information about the process to generate packages for Wazuh Indexer. For instructions about how to build Wazuh Indexer packages, refer to the instructions in the [README.md](./README.md). + +## Scripts reference + +There is a script for each of the steps, `build.sh` and `assemble.sh`. Along them, there are some utility scripts, such as `provision.sh` and `baptizer.sh`. This one takes generates the names for the packages depending of the combination of parameters given. Refer to the definition below for more details about these scripts. + +```yml +scripts: + - file: build.sh + description: | + Generates a distribution package by running the appropiate Gradle task + depending on the parameters. + inputs: + architecture: [x64, arm64] + distribution: [tar, deb, rpm] + name: the name of the package to be generated. + outputs: + package: minimal wazuh-indexer package for the given distribution and architecture. + + - file: assemble.sh + description: | + Bundles the wazuh-indexer package generated in by build.sh with plugins, + configuration files and demo certificates (certificates yet to come). + inputs: + architecture: [x64, arm64] + distribution: [tar, deb, rpm] + revision: revision number. 0 by default. + outputs: + package: final wazuh-indexer package. + + - file: provision.sh + description: Provision script for the assembly of Debian packages. + + - file: baptizer.sh + description: Generates the wazuh-indexer package name depending on the parameters. + inputs: + architecture: [x64, arm64] + distribution: [tar, deb, rpm] + revision: revision number. 0 by default. + plugins_hash: Commit hash of the `wazuh-indexer-plugins` repository. + reporting_hash: Commit hash of the `wazuh-indexer-reporting` repository. + is_release: if set, uses release naming convention. + is_min: if set, the package name will start by `wazuh-indexer-min`. Used on the build stage. + outputs: + package: the name of the wazuh-indexer package. +``` + +## Build and Assemble + +### Build + +The build process is identical for every distribution and architecture. Although the process is driven by the `build.sh` script, the compilation and bundling is perfomed by Gradle tasks. + +```bash +./build.sh -h +Usage: ./build.sh [args] + +Arguments: +-q QUALIFIER [Optional] Version qualifier. +-s SNAPSHOT [Optional] Build a snapshot, default is 'false'. +-p PLATFORM [Optional] Platform, default is 'uname -s'. +-a ARCHITECTURE [Optional] Build architecture, default is 'uname -m'. +-d DISTRIBUTION [Optional] Distribution, default is 'tar'. +-b BRANCH [Optional] Branch from wazuh/wazuh to download the index template from, default is ' By default, `ar` and `tar` tools expect the package to be in `wazuh-indexer/artifacts/tmp/deb`. + > The script takes care of creating the required folder structure, copying also the min package and the Makefile. + + Current folder loadout at this stage: + + ``` + artifacts/ + |-- dist + | |-- wazuh-indexer-min_5.0.0_amd64.deb + `-- tmp + `-- deb + |-- Makefile + |-- data.tar.gz + |-- debmake_install.sh + |-- etc + |-- usr + |-- var + `-- wazuh-indexer-min_5.0.0_amd64.deb + ``` + + Notes: + + - `usr`, `etc` and `var` folders contain `wazuh-indexer` files, extracted from `wazuh-indexer-min-*.deb`. + - `Makefile` and the `debmake_install` are copied over from `wazuh-indexer/distribution/packages/src/deb`. + - The `wazuh-indexer-performance-analyzer.service` file is also copied from the same folder. It is a dependency of the SPEC file. + +2. Install the plugins using the `opensearch-plugin` CLI tool. +3. Set up configuration files. + + > Included in `min-package`. Default files are overwritten. + +4. Bundle a DEB file with `debmake` and the `Makefile`. + + > `debmake` and other dependencies can be installed using the `provision.sh` script. + > The script is invoked by the GitHub Workflow. + + Current folder loadout at this stage: + + ``` + artifacts/ + |-- artifact_name.txt + |-- dist + | |-- wazuh-indexer-min_5.0.0_amd64.deb + | `-- wazuh-indexer_5.0.0_amd64.deb + `-- tmp + `-- deb + |-- Makefile + |-- data.tar.gz + |-- debmake_install.sh + |-- etc + |-- usr + |-- var + |-- wazuh-indexer-min_5.0.0_amd64.deb + `-- debian/ + | -- control + | -- copyright + | -- rules + | -- preinst + | -- prerm + | -- postinst + ``` + +#### RPM packages + +For RPM packages, the `assemble.sh` script will: + +1. Extract the RPM package using `rpm2cpio` and `cpio` tools. + + > By default, `rpm2cpio` and `cpio` tools expect the package to be in `wazuh-indexer/artifacts/tmp/rpm`. The script takes care of creating the required folder structure, copying also the min package and the SPEC file. + + Current folder loadout at this stage: + + ``` + /rpm/$ARCH + /etc + /usr + /var + wazuh-indexer-min-*.rpm + wazuh-indexer.rpm.spec + ``` + + Notes: + + - `usr`, `etc` and `var` folders contain `wazuh-indexer` files, extracted from `wazuh-indexer-min-*.rpm`. + - `wazuh-indexer.rpm.spec` is copied over from `wazuh-indexer/distribution/packages/src/rpm/wazuh-indexer.rpm.spec`. + - The `wazuh-indexer-performance-analyzer.service` file is also copied from the same folder. It is a dependency of the SPEC file. + +2. Install the plugins using the `opensearch-plugin` CLI tool. +3. Set up configuration files. + + > Included in `min-package`. Default files are overwritten. + +4. Bundle an RPM file with `rpmbuild` and the SPEC file `wazuh-indexer.rpm.spec`. + + > `rpmbuild` is part of the `rpm` OS package. + + > `rpmbuild` is invoked from `wazuh-indexer/artifacts/tmp/rpm`. It creates the {BUILD,RPMS,SOURCES,SRPMS,SPECS,TMP} folders and applies the rules in the SPEC file. If successful, `rpmbuild` will generate the package in the `RPMS/` folder. The script will copy it to `wazuh-indexer/artifacts/dist` and clean: remove the `tmp\` folder and its contents. + + Current folder loadout at this stage: + + ``` + /rpm/$ARCH + /{BUILD,RPMS,SOURCES,SRPMS,SPECS,TMP} + /etc + /usr + /var + wazuh-indexer-min-*.rpm + wazuh-indexer.rpm.spec + ``` + +### Epilogue: Act + +- [Install Act](https://github.com/nektos/act) + +Use Act to run the `build.yml` workflow locally. The `act.input.env` file contains the inputs for the workflow. As the workflow clones the `wazuh-indexer-plugins` repository, the `GITHUB_TOKEN` is required. You can use the `gh` CLI to authenticate, as seen in the example below. + +```bash +act -j build -W .github/workflows/build.yml --artifact-server-path ./artifacts --input-file build-scripts/act.input.env -s GITHUB_TOKEN="$(gh auth token)" +``` diff --git a/build-scripts/build.sh b/build-scripts/build.sh index 559ee09f072b4..6418dfeb8e107 100644 --- a/build-scripts/build.sh +++ b/build-scripts/build.sh @@ -7,7 +7,7 @@ # this file be licensed under the Apache-2.0 license or a # compatible open source license. -set -ex +# set -ex # ==== # Usage diff --git a/docker/README.md b/docker/README.md index a6f35832f325d..7e5d72140bb96 100644 --- a/docker/README.md +++ b/docker/README.md @@ -31,48 +31,30 @@ Once the `wi-dev:x.y.z` container is up, attach a shell to it and run `./gradlew The `builder` image automates the build and assemble process for the Wazuh Indexer and its plugins, making it easy to create packages on any system. -### Usage -1. Build the image: - ```bash - cd docker/builder && docker build -t wazuh-indexer-builder . - ``` -2. Execute the package build process - ```bash - docker run --rm -v /path/to/local/artifacts:/home/indexer/artifacts wazuh-indexer-builder - ``` - > Replace `/path/to/local/artifacts` with the actual path on your host system where you want to store the resulting package. - -#### Environment Variables -You can customize the build process by setting the following environment variables: - -- `INDEXER_BRANCH`: The branch to use for the Wazuh Indexer (default: `master`). -- `INDEXER_PLUGINS_BRANCH`: The branch to use for the Wazuh Indexer plugins (default: `master`). -- `INDEXER_REPORTING_BRANCH`: The branch to use for the Wazuh Indexer reporting (default: `master`). -- `REVISION`: The revision number for the build (default: `0`). -- `IS_STAGE`: Whether the build is a staging build (default: `false`). -- `DISTRIBUTION`: The distribution format for the package (default: `tar`). -- `ARCHITECTURE`: The architecture for the package (default: `x64`). - -Example usage with custom environment variables: +In the example below, it will generate a wazuh-indexer package for Debian based systems, for the x64 architecture, using 1 as revision number and using the production naming convention. + ```bash -docker run --rm -e INDEXER_BRANCH="5.0.0" -e INDEXER_PLUGINS_BRANCH="5.0.0" -e INDEXER_REPORTING_BRANCH="5.0.0" -v ./artifacts/dist:/home/indexer/artifacts wazuh-indexer-builder +# Wihtin wazu-indexer/docker/builder +bash builder.sh -d deb -a x64 -R 1 -s true ``` Refer to [build-scripts/README.md](../build-scripts/README.md) for details about how to build packages. -[docker]: https://docs.docker.com/engine/install -[wi-repo]: https://github.com/wazuh/wazuh-indexer - ## Building Docker images The [prod](./prod) folder contains the code to build Docker images. A tarball of `wazuh-indexer` needs to be located at the same level that the Dockerfile. Below there is an example of the command needed to build the image. Set the build arguments and the image tag accordingly. -```console +```bash docker build --build-arg="VERSION=5.0.0" --build-arg="INDEXER_TAR_NAME=wazuh-indexer-5.0.0-1_linux-x64_cfca84f.tar.gz" --tag=wazuh-indexer:5.0.0 --progress=plain --no-cache . ``` Then, start a container with: -```console +```bash docker run -it --rm wazuh-indexer:5.0.0 ``` + + + +[docker]: https://docs.docker.com/engine/install +[wi-repo]: https://github.com/wazuh/wazuh-indexer