Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve wazuh indexer package generation guide #7988

Draft
wants to merge 2 commits into
base: master
Choose a base branch
from
Draft

Improve wazuh indexer package generation guide #7988

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
eb43ffc
Checkout the right tag from github
f-galland Nov 26, 2024
b7ea426
Improve indexer package generation guide
f-galland Nov 26, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
289 changes: 101 additions & 188 deletions source/development/packaging/generate-indexer-package.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,259 +3,172 @@
.. meta::
:description: Wazuh provides an automated way of building packages for the Wazuh components. Learn how to build your own Wazuh indexer package in this section of our documentation.

=============
Wazuh indexer
=============

The packages' generation process is orchestrated by two scripts, which are found within the ``packaging_scripts`` folder of the repository:
Local packages generation
#########################

While official ``Wazuh Indexer`` packages are generated in a GitHub Actions pipeline, packages can also be compiled locally or within a docker container.

The packages' generation process is orchestrated by two scripts, found under the ``build-scripts`` directory of the repository:

- ``build.sh``: compiles the Java application and bundles it into a package.
- ``assemble.sh``: uses the package from the previous step and inflates it with plugins and configuration files, ready for production deployment.

Official packages are built through a GitHub Actions pipeline, however, the process is designed to be independent enough for maximum portability.
.. contents:: Table of contents:
:depth: 2
:local:

The building process is self-contained in the application code.
Docker environment
******************
.. raw:: html

The GitHub Actions pipeline can be tested locally with `Act <https://github.com/nektos/act>`__.
<div class="accordion-section open">

Pre-requisistes:
Pre-requisistes
===============

- Clone the ``wazuh-indexer`` repository and switch to the appropriate branch:
1. Clone the ``Wazuh Indexer`` GitHub repository and switch to the ``v|WAZUH_CURRENT|`` tag:

.. code:: console

# git clone https://github.com/wazuh/wazuh-indexer
# git clone https://github.com/wazuh/wazuh-indexer/
# git checkout v|WAZUH_CURRENT|

Build stage
-----------
2. Bring the docker environment up:

Docker environment
^^^^^^^^^^^^^^^^^^
.. code:: console

Using the provided `Docker environment <https://www.github.com/wazuh/wazuh-indexer/tree/v|WAZUH_CURRENT|/docker>`__:
# cd wazuh-indexer/docker/ci
# bash ci.sh up
# cd ../..

.. tabs::
Build a minimal package
=======================

.. group-tab:: RPM
A basic package including only the ``Wazuh Indexer`` engine without extra plugin is generated first.

.. code-block:: console

# docker exec -it wi-build_|WAZUH_CURRENT| bash packaging_scripts/build.sh -a x64 -d rpm

.. group-tab:: DEB
1. Set the environment variables:

.. code-block:: console

# docker exec -it wi-build_|WAZUH_CURRENT| bash packaging_scripts/build.sh -a x64 -d deb

.. group-tab:: TAR
.. note::

.. code-block:: console

# docker exec -it wi-build_|WAZUH_CURRENT| bash packaging_scripts/build.sh -a x64 -d tar
Replace ``<arch>`` with one of ``x64`` or ``arm64`` and ``<package-type>`` with one of ``rpm``, ``deb`` or ``tar``

Local package generation
^^^^^^^^^^^^^^^^^^^^^^^^
.. code:: console

For local package generation, use the ``build.sh`` script.
# ARCHITECTURE=<arch>
# PACKAGE_TYPE=<package-type>

Take a look at the ``build.yml`` workflow file for an example of usage.
2. Run the build script:

.. code:: console

# bash packaging_scripts/build.sh -a x64 -d tar -n $(bash packaging_scripts/baptizer.sh -a x64 -d tar -m)
# docker exec -it wi-build_$(<VERSION) bash build-scripts/build.sh -a $ARCHITECTURE -d $PACKAGE_TYPE -n $(bash build-scripts/baptizer.sh -a $ARCHITECTURE -d $PACKAGE_TYPE -m)

The generated package is sent to the ``wazuh-indexer/artifacts`` folder.
After this step, a minimal package (without plugins) will be present under the ``artifacts`` directory.

.. _full-package-assemble-stage-1:
Full package assembly
=====================

Assembly stage
--------------
1. Set the environment variables:

Docker environment
^^^^^^^^^^^^^^^^^^
.. note::

Replace ``<arch>`` with one of ``x64`` or ``arm64`` and ``<package-type>`` with one of ``rpm``, ``deb`` or ``tar``.

.. code:: console

# ARCHITECTURE=<arch>
# PACKAGE_TYPE=<package-type>

Pre-requisites:
2. Run the assembly process:

- Current directory: ``wazuh-indexer/``
- Existing package in ``wazuh-indexer/artifacts/dist/{rpm|deb}``, as a result of the *Build* stage.
- Using the `Docker environment <https://www.github.com/wazuh/wazuh-indexer/tree/v|WAZUH_CURRENT|/docker>`__:
.. code:: console

.. tabs::
# docker exec -it wi-assemble_$(<VERSION) bash build-scripts/assemble.sh -a $ARCHITECTURE -d $PACKAGE_TYPE -r 1

.. group-tab:: RPM
Native environment
******************
.. raw:: html

.. code-block:: console
<div class="accordion-section open">

# docker exec -it wi-assemble_|WAZUH_CURRENT| bash packaging_scripts/assemble.sh -a x64 -d rpm

.. group-tab:: DEB
Pre-requisistes
================

.. code-block:: console
1. Install build dependencies

# docker exec -it wi-assemble_|WAZUH_CURRENT| bash packaging_scripts/assemble.sh -a x64 -d deb
.. tabs::

.. group-tab:: RPM

.. code-block:: console

.. group-tab:: TAR
# yum install -y git curl gnupg2 gcc gcc-c++ make cpio rpm-build mesa-libGLU freeglut alsa-lib atk at-spi2-core cairo cairo-devel cups-libs libdrm libgbm nspr nspr-devel nss pango libXcomposite libXdamage libXfixes libXfixes-devel libXi libxkbcommon libXrandr libXrender libXtst rpm rpm-build maven

.. group-tab:: DEB

.. code-block:: console
.. code-block:: console

# apt-get update
# apt-get install -y git curl gnupg2 y build-essential cpio debhelper-compat debmake freeglut3 libasound2 libatk-bridge2.0-0 libatk1.0-0 libatspi2.0-dev libcairo2 libcairo2-dev libcups2 libdrm2 libgbm-dev libgconf-2-4 libnspr4 libnspr4-dev libnss3 libpangocairo-1.0-0 libxcomposite-dev libxdamage1 libxfixes-dev libxfixes3 libxi6 libxkbcommon-x11-0 libxrandr2 libxrender1 libxtst6 rpm rpm2cpio maven

# docker exec -it wi-assemble_|WAZUH_CURRENT| bash packaging_scripts/assemble.sh -a x64 -d tar
2. Clone the ``wazuh-indexer`` repository and switch to the appropriate branch

Local package generation
^^^^^^^^^^^^^^^^^^^^^^^^
.. code:: console

.. note::
# git clone https://github.com/wazuh/wazuh-indexer
# git checkout v|WAZUH_CURRENT|

Set the environment variable ``TEST=true`` to assemble a package with a minimal set of plugins, speeding up the assembly process.

.. tabs::
Build a minimal package
=======================

.. group-tab:: RPM
A basic package including only the ``Wazuh Indexer`` engine without extra plugin is generated first.

The ``assemble.sh`` script will use the output from the ``build.sh`` script and use it as a base to bundle together a final package containing the plugins, the production configuration and the service files.

The script will:

#. Extract the RPM package using ``rpm2cpio`` and ``cpio`` tools.

By default, ``rpm2cpio`` and ``cpio`` tools expect the package to be in ``wazuh-indexer/artifacts/tmp/rpm``.
The script takes care of creating the required folder structure, copying also the min package and the SPEC file.

Current folder loadout at this stage:

.. code-block:: none

/rpm/$ARCH
/etc
/usr
/var
wazuh-indexer-min-*.rpm
wazuh-indexer.rpm.spec

``usr``, ``etc`` and ``var`` folders contain ``wazuh-indexer`` files, extracted from ``wazuh-indexer-min-*.rpm``.
1. Set the environment variables:

``wazuh-indexer.rpm.spec`` is copied over from ``wazuh-indexer/distribution/packages/src/rpm/wazuh-indexer.rpm.spec``.
.. note::

The ``wazuh-indexer-performance-analyzer.service`` file is also copied from the same folder.
Replace ``<arch>`` with one of ``x64`` or ``arm64`` and ``<package-type>`` with one of ``rpm``, ``deb`` or ``tar``

It is a dependency of the SPEC file.

#. Install the plugins using the ``opensearch-plugin`` CLI tool.

#. Set up configuration files.

Included in ``min-package``. Default files are overwritten.

#. Bundle an RPM file with ``rpmbuild`` and the SPEC file ``wazuh-indexer.rpm.spec``.

``rpmbuild`` is part of the ``rpm`` OS package.
.. code:: console

``rpmbuild`` is invoked from ``wazuh-indexer/artifacts/tmp/rpm``.
# ARCHITECTURE=<arch>
# PACKAGE_TYPE=<package-type>

It creates the ``{BUILD,RPMS,SOURCES,SRPMS,SPECS,TMP}`` folders and applies the rules in the SPEC file.
2. Run the build script:

If successful, ``rpmbuild`` will generate the package in the ``RPMS/`` folder.
.. code:: console

The script will copy it to ``wazuh-indexer/artifacts/dist`` and clean: remove the ``tmp\`` folder and its contents.

Current folder loadout at this stage:

.. code-block:: none

/rpm/$ARCH
/{BUILD,RPMS,SOURCES,SRPMS,SPECS,TMP}
/etc
/usr
/var
wazuh-indexer-min-*.rpm
wazuh-indexer.rpm.spec
# bash build-scripts/build.sh -a $ARCHITECTURE -d $PACKAGE_TYPE -n $(bash build-scripts/baptizer.sh -a $ARCHITECTURE -d $PACKAGE_TYPE -m)

.. group-tab:: DEB
After this step, a minimal package (without plugins) will be present under the ``artifacts`` directory.

For DEB packages, the ``assemble.sh`` script will perform the following operations:

#. Extract the deb package using ``ar`` and ``tar`` tools.

By default, ``ar`` and ``tar`` tools expect the package to be in ``wazuh-indexer/artifacts/tmp/deb``.
Full package assembly
=====================

The script takes care of creating the required folder structure, copying also the min package and the Makefile.

Current folder loadout at this stage:

.. code-block:: none

artifacts/
|-- dist
| |-- wazuh-indexer-min_|WAZUH_CURRENT|_amd64.deb
`-- tmp
`-- deb
|-- Makefile
|-- data.tar.gz
|-- debmake_install.sh
|-- etc
|-- usr
|-- var
`-- wazuh-indexer-min_|WAZUH_CURRENT|_amd64.deb

``usr``, ``etc`` and ``var`` folders contain ``wazuh-indexer`` files, extracted from ``wazuh-indexer-min-*.deb``.
1. Set the ``ARCHITECTURE`` and ``PACKAGE_TYPE`` environment variables replacing ``<package-type>`` with one of ``tar``, ``deb`` or ``rpm`` and ``<arch>`` with ``x64`` or ``arm64`` depending on the target system the packages are being built for.

``Makefile`` and the ``debmake_install`` are copied over from ``wazuh-indexer/distribution/packages/src/deb``.
.. code:: console

The ``wazuh-indexer-performance-analyzer.service`` file is also copied from the same folder.
# ARCHITECTURE=<arch>
# PACKAGE_TYPE=<package-type>

It is a dependency of the SPEC file.

#. Install the plugins using the ``opensearch-plugin`` CLI tool.

#. Set up configuration files.

Included in ``min-package``. Default files are overwritten.

#. Bundle a DEB file with ``debmake`` and the ``Makefile``.

``debmake`` and other dependencies can be installed using the ``provision.sh`` script.
The script is invoked by the GitHub Workflow.

Current folder loadout at this stage:

.. code-block:: none

artifacts/
|-- artifact_name.txt
|-- dist
| |-- wazuh-indexer-min_|WAZUH_CURRENT|_amd64.deb
| `-- wazuh-indexer_|WAZUH_CURRENT|_amd64.deb
`-- tmp
`-- deb
|-- Makefile
|-- data.tar.gz
|-- debmake_install.sh
|-- etc
|-- usr
|-- var
|-- wazuh-indexer-min_|WAZUH_CURRENT|_amd64.deb
`-- debian/
| -- control
| -- copyright
| -- rules
| -- preinst
| -- prerm
| -- postinst

.. group-tab:: TAR
2. Run the assembly process:

.. code:: console

# bash build-scripts/assemble.sh -a $ARCHITECTURE -d $PACKAGE_TYPE

The assembly process for tarballs consists on:

#. Extraction of the minimal package
#. Bundling of plugins
#. Addition of Wazuh configuration files and tooling
#. Compression

.. code:: console

# bash packaging_scripts/assemble.sh -a x64 -d tar -r 1

Build and assemble scripts reference
------------------------------------
####################################
.. raw:: html

<div class="accordion-section open">

The packages' generation process is guided through bash scripts.

Expand Down