Skip to content

Latest commit

 

History

History
173 lines (129 loc) · 8.77 KB

README.md

File metadata and controls

173 lines (129 loc) · 8.77 KB
Raw

Non-Official Confluent Docker Images on ARM

Currently the Official Confluent Docker Images are only made available for linux/amd64 architecture.

When running in the Apple M1 ARM-based chip, these images need to be emulated and in this case break:

To help my team maintain the current labs, some of the images and versions from confluent where build on ARM using the official git repositories from Confluent:

ARM Images

Currently the images available are:

component arm image official image versions
Zookeeper ricardo7aires/cp-zookeeper confluentinc/cp-zookeeper 6.1.0, 7.1.1
Kafka (Community Version) ricardo7aires/cp-kafka confluentinc/cp-kafka 6.1.0, 7.1.1
Enterprise Kafka Distribution ricardo7aires/cp-server confluentinc/cp-server 6.1.0, 7.1.1
Kafka Connect (Community Version) ricardo7aires/cp-kafka-connect confluentinc/cp-kafka-connect 6.1.0, 7.1.1
Enterprise Kafka Connect ricardo7aires/cp-server-connect confluentinc/cp-server-connect 6.1.0, 7.1.1
Schema Registry ricardo7aires/cp-schema-registry confluentinc/cp-schema-registry 6.1.0, 7.1.1
KSQL-DB Server ricardo7aires/cp-ksqldb-server confluentinc/cp-ksqldb-server 6.1.0, 7.1.1
KSQL-DB cli ricardo7aires/cp-ksqldb-cli confluentinc/cp-ksqldb-cli 6.1.0, 7.1.1
Control Center ricardo7aires/cp-enterprise-control-center confluentinc/cp-enterprise-control-center 6.1.0, 7.1.1

Because they are being build from the source of Confluent they are compatible with the demos provided by them and charts/operators. Compatible, not the same as supported. These are for lab purpose, not production.

Build Process

Requirements

Used at the time of build:

  • macOS Monterey 12.3.1
  • Docker Desktop for Mac - Apple Chip 4.7.1 (77678)
  • socat v1.7.4.3
  • maven v3.8.5
  • tox v3.25.0

socat, maven and tox installed via brew.

Pre-Steps

After the software required is installed, we need to set some maven mirrors to avoid build errors, if maven was installed via brew this is done by editing /opt/homebrew/Cellar/maven/3.8.5/libexec/conf/settings.xml and adding in the mirror list

<mirror>
    <id>confluent</id>
    <mirrorOf>confluent</mirrorOf>
    <name>Nexus public mirror</name>
    <url>http://packages.confluent.io/maven/</url>
</mirror>

The kafka Docker build is based on the Maven plugin dockerfile-maven-plugin, which doesn't support ARM. As a workaround we need to expose the docker daemon sock in a TCP port:

socat TCP-LISTEN:2375,reuseaddr,fork UNIX-CONNECT:/var/run/docker.sock

Run the above command and leave that window open while doing the builds. On the windows performing the build make sure to change the variable for the docker host:

export DOCKER_HOST=tcp://127.0.0.1:2375

Build Base image

  1. Clone the confluentinc/common-docker git repository

  2. Change to the desired version

    git checkout tags/v7.1.1

  3. Add the next repository on the root pom.xml

    <repositories>
    <repository>
        <id>confluent</id>
        <url>https://packages.confluent.io/maven/</url>
        <releases>
            <enabled>true</enabled>
        </releases>
        <snapshots>
            <enabled>true</enabled>
        </snapshots>
    </repository>
</repositories>

  4. Beware of the versions hardcoded in the same pom.xml, some may not be available for the ARM, some required changes in time of writing:

    <ubi.openssl.version>1.1.1k-6.el8_5</ubi.openssl.version>
<ubi.zulu.openjdk.version>11.0.15</ubi.zulu.openjdk.version>

    On older versions these last step the version are hardcoded on the base/Dockerfile.ubi8

  5. On the file base/tox.ini remove the versions of pytest, pytest-xdist and pytest-cov to avoid conflicts.

  6. On older tags you may need to update pip inside the base/Dockerfile.ubi8, you can check the latest to reference, but you can add the next in the RUN statement:

...
RUN microdnf --nodocs install yum \
...
    && alternatives --set python /usr/bin/python3 \
    && python3 -m pip install --upgrade "pip${PYTHON_PIP_VERSION}" "setuptools${PYTHON_SETUPTOOLS_VERSION}" \
...
  1. Run the build
mvn clean package \
  -Pdocker -DskipTests \
  -Ddocker.registry=local/

The pom.xml has a flag to validate if the Yum/Dnf package manager detects that there is security update available and by default it will fail if it has. One can disabled or take action when detected. When I run there was an update for the zlib hence I changed the Dockerfile to updated it.

Build Zookeeper, Kafka and Kafka Connect

  1. Clone the confluentinc/kafka-images git repository

  2. Change to the desired version

    git checkout tags/v7.1.1

  3. Add the next repository on the root pom.xml

    <repositories>
    <repository>
        <id>confluent</id>
        <url>https://packages.confluent.io/maven/</url>
        <releases>
            <enabled>true</enabled>
        </releases>
        <snapshots>
            <enabled>true</enabled>
        </snapshots>
    </repository>
</repositories>

  4. Run the build

mvn clean package \
  -Pdocker -DskipTests \
  -DCONFLUENT_PACKAGES_REPO='https://packages.confluent.io/rpm/7.1' \
  -DCONFLUENT_VERSION=7.1.1 \
  -Ddocker.registry=local/

The same steps can be done on other repos in order to build some images, remember to do the common first: