Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Build Druid from source #684

Merged
merged 17 commits into from
May 23, 2024
Merged

feat: Build Druid from source #684

Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
e7e8f15
Upload script downloading tar.gz now
Maleware May 16, 2024
edac223
WIP Druid Dockerfile
Maleware May 16, 2024
a177b55
WIP First attempt of Docker build
Maleware May 16, 2024
58c848c
Adding java-devel to druid
Maleware May 16, 2024
cdf37cd
build druid tar and then untar again
labrenbe May 16, 2024
7b532a4
Removing ranger from dependencies
Maleware May 16, 2024
08482df
Building without ranger and with -Pdist
Maleware May 16, 2024
14227b5
fix docker build
labrenbe May 16, 2024
2bdfc69
clean up Dockerfile
labrenbe May 17, 2024
3bab7ff
Add hadolint ignores
labrenbe May 17, 2024
ec81298
added to changelog
labrenbe May 17, 2024
4f3dcdf
clean up Dockerfile
labrenbe May 17, 2024
f33bc21
Moving log4j env to main image
Maleware May 23, 2024
b9b72f6
Java-devel below java-base
Maleware May 23, 2024
6f39b2f
Merge branch 'main' into feat/build-druid-from-source
Maleware May 23, 2024
c09422f
Adding druid to PATH
Maleware May 23, 2024
86da5fc
Ignoring Hadoolint
Maleware May 23, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions conf.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,7 @@
"stax2_api": "4.2.1",
"woodstox_core": "6.2.1",
"authorizer": "0.5.0",
"java-devel": "11",
sbernauer marked this conversation as resolved.
Show resolved Hide resolved
},
{
"product": "27.0.0",
Expand All @@ -68,6 +69,7 @@
"stax2_api": "4.2.1",
"woodstox_core": "6.2.1",
"authorizer": "0.5.0",
"java-devel": "11",
},
{
"product": "28.0.1",
Expand All @@ -80,6 +82,7 @@
"stax2_api": "4.2.2",
"woodstox_core": "6.6.0",
"authorizer": "0.5.0",
"java-devel": "11",
},
],
},
Expand Down
92 changes: 71 additions & 21 deletions druid/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,13 +4,80 @@
# Ignoring DL4006 globally because we inherit the SHELL from our base image
# hadolint global ignore=DL3038,DL4006

FROM stackable/image/java-base
####### ToDo's ########
# Build-step
# - maven clean install (java-devel image I guess)
# - copy bin
# - test stuff

labrenbe marked this conversation as resolved.
Show resolved Hide resolved
FROM stackable/image/java-devel as druid-builder

Check warning on line 13 in druid/Dockerfile

View workflow job for this annotation

GitHub Actions / hadolint

[hadolint] druid/Dockerfile#L13 <DL3006>(https://github.com/hadolint/hadolint/wiki/DL3006) 

Always tag the version of an image explicitly
Raw output 
message:"Always tag the version of an image explicitly"  location:{path:"druid/Dockerfile"  range:{start:{line:13  column:1}}}  severity:WARNING  source:{name:"hadolint"  url:"https://github.com/hadolint/hadolint"}  code:{value:"DL3006"  url:"https://github.com/hadolint/hadolint/wiki/DL3006"}

ARG PRODUCT
ARG JACKSON_DATAFORMAT_XML
ARG STAX2_API
ARG WOODSTOX_CORE
ARG AUTHORIZER

RUN microdnf update && \
microdnf install \
# Required to install pyyaml
python-pip && \
microdnf clean all && \
rm -rf /var/cache/yum && \
# pyyaml is required for the compile Druid
pip install --no-cache-dir pyyaml==6.0.1

USER stackable
WORKDIR /stackable

RUN curl --fail -L "https://repo.stackable.tech/repository/packages/druid/apache-druid-${PRODUCT}-src.tar.gz" | tar -xzC . && \

Check warning on line 33 in druid/Dockerfile

View workflow job for this annotation

GitHub Actions / hadolint

[hadolint] druid/Dockerfile#L33 <DL3003>(https://github.com/hadolint/hadolint/wiki/DL3003) 

Use WORKDIR to switch to a directory
Raw output 
message:"Use WORKDIR to switch to a directory"  location:{path:"druid/Dockerfile"  range:{start:{line:33  column:1}}}  severity:WARNING  source:{name:"hadolint"  url:"https://github.com/hadolint/hadolint"}  code:{value:"DL3003"  url:"https://github.com/hadolint/hadolint/wiki/DL3003"}
cd apache-druid-${PRODUCT}-src && \
mvn clean install -Pdist -DskipTests -Dmaven.javadoc.skip=true && \
tar -xzf /stackable/apache-druid-${PRODUCT}-src/distribution/target/apache-druid-${PRODUCT}-bin.tar.gz && \
mv /stackable/apache-druid-${PRODUCT}-src/apache-druid-${PRODUCT} /stackable/apache-druid-${PRODUCT} && \
rm -rf /stackable/apache-druid-${PRODUCT}-src \
/stackable/apache-druid-${PRODUCT}/quickstart

# Install the Prometheus emitter extension. This bundle contains the emitter and all jar dependencies.
RUN curl --fail "https://repo.stackable.tech/repository/packages/druid/druid-prometheus-emitter-${PRODUCT}.tar.gz" | tar -xzC /stackable/apache-druid-${PRODUCT}/extensions && \
# Install OPA authorizer extension.
curl --fail "https://repo.stackable.tech/repository/packages/druid/druid-opa-authorizer-${AUTHORIZER}.tar.gz" | tar -xzC /stackable/apache-druid-${PRODUCT}/extensions && \
# Install jackson-dataformat-xml, stax2-api, and woodstox-core which are required for logging, and remove stax-ex.
rm /stackable/apache-druid-${PRODUCT}/lib/stax-ex-*.jar && \
curl --fail -L -o /stackable/apache-druid-${PRODUCT}/lib/jackson-dataformat-xml-${JACKSON_DATAFORMAT_XML}.jar \
"https://repo.stackable.tech/repository/packages/jackson-dataformat-xml/jackson-dataformat-xml-${JACKSON_DATAFORMAT_XML}.jar" && \
curl --fail -L -o /stackable/apache-druid-${PRODUCT}/lib/stax2-api-${STAX2_API}.jar \
"https://repo.stackable.tech/repository/packages/stax2-api/stax2-api-${STAX2_API}.jar" && \
curl --fail -L -o /stackable/apache-druid-${PRODUCT}/lib/woodstox-core-${WOODSTOX_CORE}.jar \
"https://repo.stackable.tech/repository/packages/woodstox-core/woodstox-core-${WOODSTOX_CORE}.jar"

# ===
# Mitigation for CVE-2021-44228 (Log4Shell)
# This variable is supported as of Log4j version 2.10 and
# disables the vulnerable feature
ENV LOG4J_FORMAT_MSG_NO_LOOKUPS=true
sbernauer marked this conversation as resolved.
Show resolved Hide resolved

# For earlier versions this script removes the .class file that contains the
# vulnerable code.
# TODO: This can be restricted to target only versions which do not honor the environment
# varible that has been set above but this has not currently been implemented
COPY shared/log4shell.sh /bin
RUN /bin/log4shell.sh "/stackable/apache-druid-${PRODUCT}"

# Ensure no vulnerable files are left over
# This will currently report vulnerable files being present, as it also alerts on
# SocketNode.class, which we do not remove with our scripts.
# Further investigation will be needed whether this should also be removed.
COPY shared/log4shell_1.6.1-log4shell_Linux_x86_64 /bin/log4shell_scanner_x86_64
COPY shared/log4shell_1.6.1-log4shell_Linux_aarch64 /bin/log4shell_scanner_aarch64
COPY shared/log4shell_scanner /bin/log4shell_scanner
RUN /bin/log4shell_scanner s "/stackable/apache-druid-${PRODUCT}"
# ===

## Final Image
labrenbe marked this conversation as resolved.
Show resolved Hide resolved
FROM stackable/image/java-base

Check warning on line 78 in druid/Dockerfile

View workflow job for this annotation

GitHub Actions / hadolint

[hadolint] druid/Dockerfile#L78 <DL3006>(https://github.com/hadolint/hadolint/wiki/DL3006) 

Always tag the version of an image explicitly
Raw output 
message:"Always tag the version of an image explicitly"  location:{path:"druid/Dockerfile"  range:{start:{line:78  column:1}}}  severity:WARNING  source:{name:"hadolint"  url:"https://github.com/hadolint/hadolint"}  code:{value:"DL3006"  url:"https://github.com/hadolint/hadolint/wiki/DL3006"}

ARG PRODUCT
ARG RELEASE

LABEL name="Apache Druid" \
Expand All @@ -22,36 +89,19 @@
description="This image is deployed by the Stackable Operator for Apache Druid."

RUN microdnf update && \
microdnf install \
findutils \
gzip \
tar \
zip && \
microdnf clean all && \
rm -rf /var/cache/yum

USER stackable
WORKDIR /stackable

COPY --from=druid-builder /stackable /stackable
COPY --chown=stackable:stackable druid/stackable /stackable
COPY --chown=stackable:stackable druid/licenses /licenses

RUN curl --fail -L https://repo.stackable.tech/repository/packages/druid/apache-druid-${PRODUCT}-bin.tar.gz | tar -xzC . && \
ln -s /stackable/apache-druid-${PRODUCT} /stackable/druid && \
RUN ln -s /stackable/apache-druid-${PRODUCT} /stackable/druid && \
# Force to overwrite the existing 'run-druid'
ln -sf /stackable/bin/run-druid /stackable/druid/bin/run-druid && \
# Install the Prometheus emitter extension. This bundle contains the emitter and all jar dependencies.
curl --fail https://repo.stackable.tech/repository/packages/druid/druid-prometheus-emitter-${PRODUCT}.tar.gz | tar -xzC /stackable/druid/extensions && \
# Install OPA authorizer extension.
curl --fail https://repo.stackable.tech/repository/packages/druid/druid-opa-authorizer-${AUTHORIZER}.tar.gz | tar -xzC /stackable/druid/extensions && \
# Install jackson-dataformat-xml, stax2-api, and woodstox-core which are required for logging, and remove stax-ex.
rm /stackable/druid/lib/stax-ex-*.jar && \
curl --fail -L -o /stackable/druid/lib/jackson-dataformat-xml-${JACKSON_DATAFORMAT_XML}.jar \
https://repo.stackable.tech/repository/packages/jackson-dataformat-xml/jackson-dataformat-xml-${JACKSON_DATAFORMAT_XML}.jar && \
curl --fail -L -o /stackable/druid/lib/stax2-api-${STAX2_API}.jar \
https://repo.stackable.tech/repository/packages/stax2-api/stax2-api-${STAX2_API}.jar && \
curl --fail -L -o /stackable/druid/lib/woodstox-core-${WOODSTOX_CORE}.jar \
https://repo.stackable.tech/repository/packages/woodstox-core/woodstox-core-${WOODSTOX_CORE}.jar
ln -sf /stackable/bin/run-druid /stackable/druid/bin/run-druid

WORKDIR /stackable/druid
CMD ["bin/run-druid", "coordinator", "conf/druid/cluster/master/coordinator-overlord/"]
18 changes: 9 additions & 9 deletions druid/upload_new_druid_version.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,33 +29,33 @@ trap cleanup EXIT

cd "$WORK_DIR" || exit

bin_file="apache-druid-${VERSION}-bin.tar.gz"
src_file="apache-druid-${VERSION}-src.tar.gz"

echo "Downloading Druid (this can take a while, it is intentionally downloading from a slow mirror that contains all old versions)"
curl --fail -LOs "https://archive.apache.org/dist/druid/${VERSION}/${bin_file}"
curl --fail -LOs "https://archive.apache.org/dist/druid/${VERSION}/${bin_file}.asc"
curl --fail -LOs "https://archive.apache.org/dist/druid/${VERSION}/${bin_file}.sha512"
curl --fail -LOs "https://archive.apache.org/dist/druid/${VERSION}/${src_file}"
curl --fail -LOs "https://archive.apache.org/dist/druid/${VERSION}/${src_file}.asc"
curl --fail -LOs "https://archive.apache.org/dist/druid/${VERSION}/${src_file}.sha512"

# It is probably redundant to check both the checksum and the signature but it's cheap and why not
echo "Validating SHA512 Checksum"
if ! (sha512sum "${bin_file}" | cut -d " " -f 1 | diff -Z - "${bin_file}.sha512"); then
if ! (sha512sum "${src_file}" | cut -d " " -f 1 | diff -Z - "${src_file}.sha512"); then
echo "ERROR: The SHA512 sum does not match"
exit 1
fi

echo "Validating signature"
echo '--> NOTE: Make sure you have downloaded and added the KEYS file (https://www.apache.org/dist/druid/KEYS) to GPG: https://www.apache.org/info/verification.html (e.g. by using "curl https://archive.apache.org/dist/spark/KEYS | gpg --import")'

if ! (gpg --verify "${bin_file}.asc" "${bin_file}" 2> /dev/null); then
if ! (gpg --verify "${src_file}.asc" "${src_file}" 2> /dev/null); then
echo "ERROR: The signature could not be verified"
exit 1
fi

echo "Uploading everything to Nexus"
EXIT_STATUS=0
curl --fail -u "$NEXUS_USER:$NEXUS_PASSWORD" --upload-file "${bin_file}" 'https://repo.stackable.tech/repository/packages/druid/' || EXIT_STATUS=$?
curl --fail -u "$NEXUS_USER:$NEXUS_PASSWORD" --upload-file "${bin_file}.asc" 'https://repo.stackable.tech/repository/packages/druid/' || EXIT_STATUS=$?
curl --fail -u "$NEXUS_USER:$NEXUS_PASSWORD" --upload-file "${bin_file}.sha512" 'https://repo.stackable.tech/repository/packages/druid/' || EXIT_STATUS=$?
curl --fail -u "$NEXUS_USER:$NEXUS_PASSWORD" --upload-file "${src_file}" 'https://repo.stackable.tech/repository/packages/druid/' || EXIT_STATUS=$?
curl --fail -u "$NEXUS_USER:$NEXUS_PASSWORD" --upload-file "${src_file}.asc" 'https://repo.stackable.tech/repository/packages/druid/' || EXIT_STATUS=$?
curl --fail -u "$NEXUS_USER:$NEXUS_PASSWORD" --upload-file "${src_file}.sha512" 'https://repo.stackable.tech/repository/packages/druid/' || EXIT_STATUS=$?

if [ $EXIT_STATUS -ne 0 ]; then
echo "ERROR: Upload failed"
Expand Down