Multi-line environment variable can make a node fail to start if RABBITMQ_CONF_ENV_FILE is set #2458

bradsease · 2020-10-01T19:26:38Z

bradsease
Oct 1, 2020

RabbitMQ server fails to start if:

the $RABBITMQ_CONF_ENV_FILE variable is defined
any environment variable contains a newline and ends with an apostrophe

This behavior can be recreated in a fresh RabbitMQ environment with the following command sequence:

echo  > env.txt
echo "'test'" >> env.txt
export TEST_VAR=`cat env.txt`
echo NODE_PORT=9999 > /test.conf
export RABBITMQ_CONF_ENV_FILE=/test.conf
bash /opt/rabbitmq/sbin/rabbitmq-server

Running this sequence in the official rabbitmq:latest docker image gives the following output:

Configuring logger redirection
18:17:32.993 [error] 

18:17:32.998 [error] BOOT FAILED
BOOT FAILED
18:17:32.998 [error] ===========
===========
18:17:32.998 [error] Exception during startup:
Exception during startup:
18:17:32.998 [error] 

18:17:32.998 [error]     supervisor:'-start_children/2-fun-0-'/3 line 355
    supervisor:'-start_children/2-fun-0-'/3 line 355
18:17:32.998 [error]     supervisor:do_start_child/2 line 371
    supervisor:do_start_child/2 line 371
18:17:32.998 [error]     supervisor:do_start_child_i/3 line 385
18:17:32.999 [error]     rabbit_prelaunch:run_prelaunch_first_phase/0 line 27
    supervisor:do_start_child_i/3 line 385
    rabbit_prelaunch:run_prelaunch_first_phase/0 line 27
18:17:32.999 [error]     rabbit_prelaunch:do_run/0 line 87
    rabbit_prelaunch:do_run/0 line 87
18:17:32.999 [error]     lists:foldl/3 line 1263
    lists:foldl/3 line 1263
18:17:32.999 [error]     rabbit_env:parse_conf_env_file_output1/3 line 1621
    rabbit_env:parse_conf_env_file_output1/3 line 1621
    rabbit_env:parse_single_quoted_literal([], [], "\"'\"'tset'\"'\"'\n") line 1650
18:17:32.999 [error]     rabbit_env:parse_single_quoted_literal([], [], "\"'\"'tset'\"'\"'\n") line 1650
18:17:32.999 [error] error:function_clause
error:function_clause

18:17:32.999 [error] 
18:17:34.000 [error] Supervisor rabbit_prelaunch_sup had child prelaunch started with rabbit_prelaunch:run_prelaunch_first_phase() at undefined exit with reason function_clause in context start_error
18:17:34.000 [error] CRASH REPORT Process <0.153.0> with 0 neighbours exited with reason: {{shutdown,{failed_to_start_child,prelaunch,function_clause}},{rabbit_prelaunch_app,start,[normal,[]]}} in application_master:init/4 line 138
{"Kernel pid terminated",application_controller,"{application_start_failure,rabbitmq_prelaunch,{{shutdown,{failed_to_start_child,prelaunch,function_clause}},{rabbit_prelaunch_app,start,[normal,[]]}}}"}
Kernel pid terminated (application_controller) ({application_start_failure,rabbitmq_prelaunch,{{shutdown,{failed_to_start_child,prelaunch,function_clause}},{rabbit_prelaunch_app,start,[normal,[]]}}})

Crash dump is being written to: erl_crash.dump...done

Removing either the TEST_VAR or RABBITMQ_CONF_ENV_FILE environment variables allows the server to start without error.

I've previously reported this issue to the Bitnami RabbitMQ project here but found that the problem seems to be in the RabbitMQ software itself. The problem appears to have started in version 3.8.4.

This docker-compose file also reproduces the problem thought it behaves slightly differently and starts the server while ignoring the config file.

version: '3'
  
services:
    rabbitmq:
        image: rabbitmq:latest
        environment:
            RABBITMQ_CONF_ENV_FILE: /test.conf
            TEST_VAR: |
                'test'
        volumes:
            - /path/to/test.conf:/test.conf:ro
        command: ["bash", "/opt/rabbitmq/sbin/rabbitmq-server"]

Versions:

RabbitMQ server: 3.8.4+
Erlang: 23
Operating system version (and distribution, if applicable): Ubuntu 18.04
All client libraries used: N/A
RabbitMQ plugins (if applicable): N/A

Answered by dumbbell

Oct 7, 2020

Here is the patch: rabbitmq/rabbitmq-common#417

@bradsease: Do you have a way to test it with your usecase? I never use Docker beside docker run -it twice a year, so I'm not sure how to test it myself.

View full answer

michaelklishin · 2020-10-01T19:38:53Z

michaelklishin
Oct 1, 2020
Maintainer

I must be missing something but

echo  > env.txt
echo "'test'" >> env.txt
export TEST_VAR=`cat env.txt`
echo NODE_PORT=9999 > /test.conf

results in the following environment variable file:

NODE_PORT=9999

which works like a charm. The following file

NODENAME='hare@warp10'

also works like a charm while it contains an apostrophe.

Env variables not related to RabbitMQ are ignored:

NODENAME='hare@warp10'
TEST_VAR='a'

also works like a charm. So does this file which contains a trailing empty line:

NODENAME='hare@warp10'
TEST_VAR='a'

I don't think we are interested in supporting newlines in environment variables values. The very basic parser we have is sufficient, or this issue would be brought up over and over again.

It is expected that if RABBITMQ_CONF_ENV_FILE is not set, then it won't be "loaded"/parsed.

5 replies

bradsease Oct 1, 2020
Author

Sorry if it wasn't clear, the problem is not any value in the RABBITMQ_CONF_ENV_FILE. The TEST_VAR is an environment variable defined on the host, outside of the config file.

You're correct that this code

echo  > env.txt
echo "'test'" >> env.txt
export TEST_VAR=`cat env.txt`
echo NODE_PORT=9999 > /test.conf

produces a config file with only

NODE_PORT=9999

but also the host environment:

RABBITMQ_DEFAULT_PASS=
LC_ALL=C.UTF-8
RABBITMQ_SSL_FAIL_IF_NO_PEER_CERT=true
LANG=C.UTF-8
RABBITMQ_CONF_ENV_FILE=/test.conf
HOSTNAME=9d4960fa2d6f
OPENSSL_VERSION=1.1.1h
OTP_VERSION=23.1
TEST_VAR=                         <------- Multiline environment variable
'test'
RABBITMQ_HOME=/opt/rabbitmq
RABBITMQ_MANAGEMENT_SSL_FAIL_IF_NO_PEER_CERT=false
RABBITMQ_MANAGEMENT_SSL_VERIFY=verify_none
PWD=/
RABBITMQ_LOGS=-
HOME=/var/lib/rabbitmq
OPENSSL_PGP_KEY_IDS=0x8657ABB260F056B1E5190839D9C4D26D0E604491 0x5B2545DAB21995F4088CEFAA36CEE4DEB00CFE33 0xED230BEC4D4F2518B9D7DF41F0DB4D21C1D35231 0xC1F33DD8CE1D4CC613AF14DA9195C48241FBF7DD 0x7953AC1FBC3DC8B3B292393ED5E9E43F7DF9EE8C 0xE5E52560DD91C556DDBDA5D02064C53641C25E5D
RABBITMQ_VERSION=3.8.9
RABBITMQ_DATA_DIR=/var/lib/rabbitmq
TERM=xterm
RABBITMQ_SSL_VERIFY=verify_peer
RABBITMQ_PGP_KEY_ID=0x0A9AF2115F4687BD29803A206B73A36E6026DFCA
SHLVL=1
LANGUAGE=C.UTF-8
OPENSSL_SOURCE_SHA256=5c9ca8774bd7b03e5784f26ae9e9e6d749c9da2438545077e6b3d755a06595d9
RABBITMQ_DEFAULT_USER=
PATH=/opt/rabbitmq/sbin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
OTP_SOURCE_SHA256=3591903503ea70be3ef1e42abc7a3e1f8af90f2c8989506bf9832175f091e6e5
_=/usr/bin/env

The server appears to be parsing the host environment variables in some way and is failing to start due to TEST_VAR.

michaelklishin Oct 1, 2020
Maintainer

With the following variable exported and visible in printenv

TEST_VAR=
'test'

I still cannot get the node to fail to start on macOS with zsh and RabbitMQ 3.8.9.

RabbitMQ prelauncher sources the RABBITMQ_CONF_ENV file using a generated script, then parses the output.

The function that fails with a missing clause (unexpected argument) is this one. In fact, it seems to be trying to handle multi-line values after splitting the line with a =. All this is new in 3.8.4, previously significantly more of this was performed in actual shell script.

bradsease Oct 1, 2020
Author

Do you have RABBITMQ_CONF_ENV_FILE defined at the same time? It seems to only trigger when both RABBITMQ_CONF_ENV_FILE points to a valid file and the multiline environment variable exists on the host.

michaelklishin Oct 1, 2020
Maintainer

Yes, I run the node as

RABBITMQ_CONF_ENV_FILE=/path/to/configs/rabbitmq-env.conf rabbitmq-server

bradsease Oct 1, 2020
Author

Hmm, interesting. In my docker environment this triggers the error

echo  > env.txt
echo "'test'" >> env.txt
export TEST_VAR=`cat env.txt`
echo NODE_PORT=9999 > /test.conf
export RABBITMQ_CONF_ENV_FILE=/test.conf
bash /opt/rabbitmq/sbin/rabbitmq-server

but this does not

echo  > env.txt
echo "'test'" >> env.txt
export TEST_VAR=`cat env.txt`
echo NODE_PORT=9999 > /test.conf
export RABBITMQ_CONF_ENV_FILE=/test.conf
/opt/rabbitmq/sbin/rabbitmq-server

michaelklishin · 2020-10-01T19:41:01Z

michaelklishin
Oct 1, 2020
Maintainer

@bradsease I cannot reproduce using several example files above. I'm afraid we need a set of specific test cases. This is not an actionable item according to the standards of our team => moved to discussions.

1 reply

bradsease Oct 1, 2020
Author

The most reliable way I've found to reproduce it is:

docker run --rm -it rabbitmq:latest bash

followed by this in the container

echo  > env.txt
echo "'test'" >> env.txt
export TEST_VAR=`cat env.txt`
echo NODE_PORT=9999 > /test.conf
export RABBITMQ_CONF_ENV_FILE=/test.conf
bash /opt/rabbitmq/sbin/rabbitmq-server

The docker-compose file from my original issue also triggers it but only if the /path/to/test.conf points to an actual path of a config file.

michaelklishin · 2020-10-01T20:07:05Z

michaelklishin
Oct 1, 2020
Maintainer

I tried to come up with a file that would fail node startup to confirm that the files are, indeed, picked up. This one does:

NODENAME='hare@warp10'
TEST_VAR///

The message is reasonably specific and mentions the file and line number:

configs/rabbitmq-env.conf: line 2: TEST_VAR///: No such file or directory

This is because the file is sourced with a shell before being "pre-formatted" and its contents parsed.

Unlike the rabbitmq.conf parser, which is a "real" parser with a syntax, data types, and per-key validators, rabbitmq-env.conf format is minimalistic and typing isn't really something that environment variables offer. So its "parser" is equally simplistic.

0 replies

lukebakken · 2020-10-02T18:07:24Z

lukebakken
Oct 2, 2020
Maintainer

I can't reproduce this issue using the 3.8.9 generic-unix package, Erlang 23.1.1 running on Arch Linux.

Here's the script I'm using to start up RabbitMQ:

https://gist.github.com/lukebakken/7ce120f566e67a6c0690ca308836d907#file-repro-sh

$ echo $BASH_VERSION
5.0.18(1)-release

Complete output:

https://gist.github.com/lukebakken/7ce120f566e67a6c0690ca308836d907#file-output-txt

2 replies

bradsease Oct 2, 2020
Author

This script still reproduces the error for me in the rabbitmq docker image:

#!/bin/bash

set -e
set -u
set -x

echo > env.txt
echo "'test'" >> env.txt
export TEST_VAR=`cat env.txt`
echo NODE_PORT=9999 > "$PWD/test.conf"
export RABBITMQ_CONF_ENV_FILE="$PWD/test.conf"

cd "/opt/rabbitmq"
LOG=debug RABBITMQ_ALLOW_INPUT=true /bin/bash ./sbin/rabbitmq-server

Note that I removed the RABBITMQ_CONFIG_FILE line because I'm not sure what's in your config file. I also notice this difference:

$ echo $BASH_VERSION
4.4.20(1)-release

lukebakken Oct 7, 2020
Maintainer

What version of /bin/sh is in the docker image? Note that the RabbitMQ team does not manage the docker image.

dumbbell · 2020-10-07T12:49:19Z

dumbbell
Oct 7, 2020
Maintainer

Hi!

I finally took the time to look at this issue. Indeed, I can reproduce the problem with the instructions from the initial comment and the docker image. I don't know what implement lf /bin/sh is in the docker image, but yes, set output differs from other tested Bourne shells (GNU Bash, dash or FreeBSD sh).

Here is how the TEST_VAR and its newline character are returned by set:

TEST_VAR='
'"'"'test'"'"

I will prepare a patch to fix this.

2 replies

lukebakken Oct 7, 2020
Maintainer

That is awful. I bet it is dash under the hood.

dumbbell Oct 7, 2020
Maintainer

This is still a perfectly valid output. Clearly not for human-being though :-) Bourne shells are really bad for literals quoting IMHO.

dumbbell · 2020-10-07T13:37:07Z

dumbbell
Oct 7, 2020
Maintainer

Here is the patch: rabbitmq/rabbitmq-common#417

@bradsease: Do you have a way to test it with your usecase? I never use Docker beside docker run -it twice a year, so I'm not sure how to test it myself.

3 replies

dumbbell Oct 7, 2020
Maintainer

I found a way to test it by scp(1)'ing the fixed & compiled rabbit_env module into the running Docker image, putting it in the rabbit_common-3.8.9.ez Zip file and trying your instructions. Now the server starts successfully.

That said, I would still love to hear from you if you can test it :-)

bradsease Oct 8, 2020
Author

Thanks for looking into this for me @dumbbell! Unfortunately I don't know much about Erlang so I don't think I can test this fix myself.

dumbbell Oct 12, 2020
Maintainer

Thank you for letting me know! I merged the branch, we'll see how it goes :-)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Multi-line environment variable can make a node fail to start if RABBITMQ_CONF_ENV_FILE is set #2458

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 6 comments 13 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

Select a reply

Multi-line environment variable can make a node fail to start if RABBITMQ_CONF_ENV_FILE is set #2458

bradsease Oct 1, 2020

Replies: 6 comments · 13 replies

michaelklishin Oct 1, 2020 Maintainer

bradsease Oct 1, 2020 Author

michaelklishin Oct 1, 2020 Maintainer

bradsease Oct 1, 2020 Author

michaelklishin Oct 1, 2020 Maintainer

bradsease Oct 1, 2020 Author

michaelklishin Oct 1, 2020 Maintainer

bradsease Oct 1, 2020 Author

michaelklishin Oct 1, 2020 Maintainer

lukebakken Oct 2, 2020 Maintainer

bradsease Oct 2, 2020 Author

lukebakken Oct 7, 2020 Maintainer

dumbbell Oct 7, 2020 Maintainer

lukebakken Oct 7, 2020 Maintainer

dumbbell Oct 7, 2020 Maintainer

dumbbell Oct 7, 2020 Maintainer

dumbbell Oct 7, 2020 Maintainer

bradsease Oct 8, 2020 Author

dumbbell Oct 12, 2020 Maintainer

bradsease
Oct 1, 2020

Replies: 6 comments 13 replies

michaelklishin
Oct 1, 2020
Maintainer

bradsease Oct 1, 2020
Author

michaelklishin Oct 1, 2020
Maintainer

bradsease Oct 1, 2020
Author

michaelklishin Oct 1, 2020
Maintainer

bradsease Oct 1, 2020
Author

michaelklishin
Oct 1, 2020
Maintainer

bradsease Oct 1, 2020
Author

michaelklishin
Oct 1, 2020
Maintainer

lukebakken
Oct 2, 2020
Maintainer

bradsease Oct 2, 2020
Author

lukebakken Oct 7, 2020
Maintainer

dumbbell
Oct 7, 2020
Maintainer

lukebakken Oct 7, 2020
Maintainer

dumbbell Oct 7, 2020
Maintainer

dumbbell
Oct 7, 2020
Maintainer

dumbbell Oct 7, 2020
Maintainer

bradsease Oct 8, 2020
Author

dumbbell Oct 12, 2020
Maintainer