Channel exception generation fails for a queue that contains non-ASCII characters

[bpint]

Describe the bug

when a shovel declares queue/exchange name that contains special characters, say '😍' or any CJK characters, status page will crashes. the shovel cannot be removed with shovel management plugin either.

this happens at least in versions of 3.12.x, 3.13.x, 4.0.x.

Reproduction steps

1. create a shovel with unicode characters in it. ensure the source and destination queues do NOT exist and are supposed to be declared by the shovel.

rabbitmqctl set_parameter shovel 'test' \
'{
    "ack-mode": "on-confirm",
    "dest-add-forward-headers": false,
    "dest-queue": "😍",
    "dest-protocol": "amqp091",
    "dest-uri": "amqp://",
    "src-delete-after": "never",
    "src-protocol": "amqp091",
    "src-queue": "😍",
    "src-uri": "amqp://"
}'

2. visit management console and navigate to Admin -> Shovel Status. a 'Management API returned status code 500 - ' banner will occur at the bottom of the browser.

3. click Admin -> Shovel Management, and then the shovel test. Delete This Shovel will fail with 'Shovel could not be deleted' and the same 500 banner.

4. delete shovel test with rabbitmqctl,

rabbitmqctl clear_parameter shovel 'test'

The Shovel Status will come back only after the rabbitmq node is restarted.
...

Expected behavior

shovel should declare source and destination automatically, no matter what characters are in their valid names.

Additional context

it may caused by a erlang:list_to_binary/1 call with the parameter in unicode encoding somehow, instead of utf-8. it crashes since elements in the list may be larger than 255. the unicode:characters_to_binary/1 should be used instead?

logs are attached. sample lines are,

test.log

2024-12-02 18:30:37.896411+08:00 [info] <0.5380.0> Shovel 'test' in virtual host '/' will now try to connect...
2024-12-02 18:30:37.902141+08:00 [error] <0.5420.0> ** Generic server <0.5420.0> terminating
2024-12-02 18:30:37.902141+08:00 [error] <0.5420.0> ** Last message in was {'$gen_cast',
2024-12-02 18:30:37.902141+08:00 [error] <0.5420.0>                            {method,
2024-12-02 18:30:37.902141+08:00 [error] <0.5420.0>                                {'queue.declare',0,<<"😍"/utf8>>,true,false,
2024-12-02 18:30:37.902141+08:00 [error] <0.5420.0>                                    false,false,false,[]},
2024-12-02 18:30:37.902141+08:00 [error] <0.5420.0>                                none,noflow}}
2024-12-02 18:30:37.902141+08:00 [error] <0.5420.0> ** When Server state == {ch,{conf,running,rabbit_framing_amqp_0_9_1,2,
2024-12-02 18:30:37.902141+08:00 [error] <0.5420.0>                                 <0.5416.0>,<0.5416.0>,<0.5400.0>,
2024-12-02 18:30:37.902141+08:00 [error] <0.5420.0>                                 <<"<port-5672@zen.1733127496.5400.0>">>,
2024-12-02 18:30:37.902141+08:00 [error] <0.5420.0>                                 {user,<<"none">>,[],
2024-12-02 18:30:37.902141+08:00 [error] <0.5420.0>                                     [{rabbit_auth_backend_dummy,none}]},
2024-12-02 18:30:37.902141+08:00 [error] <0.5420.0>                                 <<"/">>,<<>>,<0.5405.0>,
2024-12-02 18:30:37.902141+08:00 [error] <0.5420.0>                                 [{<<"publisher_confirms">>,bool,true},
2024-12-02 18:30:37.902141+08:00 [error] <0.5420.0>                                  {<<"exchange_exchange_bindings">>,bool,true},
2024-12-02 18:30:37.902141+08:00 [error] <0.5420.0>                                  {<<"basic.nack">>,bool,true},
2024-12-02 18:30:37.902141+08:00 [error] <0.5420.0>                                  {<<"consumer_cancel_notify">>,bool,true},
2024-12-02 18:30:37.902141+08:00 [error] <0.5420.0>                                  {<<"connection.blocked">>,bool,true},
2024-12-02 18:30:37.902141+08:00 [error] <0.5420.0>                                  {<<"authentication_failure_close">>,bool,
2024-12-02 18:30:37.902141+08:00 [error] <0.5420.0>                                   true}],
2024-12-02 18:30:37.902141+08:00 [error] <0.5420.0>                                 none,0,1800000,#{},infinity,1000000000},
2024-12-02 18:30:37.902141+08:00 [error] <0.5420.0>                             {lstate,<0.5419.0>,false},
2024-12-02 18:30:37.902141+08:00 [error] <0.5420.0>                             none,1,
2024-12-02 18:30:37.902141+08:00 [error] <0.5420.0>                             {0,[],[]},
2024-12-02 18:30:37.902141+08:00 [error] <0.5420.0>                             undefined,#{},#{},
2024-12-02 18:30:37.902141+08:00 [error] <0.5420.0>                             {state,fine,5000,
2024-12-02 18:30:37.902141+08:00 [error] <0.5420.0>                                 #Ref<0.1500642959.4281335809.79246>},
2024-12-02 18:30:37.902141+08:00 [error] <0.5420.0>                             false,1,
2024-12-02 18:30:37.902141+08:00 [error] <0.5420.0>                             {rabbit_confirms,undefined,#{}},
2024-12-02 18:30:37.902141+08:00 [error] <0.5420.0>                             [],[],none,undefined,[],
2024-12-02 18:30:37.902141+08:00 [error] <0.5420.0>                             {rabbit_queue_type,#{}},
2024-12-02 18:30:37.902141+08:00 [error] <0.5420.0>                             #Ref<0.1500642959.4281335809.79241>,false}
2024-12-02 18:30:37.902141+08:00 [error] <0.5420.0> ** Reason for termination ==
2024-12-02 18:30:37.902141+08:00 [error] <0.5420.0> ** {badarg,[{erlang,list_to_binary,
2024-12-02 18:30:37.902141+08:00 [error] <0.5420.0>                     ["no queue '😍' in vhost '/'"],
2024-12-02 18:30:37.902141+08:00 [error] <0.5420.0>                     [{error_info,#{module => erl_erts_errors}}]},
2024-12-02 18:30:37.902141+08:00 [error] <0.5420.0>             {rabbit_binary_generator,amqp_exception_explanation,2,
2024-12-02 18:30:37.902141+08:00 [error] <0.5420.0>                                      [{file,"rabbit_binary_generator.erl"},
2024-12-02 18:30:37.902141+08:00 [error] <0.5420.0>                                       {line,231}]},
2024-12-02 18:30:37.902141+08:00 [error] <0.5420.0>             {rabbit_binary_generator,lookup_amqp_exception,2,
2024-12-02 18:30:37.902141+08:00 [error] <0.5420.0>                                      [{file,"rabbit_binary_generator.erl"},
2024-12-02 18:30:37.902141+08:00 [error] <0.5420.0>                                       {line,223}]},
2024-12-02 18:30:37.902141+08:00 [error] <0.5420.0>             {rabbit_binary_generator,map_exception,3,
2024-12-02 18:30:37.902141+08:00 [error] <0.5420.0>                                      [{file,"rabbit_binary_generator.erl"},
2024-12-02 18:30:37.902141+08:00 [error] <0.5420.0>                                       {line,201}]},
2024-12-02 18:30:37.902141+08:00 [error] <0.5420.0>             {rabbit_channel,handle_exception,2,
2024-12-02 18:30:37.902141+08:00 [error] <0.5420.0>                             [{file,"rabbit_channel.erl"},{line,865}]},
2024-12-02 18:30:37.902141+08:00 [error] <0.5420.0>             {gen_server2,handle_msg,2,[{file,"gen_server2.erl"},{line,1056}]},
2024-12-02 18:30:37.902141+08:00 [error] <0.5420.0>             {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,241}]}]}
2024-12-02 18:30:37.902141+08:00 [error] <0.5420.0> 

[michaelklishin]

CJK characters, emojis and so on are not allowed in AMQP 0-9-1 entity names (such as queues). I am less sure about AMQP 1.0 but this is the fundamental limitation. In fact, the exception does not come from Shovel, it comes from a queue.declare method, that is, a queue declaration.

[michaelklishin]

@bpint for AMQP 0-9-1, this is a "won't fix". For AMQP 1.0, we will have to investigate what the requirements and options are. In any case, this is a very fundamental limitation that stems from the original protocol RabbitMQ implemented, so it's not a matter of swapping a few functions for their Unicode-aware equivalents.

[bpint]

@michaelklishin thanks a lot for your quick response.

as i know, many users widely use non-ascii characters in names of queues, exchanges, shovels, usernames, etc. when i create a queue with CJK names, it is totally fine. the queue can be created, deleted, bound, published to, consume from, and so on. shovel even works well with them, only if they are already declared. that give me the impression that those characters are valid for any names.

the names mentioned above are encoded in utf-8 internally and represented in binaries. the shovel plugin itself, stores source and destination queue names in JSON parameters (utf-8), and use them as strings/binaries (encoded with utf-16) to declare them, and displayed in utf-8 in web console. that is a different and strange behavior.

since there are many such names in production systems, i will investigate deeper on it.

[michaelklishin]

@bpint like I said, the AMQP 0-9-1 protocol explicitly does not allow for non-ASCII characters in entity names.

RabbitMQ has never enforced this character range and yet never in RabbitMQ docs or tutorials we use examples with non-ASCII characters. Never in the docs we promise that this is allowed, or will work, or will work with Shovel specifically or in the HTTP API.

Anyone is welcome to investigate how much work it might take to replace list_to_binary/1 and such in the protocol generator but we make no promises that we will accept, leave alone implement, the proposed solution.

It can be significantly more involved than swapping erlang:list_to_binary/1 here and there. On the hot code path, this can result in throughput drops, different resource consumption, and so on.

Message and resource representation in most cases is just a sequence of bytes, which is why you won't run into any conversion exceptions. However, the number of places where conversion is performed is also non-trivial and it is not a problem where you can solve "just N%" of, a change in accepted character range must be handled everywhere at once or not at all.

[bpint]

@michaelklishin here's my findings on this,

the stack is basicly,

1. rabbit_shovel_parameters:ensure_queue/3 tries to declare the queue passively to test whether it exists,

ensure_queue(Conn, Queue, XArgs) ->
    {ok, Ch} = amqp_connection:open_channel(Conn),
    try
        amqp_channel:call(Ch, #'queue.declare'{queue   = Queue,
                                               passive = true})
    catch exit:{{shutdown, {server_initiated_close, ?NOT_FOUND, _Text}}, _} ->
         %% ...
    end.

2. since the queue does not exist, exception is raised by rabbit_channel:handle_method/6

3. rabbit_channel:handle_exception/2 calls both

• rabbit_binary_generator:map_exception/3, and
• rabbit_channel:format_soft_error/1

4. the later rabbit_channel:format_soft_error/1 handles the unicode name correctly, so that the output text in log file is totally good.

format_soft_error(#amqp_error{name = N, explanation = E, method = M}) ->
    io_lib:format("operation ~ts caused a channel exception ~ts: ~ts", [M, N, E]).

5. on the contrary, rabbit_binary_generator:map_exception/3 calls rabbit_binary_generator:amqp_exception_explanation/2, which directly converts the lists (strings) to binaries, and subsequently crashes.

amqp_exception_explanation(Text, Expl) ->
    ExplBin = list_to_binary(Expl),
    CompleteTextBin = <<Text/binary, " - ", ExplBin/binary>>,
    if size(CompleteTextBin) > 255 -> <<CompleteTextBin:252/binary, "...">>;
       true                        -> CompleteTextBin
    end.

here Expl is,

[78,79,84,95,70,79,85,78,68,32,45,32,
110,111,32,113,117, 101,117,101,32,
39,128525,39,
32,105,110,32,118,104,111, 115,116,32,
39,47,39]

which is actually string NOT_FOUND - no queue '😍' in vhost '/'. it is the list member 128525 that causes list_to_binary failure.

it seems all other codes in rabbitmq are compatible with unicode names. rabbit_binary_generator:amqp_exception_explanation/2 is the only one i found bumpy by far.

here i propose the following change to fix it,

diff --git a/deps/rabbit_common/src/rabbit_binary_generator.erl b/deps/rabbit_common/src/rabbit_binary_generator.erl
index eb256052c0..ec0ddc4e76 100644
--- a/deps/rabbit_common/src/rabbit_binary_generator.erl
+++ b/deps/rabbit_common/src/rabbit_binary_generator.erl
@@ -228,8 +228,5 @@ lookup_amqp_exception(Other, Protocol) ->
     {ShouldClose, Code, Text, none}.
 
 amqp_exception_explanation(Text, Expl) ->
-    ExplBin = list_to_binary(Expl),
-    CompleteTextBin = <<Text/binary, " - ", ExplBin/binary>>,
-    if size(CompleteTextBin) > 255 -> <<CompleteTextBin:252/binary, "...">>;
-       true                        -> CompleteTextBin
-    end.
+    LimitedText = io_lib:format("~ts - ~ts", [Text, Expl], [{chars_limit, 255}]),
+    unicode:characters_to_binary(lists:flatten(LimitedText)).

1. unicode:characters_to_binary/1 is used instead to convert list that contains unicode characters to binary. this is coincides with the logic of rabbitmq logs.

2. just in case, direct binary truncating is replaced by io_lib:format/3 chars_limit, so that multiple utf-8 bytes of the same unicode character will not be cut in half.

by the way, i have also tested shovel with AMQP 1.0 protocol. this unicode problem does not exist for AMQP 1.0 addresses.

thanks a lot, and please consider this minor change.

[michaelklishin]

@bpint I applaud your interest in digging this one out. You can use https://www.erlang.org/docs/24/man/binary#part-3 or better yes, if the input is already a list of characters, https://www.erlang.org/docs/26/man/string#left-2 to truncate the value before using unicode:characters_to_binary/1.

You also don't need to flatten the result, this mixed list of characters is called an I/O list and many I/O-related functions handle them just fine:

%% a shell command
f(LimitedText),


LimitedText = io_lib:format("~ts - ~ts", ["An error has occurred", "причина"], [{chars_limit, 255}]),
unicode:characters_to_binary(LimitedText).

Since this is to render a protocol (channel) exception, efficiency is not particularly important but a quick test suggests that unicode functions are mostly ready to handle iolists.

Please submit a PR with a unit test in deps/rabbit_common/test/unit_SUITE.erl. CONTRIBUTING.md explains how to run a particular suite or even a particular group or tests or a single test.

Thank you.

[michaelklishin]

@bpint if you want me to make the change, I'd be happy to do it, you did 90% of the work already in any case. Please let me know, I cannot guess whether you are working on a PR or not.

[bpint]

@bpint if you want me to make the change, I'd be happy to do it, you did 90% of the work already in any case. Please let me know, I cannot guess whether you are working on a PR or not.

Thank you very much! I already submitted a PR just now.

[bpint]

@bpint if you want me to make the change, I'd be happy to do it, you did 90% of the work already in any case. Please let me know, I cannot guess whether you are working on a PR or not.

Thank you very much! I already submitted a PR just now.

sorry it took some time. my first PR to rabbitmq. #12888

@michaelklishin thanks again for your kind help!