Filesystem access error on Windows (eaccess)

[klettier]

Describe the bug

It seems we get the same symptoms as #12367 but on v4.0.3

2024-12-05 14:57:59.671000+03:00 [error] <0.546.0>     supervisor: {<0.546.0>,rabbit_amqqueue_sup}
2024-12-05 14:57:59.671000+03:00 [error] <0.546.0>     errorContext: child_terminated
2024-12-05 14:57:59.671000+03:00 [error] <0.546.0>     reason: {{badmatch,{error,eacces}},
2024-12-05 14:57:59.671000+03:00 [error] <0.546.0>              [{rabbit_classic_queue_index_v2,new_segment_file,3,
                                                                                                            
2024-12-05 14:57:59.671000+03:00 [error] <0.546.0>                   [{file,"rabbit_classic_queue_index_v2.erl"},{line,594}]},
2024-12-05 14:57:59.671000+03:00 [error] <0.546.0>               {rabbit_classic_queue_index_v2,publish,8,
2024-12-05 14:57:59.671000+03:00 [error] <0.546.0>                   [{file,"rabbit_classic_queue_index_v2.erl"},{line,583}]},
2024-12-05 14:57:59.671000+03:00 [error] <0.546.0>               {rabbit_variable_queue,maybe_write_index_to_disk,3,
2024-12-05 14:57:59.671000+03:00 [error] <0.546.0>                   [{file,"rabbit_variable_queue.erl"},{line,1841}]},
2024-12-05 14:57:59.671000+03:00 [error] <0.546.0>               {rabbit_variable_queue,publish_delivered1,5,
2024-12-05 14:57:59.671000+03:00 [error] <0.546.0>                   [{file,"rabbit_variable_queue.erl"},{line,1736}]},
2024-12-05 14:57:59.671000+03:00 [error] <0.546.0>               {rabbit_variable_queue,publish_delivered,4,
2024-12-05 14:57:59.671000+03:00 [error] <0.546.0>                   [{file,"rabbit_variable_queue.erl"},{line,542}]},
2024-12-05 14:57:59.671000+03:00 [error] <0.546.0>               {rabbit_priority_queue,publish_delivered,4,
2024-12-05 14:57:59.671000+03:00 [error] <0.546.0>                   [{file,"rabbit_priority_queue.erl"},{line,219}]},
2024-12-05 14:57:59.671000+03:00 [error] <0.546.0>               {rabbit_amqqueue_process,'-attempt_delivery/4-fun-0-',10,
2024-12-05 14:57:59.671000+03:00 [error] <0.546.0>                   [{file,"rabbit_amqqueue_process.erl"},{line,680}]},
2024-12-05 14:57:59.671000+03:00 [error] <0.546.0>               {rabbit_queue_consumers,deliver_to_consumer,4,
2024-12-05 14:57:59.671000+03:00 [error] <0.546.0>                   [{file,"rabbit_queue_consumers.erl"},{line,344}]}]}
2024-12-05 14:57:59.671000+03:00 [error] <0.546.0>     offender: [{pid,<0.547.0>},
2024-12-05 14:57:59.671000+03:00 [error] <0.546.0>                {id,rabbit_amqqueue},
2024-12-05 14:57:59.671000+03:00 [error] <0.546.0>                {mfargs,
2024-12-05 14:57:59.671000+03:00 [error] <0.546.0>                    {rabbit_amqqueue_process,start_link,
2024-12-05 14:57:59.671000+03:00 [error] <0.546.0>                        [{amqqueue,
2024-12-05 14:57:59.671000+03:00 [error] <0.546.0>                             {resource,<<"/">>,queue,
2024-12-05 14:57:59.671000+03:00 [error] <0.546.0>                                 <<"QUEUE_NAME">>},
2024-12-05 14:57:59.671000+03:00 [error] <0.546.0>                             true,false,none,
2024-12-05 14:57:59.671000+03:00 [error] <0.546.0>                             [{<<"x-queue-mode">>,longstr,<<"lazy">>},
2024-12-05 14:57:59.671000+03:00 [error] <0.546.0>                              {<<"x-queue-type">>,longstr,<<"classic">>}],
2024-12-05 14:57:59.671000+03:00 [error] <0.546.0>                             <0.542.0>,[],[],[],undefined,undefined,[],[],
2024-12-05 14:57:59.671000+03:00 [error] <0.546.0>                             stopped,0,[],<<"/">>,
2024-12-05 14:57:59.671000+03:00 [error] <0.546.0>                             #{user => <<"USER_NAME">>},
2024-12-05 14:57:59.671000+03:00 [error] <0.546.0>                             rabbit_classic_queue,#{}},
2024-12-05 14:57:59.671000+03:00 [error] <0.546.0>                         <0.545.0>]}},
2024-12-05 14:57:59.671000+03:00 [error] <0.546.0>                {restart_type,transient},
2024-12-05 14:57:59.671000+03:00 [error] <0.546.0>                {significant,true},
2024-12-05 14:57:59.671000+03:00 [error] <0.546.0>                {shutdown,600000},
2024-12-05 14:57:59.671000+03:00 [error] <0.546.0>                {child_type,worker}]
2024-12-05 14:57:59.671000+03:00 [error] <0.546.0> 
2024-12-05 14:57:59.687000+03:00 [error] <0.2616.0> Restarting crashed queue 'QUEUE_NAME' in vhost '/'.

We use both docker version and Windows exe directly installed on the server and for the moment only the windows one is facing this issue.

Any help would be appreciated to resolve this problem 🙏

Reproduction steps

1. Currently trying to reproduce in a test environment.

Expected behavior

No queue crashing

Additional context

It seems to run fine using docker RMQ management image, we only reproduced the issue using Windows machine and https://github.com/rabbitmq/rabbitmq-server/releases/tag/v4.0.3 exe file

[mkuratczyk]

This is unrelated to #12367 - the stack trace is completely different.

In your case, the key part of the error is eaccess, which is just a filesystem-level access error. Seems like something's wrong with file ownership/permissions.

[klettier]

Thx for the quick reply and sorry for the confusion
In the log we can see that the system is working for 15 minutes before getting the queue crashing for that reason, restarting the RMQ server process allows to consume the queue again without issue 🤔
Here are additional info

2024-12-05 14:57:59.609000+03:00 [error] <0.547.0> ** Reason for termination ==
2024-12-05 14:57:59.609000+03:00 [error] <0.547.0> ** {{badmatch,{error,eacces}},
2024-12-05 14:57:59.609000+03:00 [error] <0.547.0>     [{rabbit_classic_queue_index_v2,new_segment_file,3,
2024-12-05 14:57:59.609000+03:00 [error] <0.547.0>          [{file,"rabbit_classic_queue_index_v2.erl"},{line,594}]},
2024-12-05 14:57:59.609000+03:00 [error] <0.547.0>      {rabbit_classic_queue_index_v2,publish,8,
2024-12-05 14:57:59.609000+03:00 [error] <0.547.0>          [{file,"rabbit_classic_queue_index_v2.erl"},{line,583}]},
2024-12-05 14:57:59.609000+03:00 [error] <0.547.0>      {rabbit_variable_queue,maybe_write_index_to_disk,3,
2024-12-05 14:57:59.609000+03:00 [error] <0.547.0>          [{file,"rabbit_variable_queue.erl"},{line,1841}]},
2024-12-05 14:57:59.609000+03:00 [error] <0.547.0>      {rabbit_variable_queue,publish_delivered1,5,
2024-12-05 14:57:59.609000+03:00 [error] <0.547.0>          [{file,"rabbit_variable_queue.erl"},{line,1736}]},
2024-12-05 14:57:59.609000+03:00 [error] <0.547.0>      {rabbit_variable_queue,publish_delivered,4,
2024-12-05 14:57:59.609000+03:00 [error] <0.547.0>          [{file,"rabbit_variable_queue.erl"},{line,542}]},
2024-12-05 14:57:59.609000+03:00 [error] <0.547.0>      {rabbit_priority_queue,publish_delivered,4,
2024-12-05 14:57:59.609000+03:00 [error] <0.547.0>          [{file,"rabbit_priority_queue.erl"},{line,219}]},
2024-12-05 14:57:59.609000+03:00 [error] <0.547.0>      {rabbit_amqqueue_process,'-attempt_delivery/4-fun-0-',10,
2024-12-05 14:57:59.609000+03:00 [error] <0.547.0>          [{file,"rabbit_amqqueue_process.erl"},{line,680}]},
2024-12-05 14:57:59.609000+03:00 [error] <0.547.0>      {rabbit_queue_consumers,deliver_to_consumer,4,
2024-12-05 14:57:59.609000+03:00 [error] <0.547.0>          [{file,"rabbit_queue_consumers.erl"},{line,344}]}]}

We will check file system rights as you pointed 🙏

[mkuratczyk]

Then it's almost certainly what you pointed out yourself: It seems to run fine using docker RMQ management image, we only reproduced the issue using Windows machine. Most likely some security software on Windows messing stuff up.

[lukebakken]

@klettier we've seen eacces on Windows when anti-virus or other security software is in place. Please completely disable any of that and report back. Thanks.

[klettier]

Thanks a lot for the support guys, it makes sense to me 👍
I will get more information on Monday