[BUG] 4.9.1 initial startup of indexer fails due to noexec filesystems from STIG compliance + workaround

[IKatsu]

I saw another report of the same thing that said fixed in 4.9.0 but I am getting the exact same error on 4.9.1.
As per STIG/DOD policy the following filesystems are 'noexec'
/var/log
/var/log/audit
/var/tmp
/tmp (although in my case we decided to NOT do /tmp due to some legacy shenanigans , so it failed with /tmp NOT being noexec)

Oct 29 12:31:54 hostname systemd-entrypoint[47184]: WARNING: A terminally deprecated method in java.lang.System has been called
Oct 29 12:31:54 hostname systemd-entrypoint[47184]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/wazuh-indexer/lib/opensearch-2.13.0.jar)                                                                                                 
Oct 29 12:31:54 hostname systemd-entrypoint[47184]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch                                                                                                                                                    
Oct 29 12:31:54 hostname systemd-entrypoint[47184]: WARNING: System::setSecurityManager will be removed in a future release
Oct 29 12:31:55 hostname systemd-entrypoint[47184]: Oct 29, 2024 12:31:54 PM sun.util.locale.provider.LocaleProviderAdapter <clinit>
Oct 29 12:31:55 hostname systemd-entrypoint[47184]: WARNING: COMPAT locale provider will be removed in a future release
Oct 29 12:31:55 hostname systemd-entrypoint[47184]: WARNING: A terminally deprecated method in java.lang.System has been called
Oct 29 12:31:55 hostname systemd-entrypoint[47184]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/wazuh-indexer/lib/opensearch-2.13.0.jar)                                                                                                   
Oct 29 12:31:55 hostname systemd-entrypoint[47184]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security                                                                                                                                                      
Oct 29 12:31:55 hostname systemd-entrypoint[47184]: WARNING: System::setSecurityManager will be removed in a future release
Oct 29 12:32:06 hostname systemd-entrypoint[47184]: ERROR: [1] bootstrap checks failed
Oct 29 12:32:06 hostname systemd-entrypoint[47184]: [1]: system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk                                                                                                                
Oct 29 12:32:06 hostname systemd-entrypoint[47184]: ERROR: OpenSearch did not exit normally - check the logs at /var/log/wazuh-indexer/wazuh-cluster.log                                                                                                                                                 
Oct 29 12:32:06 hostname systemd-entrypoint[47184]: fatal error in thread [Thread-3], exiting
Oct 29 12:32:06 hostname systemd-entrypoint[47184]: java.lang.NoClassDefFoundError: Could not initialize class com.sun.jna.Native
Oct 29 12:32:06 hostname systemd-entrypoint[47184]:         at org.opensearch.systemd.Libsystemd.lambda$static$0(Libsystemd.java:48)
Oct 29 12:32:06 hostname systemd-entrypoint[47184]:         at java.base/java.security.AccessController.doPrivileged(AccessController.java:319)                                                                                                                                                          
Oct 29 12:32:06 hostname systemd-entrypoint[47184]:         at org.opensearch.systemd.Libsystemd.<clinit>(Libsystemd.java:47)
Oct 29 12:32:06 hostname systemd-entrypoint[47184]:         at org.opensearch.systemd.SystemdPlugin.sd_notify(SystemdPlugin.java:126)
Oct 29 12:32:06 hostname systemd-entrypoint[47184]:         at org.opensearch.systemd.SystemdPlugin.close(SystemdPlugin.java:152)
Oct 29 12:32:06 hostname systemd-entrypoint[47184]:         at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:89)
Oct 29 12:32:06 hostname systemd-entrypoint[47184]:         at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:131)
Oct 29 12:32:06 hostname systemd-entrypoint[47184]:         at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:114)
Oct 29 12:32:06 hostname systemd-entrypoint[47184]:         at org.opensearch.node.Node.close(Node.java:1690)
Oct 29 12:32:06 hostname systemd-entrypoint[47184]:         at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:89)
Oct 29 12:32:06 hostname systemd-entrypoint[47184]:         at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:131)
Oct 29 12:32:06 hostname systemd-entrypoint[47184]:         at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:81)
Oct 29 12:32:06 hostname systemd-entrypoint[47184]:         at org.opensearch.bootstrap.Bootstrap$4.run(Bootstrap.java:206)
Oct 29 12:32:06 hostname systemd-entrypoint[47184]: Caused by: java.lang.ExceptionInInitializerError: Exception java.lang.UnsatisfiedLinkError: /var/log/wazuh-indexer/tmp/jna938882129283533887.tmp: /var/log/wazuh-indexer/tmp/jna938882129283533887.tmp: failed to map segment from shared object [in >
Oct 29 12:32:06 hostname systemd-entrypoint[47184]:         at java.base/jdk.internal.loader.NativeLibraries.load(Native Method)
Oct 29 12:32:06 hostname systemd-entrypoint[47184]:         at java.base/jdk.internal.loader.NativeLibraries$NativeLibraryImpl.open(NativeLibraries.java:331)                                                                                                                                            
Oct 29 12:32:06 hostname systemd-entrypoint[47184]:         at java.base/jdk.internal.loader.NativeLibraries.loadLibrary(NativeLibraries.java:197)                                                                                                                                                       
Oct 29 12:32:06 hostname systemd-entrypoint[47184]:         at java.base/jdk.internal.loader.NativeLibraries.loadLibrary(NativeLibraries.java:139)                                                                                                                                                       
Oct 29 12:32:06 hostname systemd-entrypoint[47184]:         at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2418)
Oct 29 12:32:06 hostname systemd-entrypoint[47184]:         at java.base/java.lang.Runtime.load0(Runtime.java:852)
Oct 29 12:32:06 hostname systemd-entrypoint[47184]:         at java.base/java.lang.System.load(System.java:2025)
Oct 29 12:32:06 hostname systemd-entrypoint[47184]:         at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:1045)
Oct 29 12:32:06 hostname systemd-entrypoint[47184]:         at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:1015)
Oct 29 12:32:06 hostname systemd-entrypoint[47184]:         at com.sun.jna.Native.<clinit>(Native.java:221)
Oct 29 12:32:06 hostname systemd-entrypoint[47184]:         at java.base/java.lang.Class.forName0(Native Method)
Oct 29 12:32:06 hostname systemd-entrypoint[47184]:         at java.base/java.lang.Class.forName(Class.java:421)
Oct 29 12:32:06 hostname systemd-entrypoint[47184]:         at java.base/java.lang.Class.forName(Class.java:412)
Oct 29 12:32:06 hostname systemd-entrypoint[47184]:         at org.opensearch.bootstrap.Natives.<clinit>(Natives.java:60)
Oct 29 12:32:06 hostname systemd-entrypoint[47184]:         at org.opensearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:123)
Oct 29 12:32:06 hostname systemd-entrypoint[47184]:         at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:191)
Oct 29 12:32:06 hostname systemd-entrypoint[47184]:         at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404)
Oct 29 12:32:06 hostname systemd-entrypoint[47184]:         at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181)
Oct 29 12:32:06 hostname systemd-entrypoint[47184]:         at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172)
Oct 29 12:32:06 hostname systemd-entrypoint[47184]:         at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)                                                                                                                                                      
Oct 29 12:32:06 hostname systemd-entrypoint[47184]:         at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
Oct 29 12:32:06 hostname systemd-entrypoint[47184]:         at org.opensearch.cli.Command.main(Command.java:101)
Oct 29 12:32:06 hostname systemd-entrypoint[47184]:         at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138)
Oct 29 12:32:06 hostname systemd-entrypoint[47184]:         at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104)
Oct 29 12:32:06 hostname systemd[1]: wazuh-indexer.service: Main process exited, code=exited, status=1/FAILURE

To Reproduce
Install and try to start on STIG compliant machine.

Expected behavior
Successful starting on STIG compliant machine.

Host/Environment (please complete the following information):

• OS: RHEL
• Version: 9.4

Additional context
I can't use any of the provided automatic installation options so this was done via the step by step install documentation.
A slightly modified workaround from the older report works as a workaround:

mkdir -p /var/lib/wazuh-indexer/tmp
chown wazuh-indexer:wazuh-indexer /var/lib/wazuh-indexer/tmp
echo OPENSEARCH_TMPDIR=/var/lib/wazuh-indexer/tmp | sudo tee /etc/sysconfig/wazuh-indexer

[AlexRuiz7]

Hello @IKatsu

I assume you refer to this other issue. We'll revisit the problem.

It's good to know that the workaround works. We'll try to make that not necessary for users running wazuh-indexer under security hardened operating systems.

[IKatsu]

That's the other issue I was referring to yes.
I figured I'd report it because hardening and security practices in general are becoming more and more important :)

[QU3B1M]

This issue seems to be caused by the use of /var/log/tmp as the temporary archive storage directory. This behavior was implemented in the fix applied for 4.9.0, addressing cases where /tmp is set to noexec, we did not evaluate cases where /var/log or /var/tmp could be noexec too.

A potential solution is to store the tmp/ directory within the data directory (/var/lib/wazuh-indexer). We might consider implementing this using systemd-tmpfiles to delegate the directory creation to the tool.

[QU3B1M]

Fix applied in PR #533 is currently under testing.

This update does not utilize the systemd-tmpfiles tool due to potential inconsistencies when running on sysv-only systems, where the tool might not be installed.

While all the Linux-based OS tested so far have this tool pre-installed, we should analyze which OS might be incompatible with this implementation and the potential issues it could cause.