Unable to send messages to Sentinel

[aabuhasna]

Describe the bug
I am unable to send messages to Sentinel, I am using Ubuntu 22.04 with the latest update, syslog-ng (with the latest update) and omsagent, I have configured my machines to send CEF messages to my Ubuntu server, and it is working as a charm, using the following formula:

template t_per_host {
template("${ISODATE} ${HOST} ${MSGHDR}${MESSAGE}\n");
template_escape(no);
};

destination d_per_host {
file(
"/var/log/syslog-ng/${HOST}/${PROGRAM}"
template(t_per_host)
create-dirs(yes)
);
};

if you navigate to the actual path you will see the logs based on a specific condition I specified without any issue.

unfortunately, when I opened Sentinel I saw the agent status "not connected" (I checked firewall settings and service status and everything seems to be working fine). when I run the troubleshooter script which is designed for RHEL it says I am not using CEF (unfortunately I am not able to share the client system but all messages contain CEF).

To Reproduce
Steps to reproduce the behavior:

1. install syslog-ng and configure it to use the template I have mentioned above
2. run omsagent installation script
3. make sure that both omsagent and syslog-ng service is running
4. Make sure related ports are allowed in the firewall
5. Make sure syslog-ng is getting the related logs
6. run the troubleshooter script and see the unexpected result

Expected behavior
Sync with the agent

Desktop (please complete the following information):

• OS: Ubuntu server
• Version 22.04

Additional context
Logs:

--2024-01-23 09:47:52--  https://raw.githubusercontent.com/Azure/Azure-Sentinel/master/DataConnectors/CEF/cef_troubleshoot.py
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.111.133, 185.199.108.133, 185.199.110.133, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.111.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 39931 (39K) [text/plain]
Saving to: ‘cef_troubleshoot.py’

cef_troubleshoot.py                                  100%[===================================================================================================================>]  39.00K  --.-KB/s    in 0.03s

2024-01-23 09:47:53 (1.51 MB/s) - ‘cef_troubleshoot.py’ saved [39931/39931]

Note this script should be run in elevated privileges
Please validate you are sending CEF messages to agent machine.
Trying to use the 'locate' command to locate omsagent
Located 'omsagent'
Located security_events.conf
Validating /etc/opt/microsoft/omsagent/XXX-XXX-XXX-XXXX/conf/omsagent.d/security_events.conf content.
Current content of the daemon configuration is:
<source>
  type syslog
  port 25226
  bind 127.0.0.1
  protocol_type tcp
  tag oms.security
  format /(?<time>(?:\w+ +){2,3}(?:\d+:){2}\d+|\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}.[\w\-\:\+]{3,12}):?\s*(?:(?<host>[^: ]+) ?:?)?\s*(?<ident>.*CEF.+?(?=0\|)|%ASA[0-9\-]{8,10})\s*:?(?<message>0\|.*|.*)/
  <parse>
     message_format auto
  </parse>
</source>


<filter oms.security.**>
  type filter_syslog_security
</filter>

Omsagent event configuration content is valid
File permissions valid
omsagent security configuration supports Cisco ASA parsing

OMS Agent syslog field mapping is correct

Checking if firewalld is installed.
systemctl status firewalld
Unit firewalld.service could not be found.
Checking if security enhanced linux is enabled
getenforce
sudo: getenforce: command not found
Could not execute 'getenforce' to check if security enhanced linux is enabled
please install 'policycoreutils' package and run the troubleshoot script again
[]
Notice: rsyslog is not running but found configuration directory for it.
[]
Notice: syslog-ng is not running but found configuration directory for it.
Simulating mock data which you can find in your workspace
This will take 60 seconds.
sudo tcpdump -A -ni any port 25226 -vv
b'tcpdump: listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes\n'
Could not locate "CEF" message in tcpdump
Please make sure that traffic to the syslog daemon on port 514 and to the OMS agent on port 25226 are enabled on the internal firewall of the machine

Your machine is auto synced with the portal. In case you are using the same machine to forward both plain Syslog and CEF messages, please make sure to manually change the Syslog configuration file to avoid duplicated data and disable the auto sync with the portal. Otherwise all changes will be overwritten.
To disable the auto sync with the portal please run: "sudo su omsagent -c 'python /opt/microsoft/omsconfig/Scripts/OMS_MetaConfigHelper.py --disable'"
For more on how to avoid duplicated syslog and CEF logs please visit: https://docs.microsoft.com/azure/sentinel/connect-cef-agent?tabs=rsyslog
Validating that the OMI vulnerability patch is installed.
Protected from OMI vulnerability, patch is installed.
[]
[]
Notice: syslog-ng is not running but found configuration directory for it.
No daemon was found on the machine
Completed troubleshooting.
Please check Log Analytics to see if your logs are arriving. All events streamed from these appliances appear in raw form in Log Analytics under CommonSecurityLog type
Notice: If no logs appear in workspace try looking at omsagent logs:
tail -f /var/opt/microsoft/omsagent/XXX-XXXX-XXXX-XXX/log/omsagent.log
Warning: Make sure that the logs you send comply with RFC 5424.

[github-actions]

Thank you for submitting an Issue to the Azure Sentinel GitHub repo! You should expect an initial response to your Issue from the team within 5 business days. Note that this response may be delayed during holiday periods. For urgent, production-affecting issues please raise a support ticket via the Azure Portal.

[v-sudkharat]

Hi @aabuhasna, Thanks for flagging this issue, we will investigate this issue and get back to you with some updates by 29-01-2024. Thanks!

[v-muuppugund]

Hi @aabuhasna ,Could you please share any solution specific this connector is configured to ingest in sentinel or generic cef logs ingestion?

[aabuhasna]

Hi @v-muuppugund

I was able to make it work through the following steps:

• If syslog-ng is already configured to use port 514 TCP or UDP before installing the agent, you need to comment this out since MS Agent will try to make its own rule which will make a conflict with the connector.
• Make sure to use the build-in aliases that MS used after installing the agent otherwise, it will not work (even if the new name is matched in the Agent config file inside syslog-ng
• I could not find a way to make MS agent forward the logs inside stored logs in syslog-ng, it only forwarded logs that came from the remote machine.