Map variable within message field with parse configuration CEF logging

[krabelize]

Hi all,

Since fluentd is fundamental for Sentinel oms-agent log parsing, this Q&A section is probably where the experts are :)

I have successfully configured and installed the oms-agent and the Microsoft CEF python log forward script on a Ubuntu 20.04 x64 VM to forward Fortinet Analyzer firewall logging to Sentinel. I receive the CEF logging in Sentinel. Most fields are mapped correctly. However, not all fields within the CEF message field are mapped, such as ad.srccountry variable. This field is now in the AdditionalExtensions variable along with other data.

Within CEF -> message -> AdditionalExtensions there is the ad.srccountry variable. I want the "ad.srccountry" variable mapped to cs1 (DeviceCustomString1) in Sentinel. I got this idea from a list of all supported CEF message mapping field.

However, I cannot figure out how to configure this additional parse within the message field.

My configuration:

vi /etc/opt/microsoft/omsagent/<tenantID>/conf/omsagent.d/security_events.conf 

<source>
  type syslog
  port 25226
  bind 127.0.0.1
  protocol_type tcp
  tag oms.security
  format /(?<time>(?:\w+ +){2,3}(?:\d+:){2}\d+|\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}.[\w\-\:\+]{3,12}):?\s*(?:(?<host>[^: ]+) ?:?)?\s*(?<ident>.*CEF.+?(?=0\|)|%ASA[0-9\-]{8,10})\s*:?(?<message>0\|.*|.*)/
  <parse>
     message_format auto
  </parse>
</source>

<filter oms.security.**>
  type filter_syslog_security
</filter>

Works (raw message and primary CEF filter): https://regex101.com/r/rdf4iP/1

vi /etc/opt/microsoft/omsagent/<tenantID>/conf/omsagent.d/syslog.conf

<source>
  @type syslog
  port 25224
  bind 127.0.0.1
  protocol_type tcp
  tag oms.syslog
</source>

<filter oms.syslog.**>
  @type parser
  key_name message
  format /.*ad.srccountry=(?<cs1>.*?)\s.*$/
</filter>

Raw message and secondary ad.srccountry filter: https://regex101.com/r/v6iZHs/1

systemctl restart omsagent-<TenantID>.service && systemctl restart syslog

But the DeviceCustomString1 (cs1) field in CommonSecurityLog stays empty in Sentinel. Any tips or tricks how I can get this to work?

[fujimotos]

But the DeviceCustomString1 (cs1) field in CommonSecurityLog stays empty in Sentinel. Any tips or tricks how I can get this to work?

I took some time this evening trying to replicate this issue. Attached
is the configuration I used in my replication attempt.

As far as I can tell, it just worked, so I'm not sure what is the exact
problem here.

In particular, I poured the test data directly to Fluentd:

$ cat test.data > /dev/tcp/127.0.0.1/25226

... which resulted in the following record:

{"host":"evb-nl-ams-fw01_FG6H0E","ident":"CEF","message":"...","cs1":"Norway"}

As you can see, "cs1" got parsed properly. Isn't this what you are expecting?

fluent.conf

<source>
  @type syslog
  port 25226
  bind 127.0.0.1
  protocol_type tcp
  tag oms.**
  <parse>
     message_format auto
  </parse>
</source>

<filter oms.**>
  @type parser
  key_name message
  reserve_data true
  <parse>
    @type regexp
    expression /.*ad.srccountry=(?<cs1>.*?)\s.*$/
  </parse>
</filter>

<match oms.**>
  @type stdout
</match>

test.data

<0>Mar  6 23:10:08 evb-nl-ams-fw01_FG6H0E CEF: 0|Fortinet|FortiGate-600E|3.2.1,build1204 (DA)|1239233|app-ctrl utm pass|4|start=Mar 06 2022 23:10:08 logver=232123123 deviceExternalId=FGBHAAAAAA1 dvchost=production01 ad.vd=prod01 ad.eventtime=1646604609212138288 ad.tz=+0100 ad.logid=0000000013 cat=traffic ad.subtype=forward deviceSeverity=notice src=10.0.0.2 spt=37628 deviceInboundInterface=Dev LAN ad.srcintfrole=lan dst=70.70.70.2 dpt=443 deviceOutboundInterface=VL-Develop1 ad.dstintfrole=undefined ad.srccountry=Norway ad.dstcountry=Denmark externalID=10999383 proto=6 act=close ad.policyid=67 ad.policytype=policy ad.poluuid=dca6b9de-adc11-51e2-d100-3cc871c8edae ad.policyname=prod01 to Internet app=HTTPS ad.trandisp=noop ad.duration=1 out=1446 in=4377 ad.sentpkt=11 ad.rcvdpkt=12 ad.shapingpolicyid=1 ad.shapersentname=Production-01 ad.shaperdropsentbyte=0 ad.shaperrcvdname=Filter-WAN ad.shaperdroprcvdbyte=0 ad.appcat=unscanned ad.mastersrcmac=a1:a1:a1:a1:a1:a1 ad.srcmac=a1:a1:a1:a1:a1:a1 ad.srcserver=0 tz="+0100