Meraki Syslog Parser not parsing some events

[joseff8]

Is your feature request related to a problem? Please describe.
Meraki Syslog Parser not parsing some events

Solutions/CiscoMeraki/Parsers/CiscoMeraki.txt

Describe the solution you'd like
Would like the parser to handle additonal "LogType" events like:
"firewall"
"vpn_firewall"
"ip_flow_start"
"ip_flow_end"

Additional context
Not very good in KQL, however i was able to add the basics:
srcipaddr example:
LogType has "ip_flow_start", extract(@"src=([0-9.]+)\s",1,Substring),
LogType has "ip_flow_end", extract(@"src=([0-9.]+)\s",1,Substring),
LogType has "vpn_firewall", extract(@"src=([0-9.]+)\s",1,Substring),
LogType has "firewall", extract(@"src=([0-9.]+)\s",1,Substring),

Looks like the extract is the same for "flows" in each case

Added the LogType Events here as will
LogType !in ("urls", "airmarshal_events","security_event","ids-alerts", "events", "vpn_firewall", "firewall", "ip_flow_start", "ip_flow_end") and LogType !contains "flows", iif(isempty(LogType), "", LogType),

[github-actions]

Thank you for submitting an Issue to the Azure Sentinel GitHub repo! You should expect an initial response to your Issue from the team within 5 business days. Note that this response may be delayed during holiday periods. For urgent, production-affecting issues please raise a support ticket via the Azure Portal.

[v-muuppugund]

Hi @joseff8 , Thanks for flagging this issue, we will investigate this issue and get back to you with some updates by 29Nov23. Thanks!

[v-sudkharat]

Hi @joseff8, Sorry for the delay in response. Could you please share more detailed information about the issue? It would be helpful if you could add screenshots of the error you are encountering.

Thanks!

[v-sudkharat]

Hi @joseff8, we are waiting for your response on above comment. thanks!

[joseff8]

Hi @v-sudkharat, Could you please let me know what more information is needed? As stated the Meraki parser does not parse the events mentioned.
"firewall"
"vpn_firewall"
"ip_flow_start"
"ip_flow_end"

I added example changes to how the parser could include the events.

[v-muuppugund]

Hi @joseff8 ,Could you please share sample data in csv format for this parser issue and also screen shot of the error details.

[joseff8]

Hi @v-muuppugund,
Here is a sample.

Dec 15 07:38:10 10.47.129.253 171122330.11122336118 MX_Firewall vpn_firewall src=x.x.x.x dst=x.x.x.x protocol=udp sport=60940 dport=161 pattern: allow all

Dec 15 07:38:10 10.47.255.253 1711223390.202233603 MX_Firewall firewall src=x.x.x.x dst=x.x.x.x mac=00:xx:xx:xx:xx:xx protocol=udp sport=60944 dport=161 pattern: allow (dst x.x.x.x/24 || dst x.x.x.x/24 || (dst x.x.x.x/23 || dst x.x.x.x/24 || dst x.x.x.x/23) || dst x.x.x.x/26) && (src x.x.x.x/24)

Dec 15 07:38:10 10.47.255.253 1112233890.211223390 MX_Firewall ip_flow_start src=x.x.x.x dst=x.x.x.x protocol=udp sport=60944 dport=161 translated_dst_ip=x.x.x.x translated_port=161

As you can see the "type" of log is vpn_firewall, firewall and ip_flow_start
I was able to edit the parser to some degree to get these event "Types" parsed as well:
See attached file.
CiscoMeraki.txt

[v-muuppugund]

Hi @joseff8 ,Thanks ,will check details and do detailed analysis and will get back to you with an update

[v-muuppugund]

Hi @joseff8 ,After Analyzing the above sample data ,we are not parsing the above log types in parser,so its unable to detect,I am unable to replicate the same issue at my environment,Please add LogType has "firewall", extract(@"pattern: ([\S\s]+)", 1, Substring),
LogType has "ip_flow_start", extract(@"pattern: ([\S\s]+)", 1, Substring),
LogType has "ip_flow_end", extract(@"pattern: ([\S\s]+)", 1, Substring) and this will work ,need check whole logs can you share whole dump in csv format to v-muuppugund@microsoft.com,so will modify the query and share with you.

[v-muuppugund]

Hi @joseff8 ,I have updated the parser attached
Updated.Cisco.Meraki.txt

based on data,Could you please check and let me know if any issues

[antigrammatik]

@v-muuppugund That was really good and got most of the parsing working for the flow start and flow end events, thank you for putting this out here. There were a couple columns that weren't working correctly. The source and destination ports needed to be converted to int and the regex for the new fields needed to be updated to get the right value. Here is my updated version. It still needs a bit of work to make it perfect but it's probably 90% of the way there.

Updated Cisco Meraki 12-21-23.txt

[v-sudkharat]

@antigrammatik, thank you for your response.

[v-sudkharat]

Hi @joseff8, Could you please have a look on this comment- #9471 (comment) and let us know if your issue gets resolved. Thanks!

[v-muuppugund]

Hi @joseff8 / @antigrammatik , I have cross checked the parser ,above attached is the updated one,Gentle Reminder,Please check and let me know if any issues

[v-muuppugund]

Hi @joseff8 / @antigrammatik ,Genle Reminder,I have cross checked the parser ,above attached is the updated one, Please check and let me know if any issues, Proceeding with the changes, will raise pr

[joseff8]

@v-muuppugund, Thanks for the update. Sorry for the late reply, i have been on vacation. The code works better. We are still missing the changes for "pattern".

    Pattern = case(
                        isnotempty(column_ifexists("Pattern", "")), column_ifexists("Pattern", ""),
                        LogType has "flows", extract(@"pattern\: ([\S\s]+)", 1, Substring),
                        LogType has "vpn_firewall", extract(@"pattern\: ([\S\s]+)", 1, Substring),
                        LogType has "firewall", extract(@"pattern\: ([\S\s]+)", 1, Substring),
                        LogType has "ip_flow_start", extract(@"pattern\: ([\S\s]+)", 1, Substring),
                        LogType has "ip_flow_end", extract(@"pattern\: ([\S\s]+)", 1, Substring),
                        ""
                   ), 

[v-muuppugund]

@joseff8 ,Thanks, Will check on these changes ,will work on it and push it.

[v-muuppugund]

@joseff8 ,I have done the changes,testing with large dump of data for any issues as part of standard practice once done,will raise the PR and push it.

[v-muuppugund]

@joseff8 ,Still data ingesting is happening, Will share updates once Testing is completed and pr is pushed.

[azvella]

This really should make it into the production solution parser file. This was a great find and super helpful for me.

[abrbon]

Another issue with this parser is that it does not work for IPv6 addresses.

I fixed this by changing the following lines of code:

LogType has "flows", extract(@"src=([0-9\.]+)\s",1,Substring)

into

LogType contains "flow", extract(@"src=([^\s]+)",1,Substring)

Made the same changes for all "dst=" IP addresses.

And also it does not parse "ip_flow_start" and "ip_flow_end" LogTypes.
I fixed that also by simply changing all "flows" to "flow".

And finally i changed the code for "NetworkProtocol" from:

NetworkProtocol = case(NetworkProtocol has "tcp", "TCP",
                                NetworkProtocol has "udp", "UDP",
                                "")

to simply:

NetworkProtocol = toupper(NetworkProtocol)

and added the following to parse ICMP type protocol:

ICMPType = iif(NetworkProtocol contains "icmp", extract(@"type=([^\s]+)",1,Substring), ""),

[v-muuppugund]

Another issue with this parser is that it does not work for IPv6 addresses.

I fixed this by changing the following lines of code:

LogType has "flows", extract(@"src=([0-9\.]+)\s",1,Substring)

into

LogType contains "flow", extract(@"src=([^\s]+)",1,Substring)

Made the same changes for all "dst=" IP addresses.

And also it does not parse "ip_flow_start" and "ip_flow_end" LogTypes. I fixed that also by simply changing all "flows" to "flow".

And finally i changed the code for "NetworkProtocol" from:

NetworkProtocol = case(NetworkProtocol has "tcp", "TCP",
                                NetworkProtocol has "udp", "UDP",
                                "")

to simply:

NetworkProtocol = toupper(NetworkProtocol)

and added the following to parse ICMP type protocol:

ICMPType = iif(NetworkProtocol contains "icmp", extract(@"type=([^\s]+)",1,Substring), ""),

Sure ,will check and update you

[v-muuppugund]

Another issue with this parser is that it does not work for IPv6 addresses.
I fixed this by changing the following lines of code:
LogType has "flows", extract(@"src=([0-9\.]+)\s",1,Substring)
into
LogType contains "flow", extract(@"src=([^\s]+)",1,Substring)
Made the same changes for all "dst=" IP addresses.
And also it does not parse "ip_flow_start" and "ip_flow_end" LogTypes. I fixed that also by simply changing all "flows" to "flow".
And finally i changed the code for "NetworkProtocol" from:

NetworkProtocol = case(NetworkProtocol has "tcp", "TCP",
                                NetworkProtocol has "udp", "UDP",
                                "")

to simply:
NetworkProtocol = toupper(NetworkProtocol)
and added the following to parse ICMP type protocol:
ICMPType = iif(NetworkProtocol contains "icmp", extract(@"type=([^\s]+)",1,Substring), ""),

Sure ,will check and update you

Will be doing further analysis of Ip4 address ,Will update you post my analysis and changes.

[v-muuppugund]

Another issue with this parser is that it does not work for IPv6 addresses.
I fixed this by changing the following lines of code:
LogType has "flows", extract(@"src=([0-9\.]+)\s",1,Substring)
into
LogType contains "flow", extract(@"src=([^\s]+)",1,Substring)
Made the same changes for all "dst=" IP addresses.
And also it does not parse "ip_flow_start" and "ip_flow_end" LogTypes. I fixed that also by simply changing all "flows" to "flow".
And finally i changed the code for "NetworkProtocol" from:

NetworkProtocol = case(NetworkProtocol has "tcp", "TCP",
                                NetworkProtocol has "udp", "UDP",
                                "")

to simply:
NetworkProtocol = toupper(NetworkProtocol)
and added the following to parse ICMP type protocol:
ICMPType = iif(NetworkProtocol contains "icmp", extract(@"type=([^\s]+)",1,Substring), ""),

Sure ,will check and update you

Will be doing further analysis of Ip4 address ,Will update you post my analysis and changes.

Hi @abrbon / @joseff8 ,I have done the requested changes ,working on testing it,will update you

[v-muuppugund]

HI @joseff8 / @abrbon ,just want to update ,earlier data has been lost due to data retention period,so working on testing the new changes done once data ingestion completed

[v-rusraut]

Hi @joseff8,
We are still working on data ingestion, we will update you once testing complete.
Thanks

[v-rusraut]

Hi @joseff8,
We tried to ingest data, but we are not able to ingest data in few tables.
Please run below parser in your workspace where data available.
CiscoMeraki.txt and share working screen shots with us.
Thanks

[v-rusraut]

Hi @joseff8,
Please provide update on above comment.
Thanks

[v-rusraut]

Hi @joseff8 , Gentle Reminder: We are waiting for your response on this issue. If you still need to keep this issue active, please respond to it in the next 2 days. If we don't receive a response by 13-05-2024 date, we will be closing this issue.
Thanks!

[abrbon]

Hi @joseff8 , Gentle Reminder: We are waiting for your response on this issue. If you still need to keep this issue active, please respond to it in the next 2 days. If we don't receive a response by 13-05-2024 date, we will be closing this issue. Thanks!

Hi,
I checked and ran the parser on data from our client but unfortunately have not any data in our tables for “Pattern”.

However I still see the same issues I had with this parser like not recognizing IPv6 addresses and not parsing the “ip_flow_start” and “ip_flow_end” logtypes like I mentioned in my earlier posts.

[v-rusraut]

Hi @joseff8,
We have tried to resolve this issue in existing parser, but as it is enhancement/feature request in existing parser, so we have noted this feature request and will share this with our respective concern team. Currently we can't provide any ETA for this. but once we get any further info from our team will share update to you. So, closing this issue from GitHub. If you still need support for this issue, feel free to re-open it any time. Thank you for your co-operation.