Skip to content

Commit

Permalink
Parsers/ASimDns/Parsers/*: addressing further review findings
Browse files Browse the repository at this point in the history 
  * addressing further review findings by vakohl
  * added sample and test outputs for the parsers

Signed-off-by: Janos Szigetvari <janos.szigetvari@nxlog.org>
  • Loading branch information
@jszigetvari
jszigetvari committed Aug 25, 2023
1 parent 77d399a commit 1b01c19
Show file tree
Hide file tree
Showing 9 changed files with 1,200 additions and 8 deletions.
14 changes: 10 additions & 4 deletions Parsers/ASimDns/Parsers/ASimDnsMicrosoftNXlog.yaml
View file
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
Parser:
Title: DNS activity ASIM parser for Microsoft DNS logs collected using NXlog
Version: '0.5'
LastUpdated: Jul 10 2023
LastUpdated: Aug 25 2023
Product:
Name: MS DNS Events
Normalization:
Expand Down Expand Up @@ -237,7 +237,7 @@ ParserQuery: |
DnsResponseCode=toint(DnsResponseCode),
SrcPortNumber=toint(Port_s),
DvcHostname=Dvc,
DvcIpAddr=iff(HostIP_s == "","0.0.0.0",HostIP_s),
DvcIpAddr=HostIP_s,
EventEndTime=EventStartTime,
EventProduct = "DNS Server",
EventSchemaVersion = "0.1.7",
Expand Down Expand Up @@ -268,10 +268,16 @@ ParserQuery: |
, DnsQueryType between (261 .. 32767), 'Unassigned'
, 'Unassigned'),
EventResult=iff (EventResult == "Based on RCODE", iff(DnsResponseCode == 0, "Success", "Failure"),EventResult)
| extend
| extend
// Aliases
IpAddr = SrcIpAddr,
Src = SrcIpAddr
Src = SrcIpAddr,
// Backward compatibility
Query = DnsQuery,
QueryType = DnsQueryType,
QueryTypeName = DnsQueryTypeName,
ResponseCode = DnsResponseCode,
ResponseCodeName = DnsResponseCodeName
| project-away
*_s, *_d, QTypeName, TenantId, SourceSystem, MG, ManagementGroupName, Computer, RawData
};
Expand Down
23 changes: 19 additions & 4 deletions Parsers/ASimDns/Parsers/vimDnsMicrosoftNXlog.yaml
View file
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
Parser:
Title: DNS activity ASIM filtering parser for Microsoft DNS logs collected using NXlog
Version: '0.5'
LastUpdated: Jul 10 2023
LastUpdated: Aug 25 2023
Product:
Name: MS DNS Events
Normalization:
Expand Down Expand Up @@ -284,7 +284,7 @@ ParserQuery: |
DnsResponseCode=toint(DnsResponseCode),
SrcPortNumber=toint(Port_s),
DvcHostname=Dvc,
DvcIpAddr=iff(HostIP_s == "","0.0.0.0",HostIP_s),
DvcIpAddr=HostIP_s,
EventEndTime=EventStartTime,
EventProduct = "DNS Server",
EventSchemaVersion = "0.1.7",
Expand Down Expand Up @@ -317,8 +317,23 @@ ParserQuery: |
| extend
// Aliases
IpAddr = SrcIpAddr,
Src = SrcIpAddr
Src = SrcIpAddr,
// Backward compatibility
Query = DnsQuery,
QueryType = DnsQueryType,
QueryTypeName = DnsQueryTypeName,
ResponseCode = DnsResponseCode,
ResponseCodeName = DnsResponseCodeName
| project-away
*_s, *_d, QTypeName, TenantId, SourceSystem, MG, ManagementGroupName, Computer, RawData
};
ASimDnsMicrosoftNXLog (starttime=datetime(null), endtime=datetime(null), srcipaddr='', domain_has_any=dynamic([]), responsecodename='', response_has_ipv4='*', response_has_any_prefix=dynamic([]), eventtype='Query', disabled=false)
ASimDnsMicrosoftNXLog (
starttime=datetime(null),
endtime=datetime(null),
srcipaddr='',
domain_has_any=dynamic([]),
responsecodename='',
response_has_ipv4='*',
response_has_any_prefix=dynamic([]),
eventtype='Query',
disabled=false)
10 changes: 10 additions & 0 deletions Parsers/ASimDns/Tests/Microsoft_DNS Server_Dns_DataTest_ASimDnsMicrosoftNXLog.csv
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
Result
"(1) Warning: Empty value in 297 records (36.13%) in mandatory field [Src] (Schema:Dns)"
"(2) Info: Empty value in 20 records (2.43%) in optional field [SrcPortNumber] (Schema:Dns)"
"(2) Info: Empty value in 219 records (26.64%) in optional field [EventOriginalUid] (Schema:Dns)"
"(2) Info: Empty value in 297 records (36.13%) in recommended field [SrcIpAddr] (Schema:Dns)"
"(2) Info: Empty value in 31 records (3.77%) in optional field [DnsResponseName] (Schema:Dns)"
"(2) Info: Empty value in 318 records (38.69%) in optional field [DnsFlagsRecursionDesired] (Schema:Dns)"
"(2) Info: Empty value in 536 records (65.21%) in optional field [DnsFlagsAuthenticated] (Schema:Dns)"
"(2) Info: Empty value in 536 records (65.21%) in optional field [DnsFlagsAuthoritative] (Schema:Dns)"
"(2) Info: Empty value in 726 records (88.32%) in optional field [DnsResponseCode] (Schema:Dns)"
8 changes: 8 additions & 0 deletions Parsers/ASimDns/Tests/Microsoft_DNS Server_Dns_DataTest_vimDnsMicrosoftNXLog.csv
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
Result
"(1) Warning: Empty value in 201 records (100.0%) in mandatory field [Src] (Schema:Dns)"
"(2) Info: Empty value in 11 records (5.47%) in optional field [DnsFlagsRecursionDesired] (Schema:Dns)"
"(2) Info: Empty value in 11 records (5.47%) in optional field [DnsResponseName] (Schema:Dns)"
"(2) Info: Empty value in 201 records (100.0%) in optional field [DnsFlagsAuthenticated] (Schema:Dns)"
"(2) Info: Empty value in 201 records (100.0%) in optional field [DnsFlagsAuthoritative] (Schema:Dns)"
"(2) Info: Empty value in 201 records (100.0%) in optional field [DnsResponseCode] (Schema:Dns)"
"(2) Info: Empty value in 201 records (100.0%) in recommended field [SrcIpAddr] (Schema:Dns)"
118 changes: 118 additions & 0 deletions Parsers/ASimDns/Tests/Microsoft_DNS Server_Dns_SchemaTest_ASimDnsMicrosoftNXLog.csv
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,118 @@
Result
"(1) Warning: Missing recommended field [DnsQueryClassName]"
"(1) Warning: Missing recommended field [Dst]"
"(1) Warning: Missing recommended field [DvcDomain]"
"(1) Warning: Missing recommended field [EventUid]"
"(1) Warning: Missing recommended field [SrcDomain]"
"(1) Warning: Missing recommended field [SrcHostname]"
"(2) Info: Missing optional alias [DomainCategory] aliasing non-existent column [UrlCategory]"
"(2) Info: Missing optional alias [Duration] aliasing non-existent column [DnsNetworkDuration]"
"(2) Info: Missing optional alias [Process] aliasing non-existent column [SrcProcessName]"
"(2) Info: Missing optional alias [SessionId] aliasing non-existent column [DnsSessionId]"
"(2) Info: Missing optional alias [User] aliasing non-existent column [SrcUsername]"
"(2) Info: Missing optional field [AdditionalFields]"
"(2) Info: Missing optional field [DnsFlagsCheckingDisabled]"
"(2) Info: Missing optional field [DnsFlagsRecursionAvailable]"
"(2) Info: Missing optional field [DnsFlagsTruncated]"
"(2) Info: Missing optional field [DnsFlagsZ]"
"(2) Info: Missing optional field [DnsNetworkDuration]"
"(2) Info: Missing optional field [DnsQueryClass]"
"(2) Info: Missing optional field [DnsResponseIpCity]"
"(2) Info: Missing optional field [DnsResponseIpCountry]"
"(2) Info: Missing optional field [DnsResponseIpLatitude]"
"(2) Info: Missing optional field [DnsResponseIpLongitude]"
"(2) Info: Missing optional field [DnsResponseIpRegion]"
"(2) Info: Missing optional field [DnsSessionId]"
"(2) Info: Missing optional field [DstDescription]"
"(2) Info: Missing optional field [DstDeviceType]"
"(2) Info: Missing optional field [DstDomain]"
"(2) Info: Missing optional field [DstDvcId]"
"(2) Info: Missing optional field [DstDvcScopeId]"
"(2) Info: Missing optional field [DstDvcScope]"
"(2) Info: Missing optional field [DstFQDN]"
"(2) Info: Missing optional field [DstGeoCity]"
"(2) Info: Missing optional field [DstGeoCountry]"
"(2) Info: Missing optional field [DstGeoLatitude]"
"(2) Info: Missing optional field [DstGeoLongitude]"
"(2) Info: Missing optional field [DstGeoRegion]"
"(2) Info: Missing optional field [DstHostname]"
"(2) Info: Missing optional field [DstIpAddr]"
"(2) Info: Missing optional field [DstOriginalRiskLevel]"
"(2) Info: Missing optional field [DstPortNumber]"
"(2) Info: Missing optional field [DstRiskLevel]"
"(2) Info: Missing optional field [DvcAction]"
"(2) Info: Missing optional field [DvcDescription]"
"(2) Info: Missing optional field [DvcFQDN]"
"(2) Info: Missing optional field [DvcId]"
"(2) Info: Missing optional field [DvcInterface]"
"(2) Info: Missing optional field [DvcMacAddr]"
"(2) Info: Missing optional field [DvcOriginalAction]"
"(2) Info: Missing optional field [DvcOsVersion]"
"(2) Info: Missing optional field [DvcOs]"
"(2) Info: Missing optional field [DvcScopeId]"
"(2) Info: Missing optional field [DvcScope]"
"(2) Info: Missing optional field [DvcZone]"
"(2) Info: Missing optional field [EventMessage]"
"(2) Info: Missing optional field [EventOriginalSeverity]"
"(2) Info: Missing optional field [EventOriginalSubType]"
"(2) Info: Missing optional field [EventOwner]"
"(2) Info: Missing optional field [EventProductVersion]"
"(2) Info: Missing optional field [EventReportUrl]"
"(2) Info: Missing optional field [EventSeverity]"
"(2) Info: Missing optional field [NetworkProtocolVersion]"
"(2) Info: Missing optional field [RuleName]"
"(2) Info: Missing optional field [RuleNumber]"
"(2) Info: Missing optional field [Rule]"
"(2) Info: Missing optional field [SrcDescription]"
"(2) Info: Missing optional field [SrcDeviceType]"
"(2) Info: Missing optional field [SrcDvcId]"
"(2) Info: Missing optional field [SrcDvcScopeId]"
"(2) Info: Missing optional field [SrcDvcScope]"
"(2) Info: Missing optional field [SrcFQDN]"
"(2) Info: Missing optional field [SrcGeoCity]"
"(2) Info: Missing optional field [SrcGeoCountry]"
"(2) Info: Missing optional field [SrcGeoLatitude]"
"(2) Info: Missing optional field [SrcGeoLongitude]"
"(2) Info: Missing optional field [SrcGeoRegion]"
"(2) Info: Missing optional field [SrcOriginalRiskLevel]"
"(2) Info: Missing optional field [SrcOriginalUserType]"
"(2) Info: Missing optional field [SrcProcessGuid]"
"(2) Info: Missing optional field [SrcProcessId]"
"(2) Info: Missing optional field [SrcProcessName]"
"(2) Info: Missing optional field [SrcRiskLevel]"
"(2) Info: Missing optional field [SrcUserAWSId]"
"(2) Info: Missing optional field [SrcUserAadId]"
"(2) Info: Missing optional field [SrcUserId]"
"(2) Info: Missing optional field [SrcUserOktaId]"
"(2) Info: Missing optional field [SrcUserScopeId]"
"(2) Info: Missing optional field [SrcUserScope]"
"(2) Info: Missing optional field [SrcUserSessionId]"
"(2) Info: Missing optional field [SrcUserSid]"
"(2) Info: Missing optional field [SrcUserType]"
"(2) Info: Missing optional field [SrcUserUid]"
"(2) Info: Missing optional field [SrcUsername]"
"(2) Info: Missing optional field [TenantId]"
"(2) Info: Missing optional field [ThreatCategory]"
"(2) Info: Missing optional field [ThreatConfidence]"
"(2) Info: Missing optional field [ThreatField]"
"(2) Info: Missing optional field [ThreatFirstReportedTime]"
"(2) Info: Missing optional field [ThreatId]"
"(2) Info: Missing optional field [ThreatIpAddr]"
"(2) Info: Missing optional field [ThreatIsActive]"
"(2) Info: Missing optional field [ThreatLastReportedTime]"
"(2) Info: Missing optional field [ThreatName]"
"(2) Info: Missing optional field [ThreatOriginalConfidence]"
"(2) Info: Missing optional field [ThreatOriginalRiskLevel]"
"(2) Info: Missing optional field [ThreatRiskLevel]"
"(2) Info: Missing optional field [UrlCategory]"
"(2) Info: Missing recommended alias [Hostname] aliasing non-existent column [SrcHostname]"
"(2) Info: extra unnormalized column [Category]"
"(2) Info: extra unnormalized column [EventReceivedTime_t]"
"(2) Info: extra unnormalized column [Level]"
"(2) Info: extra unnormalized column [ProviderGuid_g]"
"(2) Info: extra unnormalized column [QueryTypeName]"
"(2) Info: extra unnormalized column [QueryType]"
"(2) Info: extra unnormalized column [Query]"
"(2) Info: extra unnormalized column [ResponseCodeName]"
"(2) Info: extra unnormalized column [ResponseCode]"
"(2) Info: extra unnormalized column [_ResourceId]"
119 changes: 119 additions & 0 deletions Parsers/ASimDns/Tests/Microsoft_DNS Server_Dns_SchemaTest_vimDnsMicrosoftNXLog.csv
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,119 @@
Result
"(1) Warning: Missing recommended field [DnsQueryClassName]"
"(1) Warning: Missing recommended field [Dst]"
"(1) Warning: Missing recommended field [DvcDomain]"
"(1) Warning: Missing recommended field [EventUid]"
"(1) Warning: Missing recommended field [SrcDomain]"
"(1) Warning: Missing recommended field [SrcHostname]"
"(2) Info: Missing optional alias [DomainCategory] aliasing non-existent column [UrlCategory]"
"(2) Info: Missing optional alias [Duration] aliasing non-existent column [DnsNetworkDuration]"
"(2) Info: Missing optional alias [Process] aliasing non-existent column [SrcProcessName]"
"(2) Info: Missing optional alias [SessionId] aliasing non-existent column [DnsSessionId]"
"(2) Info: Missing optional alias [User] aliasing non-existent column [SrcUsername]"
"(2) Info: Missing optional field [AdditionalFields]"
"(2) Info: Missing optional field [DnsFlagsCheckingDisabled]"
"(2) Info: Missing optional field [DnsFlagsRecursionAvailable]"
"(2) Info: Missing optional field [DnsFlagsTruncated]"
"(2) Info: Missing optional field [DnsFlagsZ]"
"(2) Info: Missing optional field [DnsNetworkDuration]"
"(2) Info: Missing optional field [DnsQueryClass]"
"(2) Info: Missing optional field [DnsResponseIpCity]"
"(2) Info: Missing optional field [DnsResponseIpCountry]"
"(2) Info: Missing optional field [DnsResponseIpLatitude]"
"(2) Info: Missing optional field [DnsResponseIpLongitude]"
"(2) Info: Missing optional field [DnsResponseIpRegion]"
"(2) Info: Missing optional field [DnsSessionId]"
"(2) Info: Missing optional field [DstDescription]"
"(2) Info: Missing optional field [DstDeviceType]"
"(2) Info: Missing optional field [DstDomain]"
"(2) Info: Missing optional field [DstDvcId]"
"(2) Info: Missing optional field [DstDvcScopeId]"
"(2) Info: Missing optional field [DstDvcScope]"
"(2) Info: Missing optional field [DstFQDN]"
"(2) Info: Missing optional field [DstGeoCity]"
"(2) Info: Missing optional field [DstGeoCountry]"
"(2) Info: Missing optional field [DstGeoLatitude]"
"(2) Info: Missing optional field [DstGeoLongitude]"
"(2) Info: Missing optional field [DstGeoRegion]"
"(2) Info: Missing optional field [DstHostname]"
"(2) Info: Missing optional field [DstIpAddr]"
"(2) Info: Missing optional field [DstOriginalRiskLevel]"
"(2) Info: Missing optional field [DstPortNumber]"
"(2) Info: Missing optional field [DstRiskLevel]"
"(2) Info: Missing optional field [DvcAction]"
"(2) Info: Missing optional field [DvcDescription]"
"(2) Info: Missing optional field [DvcFQDN]"
"(2) Info: Missing optional field [DvcId]"
"(2) Info: Missing optional field [DvcInterface]"
"(2) Info: Missing optional field [DvcMacAddr]"
"(2) Info: Missing optional field [DvcOriginalAction]"
"(2) Info: Missing optional field [DvcOsVersion]"
"(2) Info: Missing optional field [DvcOs]"
"(2) Info: Missing optional field [DvcScopeId]"
"(2) Info: Missing optional field [DvcScope]"
"(2) Info: Missing optional field [DvcZone]"
"(2) Info: Missing optional field [EventMessage]"
"(2) Info: Missing optional field [EventOriginalSeverity]"
"(2) Info: Missing optional field [EventOriginalSubType]"
"(2) Info: Missing optional field [EventOwner]"
"(2) Info: Missing optional field [EventProductVersion]"
"(2) Info: Missing optional field [EventReportUrl]"
"(2) Info: Missing optional field [EventSeverity]"
"(2) Info: Missing optional field [NetworkProtocolVersion]"
"(2) Info: Missing optional field [RuleName]"
"(2) Info: Missing optional field [RuleNumber]"
"(2) Info: Missing optional field [Rule]"
"(2) Info: Missing optional field [SrcDescription]"
"(2) Info: Missing optional field [SrcDeviceType]"
"(2) Info: Missing optional field [SrcDvcId]"
"(2) Info: Missing optional field [SrcDvcScopeId]"
"(2) Info: Missing optional field [SrcDvcScope]"
"(2) Info: Missing optional field [SrcFQDN]"
"(2) Info: Missing optional field [SrcGeoCity]"
"(2) Info: Missing optional field [SrcGeoCountry]"
"(2) Info: Missing optional field [SrcGeoLatitude]"
"(2) Info: Missing optional field [SrcGeoLongitude]"
"(2) Info: Missing optional field [SrcGeoRegion]"
"(2) Info: Missing optional field [SrcOriginalRiskLevel]"
"(2) Info: Missing optional field [SrcOriginalUserType]"
"(2) Info: Missing optional field [SrcProcessGuid]"
"(2) Info: Missing optional field [SrcProcessId]"
"(2) Info: Missing optional field [SrcProcessName]"
"(2) Info: Missing optional field [SrcRiskLevel]"
"(2) Info: Missing optional field [SrcUserAWSId]"
"(2) Info: Missing optional field [SrcUserAadId]"
"(2) Info: Missing optional field [SrcUserId]"
"(2) Info: Missing optional field [SrcUserOktaId]"
"(2) Info: Missing optional field [SrcUserScopeId]"
"(2) Info: Missing optional field [SrcUserScope]"
"(2) Info: Missing optional field [SrcUserSessionId]"
"(2) Info: Missing optional field [SrcUserSid]"
"(2) Info: Missing optional field [SrcUserType]"
"(2) Info: Missing optional field [SrcUserUid]"
"(2) Info: Missing optional field [SrcUsername]"
"(2) Info: Missing optional field [TenantId]"
"(2) Info: Missing optional field [ThreatCategory]"
"(2) Info: Missing optional field [ThreatConfidence]"
"(2) Info: Missing optional field [ThreatField]"
"(2) Info: Missing optional field [ThreatFirstReportedTime]"
"(2) Info: Missing optional field [ThreatId]"
"(2) Info: Missing optional field [ThreatIpAddr]"
"(2) Info: Missing optional field [ThreatIsActive]"
"(2) Info: Missing optional field [ThreatLastReportedTime]"
"(2) Info: Missing optional field [ThreatName]"
"(2) Info: Missing optional field [ThreatOriginalConfidence]"
"(2) Info: Missing optional field [ThreatOriginalRiskLevel]"
"(2) Info: Missing optional field [ThreatRiskLevel]"
"(2) Info: Missing optional field [UrlCategory]"
"(2) Info: Missing recommended alias [Hostname] aliasing non-existent column [SrcHostname]"
"(2) Info: extra unnormalized column [Category]"
"(2) Info: extra unnormalized column [EventReceivedTime_t]"
"(2) Info: extra unnormalized column [Level]"
"(2) Info: extra unnormalized column [ProviderGuid_g]"
"(2) Info: extra unnormalized column [QueryTypeName]"
"(2) Info: extra unnormalized column [QueryType]"
"(2) Info: extra unnormalized column [Query]"
"(2) Info: extra unnormalized column [ResponseCodeName]"
"(2) Info: extra unnormalized column [ResponseCode]"
"(2) Info: extra unnormalized column [_ResourceId]"
"(2) Info: extra unnormalized column [eventtype]"
0 Parsers/ASimDns/test/README.md → Parsers/ASimDns/Tests/README.md
View file
File renamed without changes.
Loading

0 comments on commit 1b01c19

Please sign in to comment.