Large ti db fixes

[blauwers]

Required items, please complete

Change(s):

• Updated .script/tests/KqlvalidationsTests/KqlValidationTests.cs
• Updated .script/tests/KqlvalidationsTests/SkipValidationsTemplates.json
• Added .script/tests/KqlvalidationsTests/CustomTables/UrlClickEvents.json
• Updated Solutions/Threat Intelligence/Analytic Rules/DomainEntity_CommonSecurityLog.yaml
• Updated Solutions/Threat Intelligence/Analytic Rules/DomainEntity_DnsEvents.yaml
• Updated Solutions/Threat Intelligence/Analytic Rules/DomainEntity_PaloAlto.yaml
• Updated Solutions/Threat Intelligence/Analytic Rules/DomainEntity_SecurityAlert.yaml
• Updated Solutions/Threat Intelligence/Analytic Rules/DomainEntity_Syslog.yaml
• Updated Solutions/Threat Intelligence/Analytic Rules/DomainEntity_imWebSession.yaml
• Updated Solutions/Threat Intelligence/Analytic Rules/EmailEntity_AzureActivity.yaml
• Updated Solutions/Threat Intelligence/Analytic Rules/EmailEntity_OfficeActivity.yaml
• Updated Solutions/Threat Intelligence/Analytic Rules/EmailEntity_PaloAlto.yaml
• Updated Solutions/Threat Intelligence/Analytic Rules/EmailEntity_SecurityAlert.yaml
• Updated Solutions/Threat Intelligence/Analytic Rules/EmailEntity_SecurityEvent.yaml
• Updated Solutions/Threat Intelligence/Analytic Rules/EmailEntity_SigninLogs.yaml
• Updated Solutions/Threat Intelligence/Analytic Rules/FileHashEntity_CommonSecurityLog.yaml
• Updated Solutions/Threat Intelligence/Analytic Rules/FileHashEntity_SecurityEvent.yaml
• Updated Solutions/Threat Intelligence/Analytic Rules/IPEntity_AWSCloudTrail.yaml
• Updated Solutions/Threat Intelligence/Analytic Rules/IPEntity_AppServiceHTTPLogs.yaml
• Updated Solutions/Threat Intelligence/Analytic Rules/IPEntity_AzureActivity.yaml
• Updated Solutions/Threat Intelligence/Analytic Rules/IPEntity_AzureFirewall.yaml
• Updated Solutions/Threat Intelligence/Analytic Rules/IPEntity_AzureKeyVault.yaml
• Updated Solutions/Threat Intelligence/Analytic Rules/IPEntity_AzureNetworkAnalytics.yaml
• Updated Solutions/Threat Intelligence/Analytic Rules/IPEntity_AzureSQL.yaml
• Updated Solutions/Threat Intelligence/Analytic Rules/IPEntity_CustomSecurityLog.yaml
• Updated Solutions/Threat Intelligence/Analytic Rules/IPEntity_DnsEvents.yaml
• Updated Solutions/Threat Intelligence/Analytic Rules/IPEntity_DuoSecurity.yaml
• Updated Solutions/Threat Intelligence/Analytic Rules/IPEntity_OfficeActivity.yaml
• Updated Solutions/Threat Intelligence/Analytic Rules/IPEntity_SigninLogs.yaml
• Updated Solutions/Threat Intelligence/Analytic Rules/IPEntity_VMConnection.yaml
• Updated Solutions/Threat Intelligence/Analytic Rules/IPEntity_W3CIISLog.yaml
• Updated Solutions/Threat Intelligence/Analytic Rules/IPEntity_imNetworkSession.yaml
• Updated Solutions/Threat Intelligence/Analytic Rules/IPEntity_imWebSession.yaml
• Updated Solutions/Threat Intelligence/Analytic Rules/Threat Intel Matches to GitHub Audit Logs.yaml
• Updated Solutions/Threat Intelligence/Analytic Rules/URLEntity_AuditLogs.yaml
• Updated Solutions/Threat Intelligence/Analytic Rules/URLEntity_OfficeActivity.yaml
• Updated Solutions/Threat Intelligence/Analytic Rules/URLEntity_PaloAlto.yaml
• Updated Solutions/Threat Intelligence/Analytic Rules/URLEntity_SecurityAlerts.yaml
• Updated Solutions/Threat Intelligence/Analytic Rules/URLEntity_Syslog.yaml
• Updated Solutions/Threat Intelligence/Analytic Rules/imDns_DomainEntity_DnsEvents.yaml
• Updated Solutions/Threat Intelligence/Analytic Rules/imDns_IPEntity_DnsEvents.yaml
• Added Solutions/Threat Intelligence/Analytic Rules/DomainEntity_DeviceNetworkEvents.yaml
• Added Solutions/Threat Intelligence/Analytic Rules/DomainEntity_EmailEvents.yaml
• Added Solutions/Threat Intelligence/Analytic Rules/DomainEntity_EmailUrlInfo.yaml
• Added Solutions/Threat Intelligence/Analytic Rules/FileHashEntity_DeviceFileEvents.yaml
• Added Solutions/Threat Intelligence/Analytic Rules/EmailEntity_EmailEvents.yaml
• Added Solutions/Threat Intelligence/Analytic Rules/IPEntity_DeviceNetworkEvents.yaml
• Added Solutions/Threat Intelligence/Analytic Rules/UrlEntityEntity_DeviceNetworkEvents.yaml
• Added Solutions/Threat Intelligence/Analytic Rules/URLEntity_EmailUrlInfo.yaml
• Added Solutions/Threat Intelligence/Analytic Rules/URLEntity_UrlClickEvents.yaml
• Updated Solutions/Threat Intelligence/Data/Solution_ThreatIntelligenceTemplateSpec.json
• Updated Solutions/Threat Intelligence/Package/createUiDefinition.json
• Updated Solutions/Threat Intelligence/Package/mainTemplate.json
• Updated Solutions/Microsoft Defender XDR/Data Connectors/MicrosoftThreatProtection.JSON
• Updated Solutions/Microsoft Defender XDR/Data/Solution_Microsoft Defender XDR.json

Reason for Change(s):

• Implement more robust prefiltering of data to optimize query and allow large TI database (>20 million active indicators) without timeout
• Fix TI query validation test to allow filtering per KQL best practices
• New analytic queries to enhance phishing detection
• Declare UrlClickEvents in MicrosoftThreatProtection
• Declare UrlClickEvents table for automated tests
• Merge changes from PR#9181 by @kfriede

Version Updated:

• Yes

Testing Completed:

• Yes

Checked that the validations are passing and have addressed any issues that are present:

• Yes

[blauwers]

@microsoft-github-policy-service agree

[blauwers]

Sorry for all the commits - it seems all the validation tests are now working.

[v-rbajaj]

Hi @blauwers, please update version of Solutions/Threat Intelligence/Analytic Rules/IPEntity_AzureKeyVault.yaml?

[blauwers]

@v-rbajaj Updated - apologies for the oversight.

[v-rbajaj]

Hi @blauwers, there is another PR (#9181) opened for Threat intelligence and there are few contents which are similar.

We would need to repackage this solution after the other PR is merged so that it doesn't raise the conflict.

[blauwers]

I just looked at that PR, and it potentially has some issues in how it runs the summarize after filtering for active. It means that if TI is updated after the initial report, the query may generate a false positive by discounting the newer TI update marking the signal as inactive.

Since I also fixed the validation issues, a better strategy may be to reject that PR and apply the changes here that make sense. I initially tried a similar approach as used in the other PR.

[blauwers]

@v-rbajaj I would be happy to port over relevant changes to this PR if that is helpful.

[v-rbajaj]

Hi @blauwers, there are couple of challenges, we can't simply close that PR unless the other author agrees to it.

And if you package this solution with 3.0.2 version than the other PR will have merge conflicts.

Can we wait for the other PR to close and then work on this ?

[blauwers]

Hello @v-rbajaj,

I am used to seeing code that passes the QA test going before code that does not. That said, the code in the other PR puts some things out of order in such a way it will break the accuracy of the TI queries. Specifically, the statement | where Active == true and ExpirationDateTime > now() needs to come after | summarize [.]

This PR also fixes the quality test that previously made analytic TI queries fail to validate - ergo why the tests are now passing, which is not addressed in the other PR.

It would be faster to merge or port the changes from the other PR to this PR and fix the query issues simultaneously. Can we agree that this is an acceptable strategy? If so, I will start working on merging the PRs and making the necessary corrections.

[blauwers]

To illustrate the point:

let table = datatable (TimeGenerated: datetime , Active: bool, ExpirationDateTime: datetime, IndicatorId: string, Description: string) [
 datetime(2023-11-17 7:00 AM), true, datetime(2023-11-20 9:00 AM), "Indicator1", "Active indicator 1 from Source 1",
 datetime(2023-11-18 7:00 AM), false, datetime(2023-11-20 9:00 AM), "Indicator1", "Inctive indicator 1 from Source 1",
 datetime(2023-11-19 7:00 AM), true, datetime(2023-11-19 9:00 AM), "Indicator2", "Active indicator 1 from Source 2",
];
table
| where TimeGenerated >= datetime(2023-11-10 9:00 AM)
| summarize LatestIndicatorTime = arg_max(TimeGenerated, *) by IndicatorId
| where Active == true and ExpirationDateTime > datetime(2023-11-19 7:00 AM)

outputs something else than the following

let table = datatable (TimeGenerated: datetime , Active: bool, ExpirationDateTime: datetime, IndicatorId: string, Description: string) [
 datetime(2023-11-17 7:00 AM), true, datetime(2023-11-20 9:00 AM), "Indicator1", "Active indicator 1 from Source 1",
 datetime(2023-11-18 7:00 AM), false, datetime(2023-11-20 9:00 AM), "Indicator1", "Inctive indicator 1 from Source 1",
 datetime(2023-11-19 7:00 AM), true, datetime(2023-11-19 9:00 AM), "Indicator2", "Active indicator 1 from Source 2",
];
table
| where TimeGenerated >= datetime(2023-11-10 9:00 AM)
| where Active == true and ExpirationDateTime > datetime(2023-11-19 7:00 AM)
| summarize LatestIndicatorTime = arg_max(TimeGenerated, *) by IndicatorId

As you can see the latter causes false positives to show up.

[kfriede]

@blauwers if you want to pull the latest changes in from #9181 again, I'd be okay with @v-rbajaj approving this in place and I can close #9181

[blauwers]

Ok, I had to back all changes as merging @kfriede's last branch caused some trouble. I had to revert the commits and will be reprocessing those changes.

[v-rbajaj]

Hi @blauwers, thanks for making these changes, will check and get back to you by 27 Nov 2023.

[v-rbajaj]

Hi @blauwers, thanks for the changes. I was going through the PR and I noticed that you have made changes in Microsoft Defender XDR's Data Connector, so for that we would need you to repackage Microsoft Defender XDR using v3 tool.

Apart from that I am reviewing the changes in Threat Intelligence.

[blauwers]

Hello @v-rbajaj, I updated the package as requested.

[v-rbajaj]

Hi @blauwers, please resolve merge conflicts from the PR.

[blauwers]

Hello @v-rbajaj as requested, I have resolved the merge conflicts by merging my branch with an up to date master.

[v-rbajaj]

Hi @blauwers, apart from that please add/update release notes for Microsoft Defender XDR
https://github.com/Azure/Azure-Sentinel/blob/master/Solutions/ReleaseNotesGuidance.md?plain=1

To fix the failing ARM TTK, please update the maintemplate for Microsoft Defender XDR solution with the change mentioned below and after updating the maintemplate, update the 3.0.2 zip

Please remove these empty arrays.

[blauwers]

@v-rbajaj I made the requested changes.

While doing so, I noticed I must not have re-merged the version bump for Microsoft Defender XDR, so I updated that to 3.0.2 again. Thank you for pointing out the issue with the empty arrays causing the ARM TTK to fail. How many issues have been in files generated by a packaging tool is odd.