Update "TI Map" analytics rules for performance #9181

kfriede · 2023-10-09T16:47:39Z

Required items, please complete

Change(s):

Changed query to put any applicable WHERE filters before any SUMMARIZE calls for performance.
Added ThreatIntelligenceTaxii data connector on missing analytics.

Reason for Change(s):

Many of these queries would fail on real-world datasets due to summarization of unnecessary data.
ThreatIntelligenceTaxii is a valid provider of ThreatIntelligenceIndicator table data and should be included

Version Updated:

Yes

Testing Completed:

Yes, as much as possible (don't have all data sources to test appropriately for some rules)

Checked that the validations are passing and have addressed any issues that are present:

Yes

kfriede · 2023-10-09T18:33:03Z

To whoever ends up picking this up, the KQL validation failures is due to Line 397 of https://github.com/Azure/Azure-Sentinel/blob/master/.script/tests/KqlvalidationsTests/KqlValidationTests.cs

The regex being used is overly-specific and not conducive for the performance improvements being made here.

v-rbajaj · 2023-10-10T12:48:21Z

Hi @kfriede, thanks for raising this PR. We will review this PR by 13 Oct 2023.

kfriede · 2023-10-10T13:32:52Z

This traces back to commit 16564be from @aprakash13. Query layouts looked largely performance-optimized before this change.

v-rbajaj · 2023-10-13T13:42:57Z

Hi @kfriede, for KQL validation, we will see what can be done in this case.

We would need you to repackage this solution using this tool https://github.com/Azure/Azure-Sentinel/tree/master/Tools/Create-Azure-Sentinel-Solution/V3

Also update the release notes for the same.

Thanks

v-rbajaj · 2023-10-17T09:08:10Z

Hi @kfriede, please provide update on above comment.

v-rbajaj · 2023-10-20T11:53:13Z

Hi @kfriede, please provide some update on this PR

kfriede · 2023-10-23T10:21:35Z

Hi @v-rbajaj, apologies I was out last week. I will look at repackaging the solution for these edits

Edit: Solution has been repackaged for 3.0.2

v-rbajaj · 2023-10-25T14:59:32Z

Hi @v-rbajaj, apologies I was out last week. I will look at repackaging the solution for these edits

Edit: Solution has been repackaged for 3.0.2

Thanks @kfriede, will review the PR by 27 Oct, 2023.

v-rbajaj · 2023-10-27T10:57:12Z

Solutions/Threat Intelligence/Package/createUiDefinition.json

@@ -6,7 +6,7 @@
    "config": {
      "isWizard": false,
      "basics": {
-        "description": "<img src=\"https://raw.githubusercontent.com/Azure/Azure-Sentinel/master/Logos/Azure_Sentinel.svg\"width=\"75px\" height=\"75px\">\n\n**Note:** Please refer to the following before installing the solution: \r \n • Review the solution [Release Notes](https://github.com/Azure/Azure-Sentinel/tree/master/Solutions/Threat Intelligence/ReleaseNotes.md)\r \n • There may be [known issues](https://aka.ms/sentinelsolutionsknownissues) pertaining to this Solution, please refer to them before installing.\n\nThe Threat Intelligence solution contains data connectors for import of threat indicators into Microsoft Sentinel, analytic rules for matching TI data with event data, workbook, and hunting queries. Threat indicators can be malicious IP's, URL's, filehashes, domains, email addresses etc.\n\n**Data Connectors:** 4, **Workbooks:** 1, **Analytic Rules:** 38, **Hunting Queries:** 5\n\n[Learn more about Microsoft Sentinel](https://aka.ms/azuresentinel) | [Learn more about Solutions](https://aka.ms/azuresentinelsolutionsdoc)",
+        "description": "<img src=\"https://raw.githubusercontent.com/Azure/Azure-Sentinel/master/Logos/Azure_Sentinel.svg\"width=\"75px\" height=\"75px\">\n\n**Note:** _There may be [known issues](https://aka.ms/sentinelsolutionsknownissues) pertaining to this Solution, please refer to them before installing._\n\nThe Threat Intelligence solution contains data connectors for import of threat indicators into Microsoft Sentinel, analytic rules for matching TI data with event data, workbook, and hunting queries. Threat indicators can be malicious IP's, URL's, filehashes, domains, email addresses etc.\n\n**Workbooks:** 1, **Analytic Rules:** 38, **Hunting Queries:** 5\n\n[Learn more about Microsoft Sentinel](https://aka.ms/azuresentinel) | [Learn more about Solutions](https://aka.ms/azuresentinelsolutionsdoc)",


Hi @kfriede, Please revert this change.

@v-rbajaj done!

v-rbajaj · 2023-10-31T10:43:18Z

Hi @kfriede, thanks. We are currently looking into the KQL validations and once that is fixed, we will revisit this PR by 2 Nov 2023.

v-rbajaj · 2023-11-02T10:51:17Z

Hi @kfriede, we are still investigating on KQL validations, will get back to by 06 Nov 2023.

kfriede · 2023-11-13T11:53:46Z

Hi @v-rbajaj, it's been over 1 week since your last update. Is this still under review?

v-atulyadav · 2023-11-15T09:20:45Z

Hi @kfriede,
Apologies for the inconvenience, but due to unavailability, the review has been delayed. We assure you that you will receive an update on this by the end of the day tomorrow. Thank you.

v-rbajaj · 2023-11-16T09:38:23Z

Hi @kfriede, we are working on the KQL changes and we will soon provide an update.

…tatements for performance

@blauwers

…sult differences noted by @blauwers

kfriede · 2023-11-22T16:00:18Z

@v-rbajaj after refactoring, files now give: Kqlvalidations.Tests.KqlValidationTests.Validate_DetectionQueries_SkippedTemplatesDoNotHaveValidKql(fileName: "IPEntity_DuoSecurity.yaml", encodedFilePath: "L2hvbWUvdnN0cy93b3JrLzEvcy9Tb2x1dGlvbnMvVGhyZWF0IE"...) in testing

v-rbajaj · 2023-11-27T12:51:39Z

Hi @kfriede, sorry for the above commits, was trying to fix few errors from KQL validations. We are checking on it, will get back to you by 29 Nov 2023.

v-rbajaj · 2023-11-28T13:00:58Z

Hi @kfriede, can we merge #9369 as the changes there look similar and all the KQL validations have been fixed by @blauwers.

If there are any changes you want to suggest on #9369 or want to contribute your changes, can you please comment on that PR?

kfriede · 2023-11-29T17:11:30Z

@v-rbajaj I’m good with #9369, and we can close this (#9181) when #9369 is merged

…

-- Kevin Friedemann

On Nov 28, 2023 at 5:01 AM -0800, v-rbajaj ***@***.***>, wrote: Hi @kfriede, can we merge #9369 as the changes there look similar and all the KQL validations have been fixed by @blauwers. If there are any changes you want to suggest on #9369 or want to contribute your changes, can you please comment on that PR? — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>

v-rbajaj · 2023-11-30T06:52:44Z

Thanks @kfriede, I have changed the PR status from open to draft and will close later once #9369 is merged.

v-rbajaj · 2023-12-12T11:16:34Z

Closing this PR as #9369 is merged.

kfriede requested review from a team as code owners October 9, 2023 16:47

kfriede mentioned this pull request Oct 9, 2023

KQLValidations function ValidateKqlForLatestTI is too restrictive #9182

Closed

v-atulyadav assigned v-rbajaj Oct 10, 2023

v-atulyadav added Solution Solution specialty review needed Analytic Rules labels Oct 10, 2023

v-rbajaj reviewed Oct 27, 2023

View reviewed changes

kfriede force-pushed the update-ti-analytics branch from 1dbaca4 to 4e2bde4 Compare October 27, 2023 12:55

v-rbajaj mentioned this pull request Nov 16, 2023

Large ti db fixes #9369

Merged

kfriede added 6 commits November 22, 2023 09:22

Updated EmailEntity_OfficeActivity for performance with large datasets

be97028

Updated IPEntity_VMConnection for performance with large dadatasets

efa7b61

Updated IPEntity_AWSCloudTrail for performance with large dadatasets

9ff0355

Updated URLEntity_SecurityAlerts for performance with large dadatasets

45459e2

Updated IPEntity_imNetworkSession for performance with large dadatasets

062cd71

Updated DomainEntity_Syslog for performance with large dadatasets

e47853a

kfriede added 11 commits November 22, 2023 09:22

Incremented version numbers

4cd24c8

Updated URLEntity_OfficeActivity for performance with large dadatasets

d3330c2

Updated IPEntity_OfficeActivity for performance with large dadatasets

4eceb29

Updated URLEntity_Syslog for performance with large dadatasets

f0570b9

Updated query to move any applicable WHERE filters before summarize s…

6ec5202

…tatements for performance

Updated missed version number

35b38b1

Added missing ThreatIntelligenceTAXII connectorId

9216155

Fixed semicolon issues

21c7d13

Repackaged solution for 3.0.2

d6e4cb9

Reverted description of createUiDefinition.json

4c822a1

Moved active and expiration where filter to after summarize due to re…

f07b36b

…sult differences noted by @blauwers

kfriede force-pushed the update-ti-analytics branch from 4e2bde4 to f07b36b Compare November 22, 2023 14:24

kfriede added 2 commits November 22, 2023 09:30

Fixed version number update issue

476b3eb

Refactored for validations

f0296a3

v-rbajaj added 4 commits November 27, 2023 17:24

Update SkipIdValidationsTemplates.json

f0e55d8

Merge branch 'master' into pr/9181

9383535

Update SkipIdValidationsTemplates.json

fe08167

Update SkipIdValidationsTemplates.json

f9164f2

v-rbajaj marked this pull request as draft November 30, 2023 06:51

v-rbajaj closed this Dec 12, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update "TI Map" analytics rules for performance #9181

Update "TI Map" analytics rules for performance #9181

kfriede commented Oct 9, 2023

kfriede commented Oct 9, 2023

v-rbajaj commented Oct 10, 2023

kfriede commented Oct 10, 2023 •

edited

Loading

v-rbajaj commented Oct 13, 2023 •

edited

Loading

v-rbajaj commented Oct 17, 2023

v-rbajaj commented Oct 20, 2023

kfriede commented Oct 23, 2023 •

edited

Loading

v-rbajaj commented Oct 25, 2023

v-rbajaj Oct 27, 2023

kfriede Oct 27, 2023

v-rbajaj commented Oct 31, 2023

v-rbajaj commented Nov 2, 2023 •

edited

Loading

kfriede commented Nov 13, 2023

v-atulyadav commented Nov 15, 2023

v-rbajaj commented Nov 16, 2023

kfriede commented Nov 22, 2023

v-rbajaj commented Nov 27, 2023

v-rbajaj commented Nov 28, 2023

kfriede commented Nov 29, 2023 via email

v-rbajaj commented Nov 30, 2023

v-rbajaj commented Dec 12, 2023

Update "TI Map" analytics rules for performance #9181

Update "TI Map" analytics rules for performance #9181

Conversation

kfriede commented Oct 9, 2023

kfriede commented Oct 9, 2023

v-rbajaj commented Oct 10, 2023

kfriede commented Oct 10, 2023 • edited Loading

v-rbajaj commented Oct 13, 2023 • edited Loading

v-rbajaj commented Oct 17, 2023

v-rbajaj commented Oct 20, 2023

kfriede commented Oct 23, 2023 • edited Loading

v-rbajaj commented Oct 25, 2023

v-rbajaj Oct 27, 2023

Choose a reason for hiding this comment

kfriede Oct 27, 2023

Choose a reason for hiding this comment

v-rbajaj commented Oct 31, 2023

v-rbajaj commented Nov 2, 2023 • edited Loading

kfriede commented Nov 13, 2023

v-atulyadav commented Nov 15, 2023

v-rbajaj commented Nov 16, 2023

kfriede commented Nov 22, 2023

v-rbajaj commented Nov 27, 2023

v-rbajaj commented Nov 28, 2023

kfriede commented Nov 29, 2023 via email

v-rbajaj commented Nov 30, 2023

v-rbajaj commented Dec 12, 2023

kfriede commented Oct 10, 2023 •

edited

Loading

v-rbajaj commented Oct 13, 2023 •

edited

Loading

kfriede commented Oct 23, 2023 •

edited

Loading

v-rbajaj commented Nov 2, 2023 •

edited

Loading