[ASIM: NXLog ASIM DNS PARSER UPDATES]

[ghost]

This PR addresses the feedback received from Microsoft / @devikamehra about contributing my changes to the ASIM DNS NXLog parser direcly, instead of as part of a solution.

CHANGES:

• Updated schema version
• Extended the list of known DNS query types
• Trying to re-introduce the DvcIpAddr field
• Removed deprecated fields

REASONS FOR CHANGE(S):

• The previous version of the parser was over a year old
• New schema versions were released since then
• Aiming to sync up the two parser versions that are in the repository

VERSION UPDATED:

• Yes, to version 0.5

TESTING COMPLETED:

• Need Help / How can I test this unreleased parser in my environment?

RAN VALIDATIONS:

• Yes, all validations passed successfully

This work is associated with NXLog's Jira Issue IN-366

[ghost]

Dear @devikamehra , as requested in PR 8154, these are the changes that I would like to contribute to the Microsoft-maintained, NXLog-related ASIM parsers.
Please let me know if there is anything that needs to be changed or updated.

[ghost]

@vakohl Could you please give me some feedback, to see if there are any necessary changes, that I'd need to make?

[vakohl]

@vakohl Could you please give me some feedback, to see if there are any necessary changes, that I'd need to make?

@jszigetvari-nxlog thankyou for your contributing in making Microsoft Sentinel better. I'll try to complete my review this week.

Can you please run the ASIM parser test functions for schema and data and attach both output with this PR? would be 4 file csv files, two for ASIM and two for vim.

Also, if possible please attach the sample logs as well, would help me with the review.

[ghost]

@vakohl Thanks for your feedback and kind words. Unfortunately I will have very little time in the upcoming few days to work on the tests. Perhaps next Monday, I can allot some time for this. Either way, I will let you know once I'm done with it, or get stuck somewhere.

[ghost]

@vakohl Hi Varun,

I tried saving the parsers into my workspace manually, by pasting the ParserQuery parts to the Sentinel Logs view/query editor, and saving them under a unique name, as functions. But unfortunately in case of the vim one, I get an error, and the save operation fails with:

Error saving function. Error code: InvalidOperationArgument. Error message: Invalid function parameters string. Encountered possible syntax errors: 'Missing expression (90,90)', 'Expected: ')' (91,93)', 'Expected: '{' (91,93)', 'Expected: '}' (91,93)'. Please see relevant documentation 'https://docs.microsoft.com/en-us/azure/kusto/query/'.

[v-rbajaj]

Hi @jszigetvari-nxlog, thanks for sharing the information, we will get back soon on this.

[vakohl]

@vakohl Hi Varun,

I tried saving the parsers into my workspace manually, by pasting the ParserQuery parts to the Sentinel Logs view/query editor, and saving them under a unique name, as functions. But unfortunately in case of the vim one, I get an error, and the save operation fails with:

Error saving function. Error code: InvalidOperationArgument. Error message: Invalid function parameters string. Encountered possible syntax errors: 'Missing expression (90,90)', 'Expected: ')' (91,93)', 'Expected: '{' (91,93)', 'Expected: '}' (91,93)'. Please see relevant documentation 'https://docs.microsoft.com/en-us/azure/kusto/query/'.

Try replacing last line:
ASimDnsMicrosoftNXLog (starttime=datetime(null), endtime=datetime(null), srcipaddr='', domain_has_any=dynamic([]), responsecodename='', response_has_ipv4='*', response_has_any_prefix=dynamic([]), eventtype='Query', disabled=false)

[ghost]

Try replacing last line:
ASimDnsMicrosoftNXLog (starttime=datetime(null), endtime=datetime(null), srcipaddr='', domain_has_any=dynamic([]), responsecodename='', response_has_ipv4='*', response_has_any_prefix=dynamic([]), eventtype='Query', disabled=false)

@vakohl Thank you Varun for your review comments! I will try your suggestion when I will have some time.
In the other cases, I will make the changes, once we agree on what to do exactly.

[ghost]

Try replacing last line: ASimDnsMicrosoftNXLog (starttime=datetime(null), endtime=datetime(null), srcipaddr='', domain_has_any=dynamic([]), responsecodename='', response_has_ipv4='*', response_has_any_prefix=dynamic([]), eventtype='Query', disabled=false)

@vakohl Hi Varun,

Thank you, your recommendation worked, I was able to save the vim parser with your changes.
I also installed and ran the parser tests, and generated the csv files for you. Please find them attached.
vimDnsMicrosoftNXlog_ASimSchemaTester_query_out.csv
vimDnsMicrosoftNXlog_ASimDataTester_query_out.csv
ASimDnsMicrosoftNXLog_ASimSchemaTester_query_out.csv
ASimDnsMicrosoftNXLog_ASimDataTester_query_out.csv

[ghost]

@vakohl Hello Varun,
I realized that in my previous comment the outputs for ASimDnsMicrosoftNXLog were wrong, as I used an older version of the function. I corrected that now and the outputs look a lot better. I included the outputs for both functions in this comment. Those can be regarded as the latest ones.

Please let us agree on the still open questions, and I will push the resulting corrections to this branch, and then hopefully we can move towards merging this PR.

ASimDnsMicrosoftNXLog_ASimSchemaTester_query_out.csv
ASimDnsMicrosoftNXLog_ASimDataTester_query_out.csv
vimDnsMicrosoftNXlog_ASimSchemaTester_query_out.csv
vimDnsMicrosoftNXlog_ASimDataTester_query_out.csv

[vakohl]

@vakohl Hello Varun, I realized that in my previous comment the outputs for ASimDnsMicrosoftNXLog were wrong, as I used an older version of the function. I corrected that now and the outputs look a lot better. I included the outputs for both functions in this comment. Those can be regarded as the latest ones.

Please let us agree on the still open questions, and I will push the resulting corrections to this branch, and then hopefully we can move towards merging this PR.

ASimDnsMicrosoftNXLog_ASimSchemaTester_query_out.csv ASimDnsMicrosoftNXLog_ASimDataTester_query_out.csv vimDnsMicrosoftNXlog_ASimSchemaTester_query_out.csv vimDnsMicrosoftNXlog_ASimDataTester_query_out.csv

I'll take a look and respond by this week.

[ghost]

@vakohl do you have any update on this?

[vakohl]

@vakohl Hello Varun, I realized that in my previous comment the outputs for ASimDnsMicrosoftNXLog were wrong, as I used an older version of the function. I corrected that now and the outputs look a lot better. I included the outputs for both functions in this comment. Those can be regarded as the latest ones.
Please let us agree on the still open questions, and I will push the resulting corrections to this branch, and then hopefully we can move towards merging this PR.
ASimDnsMicrosoftNXLog_ASimSchemaTester_query_out.csv ASimDnsMicrosoftNXLog_ASimDataTester_query_out.csv vimDnsMicrosoftNXlog_ASimSchemaTester_query_out.csv vimDnsMicrosoftNXlog_ASimDataTester_query_out.csv

I'll take a look and respond by this week.

@jszigetvari-nxlog can you upload the result as mentioned in the guidelines here?
https://learn.microsoft.com/en-us/azure/sentinel/normalization-develop-parsers#test-results-submission-guidelines

[jszigetvari]

@vakohl Thanks for the review, I will start working on it on next Monday.

[ghost]

@vakohl After the latest corrections I tried to run both of these (separately)

ASimDnsMicrosoftNXLog | limit 10000 | invoke ASimDataTester('Dns')
vimDnsMicrosoftNXlog | limit 10000 | invoke ASimDataTester('Dns')

But they both returned no results. (The time range was set to the last 4 months, and I used this very same setting during the last round of testing.)

[ghost]

@vakohl Apparently some of the tests are failing, but to me they don't seem to be tightly related to the changes I've made. Could you please help in finding out what may be wrong?

[vakohl]

But they both returned no results. (The time range was set to the last 4 months, and I used this very same setting during the last round of testing.)

when running the parser for testing results, you can use the filter/parameters you were using before i.e. giving specific values. But with PR submission, you should declare the filter as I suggested in one of my previous comment

[vakohl]

@vakohl Apparently some of the tests are failing, but to me they don't seem to be tightly related to the changes I've made. Could you please help in finding out what may be wrong?

For validation errors, Please see https://dev.azure.com/azure/Azure-Sentinel/_build/results?buildId=64957&view=logs&jobId=2bbe2297-1b5b-54f3-6060-67e7b289f6c3&j=2bbe2297-1b5b-54f3-6060-67e7b289f6c3&t=293f4a23-e36d-5efa-3a4e-ea46d4ab6d83

Please add '_ItemId' field of type string in the custom table here: https://github.com/Azure/Azure-Sentinel/blob/d798b63ced54f34a701e4893e0066b857b4ced72/.script/tests/KqlvalidationsTests/CustomTables/NXLog_DNS_Server_CL.json

the other two fields showing in the error i.e Category and Level, we should not see error for them after removing from Project-away

[ghost]

Please add '_ItemId' field of type string in the custom table here: https://github.com/Azure/Azure-Sentinel/blob/d798b63ced54f34a701e4893e0066b857b4ced72/.script/tests/KqlvalidationsTests/CustomTables/NXLog_DNS_Server_CL.json

Thanks for pointing to the source of the problem. It really helped.
Fixed.

[ghost]

But they both returned no results. (The time range was set to the last 4 months, and I used this very same setting during the last round of testing.)

when running the parser for testing results, you can use the filter/parameters you were using before i.e. giving specific values. But with PR submission, you should declare the filter as I suggested in one of my previous comment

It turned out that there was a different problem, but I sorted it now, so I am submitting the latest results for the data tests too.

[ghost]

@vakohl All the necessary changes were carried out, and the tests are passing now.

[ghost]

@vakohl This has been an exhausing process. I assume it was the same for you. Thanks for your efforts!

[vakohl]

@jszigetvari-nxlog thankyou for your contribution. I'll approve this PR once the validation are through.

[ghost]

@jszigetvari-nxlog thankyou for your contribution. I'll approve this PR once the validation are through.

@vakohl The KqlValidations task failed for some reason, and I tried to look at the error(s) but the error log doesn't seem to be too helpful: https://dev.azure.com/azure/43fcaf68-ba3d-474b-97f3-7a1e6fbb5c8d/_apis/build/builds/65361/logs/155

[vakohl]

@v-atulyadav can you please check the validation error?

[v-atulyadav]

@vakohl, validations are cleared now.

[vakohl]

@vakohl, validations are cleared now.

Thaknyou @v-atulyadav