Cisco Meraki Events via REST API overutilization and data duplication

[shaunyb93]

Hi team
I understand that this connector is in preview but we are facing an issue and would like to report it.

We are seeing the getOrganizationConfigurationChanges running ~21000 times per hour
This is resulting in excess data being logged in ASimWebSessionLogs table

We are also seeing changes from are being retrieved by the connector using the getOrganizationConfigurationChanges function are duplicated thousands of times in the ASimAuditEventLogs table.

Please can we get some help with this - I will likely need to disconnect the connector.

Thank you

[v-rusraut]

Hi @shaunyb93,
Thanks for flagging this issue, we will investigate this issue and get back to you with some updates by 26 June 2024. Thanks!

[shaunyb93]

@v-rusraut has there been any progress on this issue?
Thanks

[v-sudkharat]

Hi @shaunyb93, Sorry for delay in response. Just want to know, after clicking on disconnect button, the connector status still show as connected? Did you tried it to disconnect the connector by that option? -

Thanks!

[shaunyb93]

Hi, Yes I have disconnected the connector as it does not work properly.
We need to understand this part of my request before we reconnect it:

We are seeing the getOrganizationConfigurationChanges running ~21000 times per hour
This is resulting in excess data being logged in ASimWebSessionLogs table

We are also seeing changes from are being retrieved by the connector using the getOrganizationConfigurationChanges function are duplicated thousands of times in the ASimAuditEventLogs table.

Thanks
Shaun

[shaunyb93]

Any update here?
Thanks

[v-sudkharat]

Hey @shaunyb93, Still checking this issue with team, need some more time to investigate on it. Thanks!

[JustinGrote]

In 24 hours the connector has called the meraki dashboard 700,000 times. It is causing other applications to rate-limit.

It is causing rate limiting both with itself and with other applications:

[JustinGrote]

Several events have also been duplicated tens if not hundreds of thousands of times (this is the exact same event with the exact same timestamp, not multiple similar events) just within the last 24 hours. In the last 7 days it's approaching millions of duplicates for some records.

@v-sudkharat This needs to be addressed, it is costing us a large amount of wasted ingestion money and this connector is not ready for production, it appears its method of timestamping when the last events had arrived is not operating correctly

[shaunyb93]

@JustinGrote Somewhat relieved that someone else is seeing the same issue. We had to disconnect the connector as it was just going crazy with duplication. The connector is marked as being in a preview state so I presume some bugs are expected but it seems really difficult to get any focus on a fix - 2 months and counting on this ticket...

[v-sudkharat]

Hi @JustinGrote and @shaunyb93, We are connecting with our concern team for this issue, and once we get any update from team, we will notify you. Thanks!

[JustinGrote]

@shaunyb93 agreed, I tried to develop a codeless connector but they are such black boxes it's impossible to see what's going on, so I've been authoring an Azure Function to do this same thing instead.

[v-sudkharat]

Hi @shaunyb93 / @JustinGrote, We have received the response from our concern team for this issue, to verify the duplication, checking with you while requesting the API directly (using tool: Postman) with adding same definitions as mentioned into data connector like Time, as a result did you get the data duplication? Could you please check on this and let us know, so we can share this update with our team.
Connector API link-https://github.com/Azure/Azure-Sentinel/blob/master/Solutions/Cisco%20Meraki%20Events%20via%20REST%20API/Data%20Connectors/CiscoMerakiMultiRule_ccp/dataConnectorPoller.json
And to investigate on this issue our team required some access (like environment access) for which we are not authorized to provide. Kindly requesting you to raise support ticket in azure portal, so our support engineer can share the required information and access to the team.
Please let us know once if you raise support ticket case so we can close this issue from GitHub and this issue investigation and track will continue by our support team.

Thanks!

[JustinGrote]

No duplication from the API, in fact I made my own Azure Function to ingest this same data and it works just fine with no deduplication and vastly less API calls.

The team should probably be able to verify by signing up for a Meraki Dashboard emulation account
https://meraki.cisco.com/form/demo/ which I assume they used to do the development, and can verify using the same KQL queries I listed above.

[v-sudkharat]

@JustinGrote, Thanks for the response. We have shared this update to concern team.

@shaunyb93 / @JustinGrote, Please let us know once you open a support case. Thanks!

[shaunyb93]

@v-sudkharat can you advise which team we should be raising this with? last time I tried to raise a support case for a Sentinel connector issue, Microsoft (the developer) advised they cannot assist and just point fingers at Cisco...

[JustinGrote]

@v-amolpatil is the one who committed the solution, and has been doing other OMS/AMA migrations in his recent commit history.

[mferrellen]

Team, we would also like to report experiencing this issue as well. We are seeing 700,000+ requests a day which causes rate limiting issues with the Meraki REST API. Is there an eta on resolution regarding this bug? We are going to have to stop using the data connector until a bug fix is applied as its impacting other apps that are using the Meraki REST API.

[v-sudkharat]

Hi @shaunyb93 / @JustinGrote / @mferrellen, Please raise a support case with Data Collection team, so the ticket get transfer to our concern team. Thanks!

[v-sudkharat]

Hi @shaunyb93, Could you please confirm did you raise a support case? Thanks!

[shaunyb93]

@v-sudkharat yes
MSFT SR#2408230030007650

[Nico-WA]

Just found this thread and unhappy to report we're seeing the same thing after just setting this up this morning.

@JustinGrote - Any chance you could share that Azure function? I'd love to use that to tide us over until the connector can be fixed.

[JustinGrote]

@Nico-WA I'm exploring with my company on that but it's currently company IP unfortunately. It also does a lot more than just the 3 categories the connector uses, it parses network events for wireless logons, 802.1x logons, eap logons, nbar blocks, cf blocks, and formats them all into ASIM using a DCR, and checkpoints the last log ingested into a blob so that future checks are resumed from that date. Only thing it doesn't do is flows, which we are going to leverage Fluent Bit as a syslog ingestor for.

Works great and far less API calls with no duplicate records. In an hour it was only ~800 or so API calls, and any rate limiting issues we were seeing have completely disappeared.

So it's absolutely possible once this connector is fixed.

[Nico-WA]

@JustinGrote - Ah, got it. I understand. I may dive into that rabbit hole to do it from scratch. Fun! But I do hope the connector gets resolved sooner rather than later.

[v-sudkharat]

Thanks @shaunyb93 for open the case.

[Weeman257]

Hi everyone,

glad i've found something about this connector. I also experience the mentioned issue:

Thats roughly 24 hours running the Data-Connector

I have a question:
I am just interested in Security related Events like IDS and File Scanned. How do i configure the Connector just to poll those instead flooding me with Configuration-Changes? Is everything configured with the Functions "ASim*_CiscoMerakiV**"? I don't get it. Its a blackbox for me without any documentation

[JustinGrote]

@Weeman257 it uses the new codeless connectors format and the code is on their Github, sadly the codeless connector (which uses the user agent SCUBA which I assume is a MS codename) is very black-box in terms of how it works low-level, that part does not appear to be open source.

The only thing you could potentially do here is modify the data collection rule it uses and change the transformKQL to only collect what you want. Note that if a transform rule filters more than 50% of the logs, anything above that it filters you still get billed for.

[Weeman257]

@JustinGrote Thats why i wanted to modify the queries instead of the Collection Rules :D to only pay for what i want to have. Sad story... So we have to wait on the fix for the duplicate ingestion

Maybe i will reach out to our Customer Success Manager from Microsoft to speed things up :D

[shaunyb93]

Literally just got off the phone with Microsoft support about this - sad to report that it doesn't look like they've really even looked into the issue at the moment so wouldn't hold breath on a fix... again, the connector is in a preview state so doubt any priority will be applied to fixing it

[Weeman257]

I will add some pressure :P

[JustinGrote]

If worse comes to worst I'll discuss with my company for publishing our offering as a marketplace one for a reasonable fee, it works really well.

[v-sudkharat]

Hi @JustinGrote / @shaunyb93 / @Weeman257,
Apologies for the delay in response.
Our concern team has fixed this issue and raised the PR with corrections, the changes will get reflected in upcoming solution version.
we will notify you once the new solution version gets published into the content hub.

Thanks!

[JustinGrote]

@v-sudkharat are you referring to this PR #11195 or something internal?

[v-sudkharat]

@JustinGrote, Yes,

@shaunyb93 / @Weeman257 / @JustinGrote. The solution has been live now, Kindly update your solution and let us know if your issue gets resolve.
Note: Please delete the existing data connector and then do the fresh installation/ update. Thanks!

[v-sudkharat]

Hi @shaunyb93, Could you please confirm the update resolves your issue?

[shaunyb93]

Hi @v-sudkharat
It does look like the issue is resolved
thanks

[v-sudkharat]

@shaunyb93 , Thanks!
Closing this issue. If you still need support for this issue, feel free to re-open it any time. Thank you for your co-operation.

[Weeman257]

Hi everyone, i still have duplicates:

[Weeman257]

One Addition: The Connector maxes out the Calls per Second (10 per second) which leads to a response code of the meraki api (Code [429)]

[shaunyb93]

@Weeman257 have you updated the solution and redeployed the connector as requested by @v-sudkharat ?

[JustinGrote]

Per looking at the PR, that only maybe fixes one of the log sources that I see. We developed our own connector and aren't wasting any more time on this one, until such time it's proven to be functional.