Heapdump issue when retrieving query

[garysandler]

Greetings,

I execute: mcdev retrieve developergroup/_ParentBU_ query

This results int he following error. How do I go about resolving this? I am running the tool on a GCP Compute Engine VM.

`<--- Last few GCs --->

[4573:0x7538a30] 161967 ms: Mark-Compact 1794.6 (2085.6) -> 1788.2 (2080.1) MB, 1640.90 / 0.00 ms (average mu = 0.107, current mu = 0.012) allocation failure; scavenge might not succeed
[4573:0x7538a30] 163823 ms: Mark-Compact 1796.8 (2088.3) -> 1792.3 (2084.3) MB, 1844.59 / 0.00 ms (average mu = 0.056, current mu = 0.007) allocation failure; scavenge might not succeed

<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
1: 0xc99970 node::Abort() [node]
2: 0xb6ffcb [node]
3: 0xebe9f0 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [node]
4: 0xebecd7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [node]
5: 0x10d0785 [node]
6: 0x10d0d14 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [node]
7: 0x10e7c04 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::internal::GarbageCollectionReason, char const*) [node]
8: 0x10e841c v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
9: 0x10be721 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
10: 0x10bf8b5 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
11: 0x109ce26 v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [node]
12: 0x14f7c56 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [node]
13: 0x78710bed9ef6
Aborted (core dumped)`

[JoernBerkefeld]

hi @garysandler
I haven't personally set this up on GCP and the logs you provided seem to be generated by the server, not by mcdev. Therefore, expectation management: The following will be an educated guess but I certainly don't know this for sure nor can I provide adequate support for GCP in general.

The relevant error message seems to be JavaScript heap out of memory.

You were trying to download queries which also retrieves folders and dataExtensions to be able to resolve dependencies. My guess is, that your process wasn't allocated enough RAM to do this. I would try to beef it up a little and check if the error disappears.

[garysandler]

It turned out to be frustratingly intermittent.
Resources were increased but sometimes more or less or all was extracted - not consistently reproducible unfortunately.

What didn’t help was the sheer volume/size of queries and number of historical objects in the environment.

Ultimately I was able to run the processes component-by-component and eventually extract everything (or so I think).

I suspect when this process is automated this will be problematic. The client has not proceeded to this point yet.

[garysandler]

I’ll mark as answered as this most likely helped.

[JoernBerkefeld]

hmmmm would you remember how many queries you were downloading? I'm interested to run a test with similarly high numbers on my computer and monitor RAM & CPU usage while doing so.

the other day I successfully deployed 1000 dataExtensions with my laptop without an issue... it does have 16GB of RAM but most of that is usually used up by browsers and other apps. I suppose what I'm trying to say is that I'm surprised the process just died on you. More test data might help here.

[garysandler]

I’ll get the stats (volumes and systems resource values) and revert here.

What I observed at the time was that the extraction took at least a couple of hours when it did complete successfully. This could be resource related or that I was downloading to a GCP virtual machine - but it was slow going and very unlike using my laptop (albeit on a much smaller BU as a proof of concept).

[JoernBerkefeld]

hmmm. so what ive seen some 18 months ago was that - somehow - when I used it from my home's internet connection, or actually internet in my home town in general (Berlin!) it sometimes took forever and had lots of timeouts --> that actually when I invested a lot of time in making timeout handling more robust inside of mcdev. Months later, the problem just went away.

at the very same time though, when my partner in crime, @DougMidgley, used mcdev in the Netherlands, it was a lot faster, similar to the speeds we were used to AND he didn't face these timeout issues that I was facing. All that despite testing out the same instance of SFMC.

Our conclusion back then was that the routing to that instance's API servers must have been a lot worse for me than it was for him.

Why bring this up? My guess is, that maybe the data center of GCP that you were using is suffering from a similar issue that I experienced back then. Given that you've also seen a huge diff between your laptop and that GCP machine, maybe try out spinning it up in a different region / data center? If that ends up helping you might be able to open up a case with GCP to get the actual issue behind it fixed.

[garysandler]

The region could be an issue, I haven’t investigated too deep into that at this point. I know the GCP region is in the US-central. My client is in the Middle East but I have not confirmed what region their SFMC instance is in. Just to give an idea of the volumes, the retrieve folder for the parent BU is approx. 2.8GB. There are around 9000 queries (many of the most unwieldly sql imaginable hundreds/thousands of lines long), 20k data extensions, I didn’t try count the asset folder where which was by far the largest.

[JoernBerkefeld]

hm, my biggest INSTANCE takes up 2.9 GB on disk... but that across 100 BUs. I would strongly suggest to invest in a few sprints solely for cleanup.
nobody can tell me that 9k queries are actually used.

[JoernBerkefeld]

btw: i've seen 70k data extensions on a single BU... that was scary and it took about 80 minutes to retrieve that BU in full.
i ended up skipping DEs on that BU whenever I could

[garysandler]

You’re right – the environment is polluted with historically versioned, unused, non-working objects. What I am getting the client to realize with mcdev and git is that they don’t need keep so much bloat on SFMC. Between the Marketing teams / SFMC developers/admins and Salesforce themselves they have never considered leveraging a version control repo and deployment pipeline. Or even a basic cleanup for that matter. Therefore they have 2.8GB on a single BU!

[JoernBerkefeld]

ouch - but it's music to my ears that this helps you convince your client :)

[garysandler]

The tool is fantastic. Being command line based I could immediately see the potential. I’d be happy to chat offline and share how this tool solves some of the clients biggest challenges in this space.

[JoernBerkefeld]

sure, hit me up on linkedin, will ya? there are too many Gary Sandlers ;-)
https://www.linkedin.com/in/joernberkefeld/