Memory leak using Async Local with CommunicationClientCache

[oshvartz]

I'm running Actor Service in windows .net core 3.1 (nodes are windows VMSS in azure) I notice slow memory leak
Analyzing the process dump I notice - the memory contains data we store in AyncLocal - we use it for tracing - but this is not causing memory leaks in other place this tracing library is used
executed gc root of what is holding the memory I found the following:

   00000238451B1EE8 (ref counted handle)
    -> 0000023BDD759220 Microsoft.ServiceFabric.FabricTransport.V2.Client.FabricTransportClientConnectionEventHandlerBroker
    -> 0000023BDD758E88 Microsoft.ServiceFabric.Services.Remoting.V2.FabricTransport.Client.FabricTransportRemotingClientEventHandler
    -> 0000023BDD7592C0 System.EventHandler`1[[Microsoft.ServiceFabric.Services.Communication.Client.CommunicationClientEventArgs`1[[Microsoft.ServiceFabric.Services.Remoting.V2.FabricTransport.Client.FabricTransportServiceRemotingClient, Microsoft.ServiceFabric.Services.Remoting]], Microsoft.ServiceFabric.Services]]
    -> 0000023A45682A08 Microsoft.ServiceFabric.Services.Remoting.V2.FabricTransport.Client.FabricTransportServiceRemotingClientFactoryImpl
    -> 0000023A45682C70 Microsoft.ServiceFabric.Services.Communication.Client.CommunicationClientCache`1[[Microsoft.ServiceFabric.Services.Remoting.V2.FabricTransport.Client.FabricTransportServiceRemotingClient, Microsoft.ServiceFabric.Services.Remoting]]
    -> 0000023A45683128 System.Threading.Timer
    -> 0000023A456831A0 System.Threading.TimerHolder
    -> 00000239DD1ED3F8 System.Threading.TimerQueueTimer
    -> 00000239DD1ED2F8 System.Threading.ExecutionContext
    -> 00000239DD1ED2B8 System.Threading.AsyncLocalValueMap+ThreeElementAsyncLocalValueMap
    -> 00000239DD1ECF70 Microsoft.Azure.Sentinel.AOPOperations.OperationContext.OperationContextProvider+Wrapper

searching for related issues I found this issue aspnet/HttpClientFactory#171
not sure if it's related,but FabricTransportServiceRemotingClient is using System.Threading.Timers internally and may casue simillar issue as in the issue above

[oshvartz]

I see some other gc roots all related to system.threading.timers in code that are getting refrence to my async local such as

00000274910715A0 (ref counted handle)
    -> 0000027491B814B8 System.Fabric.KeyValueStoreReplica+KeyValueStoreEventHandlerBroker
    -> 0000027491B80E08 Microsoft.ServiceFabric.Actors.Runtime.KvsActorStateProvider+KeyValueStoreWrapper
    -> 0000027491A27CD0 System.Fabric.StatefulServiceInitializationParameters
    -> 0000027591AD32B0 System.Fabric.CodePackageActivationContext
    -> 0000027591BA3C48 System.EventHandler`1[[System.Fabric.PackageModifiedEventArgs`1[[System.Fabric.ConfigurationPackage, System.Fabric]], System.Fabric]]
    -> 0000027591BA3BF0 System.Object[]
    -> 0000027491B7FF48 System.EventHandler`1[[System.Fabric.PackageModifiedEventArgs`1[[System.Fabric.ConfigurationPackage, System.Fabric]], System.Fabric]]
    -> 0000027491B5DD20 Microsoft.ServiceFabric.Actors.Runtime.KvsActorStateProvider
    -> 0000027491B5E170 Microsoft.ServiceFabric.Actors.Runtime.VolatileLogicalTimeManager
    -> 0000027491B5E298 System.Threading.Timer
    -> 0000027491B5E310 System.Threading.TimerHolder
    -> 0000027491B5E2B0 System.Threading.TimerQueueTimer
    -> 0000027591B13350 System.Threading.TimerQueue
    -> 0000027491B986C8 System.Threading.TimerQueueTimer
    -> 0000027491B98150 System.Threading.TimerQueueTimer
    -> 0000027491B56EC8 System.Threading.TimerQueueTimer
    -> 0000027491B56950 System.Threading.TimerQueueTimer
    -> 0000027591B7CB28 System.Threading.TimerQueueTimer
    -> 0000027591B7C1A8 System.Threading.TimerQueueTimer
    -> 0000027491B8F8B0 System.Threading.TimerQueueTimer
    -> 0000027491B8F338 System.Threading.TimerQueueTimer
    -> 0000027591B906E8 System.Threading.TimerQueueTimer
    -> 0000027591B90170 System.Threading.TimerQueueTimer
    -> 0000027591BD0318 System.Threading.TimerQueueTimer
    -> 0000027491DB9A58 System.Threading.TimerQueueTimer
    -> 0000027591B934B0 System.Threading.TimerQueueTimer
    -> 0000027491B89758 System.Threading.TimerQueueTimer
    -> 0000027491B86C90 System.Threading.TimerQueueTimer
    -> 0000027491B4BB10 System.Threading.TimerQueueTimer
    -> 0000027491B32948 System.Threading.TimerQueueTimer
    -> 0000027491AFFE28 System.Threading.TimerQueueTimer
    -> 0000027491DB8B10 System.Threading.TimerQueueTimer
    -> 0000027591DD85F8 System.Threading.ExecutionContext
    -> 0000027591DD85B8 System.Threading.AsyncLocalValueMap+ThreeElementAsyncLocalValueMap
    -> 0000027591DD7C58 Microsoft.Azure.Sentinel.AOPOperations.OperationContext.OperationContextProvider+Wrapper
    -> 0000027591DD7C28 Microsoft.Azure.Sentinel.AOPOperations.OperationContext.OperationContext