New Release v4.8.4 - Query Optimizations & More Configurable Controls

[jongpie]

Core Package Changes

Synchronous Context Optimizations

Several enhancements have been made to reduce the overhead that Nebula Logger consumes when logging - these enhancements help with some platform limits, such as query limits, CPU time limits, etc. (closes #311 reported by @LawrenceLoz). Many thanks again to @LawrenceLoz for doing some stellar analysis on the codebase to identify some of these optimizations.

• Added new LoggerParameter__mdt record EnableStackTraceParsing to disable stack trace parsing org-wide. Although parsing stack traces can help provide contextual info for reporting, the code consumes CPU time in Apex - orgs can now choose if they want to disable this functionality to improve Nebula Logger's performance.
• Added new LoggerParameter__mdt records to enable/disable if LogEntryEventBuilder queries run synchronously (default behavior). When these records are disabled (Value__c = 'false'), the corresponding queries run asynchronously after platform events are published. This means that the corresponding fields aren't set on LogEntryEvent__e, but are still populated on Log__c and LogEntry__c. Four (4) new custom metadata records are included:
  1. LoggerParameter.QueryAuthSessionDataSynchronously
  2. LoggerParameter.QueryNetworkDataSynchronously
  3. LoggerParameter.QueryOrganizationDataSynchronously
  4. LoggerParameter.QueryUserDataSynchronously
• Updated Logger class to internally cached the value of UserInfo.getSessionId(). The original (erroneous) assumption was that the platform would handle caching this, but repeated calls to the method seem to consume additional CPU time - the value is now cached in a private constant.

Caching System Improvements

Platform Cache is now being leveraged to supplement the existing transaction caching (partially closes #311, and closes #389 reported by @rmccu)

• Added platform cache partion LoggerCache that comes with 3MB allocated, based on the 3MB that Salesforce provides for security-approved managed packages. Sadly, Nebula Logger has not yet gone through this process yet, but still seemed like a sensible default size to use.
  • 1MB allocated to the org partition
  • 2MB allocated to the session partition
• Updated Apex class LoggerCache to now support 3 types of caching:
  1. LoggerCache.getTransactionCache(). This is the existing caching system that's been used for a few releases. It caches values in a transaction using a private static final Map<String, Object> constant. This is still used internally throughout LoggerCache to supplement some limitations with platform cache, such as not being able to cache null values
  2. LoggerCache.getOrganizationCache(). This leverages the org partition of Platform Cache
  3. LoggerCache.getSessionCache(). This leverages the session partition of Platform Cache
• Updated all existing caching in Nebula Logger to now leverages platform caching by default.
• Added a new LoggerParameter__mdt record 'UsePlatformCache' to control if Nebula Logger leverages platform cache. When 'UsePlatformCache' is disabled (Value__c = 'false'), Nebula Logger will only use Logger.getTransactionCache() (which is existing behavior before platform cache was introduced).

Large Data Volume (LDV) Optimizations

I've included some query-related enhancements to address some of the performance problems that @rmccu brought up for orgs with large data volumes (2+ millions Log__c records created daily)

• Added new LoggerParameter__mdt records to completely disable certain queries executed by Nebula Logger (closes Flags to control potentially "expensive" functionality for high-volume implementations #390 reported by @rmccu). Some of the queries used by Nebula Logger can perform poorly in orgs with large data volumes - the new records provide a way for orgs to completely disable certain queries, with the tradeoff that the corresponding fields will be null. Seven (7) queries can now be fully disabled - there are still some other queries throughout Nebula Logger that don't have this option, but these should be the most critical queries.
  1. LoggerParameter.QueryApexClassData - when disabled, the ApexClass object is not queried, and the relevant fields on LogEntry__c are null
  2. LoggerParameter.QueryAuthSessionData - when disabled, the AuthSession object is not queried, and the relevant fields on LogEntry__c are null
  3. LoggerParameter.QueryFlowDefinitionViewData - when disabled, the FlowDefinitionView object is not queried, and the relevant fields on LogEntry__c are null
  4. LoggerParameter.QueryNetworkData - when disabled, the Network object is not queried, and the relevant fields on LogEntry__c are null. Note: this only applies to orgs that use an Experience Cloud site (stored in the Network object)
  5. LoggerParameter.QueryOrganizationData - when disabled, the Organzation object is not queried, and the relevant fields on LogEntry__c are null
  6. LoggerParameter.QueryRelatedRecordData - when disabled, the dynamic queries used to query record names (based on the field LogEntry__c.RecordId__c) are not executed, the field LogEntry__c.RecordName__c is null
  7. LoggerParameter.QueryUserData - when disabled, the User object is not queried, and the relevant fields on LogEntry__c are null

Bugfixes

• Fixed LogEntry__c.Timestamp__c being inaccurately set in some edge cases. This was an unreported bug found where LogEntryEvent__e.TimestampString__c could be out of sync with LogEntryEvent__e.Timestamp__c, resulting in inaccurate data being stored in LogEntry__c.Timestamp__c. The bug has been fixed & some additional tests have been added to ensure that this continues to behave as expected.

Data Model Changes

• Added new fields LogEntryEvent__e.RequestId__c and Log__c.RequestId__c to store the value of System.Request.getCurrent().getRequestId(). In versions v3.1.0 through v4.7.2, this value was used for Log__c.TransactionId__c, but reverted to using a more reliable custom UUID in v4.7.3 (the original solution). The new RequestId__c fields provides a way to still report on this value, which can be helpful for tying Nebula Logger's data back to platform data & logs.

Test Improvements

• Fixed LogManagementDataSelector_Tests.it_returns_profiles_for_specified_profile_ids unit test method is failing when some of user profile names begin with lower-case letter #393 (reported by @mykhaylo-partyka) by removing an unnecessary & problematic call to List.sort() on the expected query results
• Added new ApexTestSuite LoggerExtraTests (only deployed to/used by the pipeline for integration tests)
• Added more integrations tests, primarily for Experience Site (Community/Network/Portal/too-many-names-ahhh) testing around Network querying & setting of related fields
• Make LoggerEngineDataSelector and LogManagementDataSelector classes & methods virtual & implemented test implementations in the respective test classes LoggerEngineDataSelector_Tests and LogManagementDataSelector_Tests to provide a cleaner way to test query executions within Nebula Logger's codebase.
• The test implementations currently usedin test classes still execute all of the related queries, so the queries aren't returning mock values (when possible) just yet. This PR has most of the foundational changes need to support full mocking, but I'm planning to revisit some of the tests throughout the codebase in a future release to implement mock return values when possible. That will make it easier to test queried data in general, and should also give the tests classes another nice boost in test speed (in addition to some other substantial test speed improvements made in v4.7.1).

Experience Site Test Metadata Improvements

These changes only impact the pipeline to help validate logic related to orgs that use Nebula Logger for Experience Cloud sites (aka Community Cloud) - these changes don't have any direct functional changes to Nebula Logger

• Added high-volume traffic Community profile & related metadata changes for pipeline testing purposes
• Added more Apex tests for Experience Cloud scenarios in the extra-tests folder. These tests are only used in the pipeline - they are not included in the core package.

Async Failure Additions Plugin Package Changes

Created v1.0.2 of the Async Failure Additions plugin

• @jamessimone implemented an incredible, new feature for this plugin to automatically log Flow errors that occur in Screen Flows, using the platform event FlowExecutionErrorEvent (closes Add logging for FlowExecutionErrorEvent in Async Failures Addition plugin #372 - implemented in Flow Execution Error Event Handling #392)

Slack Plugin Package Changes

Created v1.5.1 of the Slack plugin

• Updated the formatting of the field 'Last Log Entry Message' to handle line breaks

Many thanks to @jamessimone , @LawrenceLoz, @rmccu, and @mykhaylo-partyka for all of the help with this release!

---

This discussion was created from the release Query Optimizations & More Configurable Controls.