NH-91603: custom otlp metrics through otlp protocol

CONFIGURATION.md

@@ -112,26 +112,27 @@ The configuration file should be Ruby code that sets key/values in the hash expo
 
 ## Reference
 
-Environment Variable | Config File Key | Description | Default
--------------------- | --------------- | ----------- | -------
-`SW_APM_AUTO_CONFIGURE` | N/A  | By default the library is configured to work out-of-the-box with all automatic instrumentation libraries enabled. Set this to `false` to custom initialize the library with configuration options for instrumentation, see [Programmatic Configuration](#programmatic-configuration) for details. | `true`
-`SW_APM_COLLECTOR` | N/A | Override the default collector endpoint to which the library connects and exports data. It should be defined using the format host:port. | `apm.collector.na-01.cloud.solarwinds.com:443`
-`SW_APM_CONFIG_RUBY` | N/A | Override the default location for the configuration file. This can be an absolute or relative filename, or the directory under which the `solarwinds_apm_config.rb` file would be looked for. | None
-`SW_APM_DEBUG_LEVEL` | `:debug_level` | Set the library's logging level, valid values are -1 through 6 (least to most verbose). <br> Setting -1 disables logging from the library. | 3
-`SW_APM_EC2_METADATA_TIMEOUT` | `:ec2_metadata_timeout` | Timeout for AWS IMDS metadata retrieval in milliseconds. | 1000
-`SW_APM_ENABLED` | N/A | Enable/disable the library, setting `false` is an alternative to uninstalling `solarwinds_apm` since it will prevent the library from loading. | `true`
-`SW_APM_LOG_FILEPATH` | N/A | Configure the log file path for the C extension, e.g. `export SW_APM_LOG_FILEPATH=/path/file_path.log`. If set, messages from the C extension are written to the specified file instead of stderr.  | None
-`SW_APM_PROXY` | `:http_proxy` | Configure an HTTP proxy through which the library connects to the collector. | None
-`SW_APM_SERVICE_KEY` | `:service_key` | API token and service name in the form of `token:service_name`, **required**. | None
-`SW_APM_TAG_SQL` | `:tag_sql` | Enable/disable injecting trace context into supported SQL statements. Set to boolean true or (or string `true` in env var) to enable, see [Tag Query with Trace Context](#tag-query-with-trace-context) for details.| `false`
-`SW_APM_TRIGGER_TRACING_MODE` | `:trigger_tracing_mode` | Enable/disable trigger tracing for the service.  Setting to `disabled` may impact DEM visibility into the service. | `enabled`
-`SW_APM_TRUSTEDPATH` | N/A | The library uses the host system's default trusted CA certificates to verify the TLS connection to the collector. To override the default, define the trusted certificate path configuration option with an absolute path to a specific trusted certificate file in PEM format. | None
-`SW_APM_LAMBDA_PRELOAD_DEPS` | N/A | This option only takes effect in the AWS Lambda runtime. Set to `false` to disable the attempt to preload function dependencies and install instrumentations. | `true`
-`SW_APM_TRANSACTION_NAME` | N/A | Customize the transaction name for all traces, typically used to target specific instrumented lambda functions. _Precedence order_: custom SDK > `SW_APM_TRANSACTION_NAME` > automatic naming | None
-N/A | `:log_args` | Enable/disable the collection of URL query parameters, set to boolean false to disable. | true
-N/A | `:log_traceId` | Configure the insertion of trace context into application logs, setting `:traced` would include the available context fields such as trace_id, span_id into log messages. | `:never`
-N/A | `:tracing_mode` | Enable/disable the tracing mode for this service, setting `:disabled` would suppress all trace spans and metrics. | `:enabled`
-N/A | `:transaction_settings` | Configure tracing mode per transaction, aka transaction filtering. See [Transaction Filtering](#transaction-filtering) for details.| None
+Environment Variable | Option | Config File Key | Description | Default
+-------------------- | ------ | --------------- | ----------- | -------
+`SW_APM_SERVICE_KEY` | required | `:service_key` | API token and service name in the form of `token:service_name`. | None
+`SW_APM_AUTO_CONFIGURE` | optional | N/A  | By default the library is configured to work out-of-the-box with all automatic instrumentation libraries enabled. Set this to `false` to custom initialize the library with configuration options for instrumentation, see [Programmatic Configuration](#programmatic-configuration) for details. | `true`
+`SW_APM_COLLECTOR` | optional | N/A | Override the default collector endpoint to which the library connects and exports data. It should be defined using the format host:port. | `apm.collector.na-01.cloud.solarwinds.com:443`
+`SW_APM_CONFIG_RUBY` | optional | N/A | Override the default location for the configuration file. This can be an absolute or relative filename, or the directory under which the `solarwinds_apm_config.rb` file would be looked for. | None
+`SW_APM_DEBUG_LEVEL` | optional | `:debug_level` | Set the library's logging level, valid values are -1 through 6 (least to most verbose). <br> Setting -1 disables logging from the library. | 3
+`SW_APM_EC2_METADATA_TIMEOUT` | optional | `:ec2_metadata_timeout` | Timeout for AWS IMDS metadata retrieval in milliseconds. | 1000
+`SW_APM_ENABLED` | optional | N/A | Enable/disable the library, setting `false` is an alternative to uninstalling `solarwinds_apm` since it will prevent the library from loading. | `true`
+`SW_APM_LOG_FILEPATH` | optional | N/A | Configure the log file path for the C extension, e.g. `export SW_APM_LOG_FILEPATH=/path/file_path.log`. If set, messages from the C extension are written to the specified file instead of stderr.  | None
+`SW_APM_PROXY` | optional | `:http_proxy` | Configure an HTTP proxy through which the library connects to the collector. | None
+`SW_APM_TAG_SQL` | optional | `:tag_sql` | Enable/disable injecting trace context into supported SQL statements. Set to boolean true or (or string `true` in env var) to enable, see [Tag Query with Trace Context](#tag-query-with-trace-context) for details.| `false`
+`SW_APM_TRIGGER_TRACING_MODE` | optional | `:trigger_tracing_mode` | Enable/disable trigger tracing for the service.  Setting to `disabled` may impact DEM visibility into the service. | `enabled`
+`SW_APM_TRUSTEDPATH` | optional | N/A | The library uses the host system's default trusted CA certificates to verify the TLS connection to the collector. To override the default, define the trusted certificate path configuration option with an absolute path to a specific trusted certificate file in PEM format. | None
+`SW_APM_LAMBDA_PRELOAD_DEPS` | optional | N/A | This option only takes effect in the AWS Lambda runtime. Set to `false` to disable the attempt to preload function dependencies and install instrumentations. | `true`
+`SW_APM_TRANSACTION_NAME` | optional | N/A | Customize the transaction name for all traces, typically used to target specific instrumented lambda functions. _Precedence order_: custom SDK > `SW_APM_TRANSACTION_NAME` > automatic naming | None
+`SW_APM_EXPORT_METRICS_ENABLED` | optional | N/A | Enable the custom metrics to export to swo backend through otlp metrics protocol | `false`
+N/A | optional | `:log_args` | Enable/disable the collection of URL query parameters, set to boolean false to disable. | true
+N/A | optional | `:log_traceId` | Configure the insertion of trace context into application logs, setting `:traced` would include the available context fields such as trace_id, span_id into log messages. | `:never`
+N/A | optional | `:tracing_mode` | Enable/disable the tracing mode for this service, setting `:disabled` would suppress all trace spans and metrics. | `:enabled`
+N/A | optional | `:transaction_settings` | Configure tracing mode per transaction, aka transaction filtering. See [Transaction Filtering](#transaction-filtering) for details.| None
 
 ### Transaction Filtering
 
@@ -151,6 +152,48 @@ SolarWindsAPM::Config[:transaction_settings] = [
 ]
 ```
 
+### OpenTelemetry Metrics Export with Custom Metrics
+
+If user want to export the opentelemetry metrics through opentelemetry otlp protocol to solarwinds observability, user can do it directly after require `solarwinds_apm`. Since we don't provide the default metrics exporter, user needs to install `opentelemetry-exporter-otlp-metrics`. However, `solarwinds_apm` agent is not responsible to decide when to export. User need to export by themselves (e.g. `OpenTelemetry.meter_provider.metric_readers.each { |reader| reader.pull if reader.respond_to?(:pull) }` or use [PeriodicMetricReader](https://github.com/open-telemetry/opentelemetry-ruby/blob/main/metrics_sdk/lib/opentelemetry/sdk/metrics/export/periodic_metric_reader.rb) to export with desired time interval.
+
+Required setting:
+
+```console
+export SW_APM_EXPORT_METRICS_ENABLED='true' # enable the metrics
+export OTEL_EXPORTER_OTLP_METRICS_ENDPOINT='your desired endpoint'
+export OTEL_EXPORTER_OTLP_METRICS_HEADERS='headers should include authentication e.g. authorization=Bearer ***'
+```
+
+Without setting `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT` and `OTEL_EXPORTER_OTLP_METRICS_HEADERS`, `solarwinds_apm` will provide the default value. These value will export metrics to solarwinds apm production.
+
+```console
+OTEL_EXPORTER_OTLP_METRICS_ENDPOINT: https://otel.collector.na-01.solarwinds.com:443/v1/metrics
+OTEL_EXPORTER_OTLP_METRICS_HEADERS: authorization=Bearer <SW_APM_SERVICE_KEY (without service name)>
+```
+
+Example:
+
+```ruby
+require 'opentelemetry-exporter-otlp-metrics'
+require 'solarwinds_apm'
+
+# initialize the metrics exporter
+otlp_metric_exporter = OpenTelemetry::Exporter::OTLP::Metrics::MetricsExporter.new
+
+# add metrics exporter to meter_provider
+OpenTelemetry.meter_provider.add_metric_reader(otlp_metric_exporter)
+
+# initialize meter
+meter = OpenTelemetry.meter_provider.meter("SAMPLE_METER_NAME")
+
+# create a new metrics instrument
+histogram = meter.create_histogram('sample_histogram', unit: 'smidgen', description: 'desscription')
+
+histogram.record(123, attributes: {'foo' => 'bar'})
+
+# you should see the metrics in swo backend
+```
+
 ### Tag Query with Trace Context
 
 You can set the environment variable `SW_APM_TAG_SQL` or configuration file option `:tag_sql` to true to enable appending the current trace context into a database query as a SQL comment. For example:

lib/solarwinds_apm.rb

@@ -20,6 +20,7 @@
 
   begin
     if RUBY_PLATFORM.include?('linux')
+      require 'solarwinds_apm/constants'
       require 'solarwinds_apm/config'
       require 'solarwinds_apm/oboe_init_options' # setup oboe reporter options
       if !SolarWindsAPM::OboeInitOptions.instance.service_key_ok? && !SolarWindsAPM::OboeInitOptions.instance.lambda_env
@@ -56,7 +57,6 @@
         SolarWindsAPM::Reporter.start # start the reporter, any issue will be logged
 
         if SolarWindsAPM.loaded
-          require 'solarwinds_apm/constants'
           require 'solarwinds_apm/api'
           require 'solarwinds_apm/support'
           require 'solarwinds_apm/opentelemetry'

lib/solarwinds_apm/constants.rb

@@ -32,5 +32,9 @@ module Constants
     INTL_SWO_OTEL_STATUS             = 'otel.status_code'
     INTL_SWO_OTEL_STATUS_DESCRIPTION = 'otel.status_description'
     INTERNAL_TRIGGERED_TRACE         = 'TriggeredTrace'
+
+    APPOPTICS_ENDPOINT               = ['collector.appoptics.com', 'collector-stg.appoptics.com', 'collector.appoptics.com:443', 'collector-stg.appoptics.com:443'].freeze
+    SW_OTEL_METRICS_ENDPOINT         = 'https://otel.collector.na-01.solarwinds.com:443/v1/metrics'
+    SW_APM_ENDPOINT                  = 'apm.collector.cloud.solarwinds.com'
   end
 end

lib/solarwinds_apm/oboe_init_options.rb

@@ -170,10 +170,7 @@ def determine_the_metric_model
     end
 
     def appoptics_collector?
-      allowed_uri = ['collector.appoptics.com', 'collector-stg.appoptics.com',
-                     'collector.appoptics.com:443', 'collector-stg.appoptics.com:443']
-
-      (allowed_uri.include? ENV.fetch('SW_APM_COLLECTOR', nil))
+      (SolarWindsAPM::Constants::APPOPTICS_ENDPOINT.include? ENV.fetch('SW_APM_COLLECTOR', nil))
     end
 
     def java_collector?(uri)

lib/solarwinds_apm/opentelemetry.rb

@@ -8,6 +8,7 @@
 
 require 'opentelemetry/sdk'
 require 'opentelemetry/instrumentation/all'
+require 'opentelemetry-metrics-sdk'
 
 # TODO: in future, it should add opentelemetry-metrics-sdk and require it here
 

lib/solarwinds_apm/otel_config.rb

@@ -91,6 +91,50 @@ def self.validate_propagator(propagators)
       disable_agent(reason: 'Missing tracecontext propagator.') unless ([::OpenTelemetry::Trace::Propagation::TraceContext::TextMapPropagator, ::OpenTelemetry::Baggage::Propagation::TextMapPropagator] - propagators.map(&:class)).empty?
     end
 
+    # use the default OTEL_EXPORTER_OTLP_METRICS_ENDPOINT or OTEL_EXPORTER_OTLP_ENDPOINT if exist
+    # otherwise, use default SW_OTEL_METRICS_ENDPOINT if the collector is not appoptics
+    def self.determine_otlp_metrics_endpoint
+      return unless ENV['OTEL_EXPORTER_OTLP_METRICS_ENDPOINT'].to_s.empty? && ENV['OTEL_EXPORTER_OTLP_ENDPOINT'].to_s.empty?
+
+      if SolarWindsAPM::Constants::APPOPTICS_ENDPOINT.include?(ENV['SW_APM_COLLECTOR'])
+        SolarWindsAPM.logger.warn { 'Endpoint is AppOptics. AppOptics does not support OTLP metrics export. No custom metrics will be exported.' }
+      else
+        # SW_OTEL_METRICS_ENDPOINT is the default endpoint for production
+        # for staging, just use OTEL_EXPORTER_OTLP_METRICS_ENDPOINT
+        ENV['OTEL_EXPORTER_OTLP_METRICS_ENDPOINT'] = SolarWindsAPM::Constants::SW_OTEL_METRICS_ENDPOINT
+      end
+
+      SolarWindsAPM.logger.warn { "OTLP metrics endpoint: #{ENV['OTEL_EXPORTER_OTLP_METRICS_ENDPOINT'] || ENV['OTEL_EXPORTER_OTLP_ENDPOINT'] || 'No valid endpoint'}." }
+    end
+
+    def self.setup_otlp_metrics
+      determine_otlp_metrics_endpoint
+
+      token, service_name = ENV['SW_APM_SERVICE_KEY'].to_s.split(':')
+
+      # if no explicit define the headers, use the composed headers from SW_APM_SERVICE_KEY
+      if ENV['OTEL_EXPORTER_OTLP_METRICS_HEADERS'].to_s.empty? && ENV['OTEL_EXPORTER_OTLP_HEADERS'].to_s.empty?
+
+        if token.nil?
+          SolarWindsAPM.logger.error { 'No valid SW_APM_SERVICE_KEY present for OTLP_METRICS_HEADERS.' }
+          return
+        end
+
+        ENV['OTEL_EXPORTER_OTLP_METRICS_HEADERS'] = "authorization=Bearer #{token}"
+        SolarWindsAPM.logger.warn { "OTLP metrics masked headers: authorization=Bearer #{mask_token(token)}" }
+      end
+
+      ENV['OTEL_RESOURCE_ATTRIBUTES'] = "sw.data.module=apm,service.name=#{service_name || ENV['OTEL_SERVICE_NAME'] || ''}" + ENV['OTEL_RESOURCE_ATTRIBUTES'].to_s
+      SolarWindsAPM.logger.warn { "OTLP metrics resource attributes: #{ENV.fetch('OTEL_RESOURCE_ATTRIBUTES', nil)}." }
+    end
+
+    def self.mask_token(token)
+      token = token.to_s
+      return '*' * token.length if token.length <= 4
+
+      "#{token[0, 2]}#{'*' * (token.length - 4)}#{token[-2, 2]}"
+    end
+
     def self.initialize
       unless defined?(::OpenTelemetry::SDK::Configurator)
         disable_agent(reason: 'missing OpenTelemetry::SDK::Configurator; opentelemetry seems not loaded.')
@@ -114,6 +158,8 @@ def self.initialize
       # for dbo, traceparent injection as comments
       require_relative 'patch/tag_sql_patch' if SolarWindsAPM::Config[:tag_sql]
 
+      setup_otlp_metrics if ENV['SW_APM_EXPORT_METRICS_ENABLED'].to_s == 'true'
+
       ::OpenTelemetry::SDK.configure { |c| c.use_all(@@config_map) }
 
       validate_propagator(::OpenTelemetry.propagation.instance_variable_get(:@propagators))

solarwinds_apm.gemspec

@@ -40,7 +40,9 @@ Gem::Specification.new do |s|
 
   s.extensions = ['ext/oboe_metal/extconf.rb']
 
+  s.add_dependency('opentelemetry-exporter-otlp-metrics', '>= 0.1.0')
   s.add_dependency('opentelemetry-instrumentation-all', '>= 0.31.0')
+  s.add_dependency('opentelemetry-metrics-sdk', '>= 0.1.0')
   s.add_dependency('opentelemetry-sdk', '>= 1.2.0')
 
   s.required_ruby_version = '>= 2.7.0'

test/api/set_transaction_name_test.rb

@@ -7,7 +7,6 @@
 require './lib/solarwinds_apm/support/txn_name_manager'
 require './lib/solarwinds_apm/opentelemetry'
 require './lib/solarwinds_apm/otel_config'
-require './lib/solarwinds_apm/constants'
 require './lib/solarwinds_apm/api'
 
 describe 'SolarWinds Set Transaction Name Test' do

test/initest_helper.rb

@@ -8,6 +8,7 @@
 require 'minitest/spec'
 require 'minitest/reporters'
 require './lib/solarwinds_apm/logger'
+require './lib/solarwinds_apm/constants'
 
 ENV['SW_APM_SERVICE_KEY'] = 'this-is-a-dummy-api-token-for-testing-111111111111111111111111111111111:test-service'
 

test/minitest_helper.rb

@@ -21,6 +21,7 @@
 
 require './lib/solarwinds_apm/version'
 require './lib/solarwinds_apm/logger'
+require './lib/solarwinds_apm/constants'
 
 # simplecov coverage information
 require 'simplecov'

test/opentelemetry/otlp_processor_test.rb

@@ -5,7 +5,6 @@
 
 require 'minitest_helper'
 require './lib/solarwinds_apm/opentelemetry'
-require './lib/solarwinds_apm/constants'
 require './lib/solarwinds_apm/support/txn_name_manager'
 require './lib/solarwinds_apm/otel_config'
 require './lib/solarwinds_apm/api'

test/opentelemetry/solarwinds_exporter_test.rb

@@ -10,7 +10,6 @@
 require './lib/solarwinds_apm/support/txn_name_manager'
 require './lib/solarwinds_apm/oboe_init_options'
 require './lib/solarwinds_apm/config'
-require './lib/solarwinds_apm/constants'
 
 describe 'SolarWindsExporterTest' do
   before do

test/opentelemetry/solarwinds_processor_test.rb

@@ -5,7 +5,6 @@
 
 require 'minitest_helper'
 require './lib/solarwinds_apm/opentelemetry'
-require './lib/solarwinds_apm/constants'
 require './lib/solarwinds_apm/support/txn_name_manager'
 require './lib/solarwinds_apm/otel_config'
 require './lib/solarwinds_apm/api'

test/opentelemetry/solarwinds_propagator_test.rb

@@ -7,7 +7,6 @@
 require 'minitest/mock'
 require './lib/solarwinds_apm/opentelemetry'
 require './lib/solarwinds_apm/support/x_trace_options'
-require './lib/solarwinds_apm/constants'
 require './lib/solarwinds_apm/support/utils'
 require './lib/solarwinds_apm/support/transaction_cache'
 require './lib/solarwinds_apm/support/transaction_settings'

test/opentelemetry/solarwinds_sampler_test.rb

@@ -6,7 +6,6 @@
 require 'minitest_helper'
 require './lib/solarwinds_apm/opentelemetry'
 require './lib/solarwinds_apm/support/x_trace_options'
-require './lib/solarwinds_apm/constants'
 require './lib/solarwinds_apm/support/utils'
 require './lib/solarwinds_apm/support/transaction_cache'
 require './lib/solarwinds_apm/support/transaction_settings'

test/patch/sw_mysql2_patch_integrate_test.rb

@@ -10,7 +10,6 @@
 require './lib/solarwinds_apm/otel_config'
 require './lib/solarwinds_apm/api'
 require './lib/solarwinds_apm/support'
-require './lib/solarwinds_apm/constants'
 require './lib/solarwinds_apm/oboe_init_options'
 
 # rubocop:disable Naming/MethodName