Documentation for Provide a way to send Liberty logs to OpenTelemetry

[pgunapal]

Feature epic details

• For the title of this issue, type: Documentation, Development epic name
• Link to development epic: Provide a way to send Liberty logs to OpenTelemetry open-liberty#27711
• Target GA release: 24.0.0.9

Operating systems

Does the documentation apply to all operating systems?

• Yes
• No; specify operating systems: ______

Summary

Provide a concise summary of your feature. What is the update, why does it matter, and to whom? What do 80% of target users need to know to be most easily productive using your runtime update?

As part of the mpTelemetry-2.0 feature, it will be sending both runtime and application logs to OpenTelemetry, that are correlatable with traces and metrics. The following Open Liberty log sources can be collected by OpenTelemetry: message, trace, and ffdc, where the fields will be mapped to the OpenTelemetry Logs Data Model.

Create a new topic

To create a topic, specify a first draft of the topic that you want added and the section in the navigation where the topic should go.

• Add a new document for MP Telemetry 2.0 under the Docs overview / Reference / Features / MicroProfile Telemetry section.
  Add information about the new <mpTelemetry/> server configuration element and sources attribute, along with the supported log sources : message, trace, ffdc. More information can be found in the Configuration section below.

• Add a new document to show the mapping table between OpenLiberty Log fields and OpenTelemetry Log Data Model for each supported log source, similar to the existing reference list document for JSON , under the Docs overview / Operations/ Logs section.

MicroProfile Telemetry 2.0 log events reference list

Message Events Reference List

(Refer to Slide 11 in the UFO: https://ibm.box.com/s/ihoa8t3h18e8w2t41gdcuveo90q24mrd)

Example from the OpenTelemetry Collector output for a message event from an application:

timeUnixNano=“1717512929163000000”     
observeTimeUnixNano=“1717512929602000000”
severityNumber=“13”
severityText=“W”
body=" BADAP0001W: Test Warning."
traceId=“a4fa4f34167d2bcbcfb94588d37b332”
spanId=“63522117c3492d42” 
Attributes:
            io.openliberty.type="liberty_message"
            io.openliberty.message_id=“BADAP0001W"
            io.openliberty.sequence=“1717512929163_0000000000012
            io.openliberty.module=“com.ibm.ws.lumberjack.badness.Angry”
            thread.id=“65”
            thread.name=“Default Executor-thread-2”
            io.openliberty.ext.app_name=“badapp”  

Trace Events Reference List

(Refer to Slide 11 in the UFO: https://ibm.box.com/s/ihoa8t3h18e8w2t41gdcuveo90q24mrd)

Example from the OpenTelemetry Collector output for a trace event from an application:

timeUnixNano=“11234512231363000000”     
observeTimeUnixNano=“17123512929602000000”
severityNumber=“1”
severityText=“3”
body="finest trace"
traceId=“3fced7fbfbc062349c66a8c1a37c31b2”
spanId=“4be39c323e375d9e” 
Attributes:
            io.openliberty.type="liberty_trace"
            io.openliberty.sequence=“12312363_0000000000021”
            io.openliberty.module=“com.test.app”
            thread.id=“79”
            thread.name=“Default Executor-thread-12”
            io.openliberty.ext.app_name=“testapp”  

FFDC Event Reference List

(Refer to Slide 12 in the UFO: https://ibm.box.com/s/ihoa8t3h18e8w2t41gdcuveo90q24mrd)

Example from the OpenTelemetry Collector output for a ffdc event from an application:

timeUnixNano=“1232335231363000000”     
observeTimeUnixNano=“123455512929602000000”
severityNumber=“13”
body="Cannot invoke "java.lang.String.toString()" because "myString" is null"
traceId=“3fced7fbfbc062349c66a8c1a37c31b2”
spanId=“4be39c323e375d9e” 
Attributes:
            io.openliberty.type="liberty_ffdc"
            exception.message="Cannot invoke "java.lang.String.toString()" because "myString" is null"
            exception.stacktrace="java.lang.NullPointerException: Cannot invoke "java.lang.String.toString()" because "myString" is null\n\tat io.openliberty.microprofile.telemetry.logging.internal.container.fat.MpTelemetryLogApp.MpTelemetryServlet.doGet(MpTelemetryServlet.java:53) ..."
            exception.type= "java.lang.NullPointerException"
            io.openliberty.class_name="io.openliberty.microprofile.telemetry.logging.internal.container.fat.MpTelemetryLogApp"
            io.openliberty.object_details="Object type = io.openliberty.microprofile.telemetry.logging.internal.container.fat.MpTelemetryLogApp\n tc = class             com.ibm.websphere.ras.TraceComponent@7baa4ff8 ...
            io.openliberty.probe_id="62"
            io.openliberty.sequence=“1723140654466_00000000000011"
            thread.id=“92”
            thread.name=“Default Executor-thread-12”

OpenTelemetry Severity Number Mapping with Liberty Log Levels

(Refer to Slide 13 in the UFO: https://ibm.box.com/s/ihoa8t3h18e8w2t41gdcuveo90q24mrd)

Configuration

List any new or changed properties, parameters, elements, attributes, etc. Include default values and configuration examples where relevant:

Server.xml configuration
To enable the MicroProfile Telemetry 2.0 feature to collect all logs, add the following configuration to your server.xml:

<features>
   <feature>mpTelemetry-2.0</feature>
</features>

<mpTelemetry source="message, trace, ffdc"/>

The source attribute accepts a list of comma-separated log sources . The <mpTelemetry/> configuration element is optional, if it is not specified or the source attribute is not specified in the element, the default source configured will be message. If the source attribute is empty (e.g. source=“”), no sources will be collected, and nothing will be routed to the configured OpenTelemetry logs exporter. The attributes and values are case insensitive.

Updates to existing topics

To update existing topics, specify a link to the topics that are affected. Include a copy of the current text and the exact text to which it will change. For example: Change ABC to XYZ

OpenTelemetry configuration

Add these properties to the following document as well:
https://openliberty.io/docs/latest/microprofile-config-properties.html#telemetry

To collect and export runtime-level logs, enable OpenTelemetry using system properties or environment variables:

otel.sdk.disabled=false / OTEL_SDK_DISABLED=false

If you would like to separately configure multiple applications in a server, you can configure OpenTelemetry with application configuration. Note that you will not collect runtime-level logs this way.

By default, all OpenTelemetry data is exported to OTLP. You can change each exporter with the following properties:

otel.logs.exporter/OTEL_LOGS_EXPORTER

When the exporter is set to none, no logs will be forwarded. If it is set to console, all the logs will be forwarded to stdout/console.log. The console.log will contain the usual logs, along with OpenTelemetry mapped (duplicate) logs as well. This should be only done for debugging purposes, and it is not recommended, as the console.log does not roll over.

Batch LogRecord Processor
By default, the SimpleLogRecordProcessor will be enabled, where the records will be send immediately. However, if you want to send the records in batches, you can also configure the following logging specific Batch LogRecord Processor.
https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#batch-logrecord-processor

otel.blrp.schedule.delay/OTEL_BLRP_SCHEDULE_DELAY
otel.blrp.max.queue.size/OTEL_BLRP_EXPORT_TIMEOUT
otel.blrp.max.export.batch.size/OTEL_BLRP_MAX_QUEUE_SIZE
otel.blrp.export.timeout/OTEL_BLRP_MAX_EXPORT_BATCH_SIZE

[dmuelle]

Hi @pgunapal - I've updated the following docs for this issue. Updates for related telemetry epics are still in progress.

• new topic: MicroProfile Telemetry log events reference list
• updated: MicroProfile Config properties: MicroProfile Telemetry
• revised: Enable observability with MicroProfile Telemetry: Configuring Open Liberty to use MicroProfile Telemetry to collect logs (other sections are still in progress)
• added: MPT feature page example: Collect logs from specified sources This is on the 1.1 page for review purposes only. Once the 2.0 page is generated after GM, I'll move it there.

[pgunapal]

@dmuelle Thanks, looks good! I added my review comments below:

• Doc: MicroProfile Telemetry log events reference list :
  • Provide the full name for Otel in the column headers for all tables (OTel => OpenTelemetry)
  • Fix Formatting for FFDC Events table
  • In the Severity number mapping section, typo for tracSpecification should be traceSpecification
• Doc: Enable observability with MicroProfile Telemetry: Configuring Open Liberty to use MicroProfile Telemetry to collect logs
  • Grammar: To enable MicroProfile Telemetry in your Open Liberty runtime, you must add the MicroProfile Telemetry feature to your server.xml file, enable the OpenTelemetry SDK. Optionally, you can specify MicroProfile Config properties to configure how MicroProfile Telemetry collects and exports logs.
  • In 1), mention that in addition to adding the MicroProfile Telemetry 2.0 feature in the server.xml, users should also add the <mpTelemetry sources=“message, trace, ffdc”/> attribute, to collect all the logs.
  • 2a) Typo: Telemtry => Telemetry
    • Grammar: The console.log file the contains the usual logs,
    • In this section, also good to mention that setting the exporter to none, no logs will be exported.
  • 2b) Maybe add the batch properties to MicroProfile Config properties: MicroProfile Telemetry doc, as well.

[dmuelle]

Thanks for reviewing @pgunapal - I believe I've made all the requested edits and corrections:

MicroProfile Telemetry log events reference list

• Provide the full name for Otel in the column headers for all tables (OTel => OpenTelemetry)
• Fix Formatting for FFDC Events table
• In the Severity number mapping section, typo for tracSpecification should be traceSpecification

Enable observability with MicroProfile Telemetry: Configuring Open Liberty to use MicroProfile Telemetry to collect logs

• Grammar and typo edits
• In 1), mention that in addition to adding the MicroProfile Telemetry 2.0 feature in the server.xml, users should also add the <mpTelemetry sources=“message, trace, ffdc”/> attribute, to collect all the logs.
• In exporter section, also good to mention that setting the exporter to none, no logs will be exported.

MicroProfile Config properties: MicroProfile Telemetry

• add the batch properties to MicroProfile Config properties: MicroProfile Telemetry doc, as well

If any further edits are needed, just let me know. When you're satisfied with the drafts, you can add the technical reviewed label to this issue to sign off. Thanks!

[donbourne]

In https://docs-draft-openlibertyio.mqj6zf7jocq.us-south.codeengine.appdomain.cloud/docs/latest/microprofile-config-properties.html#telemetry :

in the column "When the runtime reads the property" it says "Application startup". The properties will be read at the time OpenTelemetry initializes, which can be either a runtime startup or application startup, depending on whether the server is configured to use a single shared instance with the runtime or a separate instance for each app.

When the server is configured to use a separate instance for each app it uses MicroProfile Config properties. When the server is configured to use a single shared instance with the runtime it uses environment variables or system properties for configuration. So it's technically true that the MP Config properties are always read at app startup time, but we may want to put something in the header of the MP Telemetry section to explain that the MP Config properties only apply if MP Telemetry is configured to use a separate instance for each app.

[donbourne]

https://docs-draft-openlibertyio.mqj6zf7jocq.us-south.codeengine.appdomain.cloud/docs/latest/microprofile-telemetry.html

collect n and analyze
->
collect and analyze

You can configure how MicroProfile Telemetry collects and exports logs, metrics, and traces by specifying configuration properties in any of the config sources that are available to MicroProfile Config. Alternatively, if you choose to set these configuration properties by using environment variables, make the key name uppercase and convert any punctuation to underscores. For example, the otel.sdk.disabled=false property is equivalent to the OTEL_SDK_DISABLED=false environment variable.

• this needs discussion of how to set up either a single shared instance of OpenTelemetry sdk for use by the runtime and all apps, or a separate OpenTelemetry sdk instance for each application. In the latter case, telemetry from the runtime itself (such as logs or metrics unrelated to application requests) is not collected.

It includes a name, time-related data, log messages, and metadata to collect data about what happens during a transaction.
->
It includes a name, time-related data, log messages, and metadata about what happens during a transaction.

For example, to use a Jaeger server, you might
->
For example, to use a Jaeger server or to forward data to an OpenTelemetry Collector, you might

Optionally, you can specify MicroProfile Config properties (this appears in multiple places)
->
Optionally, you can specify MicroProfile Config properties or specify environment variables or system properties

To separately configure multiple applications in a server, you can configure OpenTelemetry with application configuration. However, you cannot collect runtime-level metrics this way.

• this comment seems out of place where it is currently. It should perhaps be at the top with a more thorough explanation / comparison of single/shared instance vs. instance per app config

[dmuelle]

Hi @donbourne - I've reorganized/restructured the following topics:

DEVELOPMENT:OBSERVABILITY

• Broke out the custom metrics definition to new Define custom MicroProfile Telemetry metrics topic
• made Code instrumentation for MicroProfile Telemetry tracing peer level with other topics in section.

OPERATIONS:

• moved Enable observability with MicroProfile Telemetry here, removed custom metrics section.
• Made MicroProfile Telemetry metrics reference list a child topic here by moving mptelemetry table into a new topic.

[donbourne]

https://docs-draft-openlibertyio.mqj6zf7jocq.us-south.codeengine.appdomain.cloud/docs/latest/mptelemetry-metrics-list.html

we don't provide support for all of the monitor-1.0 metrics in OTel yet, so we should change the following example:

To disable all vendor metrics, but to keep the REST base metrics configure the server.xml, as shown in the following example:

->
To disable all vendor metrics, but to keep the Session base metrics configure the server.xml, as shown in the following example:

Open Liberty can sends metric data
->
Open Liberty can send metric data

To enable only the monitoring components that are used by MicroProfile Telemetry, add the following code to your server.xml file.

->
To enable only the monitoring components that are used by MicroProfile Telemetry, add the following code to your server.xml file.

@Channyboy , the table shows "HTTP" as a monitoring component -- is that a valid monitor-1.0 component name? The text above the table doesn't list HTTP as an option for monitor filter.

In the table, the Prometheus metric names are listed after each metric. The tables also list the metric types, metric units, and descriptions of all metrics that are available for Open Liberty. For the vendor metrics, the associated monitoring component that you can use to filter the metric is also included. The Features required column of the table includes the feature or features that must be enabled to obtain that metric data. The Version introduced column specifies the minimum version of the feature that you must enable to collect the metric.
->
The table lists the metric types, descriptions and attributes that are available for Open Liberty. Where applicable, the associated monitoring component that you can use to filter the metric is also included. The Features required column of the table includes the feature or features that must be enabled to obtain that metric data. The Version introduced column specifies the minimum version of the feature that you must enable to collect the metric.

navigation:
Microservice observability with metrics
->
Microservice observability with MicroProfile Metrics

Development/Observability:

• should add a new topic (perhaps "Write logs with MicroProfile Telemetry logging"). Topic should explain that any logs sent to java.util.logging API will automatically be collected by the MicroProfile Telemetry feature. Note also that messages logged at java.util.logging.Level INFO and above are considered messages and levels below INFO are considered trace. perhaps then provide a link to the topic that explains how to use the "source" attribute on the mpTelemetry element in server.xml so people can see how to enable messages/trace to be collected.

Operations > Metrics
this topic says "Two kinds of metrics are available to monitor your Open Liberty applications: REST endpoint-style metrics that are provided by MicroProfile Metrics, and Java Management Extensions (JMX) metrics." - need to update to clarify that mp telemetry can also handle metrics. It might make more sense to move the "Collect and export metrics with MicroProfile Telemetry" to the top of this topic -- and perhaps reword that to briefly describe that there are 3 choices now, and that in particular, people using MP Telemetry for logs or traces may want to also use it for metrics instead of the other choices.

[Channyboy]

@dmuelle @donbourne yup, the HTTP is the component name to use.

The <monitor filter="ConnectionPool,ThreadPool,RequestTiming,Session,WebContainer,REST,GrpcClient,GrpcServer"/> was the full list.

For Telemetry should be
<monitor filter="ConnectionPool,ThreadPool,RequestTiming,Session,HTTP"/>

[dmuelle]

Thanks @donbourne @Channyboy - I created

Write logs with MicroProfile Telemetry logging

But this basically just includes Don's suggestions links to other resources. Not sure if anything else is needed here.

I also updated Monitoring with metrics and
MicroProfile Telemetry metrics reference list per Don's comments.

[donbourne]

https://docs-draft-openlibertyio.mqj6zf7jocq.us-south.codeengine.appdomain.cloud/docs/latest/mptelemetry-logging.html

When you enable the MicroProfile Telemetry 2.0 or later and the OpenTelemetry SDK
->
When you enable the MicroProfile Telemetry feature 2.0 or later

https://docs-draft-openlibertyio.mqj6zf7jocq.us-south.codeengine.appdomain.cloud/docs/latest/introduction-monitoring-metrics.html

MicroPRofile
->
MicroProfile

https://docs-draft-openlibertyio.mqj6zf7jocq.us-south.codeengine.appdomain.cloud/docs/latest/mptelemetry-metrics-list.html
The concept of vendor and base metrics only applies to mpMetrics, so need to change the following:

To configure only a subset of vendor metrics
->
To configure only a subset of metrics

for each vendor metric
->
for each metric

base metrics
->
metrics

None of the links in the "See also" section apply to mpTelemetry metrics.

[dmuelle]

Thanks @donbourne - all corrections made and staged to draft

[donbourne]

navigation
Development > Observability

    we should list the 3 OpenTelemetry topics at the top of the list

https://docs-draft-openlibertyio.mqj6zf7jocq.us-south.codeengine.appdomain.cloud/docs/latest/custom-mptelemetry-metrics.html

    make OpenTelemetry metrics API be a link to https://www.javadoc.io/doc/io.opentelemetry/opentelemetry-api/1.39.0/io/opentelemetry/api/metrics/package-summary.html

    MicroPRofile
    ->
    MicroProfile

https://docs-draft-openlibertyio.mqj6zf7jocq.us-south.codeengine.appdomain.cloud/docs/latest/observability.html

    should show the "Write logs with MicroProfile Telemetry logging" (or is it limited to first 5 topics?)

https://docs-draft-openlibertyio.mqj6zf7jocq.us-south.codeengine.appdomain.cloud/docs/latest/telemetry-trace.html

    change OpenTelemetry API link url to https://www.javadoc.io/doc/io.opentelemetry/opentelemetry-api/1.39.0/io/opentelemetry/api/trace/package-summary.html

    change refs to 1.19.0 to 1.39.0

    Manual instrumentation 
    step 1 should link to "Enable observability with MicroProfile Telemetry." instead of "Configuring Open Liberty to use MicroProfile Telemetry."

    Move the "Manual instrumentation" steps 1-3, plus the line after that says "After you complete those prerequisites, you’re ready to instrument your code. " to a common topic that can be referenced by both this topic and the "Define custom MicroProfile Telmetry metrics" topic.  The common topic could be called "Prepare your development environment for MicroProfile Telemetry"

https://docs-draft-openlibertyio.mqj6zf7jocq.us-south.codeengine.appdomain.cloud/docs/latest/microprofile-telemetry.html

    remove "MicroProfile Telemetry helps you collect and analyze data on the paths that application requests take through services." (this made sense when the topic was just about trace)

[dmuelle]

Thanks @donbourne - Ive edited the docs and created a new topic per these comments. Here's a list of (almost)everything we've updated so far for these epics

New topic- Prepare your development environment for MicroProfile Telemetry

DEVELOPMENT

• Observability
• Prepare your development environment for MicroProfile Telemetry
• Define custom MicroProfile Telemetry metrics
• Code instrumentation for MicroProfile Telemetry tracing
• Write logs with MicroProfile Telemetry logging

OPERATIONS

• Enable observability with MicroProfile Telemetry
• MicroProfile Telemetry log events reference list
• MicroProfile Telemetry metrics reference list

REFERENCE
MicroProfile Telemetry (feature reference page)

[donbourne]

@dmuelle , looks good - just a couple things I noticed while doing a full read-over:

https://docs-draft-openlibertyio.mqj6zf7jocq.us-south.codeengine.appdomain.cloud/docs/latest/custom-mptelemetry-metrics.html

    custom metrics metrics
    ->
    custom metrics

https://docs-draft-openlibertyio.mqj6zf7jocq.us-south.codeengine.appdomain.cloud/docs/latest/telemetry-trace.html

    remove:
        Because Liberty supports per-application configuration, it does not support GlobalOpenTelemetry. Using that class does not produce any telemetry data.
        
    remove:
        Furthermore, be aware that this documentation includes information for the OpenTelemetry Metrics and Logging APIs, which are not supported by MicroProfile Telemetry.

[dmuelle]

Thanks Don, both those files are now updated on draft

https://docs-draft-openlibertyio.mqj6zf7jocq.us-south.codeengine.appdomain.cloud/docs/latest/custom-mptelemetry-metrics.html

https://docs-draft-openlibertyio.mqj6zf7jocq.us-south.codeengine.appdomain.cloud/docs/latest/telemetry-trace.html

[donbourne]

Thanks @dmuelle !

[ramkumar-k-9286]

Peer Review

Prepare your development environment for MicroProfile Telemetry

Add the opentelemetry API and OpenTelemetry instrumentation annotations as a provided dependency to your build path. For example, with Maven, add the following code to your pom.xml file.
->
Add the opentelemetry API and opentelemetry instrumentation annotations as a provided dependency to your build path. For example, with Maven, add the following code to your pom.xml file:

---

Observability

With MicroProfile Telemetry 2.0 and later, you can manage logs, metrics, and traces in a standardized way by using the OpenTelemetry protocol. For more information, see Enable observability with MicroProfile Telemetry.
->
Should we be adding a link to MicroProfile Telemetry 2.0 here?