Report API ML state to the MF System Programmer

[jalel01]

Use Case

As a System Programmer, I want to automate system behavior based on the state of the API Mediation Layer. I want to use WTO to capture the state.

Technical Requirements

• System Automation for z/OS needs to receive a MVS message via the SubSystem Interface to be notified of the UP status
• API ML will generate a WTO when all main services are UP

Notes

• API ML consists of 3 major components, the EUREKA-Discovery Service, API Gateway, and API Catalog. By reporting the state of these components, we enable mainframe admins to control the state of the API ML using popular automation tools. 

[jalel01]

With reference to an earlier, similar request #373 in this Repo.

Other relevant Issues in other repos:
zowe/zlux#355
zowe/zowe-install-packaging#1248

[balhar-jakub]

Relevant work within the Zowe Launcher:

• Duplicate log messages into syslog according to zowe.sysMessages launcher#93
• Duplicate log messages into syslog according to zowe.sysMessages zowe-install-packaging#3553

[balhar-jakub]

Since Zowe 2.18 it's possible to achieve this functionality. It's combination of functionality within Zowe Launcher and new message introduced in the API Mediation Layer together with functionality to change the way API Mediation Layer logs the information.

The first part is zowe.sysMessages. This is functionality introduced in Zowe 2.12 that allows user to specify messages from the standard output which will be republished to the Syslog (e.g. WTO)

zowe:
  # >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  # You can define any Zowe message portions to be checked for and the message added to the
  # system log upon its logging, truncated to 126 characters.
  sysMessages:
    - "ZWEAM001I"

In order to know which message needs to be published we introduced new message that is issued to standard log when the API Mediation Layer is fully started, e.g. the components are aware of each other and are fully functional. The message contains:
ZWEAM001I API Mediation Layer started

The current default logging implementation starts with the information about the current time unlike with the message id as is typical in the z/OS world. To change this behavior we introduced the possibility to change the structure of the messages issues by API Mediation Layer in Zowe 2.18. The general details are here: https://docs.zowe.org/stable/user-guide/api-mediation/configuration-logging

To get the correct message with the id in the beginning you need to prepare custom logback.xml based on the one in the documentation article. This is an example that duplicates the message id to the beginning of the log message, or if there is no message id provides first eight characters of the message instead.

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <property resource="application.yml" />
    <property name="MAX_INDEX" value="${rollingPolicy.maxIndex:-12}"/>
    <property name="MIN_INDEX" value="${rollingPolicy.minIndex:-1}"/>
    <property name="MAX_FILE_SIZE" value="${rollingPolicy.file.maxSize:-50MB}"/>
    <property name="STORAGE_LOCATION" value="${apiml.logs.location}" />
    <property name="apimlLogPattern" value="%msgm{8} %d{yyyy-MM-dd HH:mm:ss.SSS,UTC} %clr(&lt;${logbackService:-${logbackServiceName}}:%thread:${PID:- }&gt;){magenta} %X{userid:-} %clr(%-5level) %clr(\\(%logger{15}\\)){cyan} %msg%n"/>

    <turboFilter class="org.zowe.apiml.product.logging.UseridFilter"/>
    <if condition='property("spring.profiles.include").contains("debug")||property("spring.profiles.include").contains("diag")||property("spring.profiles.include").contains("dev")'>
        <then>
        </then>
        <else>
            <turboFilter class="org.zowe.apiml.product.logging.ApimlDependencyLogHider"/>
            <turboFilter class="org.zowe.apiml.product.logging.LogLevelInfoFilter"/>
            <turboFilter class="ch.qos.logback.classic.turbo.DuplicateMessageFilter">
                <AllowedRepetitions>0</AllowedRepetitions>
            </turboFilter>
        </else>
    </if>
    <conversionRule conversionWord="clr" converterClass="org.springframework.boot.logging.logback.ColorConverter" />
    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
        <encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
            <layout class="org.zowe.apiml.product.logging.MaskingLogPatternLayout">
                <pattern>${apimlLogPattern}</pattern>
            </layout>
        </encoder>
    </appender>

    <appender name="FILE" class="org.zowe.apiml.product.logging.ApimlRollingFileAppender">
        <file>${STORAGE_LOCATION}/${logbackServiceName}.log</file>

        <rollingPolicy class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy">
            <fileNamePattern>${STORAGE_LOCATION}/${logbackServiceName}.%i.log</fileNamePattern>
            <minIndex>${MIN_INDEX}</minIndex>
            <maxIndex>${MAX_INDEX}</maxIndex>
        </rollingPolicy>

        <triggeringPolicy class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
            <maxFileSize>${MAX_FILE_SIZE}</maxFileSize>
        </triggeringPolicy>

        <encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
            <layout class="org.zowe.apiml.product.logging.MaskingLogPatternLayout">
                <pattern>${apimlLogPattern}</pattern>
            </layout>
        </encoder>
    </appender>

    <root level="INFO">
        <appender-ref ref="STDOUT" />
        <appender-ref ref="FILE" />
    </root>
</configuration>

The last step is to configure API Mediation Layer services to use this customized logback file. If you are looking only for this specific message then it's enough to configure gateway to use this file:

components:
  gateway:
    logging:
      config: /path/to/logback.xml

After that you should see the message ZWEAM001I in the Syslog when the API Mediation Layer fully starts and is ready to tackle requests.