Startup issues in V2.16

[PavelKondratiev]

Hi support team

This is continuation of #3925
I re-installed V2 from scratch and now it starts much better, i.e. without ZWEL0029E

Though new errors appeared in the log so your help is much appreciated.
First, couple of questions, please :

1. I configured to start 3 components only : gateway, discovery and API catalog - is it valid config? Is caching service mandatory as pre-req for these ones?
2. I did not configure any certificates/keyrings etc. and set verifyCertificates: DISABLED in zowe.yaml - again, is it valid config? I do not need any security as this is internal test system (zPDT)
3. Regarding p.2 I just commented out all the lines in certificate: section of zowe.yaml

Log and config are attached, many thanks in advance!
start_log3.zip
zowe.zip

[balhar-jakub]

1. Yes, it is a valid configuration. Caching service isn't required. Without the caching service you won't be able to use Personal Access Tokens functionality.
2. It's valid but not recommended. If you go this route you also need to set the servers to http.

zowe:
  components:
    gateway:
      server:
        ssl:
          enabled: false
        internal:
          ssl:
            enabled: false
    discovery:
      server:
        ssl:
          enabled: false
    api-catalog:
      server:
        ssl:
          enabled: false

which is also what causes at least some of the errors in the attached log.

[PavelKondratiev]

Thanks for the prompt response. Structure of components: section in my config (attached here as zowe.zip) looks bit different from your example, could you, please take a look on it? Here it is :

components:

  # >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  gateway:
    enabled: true
    port: 7554
    debug: false

    apiml:
      security:
        auth:
          provider: "zosmf"
          zosmf:
            jwtAutoconfiguration: "auto"
            serviceId: "zosmf"
        authorization:
          endpoint:
            enabled: false
          provider: ""
        x509:
          enabled: false
    server:
      internal:
      # gateway supports internal connector
        enabled: false
        port: 7550
        ssl:
          enabled: false
          # internal connector can use different certificate
          # certificate:
          #   keystore:
          #     alias: ""
    # If we customize this to use different external certificate, than should also
    # define "server.internal.ssl.certificate" and enable "server.internal.ssl.enabled".
    # certificate:
    #   keystore:
    #     alias: ""

  # >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  metrics-service:
    enabled: false
    port: 7551
    debug: false

  # >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  cloud-gateway:
    enabled: false
    port: 7563
    debug: false

  # >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  api-catalog:
    enabled: true
    port: 7552
    debug: false

  # >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  discovery:
    enabled: true
    port: 7553
    debug: false
    # Define this value to match your number of Discovery StatefulSet if you are running containerized Zowe
    # replicas: 1

  # >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  caching-service:
    enabled: false
    port: 7555
    debug: false

    storage:
      evictionStrategy: "reject"
      # can be inMemory, VSAM, redis or infinispan
      mode: "VSAM"
      size: 10000
      vsam:
        # your VSAM data set created by "zwe init vsam" command or ZWECSVSM JCL
        # this is required if storage mode is VSAM
        name: ""
      infinispan:
        # this is required if storage mode is infinispan
        jgroups:
          port: 7600

One more question regarding server (sub)component in the fragment above - is it mandatory?

    server:
      internal:
      # gateway supports internal connector
        enabled: false
        port: 7550
        ssl:
          enabled: false

Initially it was disabled, yesterday I turned it on (with ssl: disabled) but still get mostly the same errors.
Also tried to run with auth.provider: "dummy" but it did not improve anything.

Log says all 3 API ML components are started,

ZWEL0006I starting components          
ZWEL0001I component gateway started    
ZWEL0001I component api-catalog started
ZWEL0001I component discovery started  
ZWEL0007I components started           
ZWEL0012I starting console listener    

I see their ZWE1AC/D/G subtasks in SDSF DA screen but NETSTAT command output shows there is no listeners on ports 7552/3/4 therefore I cannot connect to API ML via web interface.
Attached log and config for latest today's run.
log_and_config_0808.zip

[skurnevich]

Hi, regarding the second point, most of the errors in the logs seem to be related to missing certificates configuration, so can you try the certificate scenario 1?
It will create a keystore with 'self-signed' certificates and is designed specifically for test purposes, uncomment the scenario 1 in your zowe.yaml or add something like:

    certificate:
      type: PKCS12
      pkcs12:
        directory: /apps/zowe/v216/keystore

And then run
zwe init certificate -v -c /apps/zowe/v216/instance/zowe.yaml --update-config
Just double check that you do not use java build 8.0.8.10, it has a known limitation.

As for the gateway.server.internal - it is used for enabling another gateway port for internal communication, so there is no need to enable it, keep it disabled.

[PavelKondratiev]

Hello Sergei,

I reconfigured product to use PKCS12 certificate and set verifyCertificates: NONSTRICT
Now it works and API ML web interface is reachable on port 7554.

I faced couple more issues during my (re)configuration attempts, your comments are appreciated :

1. I disabled caching component during installation via zOSMF PSWI but some fields related to caching remain uncommented in zowe.yaml
   It caused the following error to appear :

 2024-08-07 09:16:21.516 <ZWELNCH:67175204> ZWESVUS2 ERROR ZWEL0070E Configuration has validity exceptions:
 Validity Exceptions(s) with object at 
   Validity Exceptions(s) with object at /zowe
     Validity Exceptions(s) with object at /zowe/setup
       Validity Exceptions(s) with object at /zowe/setup/vsam
         no matching enum value at /zowe/setup/vsam/mode; expecting one of values '[NONRLS, RLS]' of type 'string'

2. When I started main server with apiml.security.auth.provider: "zosmf" I got the following errors in log :

 2024-08-09 10:48:43.211 <ZWEAGW1:Thread-15:16843817> ZWESVUS2 ERROR (o.z.a.g.s.s.JwtSecurity) ZWEAG713E Configuration error when trying to establish JWT producer. Events: Started waiting for z/OSMF instance zosmf to be registered and known by the discovery service
 Wait for z/OSMF instance zosmf to come online before deciding who provides JWT tokens.
 2024-08-09 10:48:44.208 <ZWEAGW1:Thread-15:16843817> ZWESVUS2 ERROR (o.z.a.g.s.s.JwtSecurity) ZWEAG108E z/OSMF instance 'zosmf' not found or incorrectly configured. Gateway is shutting down.
 2024-08-09 10:50:05.483 <ZWEAGW1:DiscoveryClient-CacheRefreshExecutor-0:16843817> ZWESVUS2 ERROR (o.z.a.g.s.s.z.ZosmfService) ZWEAG182E SSL Misconfiguration, z/OSMF is not accessible. Message: I/O error on GET request for &quot;https://s0w1.dal-ebis.ihost.com:10443/zosmf/info&quot;: com.ibm.jsse2.util.j: PKIX path building failed: com.ibm.security.cert.IBMCertPathBuilderException: unable to find valid certification path to requested target; nested exception is javax.net.ssl.SSLHandshakeExce
   - CN (Common Name) and z/OSMF hostname match.
   - The certificate is valid
   - TLS version matches
   - z/OSMF server certificate is trusted in Zowe's truststore
 Enable debugging to see further details in stack trace.
 2024-08-09 10:51:21.509 <ZWEAGW1:main:16843817> ZWESVUS2 WARN  (o.s.c.l.c.LoadBalancerCacheAutoConfiguration$LoadBalancerCaffeineWarnLogger) Spring Cloud LoadBalancer is currently working with the default cache. While this cache implementation is useful for development and tests, it's recommended to use Caffeine cache in production.You can switch to using Caffeine cache, by adding it and org.springframework.cache.caffeine.CaffeineCacheManager to the classpath.
 2024-08-09 10:52:17.457 <ZWEAAC1:DiscoveryClient-CacheRefreshExecutor-0:16843808> ZWESVUS2 WARN  (o.z.a.a.s.c.CachedProductFamilyService) ZWEAC705W The home page url for service IBMZOSMF was not transformed. Not able to select route for url https://s0w1.dal-ebis.ihost.com:10443/ of the service ibmzosmf. Original url used.
 2024-08-09 11:24:52.065 <ZWEAGW1:https-jsse-nio-0.0.0.0-7554-exec-9:16843817> ZWESVUS2 ERROR (o.a.c.c.C.[.[.[.[dispatcherServlet]) Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception
 org.springframework.web.client.ResourceAccessException: I/O error on POST request for "https://s0w1.dal-ebis.ihost.com:10443/zosmf/services/authenticate": com.ibm.jsse2.util.j: PKIX path building failed: com.ibm.security.cert.IBMCertPathBuilderException: unable to find valid certification path to requested target; nested exception is javax.net.ssl.SSLHandshakeException: com.ibm.jsse2.util.j: PKIX path building failed: com.ibm.security.cert.IBMCertPathBuilderException: unable to find valid c

Do they mean some config should be done on zOSMF side? Or anything else?

Thanks in advance!

[skurnevich]

Hello Pavel,

The first issue seems to be a bug in the workflows, this section should either have default value (NONRLS) or be commented out completely. I have shared this with my colleagues so it should be fixed in the next release, thank you for reporting.
Also some parts of workflows configuration are hardcoded, so when something is unclear you can compare it to example-zowe.yaml

The second one looks like configuration issue, i can look into that, meanwhile could you please confirm that if you set apiml.security.auth.provider: "saf" it works as expected?

[PavelKondratiev]

Yes, I confirm.

[PavelKondratiev]

One more observation taken from https://docs.zowe.org/v2.16.x/user-guide/import-certificates/ :

---

The public certificate in PEM format is stored at <KEYSTORE_DIRECTORY>/local_ca/localca.cer where <KEYSTORE_DIRECTORY> is defined in a customized <RUNTIME_DIR>/bin/zowe-setup-certificates.env file

---

This description seems to remain from V1 but it is not applicable to V2 as <RUNTIME_DIR>/bin/ does not contain zowe-setup-certificates.env anymore

[skurnevich]

Hello Pavel,

1. I disabled caching component during installation via zOSMF PSWI but some fields related to caching remain uncommented in zowe.yaml
   It caused the following error to appear :

Addressed in this PR

2. When I started main server with apiml.security.auth.provider: "zosmf" I got the following errors in log :

Most likely z/OSMF certificate is not trusted by Zowe, when you ran the zwe init certificate command it should have been created the zosmf trust, but if you had incorrect config in zOSMF section of your yaml file at that moment or the auth provider was set to saf, or if z/OSMF was not available it may have failed to do so. Can you please check the truststore, like
keytool -v -list -keystore /apps/zowe/v216/keystore/localhost/localhost.truststore.p12
If it is missing please re-run the zwe init certificate command or manually import the z/OSMF certificate there.

This description seems to remain from V1 but it is not applicable to V2 as <RUNTIME_DIR>/bin/ does not contain zowe-setup-certificates.env anymore

Yes seems to be a docs issue, i've found it was already reported but never fixed - zowe/docs-site#2853
I'll share it with docs team

[PavelKondratiev]

Thanks for your responses and help, closing the ticket.