Deploy Zowe in Sysplex

This documentation only covers differences comparing to normal Zowe installation and configuration. Check https://docs.zowe.org/stable/user-guide/install-zos.html before continue.

Download test Zowe build

Download the most recent build from PR-2049 and install as usual.

Expected result: install the build and start Zowe as usual, the Zowe instance should pass all sanity test.

Customize component-level certificates

When you customize zowe-setup-certificates.env before running zowe-setup-certificates.sh, you may already notice new variables COMPONENT_LEVEL_CERTIFICATES, EXTERNAL_COMPONENT_CERTIFICATES and EXTERNAL_COMPONENT_CERTIFICATE_ALIASES. Read the instruction, uncomment and modify value of COMPONENT_LEVEL_CERTIFICATES.

To successfully run zowe-setup-certificates.sh on Sysplex, the script may not be able to find your z/OSMF configuration, or fall back to wrong value. To proper define where is z/OSMF, define ZOWE_ZOSMF_HOST and ZOWE_ZOSMF_PORT environment variables. ZOWE_ZOSMF_HOSTshould be the domain name of your Dynamic VIP Address (DVIPA). In our case, it'szhaplex.svl.ibm.com`. For example.

export ZOWE_ZOSMF_HOST=zhaplex.svl.ibm.com
export ZOWE_ZOSMF_PORT=443

Now you can run zowe-setup-certificates.sh again.

Expected result #1, you should see these extra files in <keystore>/localhost/:

localhost.keystore.app-server.cer
localhost.keystore.app-server.cer-ebcdic
localhost.keystore.app-server.csr
localhost.keystore.app-server.key
localhost.keystore.app-server_signed.cer
localhost.keystore.gateway.cer
localhost.keystore.gateway.cer-ebcdic
localhost.keystore.gateway.csr
localhost.keystore.gateway.key
localhost.keystore.gateway_signed.cer
localhost.keystore.zss.cer
localhost.keystore.zss.cer-ebcdic
localhost.keystore.zss.csr
localhost.keystore.zss.key
localhost.keystore.zss_signed.cer

These are the PEM format certificates.

Expected result #2, run below command to list content of the keystore:

$ keytool -v -list -storepass password -storetype PKCS12 -keystore localhost.keystore.p12

You should see certificates generated for each components with proper alias name like this:

Alias name: gateway
Creation date: Apr 7, 2021
Entry type: keyEntry
Certificate chain length: 2
Certificate[1]:
Owner: CN=Zowe Service - gateway, OU=API Mediation Layer, O=Zowe Sample, L=Prague, ST=Prague, C=CZ

Expected result #3, check <keystore>/zowe-certificates.env, it should show instructions like this:

# To configure certificate for gateway, you can add these entries to "components.gateway" of your YAML configuration:
# certificate:
#   keystore:
#     alias: gateway
#   pem:
#     key: /global/zowe/keystore121/localhost/localhost.keystore.gateway.key
#     certificate: /global/zowe/keystore121/localhost/localhost.keystore.gateway.cer-ebcdic
# To configure certificate for app-server, you can add these entries to "components.app-server" of your YAML configuration:
# certificate:
#   keystore:
#     alias: app-server
#   pem:
#     key: /global/zowe/keystore121/localhost/localhost.keystore.app-server.key
#     certificate: /global/zowe/keystore121/localhost/localhost.keystore.app-server.cer-ebcdic
# To configure certificate for zss, you can add these entries to "components.zss" of your YAML configuration:
# certificate:
#   keystore:
#     alias: zss
#   pem:
#     key: /global/zowe/keystore121/localhost/localhost.keystore.zss.key
#     certificate: /global/zowe/keystore121/localhost/localhost.keystore.zss.cer-ebcdic

The component level certificate instruction will overwrite part of the values defined in zowe.internalCertificate.

Configure Zowe instance

You can run zowe-configure-instance.sh as usual to create Zowe instance directory. For example, zowe-configure-instance.sh -c /global/zowe/instance121. The Zowe instance directory must be put on a shared volume where all LPARs in the Sysplex can access. /global/zowe/instance is the suggested location.

Expected result, you will see these extra information introducing YAML configuration.

As technical preview, Zowe now provides a new way to customize your instance with a YAML file.
You can convert your instance.env file by running this script:
  <instance-dir>/bin/utils/convert-to-zowe-yaml.sh > <instance-dir>/zowe.yaml
The zowe.yaml will take effect once you delete or rename your <instance-dir>/instance.env file.
Please check <instance-dir>/bin/example-zowe.yaml and see how you can customize Zowe instance.
This YAML configuration format is mandatory to deploy Zowe in a Parallel Sysplex environment.

Execute <instance-dir>/bin/utils/convert-to-zowe-yaml.sh > <instance-dir>/zowe.yaml to generate zowe.yaml.

Note: please be aware of the tagging of zowe.yaml. To allow Zowe to read the file properly, it must be IBM-1047 encoding.

Check the encoding with ls -laT <instance-dir>. If you see line like t ISO8859-1 T=on -rw-r--r-- 1 ROOT ZWEADMIN 3637 Apr 7 23:51 zowe.yaml, that means the encoding is not correct, we need to convert it to IBM-1047 encoding with these commands:

cd <instance-dir>
chtag -r zowe.yaml
iconv -f ISO8859-1 -t IBM-1047 zowe.yaml > zowe.yaml.1047
mv mv zowe.yaml.1047 zowe.yaml

Now if you run ls -laT again, it should show like - untagged T=off -rw-r--r-- 1 ROOT ZWEADMIN 3637 Apr 7 23:51 zowe.yaml.

Now you can delete your <instance-dir>/instance.env and start Zowe.

Expected result, Zowe should start as normal without issues.

Customize HA Instances

You can follow <instance-dir>/bin/example-zowe.yaml haInstances section to define your instances. Also check the example about how to customize component certificate.

Configure launcher ZWESLSTC

During installation, you may notice lines like this showing on the output:

launcher/install.sh ZWESLSTC copied to <dataset-prefix>.SZWESAMP
launcher/install.sh ZWELNCH program copied to <dataset-prefix>.SZWEAUTH

Follow these steps to enable launcher.

Copy ZWESLSTC to proclib

You can find ZWESLSTC in <dataset-prefix>.SZWESAMP. Copy over to your proclib and make sure these values are correct:

1. STEPLIB DSNAME should point to your <dataset-prefix>.SZWEAUTH. For example, //STEPLIB DD DSNAME=IBMUSER.ZWE121.SZWEAUTH,DISP=SHR.
2. INSTANCE_DIR=#instance_dir should point to your instance directory. For example, INSTANCE_DIR=/global/zowe/instance121.

You can also use utility script to copy proclib:

cd <runtime>/scripts/utils
./zowe-copy-to-JES.sh -s <dataset-prefix>.SZWESAMP -i ZWESLSTC -r USER.PROCLIB -o ZWESLSTC -f /global/zowe/logs/launcher-stc.log

Define started task

Run below command to to customize runtime user for ZWESLSTC and refresh STARTED profile.

RDEFINE STARTED ZWESLSTC.* STDATA(USER(ZWESVUSR) GROUP(ZWEADMIN) TRUSTED(NO)) DATA('ZOWE LAUNCHER')
SETROPTS RACLIST(STARTED) REFRESH

Start Zowe with ZWESLSTC

Run this command to start Zowe: S ZWESLSTC,HAINST=<ha-instance-id>. For example, S ZWESLSTC,HAINST=TIVLP14A.

To terminate Zowe, run P ZWESLSTC command.

To stop a component, run F ZWESLSTC,APPL=STOP(<component>) command. For example, F ZWESLSTC,APPL=STOP(files-api). To start a component, run F ZWESLSTC,APPL=START(<component>) command.

Troubleshooting

ZIS status not OK

We need ZWESISTC be started on every LPAR you plan to run ZSS services.

TCPIP resolver failure

If you see these errors in your Zowe job log, that means your TCP resolver is not configured properly. This is important for Sysplex deployment to make sure 2 LPARs can communicate each other with domain name.

EZZ2376I Could not determine TCPIPjobname, using default of 'INET'                                                
EZZ2377I Could not establish affinity with INET (1011/11B3005A) - can not provide the requested option information

To resolve this issue,

• if you are using instance.env, you can add line RESOLVER_CONFIG="//'USER.PARMLIB(TCPDATA)'" to it,
• if you are using zowe.yaml, you can add entry RESOLVER_CONFIG: "//'USER.PARMLIB(TCPDATA)'" to zowe.environments.

You need to find correct parmlib where your TCPDATA located.

Zombie address spaces

Reason is still unknown. To remove zombie processes after Zowe is stopped,

• in USS, issue su and use ps -elf | grep ZWESVUSR | awk '{print $2}' | xargs kill -9 to kill all processes running as ZWESVUSR,
• in SDSF.DA panel, find all ZWE address spaces and cancel them.