Z Open Editor and DevOps discussion

[fswarbrick]

I hate to ask this here, but I can't find a better place. Is there any type of discussion forum for IBM Z Open Editor, where we could ask questions about how people are using the product, what might be worthwhile enhancements, etc.?

[phaumer]

Hello @fswarbrick. I would be very happy running discussions here as well. I would just tag the issue with the Question label. However, there is an IBM Developer community with a discussion forum available as well.

[FALLAI-Denis]

Hi @fswarbrick,

Glad to see you here because I have noticed on other forums that we often share the same opinions.
This ties in with my issue #100.

We are in the process of setting up a DevOps solution on the mainframe and replacing many tools. It pulls many topics and change management is very strong.

We are replacing a development tool (IBM VisualAge Pacbase) which is (among other things) a COBOL code generator. Our teams are used to not dealing with technical code and focusing on application code.

For us, IBM Z Open Editor, Zowe CLI, and COBOL V6, came at the right time and allowed us to consider a relevant and efficient solution for the replacement of this product used for almost 40 years, with practices anchored in the heads of our developers.

We try through VS Code and IBM Z Open Editor to recreate development facilities for our teams to maintain the productivity that we currently know which is very much higher than that observed with COBOL development without special tools.

We have built a framework based on COPYBOOK which exploit the conditional compilation functionality of COBOL V6: this limits the work of developers, and hides the complexity of certain technical algorithms, (for example file pairing with management of break keys) , and incidentally protects this code from any unwelcome modification.

We are also in the process of developing an extension which makes it possible to inject (and maintain iteratively) COBOL code in a program corresponding to standardized processing. This extends the mechanism of COBOL COPYBOOKs by making it possible to manage several insertion points in the COBOL code for the same functionality.
For example, to manage a file or table Db2 resource, it is necessary to manage the functions of opening, reading, closing, calculation of the synchronization and breaking keys, managing the calls to these functions, and managing all the data associated with these functions. All of this is handled in a one-action for developer with our code injection (and maintenance) extension, and in a more conceptual than purely technical form.

[fswarbrick]

Hi @FALLAI-Denis.

I've been thinking to contact you directly as well, so I am very grateful for your reply here!

I am in very early stages of looking at using VSCode/Z Open Editor/Zowe. In fact, I don't yet have it installed on our site at all. Rather, I am using the IBM "demo" z/OS system that is used for the "Master the Mainframe" course. My intent is to have something to demo for my manager and a few others to see if they think its worthwhile pursuing.

One of the things that has disappointed me up to now is the lack of a simple "compile/build" step. I know that ZOE has integration with IBM DBB, but honestly that seems to be "overkill" for my shop. We are, I guess, a very simple shop. We don't have "builds" as seem to be common in other environments. A developer simply makes code changes and then compiles and links in to a designated test environment. There are no dependency lookups , no "build" automation or anything like that. Nor am I convinced their need be.

Anyway, I was able to create a simple REXX exec that I can invoke from the zowe zos-tso command in order to compile a program. I added this as the default "build" command in my VSCode. It works...ok.

I am curious about your use of Git. Do you use Git on z/OS, or do you do your source control outside of z/OS? One of my hopes for VSCode was to use it's built in Git support for z/OS source code. But if the source code is hosted on z/OS itself I'm no longer sure how that can be made to work. In my mind ideally the VSCode Remote - SSH extension would be available. But looking more deeply in to it, it sounds like having a VSCode Server environment running on z/OS, which would be required, is perhaps unlikely to ever happen. That component, and the client extension, are Microsoft closed-sources, and I am doubtful they would be interested in supporting z/OS. Doesn't look like they even support Linux on Z (s390x). Now if IBM were to reach out to them and make some sort of arrangement to get this support...we'll that would be great!

In the end, based on the lack of response to your issue #100 , I am not hopeful that you aren't the only major user of Z Open Editor. But I'd love to see input from others, if there are any!

[FALLAI-Denis]

Regarding the build, in a DevOps approach, using Git mechanisms, two cases must be considered:

• the user build: a build triggered from the developer workstation, from their local workspace
• the build project: a build triggered by a Jenkins-type pipeline

We (still) use Endevor (disengage action in progress, probably to go under DBB, Artifactory, and XLR / XLD).
We have developed our own build functions:

• user build: via a propriatory VS Code extension, we transfer the (COBOL) source to a temporary file, and we create a JCL that contains the Endevor SCL that we submit. Interactions with z/OS were first handled by FTP, then when we got z/OSMF (a long fight with our system team) they were handled by Zowe CLI.
• project build: this is an stage in the Jenkins pipeline that does the same thing as the VS Code extension.

We use a set of (YAML with Schema validation) files in the VS Code workspace that contain the informations needed to build the SCL Endevor : merged from element level (type / processor in Endevor), project level (ccid in Endevor), application level (lpar / system /subsystem in Endevor), Git branch level (environment / stage entry in Endevor)

[FALLAI-Denis]

Regarding Git, we do not use Git for z/OS, (we tested it, no interest).
There are no Git actions on z/OS.

Git repositories are managed under Bitbucket.
VS Code clones the Git repository for the developer's work.
Then during the referral, (via Pull Request), Bitbutcket takes care of merge operations, and validate operations through a Jenkins pipeline that activates tools like SonarQube, and triggers builds on the z/OS system (sources are taken from Git at Jenkins level and sent to the z/OS system by the build step described previously).

We have built a Git workflow inspired by GitFlow but adapted to the operation of Endevor, (several promotion mappings with as many entry points).

Git is the source repository.
Endevor is the binaries repository and promotion tool.

In the context of the replacement of Endevor, the interest of DBB is the management of the intelligent build with the use of references (dependencies) between components.
For example, recompile all COBOL sources that use a COPYBOOK that has been modified.
We see no other point in this, and depending on the architecture of the heritage to be managed, this automatic reconstruction capacity may not be necessary.

[FALLAI-Denis]

To use VS Code connected to your z/OS system you only need a Zowe CLI compatible server:

• either z/OSMF, this is the solution we have chosen, with a "light" z/OSMF, without system administration function, dedicated to development teams.
• or RSE (already used by Eclipse RDz / IDz clients, or Z / OS Explorer), with an RSE API interface layer to be addressed by Zowe CLI.

No need to install Zowe Server: no added value for us.

We have two instances of z/OSMF per partition:

• a complete one for system teams
• a light one for development teams

See also the FTP extension for Zowe Explorer, but I'm not sure if it can work with Z Open Editor (which requires z/OS access via Zowe CLI)

[FALLAI-Denis]

Our tools chain:

• current (still under construction)
  

• target to reach
  

[phaumer]

About the usage of Git on z/OS: as Git is a decentralized SCM you can push your commits from your local editor client directly to USS without the need of any server such as GitHub, GitLab, Bitbucket etc. and then start a build script on USS remotely via an ssh command. This is what we did and documented before Z Open Editor had its native DBB integration that now provides a convenient right-click.

See this document in our GitHub history of our sample repo describing the setup steps.

The main thing is to configure the z/OS repository in your USS home directory with

git init
git config --local receive.denyCurrentBranch updateInstead

which materializes a branch into the Git working directory after a push was received. Then you can use that working directory to run a build script in your preferred language (does not have to be DBB groovy), which we started from the editor by using password-less ssh that could be called via a VS Code Task. That script would also live on the USS side and could do other things such as checkout other branches etc. Our example that started groovy was here.

[fswarbrick]

Thanks, @phaumer. I had no idea that you could push from one non-server repository to another. Our z/OS system does not currently have sshd enabled, so I can't test it out there, but I did test it on my laptop between my Windows environment and a Linux VM, and it works perfectly.

Just a general comment... We have been a mainframe shop since the 80s, first on (DOS/)VSE, and on z/OS since 2010. Our environment is very basic. Developers use ISFP to edit source code and JCL, an in-house written REXX exec that builds and submits a compile job, and an in-house written REXX-based change control system. Our "dependency analysis", as it were, is the developers knowing that if they make any copybook changes they may have to recompile other programs. They then do the proper change control actions to make that happen.

Understanding how to get from there to a environment that uses VSCode/Z Open Editor, git, DBB, etc. is quite a bit to try to get in one's brain. I'm trying somehow to learn small steps at a time, but all the documentation I've seen makes a lot of assumptions that I already know the DevOps world, which I do not.

While we do have a DevOps team in-house, the mainframe people don't use it or understand it, and the DevOps team knows pretty much nothing about mainframe. It's hard for me, as someone who does not understand DevOps, to make much sense out of what Wazi/DBB/etc. are supposed to do for us.

Anyway, don't know if you can help me in that regard, but figured I'd put that "out there".

Thanks!

[FALLAI-Denis]

The use of Z Open Editor as a source code editor is not dependent on the implementation of a DevOps approach and the use of Git as a source server.

If you already have mechanisms for managing sources, you can keep it.

In our case, we made the choice to modernize our development workshop, and at the same time to set up DevOps management, and to unify all the tools by reusing what had been put in place for the development in the distributed world.

I confirm that wanting to carry out all these changes at the same time complicates the work and greatly disturbs the teams, already very worried about having to leave a development tool in place for almost 40 years and perfectly mastered.

I also take the opportunity to warn about the fact that mainframe architectures cannot be managed like distributed architectures, because of the technologies used.

Often in distributed architectures one makes use of interpreted programming languages, with a "late" binding between the components (at runtime).

In the mainframe architectures, we use compiled languages, with an "early" binding between the components (either during build, compilation, or deployment).
This creates constraints in the way of packaging and distributing.

The volumes of the number of components are not the same either.
In distributed architectures, complete applications are deployed, even if only certain components have evolved.
I can hardly imagine having to deploy complete applications in mainframe architecture: several tens or hundreds of Db2 tables, several hundreds or thousands of COBOL programs.
In addition, there are often interdependencies between applications in mainframe architectures, which are not necessarily found in distributed architectures (and especially client architectures).

To enlighten you on the management of sources and deployments, I can advise you to read this document, produced by the excellent and brilliant Nicolas Dangeville (whom I have known for a long time and whom I particularly appreciate):
Develop mainframe software with OpenSource source code managers and IBM Dependency Based Build

[fswarbrick]

@FALLAI-Denis, thanks so much for this last reply, especially the link to that document. It points out all of the "mainframe specific" issues that I've been concerned about. It will take me a while to really grasp all of its answers, but I'm glad to see it address these things.

Do you know if they ever published the part 2 of that document?

[FALLAI-Denis]

I think the second part is here:
https://www.ibm.com/support/pages/system/files/inline-files/Packaging-and-Deployment-Strategies-in-CICD-Pipelines-for-MainframeDev.pdf

[kraabrsg]

Hi, we have setup VSCODIUM + zowe + z open editor to use it for PL1 development.
Can anyone give us a working example to include copybooks from the local drive ?
"zopeneditor.propertygroups": [ { "name": "mvs-members", "type": "mvs", "syslib": [ "zzzz.ttt.xxxx", "xxxx.yyyy.zzz", ] }

i have tried this example(in the folder i have files with xxxxx.INC like set in "zopeneditor.datasets.pl1Datasets" ) from
https://ibm.github.io/zopeneditor-about/Docs/setting_propertygroup.html#_3-absolute-path-patterns :
"name": "local-files", "type": "local", "syslib": [ "C:\\Users\\<username>\\Desktop\\zopeneditor-sample\\COPYBOOK", "C:/Users/<username>/Desktop/zopeneditor-sample/INCLUDES" ],

When opening a pl1 file just copybooks from type MVS get resolved but no local files.

Am i missing something (z open editor version 1.2.1)?

Thank you!

[FALLAI-Denis]

@kraabrsg

Please do not introduce any information that is not related to the current subject (open a new issue).

[phaumer]

Let's try something new: I just enabled Discussions in this GitHub repo that could be used to for these kind of usage questions as well as discussion topics and experience reports. I have not used this feature before, but let's give it a try.

[phaumer]

Converted this from issue to discussion topic.

[FALLAI-Denis]

The documents listed above, and others just as interesting, are referenced on this site:
DevOps Accelaration Program - Resources