The language server is not a language compiler

[FALLAI-Denis]

Hi,

For COBOL, several issues have been reported that seem to be related to the way the language server works.
#321 - COBOL - Error while processing boolean compilation variables
#203 - COBOL - Independence of OUTLINE view and other Language Server functions
#202 - COBOL Parsing Error because of Period in 72 and Standard Numbering Present
#199 - COBOL - Independence of search function COPYBOOKs and other Language Server functions
#189 - COBOL - Sequence unnumber action could not be performed.. Internal error: {}
#186 - COBOL COPYBOOK Improperly reporting TOKEN errors
#181 - COBOL - COPY REPLACING Invalid Parsing Error Message
#154 - Cobol Language Server - "Full red program" after LS analysis
#113 - COBOL - Access to COPYBOOKs independent of conditional compilation directives
#96 - COBOL - Improve conditional compilation support (COPYBOOKs analysis)
#83 - COBOL - Improve conditional compilation support (conditional / dynamic syntax check)

It seems to me that this is because the language server tries to work like the compiler and parses the source code like the compiler would.
I think this is a mistake.
The language server cannot have the same knowledge as the compiler and cannot decide the validity of the source code like the compiler would.

At the language server level, the code can be fragmented, with incomplete declarations, one end in the source code of the program and one end in a COPYBOOK, or redundant but under conditional compilation control and therefore not all active at the same time during of the compilation.
The language server is also not aware of external informations, such as, for example, the passing of the value of conditional compilation variables by the DEFINE compilation option, or COPYBOOKs not accessible by the language server (either by configuration error, or intentionally).

For these reasons, the language server must be much more tolerant than the compiler, and must not try to have a complete analysis of a program, but must limit the scope of the analysis to the smallest possible part of the source code (which seems to be the case for COPYBOOKs which by definition contain an incomplete code).
Likewise, an error detected in one place in the source code should not impact other places in the source code.

It is also necessary to separate the functionalities in the language server and to be able to deactivate them selectively.
Currently, because conditional compilation is problematic for the language server which then declares almost all of the source code in error, we have to deactivate the language server. By doing this we also deactivate the search for COPYBOOKs, which is unrelated... we also deactivate the advanced search and replace functions... we then lose all the advantages of Z Open Editor ...

[kkrein]

[gecolbert]

I'm new to this project, so forgive my ignorance.

Exactly what is the language server? I need the editor to identify coding errors properly (as determined by the Enterprise Cobol compiler). To me, this needs to be the case if the data is defined in working-storage or is included from a copy book. The reality that I have to live with is that our copybooks have been coded with 'the flexibility' to allow their contents to be used multiple ways. As it sits, z Open Editor has problems with copybooks that have been in my production system for over 40 year because it get confused parsing the COPY REPLACING statement.

There is so much good work that has been done here that I feel out-of-line reporting issues. Is this a project where the code is shared and can be contributed to? If so, how do I go about accessing the code to see if I can address my copy replacing issues?

Glen

[FALLAI-Denis]

Hi @gecolbert

For an explanation of language servers you can refer to the VS Code site:
Language Server Extension Guide
or the site dedicated to the language server protocol
What is the Language Server Protocol?

In a nutshell, a language server provides understanding of the code, and therefore identifying the nature of the information present in the code, where a simple parser (textmate grammar) can almost only do syntax highlighting.
With a language server you can for example find all the uses of a variable, because a fragment text has been recognized as being a variable, and not do a simple search and replace of strings.

I think that the majority of the problems encountered today when parsing COBOL source code by the language server come precisely from the fact that it tries to function like the compiler, which is not possible because fragments of text such as COPYBOOKs cannot be understood outside their context of use which only exists at compile time and not when editing code in Z Open Editor.

You take the example of the data that must be defined in Data Division. You will see in problem #83 a use case where this cannot be applied.

[phaumer]

Sorry, for the slow response. There are certainly pros and cons for this and perhaps lots of grey areas, but our ambition for our language server is to come close to the compiler, i.e. finding as many issues as possible and parsing everything correctly for different context settings.

We are certainly not there, but our flag ship IDE IBM Developer for z/OS (IDz) provides already many solutions for some of the problems that we want to address in the same way. The notion of property groups was driven by IDz and it uses those to provide the missing context. For conditional compilation our thinking is to utilize property groups there further. We already have a compiler options property and want to use that to declare the DEFINE values to be used for preprocessing and then parsing, so the user can edit one set of branches based on these values at the time. Also property groups could be used to narrow down the scope for editing copybooks in the scope of a specific program and settings.

For third party preprocessor IDz also has a solution that we are also looking to adopt in a similar fashion.

[FALLAI-Denis]

Extract from #189

Today the functioning of the server language is of the "all or nothing" type, one cannot activate / deactivate by functionalities, but it is probable that certain functionalities are independent of the others.

It also seems to me that the language server tries to parse the source code as a whole and that this is not always possible due to the use of COPYBOOK and conditional compilation. The language server would have to do local analyzes of the source code, even if it means being less complete or less powerful.

Apparently when parsing a COPYBOOK (.cpy) the language server is less demanding than when parsing a COBOL (.cbl). This suggests that mechanisms limiting the scope of the analysis must therefore be in place in the language server.

Would it be possible to introduce a level of analysis in the language server, going from "all functions" to "minimal functions" to selectively deactivate the functions that could cause problems?
It seems to me that this is what is done for the detection of unreachable code.

To help debug the language server, it would be necessary to be able to activate logs or traces to capture information on the problems encountered.

[FALLAI-Denis]

Hi @phaumer

If you set up a system of properties groups, it will have to allow a certain flexibility and does not impose to organize the source files in folders by compilation typologies because this does not necessarily make sense for the developer: today our source files are rather distributed in folders by business functionality.

As we are not (yet) using IBM Dependency Based Build (but it could be in the future), we have implemented an equivalent of properties groups, with one companion file per source file (COBOL or other) : the companion file has the same name as the source file to which we add a second extension, (for example: PROGRAM.cbl source file, PROGRAM.cbl.build companion file).
The companion file, internal YAML format, validated by Schema, contains the information needed to build the source file.
We also have files that describe the different build targets, containing the information attached to the targets. The companion file can reference one or more targets, which will trigger one or more builds (our information system is made up of several physical servers and we have to distribute certain components on several servers simultaneously).
It's a proprietary solution, but we could adopt a standard solution if it was offered, and if it met our expectations. If you find it interesting, we could discuss expected features before you propose a solution through Z Open Editor.

[FALLAI-Denis]

Hi,

Z Open Editor does not edit "true" COBOL source code.
Z Open Editor edits a text file which after several operations carried out by the various stages of the COBOL compiler will become a "correct" COBOL source code.

The various transformations applied to the text file by the compiler, during the text manipulation stage, before this text file becomes a "correct" COBOL source code processed by the compilation stage, are as follows (extracted from the draft of the ISO COBOL standard):

Step 1:
An expanded compilation group is created in logical free-form reference format — input lines are accepted sequentially; logically converted to free-form reference format and placed in the expanded compilation group.
Library text identified in COPY statements is incorporated; replacing actions associated with the REPLACING phrase of the COPY statement are deferred to processes associated with step 2. COPY statements and their incorporated text shall be identifiable in the expanded compilation group for purposes of any logically subsequent processing associated with a REPLACING phrase.
Lines that appear in the false path of an IF or EVALUATE directive, including library text identified in COPY statements, may be omitted from the expanded compilation group. SOURCE FORMAT directives in the false path shall be processed to correctly interpret input lines.
NOTE: Recognition of true and false paths during logical conversion is neither required nor precluded.
The resulting lines constitute an expanded compilation group.

Step 2:
A conditionally-processed compilation group is created — the expanded compilation group is read and the following compiler directives and substitutions are processed in the order encountered in the expanded compilation group:
a) DEFINE, IF, and EVALUATE directives
b) Substitution of compilation-variable values into constant entries
c) Saving and restoring directives with PUSH and POP directives
d) The replacing actions of COPY statements.
The resulting lines constitute a conditionally-processed compilation group.

Step 3:
The conditionally-processed compilation group is read and the replacing actions of REPLACE statements are applied in order.

Step 4:
The results of Step 3 are read, and the actions of the COBOL-WORDS directive are applied in order.
The resulting lines constitute a structured compilation group.

References to a compilation group after text manipulation processing are to the structured compilation group, which contains the lines to be used in the compilation stage.

Added to this is the processing of compilation options, and in particular the DEFINE option which allows you to value conditional compilation variables which influences the different steps of the text manipulation step.

All this to say that the syntactic/semantic checks performed by Z Open Editor should be much more permissive than those performed by the COBOL compiler.