How to get Feature File commented lines in Cucumber Json Test Report

[rummitta]

In Cucumber version 1.2.5, comments used to appear in test report json:

However, in our framework, we upgraded to Cucumber version 5.7.0 and now we are not able to see the comments in test report.

This is causing lot of trouble for our users - who have used comments to express lot of information about the test case they are trying to test. Please let us know how to get back the comments in the cucumber report - so that subsequently, we can show the comments in the report generated by our framework.

We are using selenium java based framework. any pointers will be helpful.

[rummitta]

Attaching the cucmber json report before (with comments) and after (missing comments)
FeatureFileWithSel4.txt
FeaturefilebeforeSel4.txt

[mpkorstanje]

Out of curiosity, the latest version of Cucumber JVM is v7.15.0. Why are you upgrading only to v5.7.0?

And I don't think there is a way to get those comments back using the current json formatter. The change in architecture between v1 and v2 was pretty drastic. And one of the changes, is that to wit comments are no longer associated to an element.

You didn't describe your setup in much detail, but assuming your report is completely custom, you could consider building your report based on Cucumber Messages instead. The GherkinDocument message has all the comments.

Cucumber v6 introduced a messages format and we're slowly migrating the cucumber provided plugins to that. For example see the cucumber-junit-xml-formatter and html-formatter already use the message protocol.

If your report is made by another open source tool, you may want to consider migrating that tool to use messages.

[rummitta]

Hi Team,
Tried adding Cucumber options like this:

Also tried setting it in Cucumber options pojo and then setting it in pair:

It is showing the plugin correctly in cucumber options printed in the logs

However, there is no such json file as cucumber.json or no such html file as cucumber.html in the resources folder. The folder specified looks like this

In the final.json file, comments are not present - other step details are present:

Please let me know how to use this plugin format to get the comments in the report. This is very critical for our clients.

[mpkorstanje]

I can't make much sense of your reply. I was looking for an explanation of how you use the output of the .json file to better understand what steps you might have to take to get your reports in working order.

So just to restate this for clarity.

1. The Json formatter does not retain the comments for any version of Cucumber after v1.

2. Should you want to get the comments back, they're available as part of the message format introduced in v6. You can consume the messages by writing your own plugin.

3. For hints on consuming the message format, I would recommend looking at the implementation of the cucumber-junit-xml-formatter and html-formatter and their usage in the source code of Cucumber-JVM.

If you have specific questions feel free to ask.

[rummitta]

Was able to get the comments inside a cucumber generated ndjson file... However, if my FF has 3 scenarios and all of them have different set of comments, the ndjson file generated seem to have all the comments put together - without distinguishing which comment belongs to which scenario. This is not helping in the correct test report generation.
Also the ndjson file generated is too huge - for just 3 scenarios (each scenario having 6-7 steps), the ndjson file is 572 KB. In our client's case, since each scenario itself has as many as 60-80 steps, the corresponding ndjson file is likely to cause issues. Is there any other way to generate a smaller file which has comments in the correct position.

[mpkorstanje]

However, if my FF has 3 scenarios and all of them have different set of comments, the ndjson file generated seem to have all the comments put together - without distinguishing which comment belongs to which scenario. This is not helping in the correct test report generation.

The design of the Gherkin parser changed and comments are no longer associated with scenarios. I can't immediately find the associated change with the reasoning though.

Currently you will have to re-associate them yourselves.

Also the ndjson file generated is too huge - for just 3 scenarios (each scenario having 6-7 steps), the ndjson file is 572 KB.

The bulk of the output is made up of step definitions, hooks and parameter types which re-instantiated for every scenario. This unfortunately can not be fixed until cucumber/cucumber-jvm#2035 gets resolved. Possibly through cucumber/cucumber-jvm#2279 which is a major undertaking. In the mean time you could filter hooks, step definitions and parameter type messages from the output as done by the html formatter here:

https://github.com/cucumber/cucumber-jvm/blob/main/cucumber-core/src/main/java/io/cucumber/core/plugin/HtmlFormatter.java#L25-L31

Note: You can ask Cucumber to run your own plugins through cucumber.plugin=com.example.MyPlugin:optional-output.txt.

[mpkorstanje]

I'm assuming you've figured things out. If not, feel free to ask more questions.