[BUG]: Summary not generated unless script is terminated

[tekno0ryder]

Project

extension-checker

Description

I am working on a JS server that utilizes accessibility-checker to intercept chrome debugging session whenever needed to run getCompliance() on the page and save result.

Issue

I had really hard time trying to generate summary reports either in xlsx or csv until I figured out the script has to be terminated in order to generate them, closing the engine was not enough.

Solution

There was emphasize on closing the engine but terminating script was never mentioned in the docs.
1- Either generate summary when closing engine itself.
2- If above is a huge breaking change then improving the docs to demonstrate this point.

Thanks

Steps to reproduce

1- Run getCompliance().
2- Close the engine.
3- Keep the server running while(true){}

[shunguoy]

@tekno0ryder If an error occurs before the engine is closed, the summary may not be created, so it's best to close the engine in a 'finally' clause to cover both success and failure cases. See more examples in the README.

[tombrunet]

@tekno0ryder Just to avoid ambiguity, you have to call the close method shown in the examples referenced above. When close is called, it indicates that you are done with the checker and it will run the report generation actions that normally run on exit.

[tekno0ryder]

@shunguoy @tombrunet Thanks for the response, I am sure no exception is thrown and I made sure to call "close" but that was not enough to generate summaries.

The example I tried is from boilerplate "batch-scan" and by just adding an infinite loop at the end you can reproduce it

[shunguoy]

Need to update the document to point out that the report is generated only if the engine is properly closed. If the script keeps running (such as a while (true)) without closing the engine, no report will be generated.

[tombrunet]

@tekno0ryder Note that while (true) {} will prevent any code from executing. Node is single-threaded, so if you create an infinite loop that isn't asynchronous, you'll hang everything in a Node environment. This isn't a realistic scenario where you'd perform testing.

[tekno0ryder]

@tombrunet True but having a service like Express.js or ZeroMQ like in my case it's expected to be a long running process that never terminates.

[shunguoy]

@tekno0ryder have you tried in your while (true) to construct the checker, run getCompliance(), then close the engine for each test case? in this case, each test is run independently though it's not desirable performance-wide.

[tombrunet]

A more realistic test case would be something like:

(async () => {
  while (true) {
    await new Promise(resolve => setTimeout(resolve, 1000));
  }
})()

So, it would keep the Node environment running, but also allow Node to be able to process other things on the call stack.

That said, digging into the code, it looks like the report generation isn't currently happening as part of the close function. I'm trying to figure out when and why that happened.