test: add script to test all implementations #96
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The script tests all implementations and prints PASS or FAIL status. In case of failure it also prints implementation output to stderr. This will be handy for adding new test samples. Show test statuses and omit failing output: ```sh $ ./test_all.sh 2>/dev/null PASS artsiomkorzun PASS baseline PASS bjhara PASS criccomini FAIL ddimtirov FAIL ebarlas PASS filiphr FAIL itaske PASS khmarbaise FAIL kuduwa-keshavram FAIL lawrey PASS padreati FAIL palmr PASS richardstartin FAIL royvanrijn FAIL seijikun PASS spullara PASS truelive ``` Show only passing implementations: ``` $ ./test_all.sh 2>/dev/null | grep PASS | cut -d' ' -f2 artsiomkorzun baseline bjhara criccomini filiphr khmarbaise padreati richardstartin spullara truelive ``` For gunnarmorling#61
AlexanderYastrebov
force-pushed
the
test_all.sh
branch
from
553a0f0
to
0d4d8c0
Compare
|
Ah, that is really good! Thank you so much, really appreciate it. I'm surprised to see lawrey fail, that LGTM here 🤔 . |
Might be due to my setup. E.g. in my fork I had to fix scripts that use sdk like 71a4ee8 |
|
It was caused by the GC logging he did. I've removed that one, so it's showing |
|
Do others pass in your environment? Maybe you could run test_all.sh and file an issue for each failure? |
|
Done, see #284 ff. (there's a few more failures but these authors have pending PRs already, so I reckon any failures have been fixed there already. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The script tests all implementations and prints PASS or FAIL status. In case of failure it also prints implementation output to stderr.
This will be handy for adding new test samples.
Show test statuses and omit failing output:
$ ./test_all.sh 2>/dev/null PASS artsiomkorzun PASS baseline PASS bjhara PASS criccomini FAIL ddimtirov FAIL ebarlas PASS filiphr FAIL itaske PASS khmarbaise FAIL kuduwa-keshavram FAIL lawrey PASS padreati FAIL palmr PASS richardstartin FAIL royvanrijn FAIL seijikun PASS spullara PASS trueliveShow only passing implementations:
For #61