Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Testing/jenkins integration #862

Merged
merged 67 commits into from
Oct 24, 2023
Merged

Testing/jenkins integration #862

merged 67 commits into from
Oct 24, 2023

Conversation

fionnodonohoe-xlnx
Copy link
Collaborator

@fionnodonohoe-xlnx fionnodonohoe-xlnx commented Jul 24, 2023

Main changes include:

  • test support for 4 boards: Pynq-Z1, Alveo U250, Kria SOM KV260 and ZCU104
  • added a bnn sanity test option (creates a single bitstream for the now 4 supported boards)
  • broke up existing bnn end2end tests into a per-board option
  • refactored existing Jenkins setup to a declarative Jenkins pipeline
  • XML results captured for tests - for jenkins GUI
  • HTML results captured for docker & hardware tests
  • HTML code coverage results captured for test suites: util, streamline, transform, brevitas_export, notebooks, fpgadataflow

When testing on HW, various issues were seen:

  • the following commit was needed as U250 tests were failing: e76f20d
  • there were some erroneous results due to incorrect data generated for the HW tests.
    A workaround was put in just to get some tests passing - so a temporary hack script is seen in this PR.
    I removed this script when I correctly adjusted the test data here: 61cba65 and 10d34b5
  • The Jenkinsfile became too large at one point so testing had to be done with multiple job runs. I put the end2end testing and html report code on separate 'feature' branches until I could shrink the Jenkinsfile appropriately and retest

Signed-off-by: Fionn O'Donohoe <fionno@xilinx.com>
Signed-off-by: Fionn O'Donohoe <fionno@xilinx.com>
Signed-off-by: Fionn O'Donohoe <fionno@xilinx.com>
Signed-off-by: Fionn O'Donohoe <fionno@xilinx.com>
…ew directory

Signed-off-by: Fionn O'Donohoe <fionno@xilinx.com>
…r more boards to be parameterized

Signed-off-by: Fionn O'Donohoe <fionno@xilinx.com>
Signed-off-by: Fionn O'Donohoe <fionno@xilinx.com>
…y board marker

Signed-off-by: Fionn O'Donohoe <fionno@xilinx.com>
Signed-off-by: Fionn O'Donohoe <fionno@xilinx.com>
Signed-off-by: Fionn O'Donohoe <fionno@xilinx.com>
Signed-off-by: Fionn O'Donohoe <fionno@xilinx.com>
Signed-off-by: Fionn O'Donohoe <fionno@xilinx.com>
Signed-off-by: Fionn O'Donohoe <fionno@xilinx.com>
Signed-off-by: Fionn O'Donohoe <fionno@xilinx.com>
…djust test scripts to workaround these changes

Signed-off-by: Fionn O'Donohoe <fionno@xilinx.com>
Signed-off-by: Fionn O'Donohoe <fionno@xilinx.com>
Signed-off-by: Fionn O'Donohoe <fionno@xilinx.com>
… from the pipeline

Signed-off-by: Fionn O'Donohoe <fionno@xilinx.com>
Signed-off-by: Fionn O'Donohoe <fionno@xilinx.com>
Signed-off-by: Fionn O'Donohoe <fionno@xilinx.com>
The fpgadataflow tests were placed in their own stage with their own build parameter as the test takes longer than a day to run.
This means that this suite cannot sensibly be used in daily CI test runs.

Some notes on the stages and their setup:
- the when{} block is used as an 'if' statement, checking if a certain input parameter to the pipeline has been set. By default - the fpgadataflow stage will not run unless explicitly set to true by the tester/CI system
- FINN_HOST_BUILD_DIR is set to a unique directory per stage for ease of use/test cleanup
- catchError is used in order to allow the pipeline to continue to possible future stages if a stage along the way fails. Otherwise the first failed stage found would end the test run

Signed-off-by: Fionn O'Donohoe <fionno@xilinx.com>
Signed-off-by: Fionn O'Donohoe <fionno@xilinx.com>
…st scripts

Signed-off-by: Fionn O'Donohoe <fionno@xilinx.com>
Signed-off-by: Fionn O'Donohoe <fionno@xilinx.com>
…d is offline

Signed-off-by: Fionn O'Donohoe <fionno@xilinx.com>
docker/jenkins/Jenkinsfile Show resolved Hide resolved
docker/jenkins/Jenkinsfile Show resolved Hide resolved
docker/jenkins/Jenkinsfile Show resolved Hide resolved
docker/jenkins/Jenkinsfile Show resolved Hide resolved
docker/jenkins/Jenkinsfile Outdated Show resolved Hide resolved
docker/jenkins/Jenkinsfile Outdated Show resolved Hide resolved
docker/jenkins/Jenkinsfile Outdated Show resolved Hide resolved
docker/jenkins/Jenkinsfile Outdated Show resolved Hide resolved
johnnoel and others added 6 commits August 23, 2023 09:54
[CI] Fix bug with build parameters and result flags sharing common names
…mmands

Jenkins unexpectedly trims trailing spaces from env variables. This leads to badly formed inputs for docker. Appending an extra space solves this issues and causes no further problems.
@smtalds
Copy link

smtalds commented Oct 6, 2023

@fionnodonohoe-xlnx @auphelia Hello, can I get information about Kria KV260? Can we use this board for FINN? Do we submit this information to the FINN documentation if it is available?
image

@jmonks-amd jmonks-amd force-pushed the testing/jenkins-integration branch 4 times, most recently from 46767dd to 1787ec5 Compare October 17, 2023 14:14
@auphelia
Copy link
Collaborator

Thanks @fionnodonohoe-xlnx and @johnnoel-xlnx !

@auphelia auphelia merged commit 69bd72f into dev Oct 24, 2023
2 checks passed
@auphelia auphelia deleted the testing/jenkins-integration branch October 24, 2023 09:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants