Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DX Streamline] LOTUS_RUN_EXPENSIVE_TESTS periodically outside of PR workflow #12136

Open
BigLep opened this issue Jun 22, 2024 · 3 comments · May be fixed by #12234
Open

[DX Streamline] LOTUS_RUN_EXPENSIVE_TESTS periodically outside of PR workflow #12136

BigLep opened this issue Jun 22, 2024 · 3 comments · May be fixed by #12234
Milestone

Comments

@BigLep
Copy link
Member

BigLep commented Jun 22, 2024

Done Criteria

  1. LOTUS_RUN_EXPENSIVE_TESTS are run periodically (e.g., nightly) against master automatically. They aren't run as part of the normal PR flow because of how long they take.
  2. There is notification via github issue or Slack if there are any failures so that maintainers can try and identify the culprit change.
  3. There is a self-service URL to find the previous results of LOTUS_RUN_EXPENSIVE_TESTS.

Why Important

There are "expensive" tests that test key functionality. This is getting exasperated by NI-PoRep tests which can take 10+ minutes on CI. We want to avoid compromising on test quality because of long CI runs. We want the test coverage but not to be slowed down by it for every CI run.

User/Customer

Contributors and maintainers

Notes

  1. This was discussed in #fil-lotus-dev on 2024-06-20 in https://filecoinproject.slack.com/archives/CP50PPW2X/p1718859677050149
@BigLep BigLep added this to the DX-Streamline milestone Jun 22, 2024
@rvagg
Copy link
Member

rvagg commented Jun 22, 2024

Currently LOTUS_RUN_EXPENSIVE_TESTS=1 is the same as CI=true, which is why I proposed that LOTUS_RUN_VERY_EXPENSIVE_TESTS introduces a new level that we would run periodically.

@rvagg rvagg mentioned this issue Jun 25, 2024
10 tasks
@rvagg
Copy link
Member

rvagg commented Jun 25, 2024

master now includes one test that's behind LOTUS_RUN_VERY_EXPENSIVE_TESTS, so this has become a slightly higher priority now

@galargh
Copy link
Contributor

galargh commented Jun 28, 2024

I think it'd be good if there was also a way to request the run of very expensive tests on a PR (e.g. through a label) and if they run when the test itself was modified

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: 📌 Triage
Development

Successfully merging a pull request may close this issue.

3 participants