Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add Fork Choice compliance test generator #3831

Draft
wants to merge 112 commits into
base: dev
Choose a base branch
from
Draft

Add Fork Choice compliance test generator #3831

wants to merge 112 commits into from

Conversation

ericsson49
Copy link
Contributor

@ericsson49 ericsson49 commented Jul 3, 2024 •
edited by mkalinin
Loading

An implementation of the Fork Choice compliance test generator, which is a part of the initial phase of Fork Choice compliance testing methodology.

More information, including links to generated test suites, is available in the preliminary announcement.

This work was supported by a grant from the Ethereum Foundation.

Test suites

Test suite size Purpose Link
Tiny 135 tests Demonstration, smoke testing link
Small 1472 tests Initial adoption, smoke testing link
Standard 13240 tests Main testing link

TODO list

  • Code refactoring
  • Decide where to put the test generator, as it's not supposed to be a part of regular runs
  • Decide how to integrate the test generator into existing release procedures

mkalinin and others added 30 commits June 28, 2024 14:51
@mkalinin
Implement basic version of SM link generator
68deb2d
@mkalinin
Remove numpy as a dependency
f989f37
@mkalinin
Clean up generator's code
967445f
@mkalinin
Add debug output
2101e9f
@mkalinin
Add anchor_epoch as a parameter and debug checks
507bac8
@mkalinin
Switch sm_links generator to minizinc model
5885bf4
@mkalinin
Rename sm_links test and its deps
0c7ff17
@mkalinin
Disable debug mode
a6e9dd5
@mkalinin
Fixes and polishing
e00346a
@mkalinin
Update sm_links test and generator
cf38551
@mkalinin
Fix contraint reachability problem
ae322c7
@mkalinin
Update debug output
3a2d5fe
@mkalinin
Add CLI args to fork choice generator
995cab2
@mkalinin
Fix bugs, inrease verbosity
beac058
@mkalinin
Add basic block tree model instantiator
e07f580
@mkalinin
Minor fixes
aab9f4b
@mkalinin
Add block cover instantiator
c46b9a3
@mkalinin
Update block_cover3 model and fix instantiator
84c658b
@ericsson49 @mkalinin
Sort tips before sampling
e4b4556
@ericsson49 @mkalinin
Added an option to pre-generated model instances: `--fc-gen-instances…
71e70ad 
…-path`
@ericsson49 @mkalinin
Sample yaml file with pre-generated instances
0f358da
@ericsson49 @mkalinin
minor fixes
c819ef4
@ericsson49 @mkalinin
rename params
9b5f6ec
@ericsson49 @mkalinin
Modified filter_block_tree_generator.py: added an option to loa pre…
a0999cd 
…-generated MiniZinc instances.
@mkalinin
Implement on_attestation
5f00017
@mkalinin
Implement attester slashings
310893c
@mkalinin
Update block tree model
cc852ca
@ericsson49 @mkalinin
Added test_gen.py driver to generate all tests from a single confi.
d477b33 
NB: to be renamed to `main.py` in future.
@mkalinin
Implement invalid blocks
b0be1ae
@mkalinin
Omit valid: true for on_attestation
b563381
ericsson49 and others added 26 commits June 28, 2024 14:51
@ericsson49 @mkalinin
split yield_fork_choice_test_case into two steps:
fd21a46 
- make steps/events (tick|block|att|slashing) from messages
- execute the steps/events
@ericsson49 @mkalinin
refactoring of mutated test generation
aa91e8d
@ericsson49 @mkalinin
re-worked test generation: skip duplicate tick events
7fe26e6
@ericsson49 @mkalinin
Links added:
8233c8c 
- previous repo
- testing methodoolgy description
@ericsson49 @mkalinin
Test runner added
05bbbc7
@ericsson49 @mkalinin
Typos fixed - increase amount of tests for shuffling test
3cec3f1
@ericsson49 @mkalinin
Fixed some bugs with scheduling
c947d0c
@ericsson49 @mkalinin
Move mutation_operators to instantiators
7c9ddbb
@mkalinin
Extend fork choice test format
f184254
@ericsson49
remove with_viable_for_head_weights=True (kept by mistake)
47199a3
@ericsson49
put back MutationOps (dropped by mistake)
02a0287
@ericsson49
switch to DENEB as the default fork
423515e
@ericsson49
remove debug prints and recovery property from test steps
9930464
@ericsson49
drop unnecessary comments
6a27e1e
@ericsson49
Drop checks_with_viable_for_head_weights which is not a part of `on…
5a14ef6 
…_tick_and_append_step` yet
@mkalinin
Merge pull request ethereum#17 from ericsson49/fc-compliance2
8545060 
Some fixes
@ericsson49
added viable_for_head_roots_and_weights check description
4ed16ab
@ericsson49
fixed names (small -> tiny, smoke -> small)
fb3ff0f
@ericsson49
added instructions how to generate and run tests
c494a96
@ericsson49
Update README.md
fb231de 
Fix typo
@ericsson49
Update README.md
eeafb1c 
Fixed typos
@ericsson49 @mkalinin
Update tests/formats/fork_choice/README.md
959274f 
Co-authored-by: Mikhail Kalinin <noblesse.knight@gmail.com>
@mkalinin
Fix viable_for_head_roots_and_weights in example
ce897c8
@mkalinin
Merge pull request ethereum#19 from ericsson49/fc-compliance2
4a0745b 
Add new check description to F/C test format
@ericsson49
Added a line about EF grant
543ec86
@ericsson49
Changed viable_for_head_roots_and_weights format
1854266
@mkalinin mkalinin mentioned this pull request Jul 3, 2024
Draft
2 tasks
@ericsson49 ericsson49 mentioned this pull request Jul 9, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
scope:CI/tests/pyspec scope:fork-choice
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@ericsson49 @hwwhww @mkalinin