Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rework rlpx-simulator.spec.ts tests / Fix devp2p/genesis/wallet Coverage Reporting #3760

Merged
merged 19 commits into from
Oct 31, 2024

Conversation

scorbajio
Copy link
Contributor

This change looks into the rlpx-simulator.spec.ts tests in the devp2p package and attempts to understand and fix the test cases and if any expected bugs are found, they will be discussed here and fixed in followup PRs. Also see #3755.

Copy link

codecov bot commented Oct 21, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 49.05%. Comparing base (dc7169c) to head (74da3dc).
Report is 1 commits behind head on master.

Additional details and impacted files

Impacted file tree graph

Flag Coverage Δ
block 67.57% <ø> (ø)
blockchain 83.50% <ø> (ø)
client 0.00% <ø> (ø)
common 89.88% <ø> (ø)
devp2p 71.95% <ø> (+71.95%) ⬆️
evm 64.89% <ø> (ø)
genesis 100.00% <ø> (+100.00%) ⬆️
mpt 52.30% <ø> (+0.50%) ⬆️
statemanager 68.84% <ø> (ø)
tx 76.70% <ø> (ø)
util 73.46% <ø> (ø)
vm 57.65% <ø> (ø)
wallet 79.67% <ø> (+79.67%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

it('RLPX: remove node', async () => {
const { rlpxs, peer } = util.initTwoPeerRLPXSetup(undefined, undefined, undefined, 40504)
rlpxs[0]
['_dpt']!.addPeer(peer)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reason I moved the addPeer call out of initTwoPeerRLPXSetup is to be able to chain the disconnect here before checking for peer:removed events in the awaited promise that follows.

assert.equal(rlpxs[0]._getOpenSlots(), 9, 'should have maxPeers - 1 open slots left')
await util.delay(500)
util.destroyRLPXs(rlpxs)
const { rlpxs, peer } = util.initTwoPeerRLPXSetup(undefined, undefined, undefined, basePort + 1)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe basePort + 1 is correct here since the following assert is checking if the peer's port is connecting as expected.

throw new Error(`Peering failed: ${e}: ${e.stack}`)
})
await new Promise((resolve) => {
rlpxs[0].events.once('peer:removed', (_, reason: any) => {
assert.equal(
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This assert is failing, expecting a CLIENT_QUITTING but getting a TIMEOUT.

@holgerd77
Copy link
Member

Can you add a readiness label (+ drop a comment on that)?

@holgerd77
Copy link
Member

Could you give a summary of what you finally did here and how things played out?

Copy link
Contributor Author

@scorbajio scorbajio left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I left some comments detailing what I did with the last few commits to get the tests running as expected. Other than that, I also fixed some typing errors.

const basePort = 60661
const rlpxs = util.getTestRLPXs(3, 1, basePort)
const peer = { address: util.localhost, udpPort: basePort + 1, tcpPort: basePort + 1 }
rlpxs[0]['_dpt']!.addPeer(peer)
try {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reworked this test case to be a lot more similar to how the test was originally written to preserve the test coverage it gives for queuing/refilling connections while also updating it to run asynchronously.

}
})
}, 30000)
it('RLPX: remove node', async () => {
Copy link
Contributor Author

@scorbajio scorbajio Oct 29, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I both swapped around the final two test cases and increased the timeout of one of them since they have some interference happening with one another: They both pass in isolation and when ran in this new relative order, but fail with the old order they were in. Not sure if the cleanup is experiencing issues.

@holgerd77
Copy link
Member

Thanks for the insights! 🙏 And on the more general level, so this re-activates test running in general, is this correct?

Can we "prove" this is working by coverage already? If not we should get the coverage running again as a first step maybe and then test here with the fix, at least that's what I would suggest if I get everything "right" here and over on the coverage front. 🙂

@scorbajio
Copy link
Contributor Author

Thanks for the insights! 🙏 And on the more general level, so this re-activates test running in general, is this correct?

Can we "prove" this is working by coverage already? If not we should get the coverage running again as a first step maybe and then test here with the fix, at least that's what I would suggest if I get everything "right" here and over on the coverage front. 🙂

Yes, this should activate tests that were previously not running fully to show lines covered increase in codecov relative to the base commit. devp2p is however one of a number of our packages that has switched over fully to using vitest instead of tape, which results in lines covered not showing up in the uploaded lcov file generated with coverage runs. I'll prioritize getting coverage results for the devp2p package as a part of #3754 and come back to test the theory with these tests being fixed and reintroduced 🙂

@scorbajio
Copy link
Contributor Author

scorbajio commented Oct 30, 2024

I switched devp2p, genesis, and wallet over to using vitest's v8 provider as their coverage provider, it seems to be doing the trick. The new coverage report for rlpx.ts can be seen to show lines covered by the fixed tests, than the original (I locally calculated the coverage for the file on master as well to confirm this). This just leaves client as the last major package to not be showing any lines covered, likely due to issues with vitest, and a similar issue we have is two packages are uploading coverage files that are erroring as "unreadable," which I wonder if that is also related to changing our testing framework to vitest. You can also see here that the devp2p flag is showing suddenly a large percentage increase in coverage, as further confirmation to switching over to the vitest coverage provider is a working fix for now.

UPDATE: I wasn't able to get coverage reports generated for the wallet package locally because the tests in index.spec.ts aren't finishing running after a long time. This might not be an issue when run in the CI, otherwise, it would be an issue with the wallet tests rather than code coverage reporting.

Copy link
Member

@holgerd77 holgerd77 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! 🙏🎉

Thanks for all the additional info!

@holgerd77 holgerd77 changed the title Rework rlpx-simulator.spec.ts tests Rework rlpx-simulator.spec.ts tests / Fix devp2p/genesis/wallet Coverage Reporting Oct 31, 2024
@holgerd77 holgerd77 merged commit 987a855 into master Oct 31, 2024
40 of 41 checks passed
@holgerd77 holgerd77 deleted the rlpx-simulator-tests-fixes branch October 31, 2024 16:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants