Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[executor] Introduce on-the-fly scheduling for Read/Allocate/Write keys #814

Merged
merged 52 commits into from
Apr 17, 2024
Merged

[executor] Introduce on-the-fly scheduling for Read/Allocate/Write keys #814

Show file tree
Hide file tree
Changes from 47 commits
Commits
Show all changes
52 commits
Select commit Hold shift + click to select a range
7cabcd7
[executor] refactor worker func
wlawt Mar 28, 2024
8f1d6f9
[executor] renaming
wlawt Mar 28, 2024
af221fc
[executor] use blocking/dependency type from programmatic-deploy fork
wlawt Mar 28, 2024
38d0d12
[executor] add key type support
wlawt Mar 28, 2024
adcedb6
wip
wlawt Apr 2, 2024
e376a81
wip
wlawt Apr 2, 2024
35c7936
switch statements
wlawt Apr 3, 2024
54d739a
programmatic deploy fork + concurrent reads work
wlawt Apr 3, 2024
4f713d2
sequential writes work
wlawt Apr 3, 2024
46bbffb
w->r->r works
wlawt Apr 3, 2024
2cb8686
r->r->w works
wlawt Apr 3, 2024
f0a6a7f
w->r->r...w->r->r works
wlawt Apr 3, 2024
171c25c
r->r->w...r->r->w works
wlawt Apr 3, 2024
d22fd32
all unit tests pass
wlawt Apr 3, 2024
5665181
cleanup
wlawt Apr 3, 2024
c2d0a92
coverage 97%
wlawt Apr 3, 2024
95555a5
cleanup switch
wlawt Apr 3, 2024
05fd669
switch comments
wlawt Apr 3, 2024
c892947
self review
wlawt Apr 3, 2024
f50423a
go mod tidy
wlawt Apr 4, 2024
17ad618
add done prints to tests
wlawt Apr 9, 2024
bf5ffa1
rename isAllocateWrite
wlawt Apr 9, 2024
a64d5db
remove id from task
wlawt Apr 9, 2024
eac02e1
use dummy dep to keep track of all keys
wlawt Apr 9, 2024
71975ad
simplify for loop in adding bt dep
wlawt Apr 9, 2024
d3074af
fix integration bug and add unit test for it
wlawt Apr 9, 2024
8783dc8
fix race condition with write-after-read(s) potentially
wlawt Apr 10, 2024
bd5bbab
run unit tests over multiple iterations
wlawt Apr 10, 2024
b3b2353
cut down on unit testing time and add comments
wlawt Apr 10, 2024
553c5d7
simplify num times we call add blocking if not exec
wlawt Apr 10, 2024
2a9f12e
new way to keep track of concurrent Reads
wlawt Apr 11, 2024
1d2f636
self review
wlawt Apr 11, 2024
dd0acff
have unit tests run under max time
wlawt Apr 11, 2024
dafea5b
[executor] Simplify Logic + Speed Up Tests (#831)
patrick-ogrady Apr 12, 2024
6398126
better var names
wlawt Apr 15, 2024
42e2250
add larger unit tests
wlawt Apr 15, 2024
b1bbf9e
ignore lint for rand
wlawt Apr 16, 2024
09a732b
add unique and conflicting keys randomly to txs
wlawt Apr 16, 2024
6d084f7
fix for loops
wlawt Apr 16, 2024
08362f1
use max function
wlawt Apr 16, 2024
159e9a2
make conflict keys 100 and pick 1-5
wlawt Apr 16, 2024
ad56cbd
make num slow chan consistent
wlawt Apr 16, 2024
2d346d4
use set.Contains to speed up tests
wlawt Apr 16, 2024
d3b4643
random perm for each unique key
wlawt Apr 16, 2024
ef41d6a
group var names
wlawt Apr 16, 2024
53aeadd
use numTxs in generating blocking txs
wlawt Apr 16, 2024
79cef56
increase num conflict keys for concurrent Reads and Writes test
wlawt Apr 16, 2024
fd2bdde
[executor] multi-key conflict bug (#837)
wlawt Apr 17, 2024
180d43f
random perm per conflict key
wlawt Apr 16, 2024
6a9b6b3
make maxDep a param
wlawt Apr 17, 2024
2009f95
add maxDep as const in chain
wlawt Apr 17, 2024
a14e434
placement of maxDep comment
wlawt Apr 17, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion chain/builder.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,7 @@ func HandlePreExecute(log logging.Logger, err error) bool {
}
}

// Note: This code is terrible and will be removed during the Vryx integration.
func BuildBlock(
ctx context.Context,
vm VM,
Expand Down Expand Up @@ -135,12 +136,15 @@ func BuildBlock(
// prepareStreamLock ensures we don't overwrite stream prefetching spawned
// asynchronously.
prepareStreamLock sync.Mutex

// stop is used to trigger that we should stop building, assuming we are no longer executing
stop bool
)

// Batch fetch items from mempool to unblock incoming RPC/Gossip traffic
mempool.StartStreaming(ctx)
b.Txs = []*Transaction{}
for time.Since(start) < vm.GetTargetBuildDuration() {
for time.Since(start) < vm.GetTargetBuildDuration() && !stop {
prepareStreamLock.Lock()
txs := mempool.Stream(ctx, streamBatch)
prepareStreamLock.Unlock()
Expand Down Expand Up @@ -372,6 +376,7 @@ func BuildBlock(
// stop building. This prevents a full mempool iteration looking for the
// "perfect fit".
if feeManager.LastConsumed(dimension) >= targetUnits[dimension] {
stop = true
return errBlockFull
}
}
Expand Down
1 change: 1 addition & 0 deletions examples/morpheusvm/go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -125,6 +125,7 @@ require (
go.opentelemetry.io/otel/sdk v1.11.2 // indirect
go.opentelemetry.io/otel/trace v1.11.2 // indirect
go.opentelemetry.io/proto/otlp v0.19.0 // indirect
go.uber.org/atomic v1.11.0 // indirect
go.uber.org/mock v0.4.0 // indirect
go.uber.org/multierr v1.11.0 // indirect
golang.org/x/crypto v0.17.0 // indirect
Expand Down
2 changes: 2 additions & 0 deletions examples/morpheusvm/go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -636,6 +636,8 @@ go.opentelemetry.io/otel/trace v1.11.2/go.mod h1:4N+yC7QEz7TTsG9BSRLNAa63eg5E06O
go.opentelemetry.io/proto/otlp v0.7.0/go.mod h1:PqfVotwruBrMGOCsRd/89rSnXhoiJIqeYNgFYFoEGnI=
go.opentelemetry.io/proto/otlp v0.19.0 h1:IVN6GR+mhC4s5yfcTbmzHYODqvWAp3ZedA2SJPI1Nnw=
go.opentelemetry.io/proto/otlp v0.19.0/go.mod h1:H7XAot3MsfNsj7EXtrA2q5xSNQ10UqI405h3+duxN4U=
go.uber.org/atomic v1.11.0 h1:ZvwS0R+56ePWxUNi+Atn9dWONBPp/AUETXlHW0DxSjE=
go.uber.org/atomic v1.11.0/go.mod h1:LUxbIzbOniOlMKjJjyPfpl4v+PKK2cNJn91OQbhoJI0=
go.uber.org/goleak v1.2.1 h1:NBol2c7O1ZokfZ0LEU9K6Whx/KnwvepVetCUhtKja4A=
go.uber.org/goleak v1.2.1/go.mod h1:qlT2yGI9QafXHhZZLxlSuNsMw3FFLxBr+tBRlmO1xH4=
go.uber.org/mock v0.4.0 h1:VcM4ZOtdbR4f6VXfiOpwpVJDL6lCReaZ6mw31wqh7KU=
Expand Down
1 change: 1 addition & 0 deletions examples/tokenvm/go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -140,6 +140,7 @@ require (
go.opentelemetry.io/otel/sdk v1.11.2 // indirect
go.opentelemetry.io/otel/trace v1.11.2 // indirect
go.opentelemetry.io/proto/otlp v0.19.0 // indirect
go.uber.org/atomic v1.11.0 // indirect
go.uber.org/mock v0.4.0 // indirect
go.uber.org/multierr v1.11.0 // indirect
golang.org/x/crypto v0.17.0 // indirect
Expand Down
2 changes: 2 additions & 0 deletions examples/tokenvm/go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -670,6 +670,8 @@ go.opentelemetry.io/otel/trace v1.11.2/go.mod h1:4N+yC7QEz7TTsG9BSRLNAa63eg5E06O
go.opentelemetry.io/proto/otlp v0.7.0/go.mod h1:PqfVotwruBrMGOCsRd/89rSnXhoiJIqeYNgFYFoEGnI=
go.opentelemetry.io/proto/otlp v0.19.0 h1:IVN6GR+mhC4s5yfcTbmzHYODqvWAp3ZedA2SJPI1Nnw=
go.opentelemetry.io/proto/otlp v0.19.0/go.mod h1:H7XAot3MsfNsj7EXtrA2q5xSNQ10UqI405h3+duxN4U=
go.uber.org/atomic v1.11.0 h1:ZvwS0R+56ePWxUNi+Atn9dWONBPp/AUETXlHW0DxSjE=
go.uber.org/atomic v1.11.0/go.mod h1:LUxbIzbOniOlMKjJjyPfpl4v+PKK2cNJn91OQbhoJI0=
go.uber.org/goleak v1.2.1 h1:NBol2c7O1ZokfZ0LEU9K6Whx/KnwvepVetCUhtKja4A=
go.uber.org/goleak v1.2.1/go.mod h1:qlT2yGI9QafXHhZZLxlSuNsMw3FFLxBr+tBRlmO1xH4=
go.uber.org/mock v0.4.0 h1:VcM4ZOtdbR4f6VXfiOpwpVJDL6lCReaZ6mw31wqh7KU=
Expand Down
Loading
Loading