[ENH] wal3 is the write-ahead logging lightweight library #3028
Conversation
Reviewer ChecklistPlease leverage this checklist to ensure your code review is thorough before approving Testing, Bugs, Errors, Logs, Documentation
System Compatibility
Quality
|
| - A reader writing a _new_ cursor, or a cursor that goes back in time must complete the operation in | ||
| less than the garbage collection interval and then check for a concurrent garbage collection | ||
| before it considers the operation complete. If the reader somehow hangs between loading a log | ||
| offset and writing the cursor for more than the garbage collection interval, the cursor will | ||
| reference garbage collected data. The reader will fail. |
There was a problem hiding this comment.
nit: reader abstraction (and cursor) has not been introduced so far
| use crate::{Error, LogPosition}; | ||
|
|
||
| ////////////////////////////////////////////// Cursor ////////////////////////////////////////////// | ||
|
|
There was a problem hiding this comment.
note for reviewer: the read path is WIP
| // allows. If we hit throttle errors at this throughput we have a case for support. | ||
| throughput: 2_000, | ||
| // How much headroom we have for retries. | ||
| headroom: 1_500, |
There was a problem hiding this comment.
I'm wondering if throughput + headroom = <s3_throughput>?
There was a problem hiding this comment.
That's how I picked these constants.
There was a problem hiding this comment.
In this case the comments seems suspicious: if a throughput of 2000 is 5/7 of max throughput, doesn't this imply the head room is 800?
| .map(|f| f.seq_no) | ||
| .max() | ||
| .unwrap_or(ShardSeqNo(0)); | ||
| max_seq_no < max_seq_no + 1 && max_seq_no + 1 == *seq_no && *start < *limit |
There was a problem hiding this comment.
May I ask what is max_seq_no < max_seq_no + 1 checking for here? Is this equivalent to max_seq_no < u64::Max?
There was a problem hiding this comment.
That check would do it, too, but precludes allowing u64::MAX as a value. I'm fine to throw that value away. Would you find the check more readable that way?
There was a problem hiding this comment.
I'm fine both way. I'm wondering if it is ever possible to violate this check?
Documented in
rust/wal3/README.md.