Hi I wanted to know if this project is dead?

[Raj2032]

Hi I wanted to know if this project is dead?

[pangzhenzhou]

Yeah, I wanna know too

[roignpar]

I don't have an official answer, but the README mentions:

A full rewrite of sled's storage subsystem is happening on a modular basis as part of the komora project, in particular the marble storage engine. This will dramatically lower both the disk space usage (space amplification) and garbage collection overhead (write amplification) of sled.

It seems like sled's author, spacejam is still actively contributing to various projects including komora repos which AFAICT will become part of sled when ready.

You might also find this relevant: #1417 (comment)

[Raj2032]

Interesting, thanks mate. ------- Original Message -------

…

On Tuesday, April 11th, 2023 at 5:46 AM, roignpar ***@***.***> wrote: I don't have an official answer, but the README mentions: > A full rewrite of sled's storage subsystem is happening on a modular basis as part of the [komora project](https://github.com/komora-io), in particular the marble storage engine. This will dramatically lower both the disk space usage (space amplification) and garbage collection overhead (write amplification) of sled. It seems like sled's author, [spacejam](https://github.com/spacejam) is still actively contributing to various projects including komora repos which AFAICT will become part of sled when ready. You might also find this relevant: [#1417 (comment)](#1417 (comment)) — Reply to this email directly, [view it on GitHub](#1444 (comment)), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/AG7P2BSOPPGEXQC7G3ZOOZ3XARPRRANCNFSM6AAAAAAWXA5ZLY). You are receiving this because you authored the thread.Message ID: ***@***.***>

[spacejam]

I've been working on a rewrite of smaller components in the komora github org, which will be released as part of sled's new storage engine, solving the current space issues. It's much easier for me to work on it in a quiet way off to the side like that, but it is coming together nicely.

[physics515]

@spacejam is there a writeup on what features the new sled will have? I am mostly wondering if multi-threaded support will be improved?

[vi]

Will the renewed Sled have the same license (and approach towards distribution) as current one?

Or will there be any scheme like a perpetual beta "Community edition" plus production-ready "Enterprise edition"?

[spacejam]

@vi generally I see sled remaining MIT/Apache-2.0. There are a variety of non-open systems that build on top of sled in interesting ways, some of which I've built myself, but I don't plan to monetize sled directly. sled's openness is good for the reliability of the core.

@physics515 could you elaborate on which multi-threaded aspects could help your use cases if they were changed? In general, I see sled's multi-threaded aspects as one of its strong points, given the high throughput lock-free indexing etc... and the next version is more about operational metric improvements around space and efficiency than new features. Although there are some aspects of transactions, merge operators, and subscription that I feel could be unified into a higher consistency unified concept, so it's possible that the pre-release will not have transactions/merge operators/subscription (at least at first, while the replacements are experimented with) but for the stable release the replacement / improvements for these 3 features should be in-place. Async is likely to see more attention, too.

[physics515]

@spacejam maybe I misspoke. I meant more along the lines of being able to have multiple instances. I have run into a few use cases where it would have been handy to access the same "data" from multiple programs simultaneously.

As I understand it, this is not currently possible, but I could be wrong.

[spacejam]

@physics515 that makes sense! multi-process is definitely one of the major draws for embedded db's like LMDB or SQLite, and I see what they enable as being very valuable in certain use cases, but they require a very heavy syscall : write operation ratio to ensure consistency, which severely limits write throughput. RocksDB also supports a form of multi-process access, but it is much more restrictive. Multi-process has been elusive to me, because I feel like it always involves some sharp performance and complexity trade-offs that haven't felt right for sled. When sled supported a read-only mode, it was a constant source of bugs, so I decided to remove it to avoid causing data loss for users by exposing them to bug densities that I didn't feel comfortable about.

The thought that has kept my attention over the years is to include a server binary with sled that can be connected to from the sled library instead of opening a file, and support a more restrictive set of features, but once sockets start to become involved, users start to have far more specific requirements that are difficult to meet well for 80%+ of users. Operations that involve multiple communications with a server start to become interesting from a distributed systems perspective, which entail even sharper performance and complexity trade-offs.

So, I tend to suggest that people who want multi-process support to achieve it by using sockets in a way that meets their own goals. And it's possible that I'll provide a network access primitive for assisting users with this over time. But I don't see it as something that I'm likely to release in the short term.

[physics515]

@spacejam that has been exactly my solution a few times with pet projects. I have implemented lightweight Axum wrappers around sled and have had a bit of success. Maybe when I get around to formalizing something I'll publish a separate crate.

[santoshakil]

@spacejam Is there any expected date for the new rewritten version of sled?

[CosminPerRam]

Is there any expected date for the new rewritten version of sled?

Bump.

[vi]

There is activity in a non-main branch.

[snaggen]

As a happy user of sled, it is nice to see that there is things going on behind the curtain (well, on another branch).
Just out of curiosity, it would be good to have a Milestone or just a roadmap, to get an idea of what the criterias for a release of that is. And to get an idea of what the major changes from the current releases are going to be.

[spacejam]

@snaggen the main thing that I need to improve in the new storage system before moving it forward toward release candidates basically relates to random write throughput. The new storage engine is in a great place when writes have a lot of locality, but some of the initial folks who have tested the alphas have highlighted that there is still some room for improvement when it comes to randomized write workloads. I'm definitely not aiming for perfection with the next stable release, but I'd like random write patterns to at least not get any worse compared to the current stable release, and be in a position to make nice incremental progress on from there. I agree that a roadmap would be great to help folks to follow along and have some more visibility, and I think the readme and other docs are ready for a face-lift in general.