-
-
Notifications
You must be signed in to change notification settings - Fork 15
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
LevelDB & Node.js 'real world' use cases #49
Comments
It empowers you to write your own db, from a super light but fast filesystem abstraction to a beefed up server with custom replication schemes and other application-logic right at the heart of this super fast thing. And all in one language, on client, server and database. |
You could implement something like lambda architecture completely on top of leveldb. it's a basic building block, you can build your own db abstraction with your own trade offs on top of it. |
@navaru LevelDB makes data persistence easy. Instead of storing your data structures in memory, store them in a LevelDB database and fetch when neede--this way you can store a ton of data and not have to worry about RAM and you get to keep that data across restarts. This is how I mainly use it, LevelUP makes it super simple and fetching & processing large amounts via a |
I propose leaving this option open as a 'discussion issue' |
I've read the article on 'lambda architecture', and I think I've worked with those concepts when using other DBs, batch layer on CouchDB views, Serving layer on Riak, etc, am I correct? Because there are no examples, if I need to implement a namespace structure (collections in MongoDB or buckets in Riak), how should I proceed? I need to create an architecture first as the one advised in 'lambda architecture' or I can create a simpler layer to work with? Do you have any architecture layer available as Open Source or any apps where you've used LevelDB (LevelUP)? Most of my apps are targeted at small companies, where I don't need to setup a cluster, a single server almost always does more than I need, when I deliver an app I don't want to include an external DB like Mongo or Riak. I want to build a layer on top of LevelDB that suits my needs, like a custom CMS or something in this area. Thank you for your replies :) |
There are two modules available for generating namespaces, The way that leveldb is set up internally, it's pretty much it's own lambda architecture. also, see the module listing! https://github.com/rvagg/node-levelup/wiki/Modules it's brand new! |
It occurred to me after writing it that the LevelDB code in Level Session might be a good example of the basics of using LevelDB in Node. It covers the basic operations and a simple https://github.com/rvagg/node-level-session/blob/master/lib/level-store.js |
Hi there, Not sure that this is the good place to post that but whatever... I've put together a small app based on express which relies on leveldb and which might be a good illustration of what you can achieve pretty quickly w/ level-up (and level-namespace) https://github.com/jeremybenaim/express-leveldb Any feedback is more than welcome :) Thanks folks |
looks good, I'd only stream the results of findall to the client instead of buffering them |
Thanks , will do that. Btw, do not hesitate to open issues directly on the repo if you got any other suggestions! :) |
@juliangruber you can't stream results to the client with express, express is not build for streams. |
Right, but through sockets? |
Sure through sockets but then you either need to parse stream JSON over XHR or websockets or build a streaming templating engine |
@Raynos Express only augments the response object, res.write still is there. Express is really unobstrusive. |
@juliangruber the fact that it augments node is really obtrusive. The fact that it enforces a global lockstepped waterfall pipeline is about as anti-stream as you can get. |
@Raynos it does neither. |
This is not an issue about express, guys. |
Thank you for your help and examples. I've tried most of the modules that I could find on NPM. Taking an example like a 'users' collection, based on the examples I've got something like: key: How would I search for all the users under 22? I could do a map-reduce but is there a better way when the number of records is very large? I've tried to look at how Riak uses leveldb and how search goes there but I found this comment on a post about Riak: "To be clear, search uses its own storage engine, merge_index, which is separate from the KV backend... The search indexes themselves are stored in a merge_index backend, and in the future we may also support a leveldb backend for those search indexes." Search indexes == views? TL;DR |
you can use level-hooks to store young users under a special field. Or store your own indexes, e.g. with b+ trees. Search is kinda hard and we don't have that implemented yet. This could be sth where you have to resort to classical computer sience. |
@navaru There are two ways you can do searching, incremental map reduce or re-indexing. You can create a |
@navaru or you can use levelhooks to index your documents in elasticsearch, which might be not the better way (since you need an elasticsearch server) but sounds like a solid solution. Anyway, a full leveldb search layer would surely be better. |
using map-reduce gives you basically the same thing as couch db has, except it's incremental and realtime. If you are gonna use a hook type thing to index in elastic search (or any other db sort of thing) use level-trigger instead, because it will be reliably eventually consistent, even if your process crashes. if you want to make a separate index you can do it like this with hooks: db.hooks.pre(function (batch) {
var l = batch.length
while(l--) {
var v = JSON.parse(batch[l].value)
if('number' === typeof v.age) {
batch.push({type: 'put', key: ['AGES', pad(v.age, 10), v.name].join('!'), value: batch[l].value}
}
}
return batch
}) Iterate over the batch backwards (so you don't hit the things you are adding) join the key with ! so you can sort by range. ! is the first printable ascii character. for this to work you must not have ! in the key. you could also use null ('\x00') but I am using This will save it twice, but each record will be the same. Then you can query an age range by
However, age changes every year, so this will end up with records saved twice.
|
I just released a library for bytewise structural sorting [1] to avoid having to use these kinds of hacks. It will sort numbers and arrays and such correctly, and exactly like couchdb -- except it supports an even wider range of types. |
levelup has support for custom comparator functions, but they need to be implemented in C. |
Great -- I looked for that in the docs but hadn't come across it. Still, this approach allows you to build pretty much any kind of sorting you could possibly need using a fast, simple binary serialization. It'll let you store anything you can serialize with json (and a lot more), except better, because it'll sort properly, including for numbers and component-wise for arrays. |
hmm, the only problem is that a range query on those values will not comeout in the same ordering... |
Ooops, I just read your docs and realized that was indeed what you had done! |
@deanlandolt I am curious what the string versions look like, can you add a section showing that to the readme? |
I'm not sure I understand exactly what you mean by "string versions" but I put a bunch of examples in the README with their corresponding hex values. |
Oh, right - that was what I meant. How come you used hex instead of making it (semi) human-readable? |
What encoding do you prefer? The type tags are just arbitrary (sometimes non-printable) bytes, and those bytes are actually shifted by one when nested in arrays (so 'abc' becomes 'bcd' + a null terminating byte). I figured anything other than hex would just look like noise, but maybe a few examples that are printable (even if a little confusing) would be in order. |
Okay, I added a few examples showing the raw binary encoding. Thanks for the feedback. I'm really curious what the levelup community thinks about this lib -- my hope is that it makes some really difficult use cases a lot easier. |
this would work nicely with #51, we really need a simple way of hooking in to encode & decode operations for all reads and writes |
It would be great to get something like https://github.com/karlseguin/the-little-redis-book for LevelDB. After reading that book (1.5 hours) I now am equipped with the knowledge of what redis does, how it does it, and what I can (and shouldn't) use it for. Right now, the leveldb stuff just talks about the API, it's good for people who already have the problem, and know what their looking for, but for people evaluating what's out there, and where it fits in, leveldb is in the dark. |
I realize this issue is closed, but here's a "real world use case": Groove Basin - music player server and client |
@andrewrk Awesome! Do you accept bitcoin donations? |
Sure, if you feel so inclined. https://www.gittip.com/superjoe30/ |
Merging into #88 |
I've read the docs on LevelDB and some topics on the LevelDB Google Group & StackOverflow, I understand for what it was built.
What I want to know is what are some of your use cases and on what scenario do you believe LevelDB & Node.js is a good fit.
I am not very experienced with DBs but I would like to learn and there's not too much info on LevelDB & Node.js
Thank you :)
The text was updated successfully, but these errors were encountered: