module system

[juliangruber]

rvagg:

I don't feel strongly either way about this. I'm not a strict minimalist but like Dominic I tend to use batch() programatically so a chaining API is less helpful (but I may use it).

What we really need is a proper plugin system for Node so a project like LevelUP can load optional plugins that may be installed in the NODE_MODULES path(s) and do things like expose the main LevelUP.prototype to plugins that may want to augment it. Then we could say something on our README like: "If you npm install levelup-batch, LevelUP will make the chaining batch() API available. And levelup-batch can monkey-patch LevelUP.prototype.batch to provide chaining.

I need something like this for Ender too so maybe I'll actually build a generic plugin helper system some day..

rvagg:

Re plugins, Grunt and DocPad do something similar already but certainly not in a modular way that you can pull out and re-use, we need something that smaller projects like LevelUP can easily include. We should collaborate on it so we get the API right!

[juliangruber]

• it should detect collisions
• it should load all plugins from a specified folder
• it should extend the prototype

Will it do the extending automatically or should we say db.use('batch', require('cbatch'))

[rvagg]

I'm thinking that package.json specifies the extension 'types' that the main project exports. So in the case of LevelUP we may have an extension 'type' of "prototype" and if the package.json includes that then LevelUP needs to expose it to the plugin system somehow which then passes it in a predefined manner to the plugin(s) that need it.

[ralphtheninja]

I think extending should be done automatically. How about:

db.require('cbatch')

or perhaps .use() is common practice?

db.use('cbatch')

[rvagg]

So, levelup-batch has a "levelup": { "pluginType": [ "prototype" ] } in package.json, then the plugin system scans NODE_MODULES and finds this module with a "levelup" descriptor in package.json and adds it to the list of plugins. Then LevelUP somewhere does something like plugins.expose('prototype', LevelUP.prototype) (this is simplistic, you'd want to expose other stuff that wouldn't fit nicely into this pattern). and then somewhere else can do 'plugins.load(callback) which does all the exposing/extending stuff by working through all the plugins it found.

Make sense? It does in my head at least.

[rvagg]

@ralphtheninja I want something that doesn't require LevelUP to be aware of the various plugins that it can load; that way you can write a plugin without LevelUP-core even knowing about it, you could even write a completely private one that never sees the light of day outside your company.

[ralphtheninja]

@rvagg Aye, makes sense. It feels like a plugin pattern :)

[ralphtheninja]

I think we could scratch the "pluginType" for now and just assume one way of doing it. Pick the simplest use case and make it work, then introduce other ways of extending.

[dominictarr]

automatic plugins will come back and bite us on the ass.
in my levelup stuff (see all my modules that are prefixed level-*) i initially used a db.use(plugin) pattern
but then abandoned that, and just went with plugin(db) the plugin is responsible for checking that it's not already attached, and to make sure that it only minimally alters the db object - I stuck with only adding one patch that has the same name as the plugin itself.

This method is rather ugly, but at least it's simple.

So, I built a bunch of stuff with this pattern, some plugins just patched the db object, but others actually inserted data into the database. This meant prefixing the keys of inserted data.

I did that enough, that now I'm thinking that a much better approach would be to abstract that all away,
go db2 = db.namespace('PREFIX') to get another db, and then do something like
map(db1, db2, function map(k, v, emit) { emit(k, 1) }) etc...

this could be much more flexible, and also cleaner - but there is a bit of thinking on how to get everything working together. Most of my stuff is based at some point on hooking into a put and turning it into a batch that inserts a job, or other keys atomically. it would be nice to cleanly isolate these sections, but i'd need also need to be able to insert into several sections atomically to get consistency which is the objective at the end of the day.

[Raynos]

There's no need for a plugin systems. We already have commonJS

var myPlugin = require("level-plugin")

myPlugin(db, "do shit", function (err, result) {

})

Functions are really, really simple and work universally.

Now if you want a plugin system because you want to hook into put / del / batch etc, then we should have a single module that uses hooks and everyone uses that.

[Raynos]

As for detecting collisions. We already have a namespaces enforced by npm. It'll be easier to use those.

[dominictarr]

@Raynos calm down and clearly explain what you think, and why you think it.
I know you, so I know how you really mean it, but carrying on like this does not make for a pleasant and constructive discussion.

[Raynos]

I'll be sensible.

Why do we need a plugin system? What benefit does it give us over just using require ? Can we enumerate the use-cases for plugins and the scenarios where they are useful, as I can't think of any.

[dominictarr]

we have a collection of modules, which are extensions to leveldb - they are firmly coupled, because they all interact with levelup some how.

We want them to be as loosly coupled as possible, so that it's possible to iterate on the levelup implementation without breaking them, if they depend on incidental implementation details of levelup then they will be fragile.

So, we already have a partial list here https://github.com/rvagg/node-levelup/wiki/Modules

And there are more that we want to build, but havn't gotten around to implementing yet. I think what this discussion really wants to be about is 'what is the interface that extensions should interact with?'.

certainly speaking for myself, getting the plugins that I wrote correct was quite difficult, and I think there is a lot of room for improvement here.

[Raynos]

@dominictarr the interface is the same as the interface for levelup.

If we make a breaking change in levelup then

a) we need to update all apps that use levelup to use the new API
b) we need to update all levelup modules to use the new API

We should be using peer-dependencies in our levelup modules to lock the version.

I agree that making a breaking improvement in levelup is a pain for module owners but that's how it works. This the downside of modular javascript, if you have many intrarelated modules then you need to update them all in one batch. I don't know any answer for this.

I have had issues with it with Raynos/signal-channel, Raynos/read-write-stream and @Gozala has issues with it for gozala/reducers

[Gozala]

Since I have being mentioned in this thread so here is my two cents. I'm not very familiar with levelup specefics so bare with me if my comment are little off:

Method chaining

From my personal experience I find implicit extensions more problematic than helpful, specially considering the weights they carry. In most of the cases libs tend do this just for the sake of method chaining but that problem
is a different domain and should be solved independently IMO. For example reducers and all the libraries that work with it don't come with any method chaining support instead they embrace alternative drop in dsl libraries like:

• https://github.com/Gozala/enchain
• https://github.com/Gozala/synthesize
• https://github.com/gozala/gum
• https://github.com/gordonbrander/choochoo

As a matter of fact decision on which methods should be included in the chaining API is a domain of the app
code using it. Making decisions at the library level is harmful as the non-core functions become second class and users tend to hesitate using them.

Updates

Making breaking changes in big ecosystem is problematic and requires quite an effort regardless. If work is distributed across diff libraries well you're facing same distributed architecture challenges... If you have a lot of small libraries that require updates on every breaking change, you'll end with some additional npm related boilerplate for updates but it's not being too bad for me & hopefully npm will get better at this too.

[dominictarr]

agree - so I think the way to go about this is to experiment rapidly, and then discuss the our experiments in detail - we are already well into the experimentation phase, hmm, I guess I should explain in detail how all my stuff interacts... I'll try and do that today...

[rvagg]

Yes please @dominictarr, would be interesting to see how you're extending and what kind of extension points might be needed.

Slight digression:

While I'm sympathetic to the 'everything as a separate module' argument championed mainly by folks in #stackvm, I'm sceptical that it's actually the best approach to making libs friendly to those outside the Node-elite. The main problems I see for the average user are:

• discovery (!!)
• trust (i.e. can I trust this author's code without having to read through it myself? can I trust that it actually works with these other components?)
• understanding how the pieces can fit together
• making sure all the pieces continue to fit together through release cycles that don't match up
• documentation (it still amazes me how many awesome libs exist in the Node ecosystem that have garbage documentation and therefore remain out of reach to people who just want to get stuff done)

It's usually pretty difficult for us "authors" to think more as "users" even tho we do plenty of using; we're too used to being able to churn out our own solutions if we don't like what's already on offer (or just because we feel like it!). I don't believe it's reasonable to expect that of most people who are using Node. I suspect that the number of people using Node every day to build stuff is many times larger than the number of people who have packages in npm.

Sooo; that's the perspective that I'm attempting to see things from.

[Raynos]

discovery

Discovery is a difficult problem. One way is the framework way, jQuery, ember, bla bla handle discovery by bundling everything in one thing and having all the docs & tutorials and shit in one place.

The downside of that is that it's not modular and you buy either the entire thing or not. That sounds very much like mongoDB and postgres. LevelDB is the most modular DB ever, I think it's worthwhile to embrace that and do a modular JS thing

[dominictarr]

maybe someone needs to write a book on the stackvm way?

[Gozala]

maybe someone needs to write a book on the stackvm way?

Most books about functional programing embrace and cover modularity part in great detail.
For starters I would really recommend SICP

[rvagg]

modularity can mean a number of things, it doesn't necessarily mean that your modules need to be in completely separate places (i.e. individually downloaded from npm).

[rvagg]

(note I'm not advocating bundling everything into LevelUP, that's the point of this discussion now; to figure out how we can foster an easy to approach ecosystem of modules & plugins around LevelDB in Node).

[Raynos]

trust

Trust is also a difficult problem. For me I solve this by do I know the author or does someone I know recommend them. I also solve this by travis badge / testling badge / good docs.

understanding how the pieces can fit together

This can be solved with good blog posts / tutorials / wiki articles / talks about leveldb etc.

making sure all the pieces continue to fit together through release cycles that don't match up

Upto individual authors to make sure their shit still runs. If an author maintains a module it will continue to fit. If he doesnt maintain it then it does or another author maintains it.

documentation

I know how you feel. Both me and @dominictarr don't write enough docs. There can be some kind of levelup community curated list of modules where each module needs to meet a good standard of documentation / tests to be on the list. This may help or may cause a walled garden instead.

[rvagg]

I think this would be an appropriate time to insert a link to a dramatic reading of this.

[Raynos]

For the record I am trying to be constructive and not a dick.

[rvagg]

@Raynos I think most of us understand where you're coming from, all good.

[Raynos]

For the other record, yesterday I was being a complete dick

[dominictarr]

This module trust and discoverability problem is out of scope, there arn't gonna be 20k levelup modules, it's closer to the scale of connect-middleware -- whether or not you approve of the fact they exist --

To work as a community, we need to agree on certain things, like in node, we have Callbacks, EventEmitters, and Streams.

If your module doesn't follow that pattern - well, I'm not gonna use it, for one.

But, with leveldb, the module have a lot more in common, because it's specifically an embedded database.

The more we can agree on, the more we can make interoperate, but paradoxically,
the less we need to agree on the easier it will be to interoperate.

[dominictarr]

https://github.com/rvagg/node-levelup/wiki/Plugin-Pattern-Discussion

[dominictarr]

this wiki page outlines what I want to use levelup plugins for... please comment if you have questions, or want me to clarify something

[Raynos]

Why was the following api chosen

var hooks = require('level-hooks')
hooks(db)
db.hooks.stuff()

instead of

var hooks = require('level-hooks')(db)
hooks.stuff()

[ralphtheninja]

Thanks for the writeup. Have the same question as Raynos.

On 30 January 2013 02:43, Raynos notifications@github.com wrote:

Why was the following api chosen

var hooks = require('level-hooks')hooks(db)db.hooks.stuff()

instead of

var hooks = require('level-hooks')(db)hooks.stuff()

—
Reply to this email directly or view it on GitHubhttps://github.com/rvagg/node-levelup/issues/68#issuecomment-12869576.

[dominictarr]

Because I want to ensure that there is only one instance of hooks.
I know it's ugly, but since it actually inserts data,
(with a prefix, but basically into a global namespace)
then you basically have monkey-patched the data, inside the database.

Doing it some overly clever way, like weak maps, or something will probably lead to data collisions or something.

However, if there was a better way of separating prefixed sections - like a built in namespace thing, and here the database object would STILL need to keep track of what prefixes things are using. - there may be a way to do it as you describe.

So, I agree with your objections @Raynos and I'd never normally do it like this, I only did it this way this time because I couldn't not monkey-patch the data, so this was was simpler.

If there was a really clean way to partition the data, then I could remove that stuff.

[Raynos]

@dominictarr i.e. this would be a problem

var hooks1 = hooks(db)
var hooks2 = hooks(db)

hooks1.pre(mutate1)
hooks2.pre(mutate2)

Because the intercepted put would be persisted to the db twice?

One thing you can do is memoize hooks to return the same thing given a db but that would still cause dedup problems if there are TWO seperate hooks modules.

@dominictarr what about a compromise?

function hooks(db) {
  if (db.__hooks__exists) {
    return db.__hooks__exists
  }

  ... 

  db.__hooks__exists = hooksThing
  return hooksThing
}

That way you keep the invariant but you can have the nicer API of just returning the thing.

This is only worthwhile if most other plugins don't need to put their methods as a namespace on the db and can just return themself.

[dominictarr]

@Raynos that is exactly what I am doing, except without the __ prefix. I didn't try to hide it, because I wanted it to be apparent what was going on, and so that people would notice. Looks like it worked! :)

hooks is an exception, because it doesn't actually insert any data into the database, it just patches the db object.

level-trigger is a better example, the put wouldn't be persisted twice, because hooks doesn't handle that, db still does. If two triggers where created with the same prefix, but different rules... things will break, because the order of parallel inserts is not reliable (i've tested this)

A plugin needs to get the same prefix after the process restarts, but a different prefix than any other plugin.

so in summary, I think that a leveldb plugin system needs

• reliably separated areas of data.
• pre/post hooks on put, del, batch

[Raynos]

👍 to reliably seperated areas of data.

Hooks are needed but may work as a userland module

[dominictarr]

well, hooks monkey patch put, del, and batch. And depends on fairly tight implementation details, that is why it might be good to have it tied into levelup. Especially now that we have leveldown, and if you want a super light weight binding, you can just use that!

[ralphtheninja]

isaacs/npm#1400

[heapwolf]

I'm using hooks right now, they are awesome and im totally happy with them being a module.

[dominictarr]

The situation with levelup has changed quite a lot since we started this thread, since we now have levelDown.
This is a new thing, prehaps, because we have an inner-core, and and outer-core.

I think this relaxes some of our initial concerns - because we have levelDown for a minimum binding, and levelup for convenience.

It may be a good idea to bundle everything needed to create a stable platform for extending the database - but that will have to wait until we've really figured out what the generalized foundations are.