@most/core 2.0 Ideation

[TylorS]

Within #676, we've identified the first task we will defer to 2.0 as it is a breaking change for the JS API of @most/core.

I thought it'd be a good idea to create a place for us and the community to brainstorm on what we envision what 2.0 could be like.

At a very high level there may be two general approaches

1.) Incremental changes: 2.0 will be pretty similar to 1.x, some minor breaking changes for at() and maybe an update to use the ES native Disposable interface

2.) More dramatic changes: 2.0 will be similar but also quite different from 1.x.

Under the "dramatic" changes some thoughts I have myself, in no order of importance

• Supporting AsyncDisposable native interface to allow aynchronous disposal
• Adding an E type parameter to track expected errors
• Adding an R type parameter to track the context in which a Stream is running within.

We'd absolutely love any ideas everyone else has for the future of most, no matter how big or small!

[briancavalier]

In my mind, an important overarching question is: what will be the goals of most/core 2.0. IOW, what problem(s) does it intend to solve for its users?

Since it's inception, most has been becoming more specialized in what problems it intends to solve, moving from data processing (very early versions) toward declarative reactive programming, primarily for declarative reactive UI programming. Do we want to continue in that direction?

In addition to (or perhaps, as a result of) becoming more specialized, most's core API has actually gotten smaller and, imho, simpler over time. That's definitely a trend I'd like to continue. We could drop problematic functions and/or move less-used functions to separate packages.

Supporting AsyncDisposable native interface to allow aynchronous disposal

Yeah, this is a great idea, especially since it's already stage 3.

Adding an E type parameter ...
Adding an R type parameter ...

I like both of these. I think Stream<A, E, R> could work pretty well, and R can replace the explicit scheduler parameter. If we're going revolutionary, changing the main types (Stream, Sink, etc.) is totally fair game.

[TylorS]

No I'm not really sure how it'd be done to be honest, just a bit of a pie-in-the-sky hope. It helps a lot that our Sinks are mostly named corresponding to the operators you used for general understanding, but the stack traces still point to code that you didn't write yourself and oftentimes anonymous functions you did write

[briancavalier]

Ah, yeah, great point about stack traces. Seeing a stack trace of a bunch of mostjs internals probably isn't helpful for app developers.

Have you thought about what you'd want to see in a stack trace? For example, say you have a map(f, stream) and f throws. Would you want to see a stack trace that points to the map(f, stream call? Something else?

[TylorS]

Yeah, I'd essentially want to replace any <operator>Sink with operator(...) in a stack trace, so map(f, stream) in your example above

[briancavalier]

The approach creed takes may work well. It maintains a context stack by pushing each "action" when its created. An action is similar to a Stream (not really, since creed is eager, but close enough). In creed's case, the Action class is basically an abstract base class of all actions, so its constructor does the pushing part of the maintenance. There's some context swapping that happens in the coroutine runner, and I'm not sure yet how that will factor in mostjs's case, but we may need to swap context whenever a source (or async combinator) runs.

@TylorS Do you know if Effect-TS does anything interesting with stack traces? Any lessons or techniques we might learn from it?

[TylorS]

Effect does at least 2 things regarding stack traces, I dunno how interesting they are for us though

• It has a TaggedError type which extends Error, but also does a bit of work to trim the stack track of useless information from the Fiber runtime
• It has built-in support for tracing, so you can attach spans and such to Effects and those get added to logs and exceptions

[briancavalier]

Here's TS playground with some of the things we were discussing in slack, including the additional type parameters, and dropping the Time argument.

[briancavalier]

In my mind, there are some characteristics that are fundamental to mostjs's identity. Together, these make it what it is, and differentiate it from other packages with similar goals. Hopefully, they're also things its users and contributors associate with it:

1. Do more with less, small(ish) declarative API
2. Referentially transparent (including time and errors), with a few exceptions
   • some non-ref-transparent features have been moved out of @most/core intentionally, like multicast
3. Simple architecture
   • Stream and Sink represent practically the whole architecture
   • This is probably more important to contributors than to users
4. Speed
   • As @TylorS has pointed out, we can probably do better if we kill the Time parameter
   • We can probably do better on memory stability
     • mem usage is typically low, but can have a pretty wide variance

[briancavalier]

Having just replied to #682, I think we should probably reevaluate optimizations like map/filter fusion, slice fusion, merge squashing, etc.. Maybe we start without them, and introduce some over time as we can support them with some data.

[TylorS]

That seems fair to me, it would be good to get some data on how valuable it all is.

If it is valuable, we might get some benefit from avoiding our stream instanceof Impl checks for stream.constructor === Impl or having some kind of .type === name convention as equality checks are much faster than instanceof.

Unrelated thought but some kind of Stream.type: string API might be nice for runtime analysis. I'd really love to build a UI for visualizing your stream graph and seeing the events flowing through them, and the tricky bit is relating them back to something familiar like "map", "filter", etc.

[briancavalier]

I like the .type idea, or at least some sort of string tag 👍 We could potentially rely on stream.constructor.name if we stick with classes for streams (I'm not 100% convinced we need to)

[briancavalier]

What are the pros and cons of dropping Sink's error channel?

That may seem like a crazy question, but this seems like the right thread for crazy questions 😂

[TylorS]

I think that's probably it for any merging of error/end worth mentioning at least, but maybe that's even an assumption we should reconsider? Should an error implicitly be the end of the stream?

Would it might be semantically strange for an operator like recoverWith to intercept an end() call instead of an error() call? I think maybe but I'm not that confident in my intuition there

FWIW, I've been trying out the behavior of multiple errors in @typed/fx, and it seems to match well with my own intuition around reactive UIs. It acts more as just a separate event channel, where "success" events are used to update the UI, and "error" events can be either notifications or hard failures depending on the operators utilized. e.g. Fx<Resources, Redirect, RenderEvent> is what I might use for a component which may ask for a redirect to occur.

It does lead to wanting all of the same combinators we use today for events, to also work on the "error channel", and for some that operate on both at the same time. For example, I have a switchMatch(stream, onError, onSuccess) operator which has the behaviors of switchMap, but ensures just 1 "inner" stream is running between both channels rather than only the success channel

[briancavalier]

Yeah, it definitely doesn't hurt to ask ourselves what error should mean.

As I remember it (and my memory isn't always great!), the original intent was to signal that the stream cannot produce more events because something unexpected happened. It wasn't intended to be a parallel channel for a second type of events. And the recoverWith combinator was intended to provide a way to substitute another stream in place of the one that failed.

I remember writing the Application errors vs Stream failures section of the docs about it, and that the error channel was intended to represent a Stream failure.

That doesn't mean it has to be that way, but I think this distinction is still valid: it is helpful to distinguish Stream failures (cannot produce more events) from application errors. Using your own Either in a Stream<Either<E, A>> still makes sense to me for application errors.

Your switchMatch case is interesting. It seems like a similar thing can be achieved with an Either type and map + switchLatest, where the map pattern matches on the Either.

[briancavalier]

Would it might be semantically strange for an operator like recoverWith to intercept an end() call instead of an error() call? I think maybe but I'm not that confident in my intuition there

Good question. I think the idea of being able to recover from a stream failure is still valid, and that we'd just be changing how we represent a stream failure. So, to me, it's still valid.

I just noticed, though, that it might mean that continueWith and recoverWith could also be merged. For example, if end's signature is end(e?: E): void, then continueWith could have the analogous signature: continueWith(f: (e?: E) => Stream<B>, s: Stream<A>): Stream<A | B>.

Still not sure whether that's a good way to go or not, though.

[semmel]

TLDR; I agree that sink.error(E) can be absorbed by sink.end(E | EndOk) (point 4 above). Also that recoverWith can be absorbed by continueWith as you described @briancavalier

I really like how the error "channel" is modelled in most streams:

… that the error channel was intended to represent a Stream failure.

Essentially it corresponds to catch(error) after the try{} block in conventional JS code. Or how Promise.all([promises]) shortcuts single promise rejections into a rejected resulting promise.
So in most, throwError(error) is named exactly right.

However, for newcomers, it is tempting to "abuse" the error "channel" for program logic. As new Vanilla JS coder your experience is that Error (i.e. exceptions) are used everywhere for program logic.
(E.g. when you AbortController.abort() on a timeout or if no longer needed, it rejects the pending fetch() with a nice AbortError. In the same way as it would reject fetch() if fatally the url is malformed.)

After years of working with reactive streams (mostly BaconJS) I am convinced that parallel channels lead to complicated code. With BaconJS now I always stream.endOnError(). I cherish that most has it implicitly guaranteed.

… it is helpful to distinguish Stream failures (cannot produce more events) from application errors. Using your own Either in a Stream<Either<E, A>> still makes sense to me for application errors.

Yes. But that's the hard part for newcomers:

• You need to pick and learn decent library which offers an Either.
• You need to write map(map(fn), s), map(cata(onLeft, onRight), s) when otherwise it's just map(fn, s) map(x => x instanceof Error ? onLeft(x): onRight(x))

I think the most documentation should include more examples using Either events. And it should recommend some Either implementations.

[briancavalier]

@semmel Thanks very much for sharing your perspective.

tl;dr I think it's important to keep the design that streams end when either there are no more events to be produced, or an error prevents the stream from producing more.

I think there may be a slippery slope: if mostjs provides 2 channels of events (i.e. an error channel that doesn't end the stream), why not 3? What if your application needs 4? I know, I know, 1 is probably the vast majority of cases with 2 is probably the second most common, but mostjs can't predict what every application will need. To me this is all a strong signal that mostjs should stick with a single parametric event channel, and it's up to application developers to determine how to represent their needs in that channel.

I agree that sink.error(E) can be absorbed by sink.end(E | EndOk) (point 4 above). Also that recoverWith can be absorbed by continueWith

Coincidentally, I started this repo to dome some brainstorming in code. It includes merging error+end and continueWith+recoverWith. I'm glad I tried it. After writing the code, I'm less confident that merging them provides any code-level benefit.

I do still like that it makes it painfully obvious that a stream will end either successfully or with a failure, but I'm finding it hard to justify the change.

As new Vanilla JS coder your experience is that Error (i.e. exceptions) are used everywhere for program logic.

Yes. I agree this is a problem in general. My experience has also been that every kind of error ends up being thrown or Promise.reject()ed, at which point it becomes more tedious and difficult to see and understand them, and to separate and
handle them differently later. Many codebases just lump all errors together and handle them in the same way (often by crashing). It's even a problem in TS where try/catch/throw and Promises destroy error type information.

I cherish that most has it implicitly guaranteed.

Thanks. I'm glad to hear that it's been helpful!

Yes. But that's the hard part for newcomers:

It is, and I always try to remember that. That said, I don't mind helping people to learn functional techniques. I also don't mind if mostjs tends to appeal to an audience who also value and/or want to learn those.