Effect scheduling improvements

The existing Schedule type has been massively upgraded to support even more complex scheduling for repeating, retrying, and folding of Aff and Eff types.

A huge thanks to @bmazzarol who did all of the heavy lifting to make this feature a reality!

It has been refactored from the ground up, a Schedule now is a (possibly infinite) stream of durations. Each duration indicates to the retry, repeat, and fold behaviours how long to wait between each action. The generation of those streams comes from:

• Schedule.Forever - infinite stream of zero length durations
• Schedule.Once - one item stream of zero length duration
• Schedule.Never - no durations (a schedule that never runs)
• Schedule.TimeSeries(1, 2, 3 ...) - pass in your own durations to build a bespoke schedule
• Schedule.spaced(space) - infinite stream of space length durations
• Schedule.linear(seed, factor) - schedule that recurs continuously using a linear back-off
• Schedule.exponential(seed, factor) - schedule that recurs continuously using a exponential back-off
• Schedule.fibonacci(seed, factor) - schedule that recurs continuously using a fibonacci based back-off
• Schedule.upto(max) - schedule that runs for a given duration
• Schedule.fixedInterval(interval) - if that action run between updates takes longer than the interval, then the action will run immediately
• Schedule.windowed(interval) - a schedule that divides the timeline into interval-long windows, and sleeps until the nearest window boundary every time it recurs.
• Schedule.secondOfMinute(second) - a schedule that recurs every specified second of each minute
• Schedule.minuteOfHour(minute) - a schedule that recurs every specified minute of each hour
• Schedule.hourOfDay(hour) - a schedule that recurs every specified hour of each day
• Schedule.dayOfWeek(day) - a schedule that recurs every specified day of each week

These schedules are mostly infinite series, and so to control their 'length' we compose with ScheduleTransformer values to create smaller series, or to manipulate the series in some way (jitter for example). The following functions generate ScheduleTransformer values.

• Schedule.recurs(n) - Clamps the schedule durations to only recur n times.
• Schedule.NoDelayOnFirst - Regardless of any other settings, it makes the first duration zero
• Schedule.RepeatForever - Repeats any composed schedules forever
• Schedule.maxDelay(max) - limits the returned delays to max delay (upper clamping of durations).
• Schedule.maxCumulativeDelay(Duration max) - keeps a tally of all the delays so-far, and ends the generation of the series once max delay has passed
• Schedule.jitter(minRandom, maxRandom, seed) - adds random jitter to the durations
• Schedule.jitter(factor, seed) - adds random jitter to the durations
• Schedule.decorrelate(factor, seed) - transforms the schedule by de-correlating each of the durations both up and down in a jittered way.
• Schedule.resetAfter(max) - resets the schedule after a provided cumulative max duration
• Schedule.repeats(n) - not to be confused with recurs, this repeats the schedule n times.
• Schedule.intersperse(schedule) - intersperse the provided schedule between each duration in the schedule.

Schedule and ScheduleTransformer can be composed using | (union) or & (intersection):

    var schedule = Schedule.linear(1 * sec) | Schedule.recurs(3) | Schedule.repeat(3);

    // [1s, 2s, 3s, 1s, 2s, 3s, 1s, 2s, 3s]

Union | will take the minimum of the two schedules to the length of the longest, intersect & will take the maximum of the two schedules to the length of the shortest.

One thing remaining to-do is to bring HasTime<RT> back into the Core and allow these schedules to use injectable time. Some of the functions already take a Func<DateTime> to access 'now', this will be expanded so time can be sped up or slowed down, with the schedules 'just working'. That'll be in the next few weeks I'm sure, and is related to this issue.

Check out the API documentation to see what's what. And again, thanks to @bmazzarol for the hard work 👍

BREAKING: New LanguageExt.Transformers package

The transformers extensions, which are a big set of T4 templates for generating extension methods for nested monadic types have now been broken out into their own package: LanguageExt.Transformers

If you use the following functions: BindT, MapT, FoldT, FoldBackT, ExistsT, ForAllT, IterT, FilterT, PlusT, SubtractT, ProductT, DivideT, SumT, CountT, AppendT, CompareT, EqualsT, or ApplyT - then you will get compile errors, and will need to add a reference to the LanguageExt.Transformers package.

I've done this for a couple of reasons:

• There's been an ongoing concern from a number of users of this library about the size of the LanguageExt.Core library. This change takes the Core package from 3,276 kb to 2,051 kb.
  • The Core library will always be quite chunky because of the sheer amount of features, but the transformer extension methods definitely aren't always needed, so breaking them out made sense
• I suspect issues around the .NET ReadyToRun usage will be alleviated somewhat by this change.
  • I can't prove this, but the C# tooling has had a hard time with those 10,000s of generated extension methods before - so rather than wait for Microsoft to fix their tooling, I'm trying to be proactive and see if this will help.

The main transformer extensions that remain in the Core library are:

• Traverse
• Sequence

These are so heavily used that I believe moving them out into the Transformers library would mean everyone would be obliged to use it, and therefore it wouldn't achieve anything. There may be an argument for bringing BindT and MapT back into the core at some point. I will see how this plays out (it wouldn't be a future breaking change if that were the case).

Any problems, please report via the Issues in the usual way.

Aff, Eff, AtomHashMap, TrackingHashMap + fixes [RTM]

Language-ext had been in beta for a few months now. Today we go back to full 'RTM' mode.

The recent changes are:

AtomHashMap and Ref change-tracking

A major new feature that allows for tracking of changes in an AtomHashMap or a Ref (using STM). The Change event publishes HashMapPatch<K, V> which will allow access to the state of the map before and after the change, as well as a HashMap<K, Change<V>> which describes the transformations that took place in any transactional event.

In the LanguageExt.Rx project there's various observable stream extensions that leverage the Change event:

• OnChange() - which simply streams the HashMapPatch<K, V>
• OnEntryChange() - which streams the Change<V> for any key within the map
• OnMapChange() - which streams the latest HashMap<K, V> snapshot

Ref which represents a single value in the STM system, has a simpler Change event that simply streams the latest value. It also has an Rx extension, called Change().

Documented in previous beta release notes

TrackingHashMap<K, V>

This is a new immutable data-structure which is mostly a clone of HashMap; but one that allows for changes to be tracked. This is completely standalone, and not related to the AtomHashMap in any way other than it's used by the AtomHashMap.Swap method. And so, this has use-cases of its own.

Changes are tracked as a HashMap<K, Change<V>>. That means there's at most one change-value stored per-key. So, there's no risk of an ever expanding log of changes, because there is no log! The changes that are tracked are from the 'last snapshot point'. Which is from the empty-map or from the last state where tracking HashMap.Snapshot() is called.

Documented in previous beta release notes

Fixes to the Aff and Eff system

The Aff and Eff system had some unfortunate edge-cases due to the use of memoisation by-default. The underlying system has been simplified to be more of a standard reader-monad without memoisation. You can still memoise if needed by calling: ma.Memo().

Future changes will make the Aff and Eff into more of a DSL, which will allow for certain elements of the system to be 'pure', and therefore safely memoisable, and other elements not. My prototype of this is in too early a stage to release though, so I've taken the safer option here.

Breaking change: Both Clone and ReRun have been removed, as they are now meaningless.

New package LanguageExt.SysX - for .NET5.0+ features

The LanguageExt.Sys package is a wrapper for .NET BCL IO functionality, for use with the runtime Eff and Aff monads. This is going to stay as netstandard2.0 for support of previous versions of the .NET Framework and .NET Core. This new package adds features that are for net5.0+.

The first feature to be supported is the Activity type for Open-Telemetry support.

ScheduleAff and ScheduleEff usage was inconsistent

Depending on the monad they were used with, you might see a 'repeat' that was 1 greater than it should have been. This is now fixed.

Lazy Seq equality fix

Under certain rare circumstances it was possible for the equality operator to error with a lazy Seq. Fix from @StefanBertels - thanks!

No more empty array allocations

It seams c# isn't smart enough to turn a new A[0] into a non-allocating operation. And so they have all been replaced with Array.Empty(). Fix from @timmi-on-rails - thanks!

Any problems, please report them in the Issues as usual. Paul 👍

Change events for AtomHashMap and the STM system (via Ref)

As requested by @CK-LinoPro in this Issue. AtomHashMap and Ref now have Change events.

Change<A>

Change<A> is a new union-type that represents change to a value, and is used by AtomHashMap and TrackingHashMap.

You can pattern-match on the Change value to find out what happened:

public string WhatHappened(Change<A> change) =>
    change switch
    {
        EntryRemoved<A> (var oldValue)      => $"Value removed: {oldValue}",
        EntryAdded<A> (var value)           => $"Value added: {value}",
        EntryMapped<A, A>(var from, var to) => $"Value mapped from: {from}, to: {to}",
        _                                   => "No change"
    };

EntryMapped<A, B> is derived from EntryMappedFrom<A> and EntryMappedTo<B>, so for any A -> B mapping change, you can just match on the destination value:

public string WhatHappened(Change<A> change) =>
    change switch
    {
        EntryRemoved<A> (var oldValue)      => $"Value removed: {oldValue}",
        EntryAdded<A> (var value)           => $"Value added: {value}",
        EntryMappedTo<A>(var to)            => $"Value mapped from: something, to: {to}",
        _                                   => "No change"
    };

That avoids jumping through type-level hoops to see any changes!

There are also various 'helper' properties and methods for working with the derived types:

Member	Description
HasNoChange	true if the derived-type is a NoChange<A>
HasChanged	true if the derived-type is one-of EntryRemoved<A> or EntryAdded<A> or EntryMapped<_, A>
HasAdded	true if the derived-type is a EntryAdded<A>
HasRemoved	true if the derived-type is a EntryRemoved<A>
HasMapped	true if the derived-type is a EntryMapped<A, A>
HasMappedFrom<FROM>()	true if the derived-type is a EntryMappedFrom<FROM>
ToOption()	Gives the latest value from the Change, as long as the Change is one-of EntryAdded or EntryMapped or EntryMappedTo

There are also constructor functions to build your own Change values.

AtomHashMap<K, V> and AtomHashMap<EqK, K, V>

The two variants of AtomHashMap both now have Change events that can be subscribed to. They emit a HashMapPatch<K, V> value, which contains three fields:

Field	Description
From	HashMap<K, V> that is the state before the change
To	HashMap<K, V> that is the state after the change
Changes	HashMap<K, Change<V>> that describes the changes to each key

There are three related Rx.NET extensions in the LanguageExt.Rx package:

AtomHashMap Extension	Description
OnChange()	Observable stream of HashMapPatch<K, V>
OnMapChange()	Observable stream of HashMap<K, V>, which represents the latest snapshot of the AtomHashMap
OnEntryChange()	Observable stream of (K, Change<V>), which represents the change to any key within the AtomHashMap

Example

var xs = AtomHashMap<string, int>();

xs.OnEntryChange().Subscribe(pair => Console.WriteLine(pair));
            
xs.Add("Hello", 456);
xs.SetItem("Hello", 123);
xs.Remove("Hello");
xs.Remove("Hello");

Running the code above yields:

(Hello, +456)
(Hello, 456 -> 123)
(Hello, -123)

Swap method (potential) breaking-change

The implementation of Swap has changed. It now expects a Func<TrackingHashMap<K, V>, TrackingHashMap<K, V>> delegate instead of a Func<HashMap<K, V>, HashMap<K, V>> delegate. This is so the Swap method can keep track of arbitrary changes during the invocation of the delegate, and then emit them as events after successfully committing the result.

Ref<A>

Refs are used with the atomic(() => ...), snapshot(() => ...), and serial(() => ...) STM transactions. Their changes are now tracked during a transaction, and are then (if the transaction is successful) emitted on the Ref<A>.Change event. These simply publish the latest value.

As before there's Rx extensions for this:

Ref Extension	Description
OnChange()	provides an observable stream of values

Example

var rx = Ref("Hello");
var ry = Ref("World");

Observable.Merge(rx.OnChange(), 
                 ry.OnChange())
          .Subscribe(v => Console.WriteLine(v));

atomic(() =>
{
    swap(rx, x => $"1. {x}");
    swap(ry, y => $"2. {y}");
});

This outputs:

1. Hello
2. World

TrackingHashMap

This is a new immutable data-structure which is mostly a clone of HashMap; but one that allows for changes to be tracked. This is completely standalone, and not related to the AtomHashMap in any way other than it's used by the AtomHashMap.Swap method. And so, this has use-cases of its own.

Changes are tracked as a HashMap<K, Change<V>>. That means there's at most one change-value stored per-key. So, there's no risk of an ever expanding log of changes, because there is no log! The changes that are tracked are from the 'last snapshot point'. Which is from the empty-map or from the last state where trackingHashMap.Snapshot() is called.

Example 1

var thm = TrackingHashMap<int, string>();

Console.WriteLine(thm.Changes);

thm = thm.Add(100, "Hello");
thm = thm.SetItem(100, "World");

Console.WriteLine(thm.Changes);

This will output:

[]
[(100: +World)]

Note the +, this indicates an Change.EntryAdded. And so there has been a single 'add' of the key-value pair (100, World). The "Hello" value is ignored, because from the point of the snapshot: a value has been added. That's all we care about.

If I take a snapshot halfway through, you can see how this changes the output:

var thm = TrackingHashMap<int, string>();

thm = thm.Add(100, "World");
thm = thm.Snapshot();
thm = thm.SetItem(100, "Hello");

Console.WriteLine(thm.Changes);

This outputs:

[(100: World -> Hello)]

So the snapshot is from when there was a (100, World) pair in the map.

Hopefully that gives an indication of how this works!

v4.0.2 ROLLBACK - Reverting change to Seq1

@StefanBertels has rightly highlighted some possible edge-cases with yesterday's rollout. These edge cases could turn into very real bugs in your application, and so I am:

• Rolling back the change to Seq1: Seq1 will continue to be the way you create singleton Seq<A> collections.
• Keeping the migration of the Seq conversion functions to toSeq. This will be consistently be a compilation error, which you can fix up by just renaming the Seq(blah) to toSeq(blah) or blah.ToSeq().

It isn't ideal that Seq1 is still hanging around, being all inconsistent with the rest of the library, but neither is putting a burden of tracking all possible edge cases. And so, this is the right thing to do.

What I will do is pause the migration of Seq1 to Seq until June 2022. That should give everyone enough time to get to version 4 and migrate their Seq conversion function usage to toSeq. Then the subsequent change will have zero impact.

For those that don't upgrade in that time and want the safest possible migration, they can upgrade to a pre-June 2022 version, fix the conversion functions and then upgrade to the latest.

Apologies to anyone who started the migration yesterday.

Breaking change: Seq1 to Seq

This is the backstory: When language-ext was still in version 1, there was no Seq<A> collection type. There were however Seq(...) functions in the Prelude. Those functions coerced various types into being IEnumerable, and in the process protected against any of them being null.

These are they:

	Seq <A> (A? value)
	Seq <A> (IEnumerable<A> value)
	Seq <A> (A[] value)
	Seq <A> (Arr<A> value)
	Seq <A> (IList<A> value)
	Seq <A> (ICollection<A> value)
	Seq <L, R> (Either<L, R> value)
	Seq <L, R> (EitherUnsafe<L, R> value)
	Seq <A> (Try<A> value)
	Seq <A> (TryOption<A> value)
	Seq <T> (TryAsync<T> value)
	Seq <T> (TryOptionAsync<T> value)
	Seq <A> (Tuple<A> tup)
	Seq <A> (Tuple<A, A> tup)
	Seq <A> (Tuple<A, A, A> tup)
	Seq <A> (Tuple<A, A, A, A> tup)
	Seq <A> (Tuple<A, A, A, A, A> tup)
	Seq <A> (Tuple<A, A, A, A, A, A> tup)
	Seq <A> (Tuple<A, A, A, A, A, A, A> tup)
	Seq <A> (ValueTuple<A> tup)
	Seq <A> (ValueTuple<A, A> tup)
	Seq <A> (ValueTuple<A, A, A> tup)
	Seq <A> (ValueTuple<A, A, A, A> tup)
	Seq <A> (ValueTuple<A, A, A, A, A> tup)
	Seq <A> (ValueTuple<A, A, A, A, A, A> tup)
	Seq <A> (ValueTuple<A, A, A, A, A, A, A> tup)

When Seq<A> the type was added, I needed constructor functions, which are these:

    	Seq <A> ()
	Seq1 <A> (A value)
	Seq <A> (A a, A b)
	Seq <A> (A a, A b, A c, params A[] ds)

The singleton constructor needed to be called Seq1 because it was clashing with the original single-argument Seq functions from v1 of language-ext.

This has been bothering me for a long time. So, it's time to take the hit.

All of the legacy coercing functions (the first list) are now renamed to toSeq. And Seq1 has been renamed to Seq (well, Seq1 still exists, but it's marked [Obsolete].

The breaking change is that if you use any of those legacy Seq functions, you'll either need to change, this:

    Seq(blah);

Into,

    toSeq(blah);

Or,

    blah.ToSeq();

There were quite a lot of the coercion functions used in language-ext, and it took me about 10 minutes to fix it up. The good thing is that they all turn into compilation errors, no hidden behaviour changes.

Apologies for any inconvenience with this. I figure that as we've switched into v4 that this would be a good time for a clean break, and to tidy up something that I have actively seen cause confusion for devs. Seq is the most performance immutable list in the .NET sphere, so it shouldn't really be confusing to use!

Language-Ext Version 4.0.0 (finally out of beta!)

Here it is, version 4.0 of language-ext - we're finally out of beta. I had hoped to get some more documentation done before releasing, but I have ever decreasing free time these days, and I didn't want to delay the release any further. A lot has happened since going into beta about a year or so ago:

Effects

How to handle side-effects with pure functional code has been a the perennial question. It is always coming up in the issues and it's time for this library to be opinionated on that. Language-ext already has Try and TryAsync which are partial solutions to this problem, but they only really talk to the exception catching part of IO. Not, "How do we inject mocked behaviours?", "How do we deal with configuration?", "How do we describe the effects that are allowed?", "How do we make async and sync code play well together?" etc.

Language-Ext 4.0 has added four new monadic types:

• Eff<A> - for synchronous side-effects. This is a natural replacement for Try<A>
• Aff<A> - for asynchronous side-effects. This is a natural replacement for TryAsync<A>
• Eff<RT, A> - for synchronous side-effects with an injectable runtime. This is like the Reader<Env, A> monad on steroids!
• Aff<RT, A> - for asynchronous side-effects with an injectable runtime. This is like the ReaderAsync<Env, A> monad that never existed!

These four monads have been inspired by the more bespoke monads I'd been building personally to deal with IO in my private projects, as well as the Aff and Eff monads from PureScript, and also a nod to ZIO from the Scala eco-system. Together they form the Effects System of language-ext, and will be the key area of investment going forward.

These four monads all play well together, they can be used in the same LINQ expression and everything just works.

There is now a whole section of the wiki dedicated to dealing with side-effects from first principles. I ran out of steam to complete it, but I hope to get back to it soon. It does go into depth on the Aff and Eff monads, and how to build runtimes.

On top of that, there's a project called EffectsExamples in the Samples folder. It is a whole application, with an effects driven menu system, for showing what's possible.

Finally, there's a new library called LanguageExt.Sys that wraps up the core side-effect producing methods in .NET and turns them into Aff or Eff monads. So far it covers:

• Time
• Environment
• Encoding
• Console
• IO.Directory
• IO.File
• IO.Stream
• IO.TextRead

This library will grow over time to cover more of .NET. You are not limited to using these: the runtime system is fully extensible, and allows you to add your own side-effecting behaviors.

Pipes

Pipes is a C# interpretation of the amazing Haskell Pipes library by Gabriella Gonzalez. It is a streaming library that has one type: Proxy. This one type can represent:

• Producers
• Pipes
• Consumers
• Client
• Server

Each Proxy variant can be composed with the others, and when you've got it correct they all fuse into an Effect that can be run. For example, a Producer composed with a Pipe, and then composed with a Consumer will fuse into a runnable effect. Each component is standalone, which means it's possible to build reusable effects 'components'

It also supports either uni or bi-directional streaming and can work seamlessly with the effects-system.

It's actually been in language-ext for a long time, but it was borderline unusable because of the amount of generic arguments that were needed to make it work. Since then I've augmented it with additional monads and 100s of SelectMany overrides that make it easier to use. I have also removed all recursion from it, which was a problem before.

The original Haskell version is a monad transformer, meaning that it can wrap other monads. Usually, you'd wrap up the IO monad, but it wasn't limited to that. However, the C# version can't be as general as that, so it only wraps the Aff and Eff monads. However, I have managed to augment the original to also support:

• IEnumerable
• IObservable
• IAsyncEnumerable

This means we can handle asynchronous and synchronous streams. All of which play nice with the effects system. Here's a small example:

    public class TextFileChunkStreamExample<RT> where RT : 
        struct, 
        HasCancel<RT>,
        HasConsole<RT>,
        HasFile<RT>,
        HasTextRead<RT>
    {
        public static Aff<RT, Unit> main =>
            from _ in Console<RT>.writeLine("Please type in a path to a text file and press enter")
            from p in Console<RT>.readLine
            from e in mainEffect(p)
            select unit;
        
        static Effect<RT, Unit> mainEffect(string path) =>
            File<RT>.openRead(path) 
               | Stream<RT>.read(80) 
               | decodeUtf8 
               | writeLine;

        static Pipe<RT, SeqLoan<byte>, string, Unit> decodeUtf8 =>
            from c in awaiting<SeqLoan<byte>>()         
            from _ in yield(Encoding.UTF8.GetString(c.ToReadOnlySpan()))
            select unit;
        
        static Consumer<RT, string, Unit> writeLine =>
            from l in awaiting<string>()
            from _ in Console<RT>.writeLine(l)
            select unit;
    }

• This example asks the user to enter a filename, then reads the file in 80 byte chunks, converts those chunks into strings, and then writes them to the console
• mainEffect is where the Effect is fused by composing the File.openRead producer, the Stream.read pipe, the decodeUtf8 pipe, and the writeLine consumer.
• You may notice the new type SeqLoan. This is a version of Seq that has a borrowed backing array from an ArrayPool. The Pipes system automatically cleans up resources for you. And so, this stream doesn't thrash the GC

NOTE: The client/server aspect of this still has the generics problem, and is difficult to use. I will be doing the same work I did for Producers, Pipes, and Consumers to make that more elegant over time.

Concurrency

Although this library is all about immutability and pure functions, it is impossible (at least in C#) to get away from needing to mutate something. Usually it's a shared cache of some sort. And that's what the concurrency part of language-ext is all about, dealing with that messy stuff. We already had Atom for lock-free atomic mutation of values and Ref for use with the STM system.

In this release there have been some major improvements to Atom and Ref (no more deadlocks). As well as the addition of three new lock-free atomic collections:

• AtomHashMap
• AtomSeq
• VersionHashMap

And two other data-structures:

• VersionVector
• VectorClock

All but VectorClock, VersionVector, and VersionHashMap were detailed in previous release notes, so I won't cover those again. VectorClock, VersionVector, and VersionHashMap allow you to easily work with distributed versioned changes. Vector-clocks comes from the genius work of Leslie Lamport amongst others, and allow for tracking of causal or conflicting events; when combined with some data in VersionVector it's possible to resolve many clients wanting to change the same piece of data, knowing who did what first. When VersionVector is combined with an atomic-hash-map, you get VersionHashMap. This manages multiple keys of VersionVectors, making it a short hop to a distributed database.

There's some good blog posts on the Riak site that explain vector clocks.

when and unless

These two functions make working with alternative values easier in LINQ expressions.

    from txt1 in File<RT>.readAllText(path)
    from txt2 in when(txt1 == "", SuccessEff("There wasn't any text in the file, so use this instead!"))
    select txt;

The dual of when is unless, it simply reverses the predicate.

guard and guardnot

These are similar to when and unless in that they take a predicate, but instead they return the monad's alternative value (Left for Either, Error for Fin, and Aff, for example). This makes it easy to catch problems and generate contextual error messages.

Pretty

Another Haskell library I've taken inspiration from is Pretty. It's synopsis is:

Pretty is a pretty-printing library, a set of API's that provides a way to easily print out text in a consistent format of your choosing. This is useful for compilers and related tools.
It is based on the pretty-printer outlined in the paper 'The Design of a Pretty-printing Library' by John Hughes in Advanced Functional Programming, 1995

It's a fully composable text document building system that can understand layouts and can do lovely reformatting functions depending on context (like the desired width of text, the tabs, etc.). If you've ever had to manage printing of text with indented tabs, this is the feature for you.

It's pretty new in the library, I needed it for some compiler work I was working on, and it seemed generally useful - so I've added it. However, there's no documentation [other tha...

Concurrency improvements (AtomHashMap, Atom, Ref)

This release features improvements to the concurrency elements of language-ext:

• Atom - atomic references
• STM / Ref - software-transactional memory system
• AtomHashMap - new atomic data-structure

Atom and STM

The atomic references system (Atom) that wraps a value and allows for atomic updates; and the STM system that allows many Ref values (that also wrap a value, but instead work within an atomic sync transaction), have both been updated to never give up trying to resolve a conflict.

Previously they would fail after 500 attempts, but now any conflicts will cause the conflicting threads to back-off and eventually yield control so that the other thread(s) they were in conflict with eventually win and can update the atomic reference. This removes the potential time-bomb buried deep within the atomic references system, and creates a robust transactional system.

AtomHashMap<K, V>

One pattern I noted I was doing quite a lot was wrapping HashMap in an atom, usually for shared cached values:

   var atomData = Atom(HashMap<string, int>());

   atomData.Swap(static hm => hm.AddOrUpdate("foo", 123));

It struck me that it would be very useful to have atomic versions of all of the collection data-structures of language-ext. The first one is AtomHashMap. And so, instead of the code above, we can now write:

    var atomData = AtomHashMap<string, int>();

    atomData.AddOrUpdate("foo", 123);

All operations on AtomHashMap are atomic and lock-free. The underling data structure is still an immutable HashMap. It's simply the reference to the HashMap that gets protected by AtomHashMap, preventing two threads updating the data structure with stale data.

The main thing to understand with AtomHashMap is that if a conflict occurs on update, then any transformational operation is re-run with the new state of the data-structure. Obviously conflicts are rare on high-performance CPUs, and so we save processing time from not taking locks on every operation, at the expense of occasional re-running of operations when conflicts arise.

Swap

AtomHashMap also supports Swap, which allows for more complex atomic operations on the underlying HashMap. For example, if your update operation relies on data within the AtomHashMap, then you might want to consider wrapping everything within a Swap call to allow for fully idempotent transformations:

    atomData.Swap(data => data.Find("foo").Case switch
                          {
                              int x => data.SetItem("foo", x + 1),
                              _     => data.Add("foo", 1)
                          });

NOTE: The longer you spend inside a Swap function, the higher the risk of conflicts, and so try to make sure you do the bare minimum within swap that will facilitate your idempotent operation.

In Place Operations

Most operations on AtomHashMap are in-place, i.e. they update the underlying HashMap atomically. However, some functions like Map, Filter, Select, Where are expected to process the data-structure into a new data-structure. This is usually wanted, but we also want in-place filtering and mapping:

    // Only keeps items with a value > 10 in the AtomHashMap
    atomData.FilterInPlace(x => x > 10);

    // Maps all items in the AtomHashMap 
    atomData.MapInPlace(x => x + 10);

The standard Map, Filter, etc. all still exist and work in the 'classic' way of generating a new data structure.

ToHashMap

At any point if you need to take a snapshot of what's in the AtomHashMap you can all:

    HashMap<string, int> snapshot = atomData.ToHashMap();

This is a zero allocation, zero time (well in the order of nanoseconds), operation. And so we can easily take snapshots and work on those whilst the atomic data structure can carry on being mutated without consequence.

The Rest

As well as AtomHashMap<K, V> there's also AtomHashMap<EqK, K, V> which maps to HashMap<EqK, K, V>.

This is just the beginning of the new Atom based data-structures. So watch this space!

Code-gen improvements - discriminated unions

This release features updates to the [Union] feature of language-ext (discriminated union generator) as well as support for iOS Xamarin runtime code-gen

• Generates a Match function for each union
• Generates a Map function for each union.
• Unions ignore ToString and GetHashCode - allowing for your own bespoke implementations
• iOS Xamarin doesn't allow usage of IL, and so for platforms that don't support IL the runtime code-gen (for Record<A>, Range, and other types) now falls back to building implementations with Expression.

Match

Unions can already be pattern-matched using the C# switch statement and switch expression. Now you can use the generated Match function like other built-in types (such as Option, Either, etc.). This match function enforces completeness checking, which the C# switch can't do.

There are two strategies to generating the Match function:

1. If the [Union] type is an interface, then the generated Match function will be an extension-method. This requires your union-type to be a top-level declaration (i.e. not nested within another class). This may be a breaking change for you.

It looks like this:

public static RETURN Match<A, RETURN>(this Maybe<A> ma, Func<Just<A>, RETURN> Just, Func<Nothing<A>, RETURN> Nothing) =>
    ma switch
    {
        Just<A> value => Just(value), 
        Nothing<A> value => Nothing(value), 
        _ => throw new LanguageExt.ValueIsNullException()
    };

2. If the [Union] type is an abstract partial class, then the generated Match function will be an instance-method. This doesn't have the limitation of the interface approach.

Then the Match instance methods look like this:

// For Just<A>
public override RETURN Match<RETURN>(Func<Just<A>, RETURN> Just, Func<Nothing<A>, RETURN> Nothing) => 
    Just(this);

// For Nothing<A>
public override RETURN Match<RETURN>(Func<Just<A>, RETURN> Just, Func<Nothing<A>, RETURN> Nothing) => 
    Nothing(this);

And so, if you still need unions nested within other classes, switch the interface to an abstract class:

This is an example of an interface based union:

    [Union]
    internal interface Maybe<A>
    {
        Maybe<A> Just(A value);
        Maybe<A> Nothing();
    }

This is the equivalent as an abstract class:

    [Union]
    public abstract partial class Maybe<A>
    {
        public abstract Maybe<A> Just(A value);
        public abstract Maybe<A> Nothing();
    }

Map

One thing that is true of all algebraic-data-types (of which category discriminated-unions fall into); is that there is exactly one way to generate a functor Map function (Select in LINQ) it's known as a theorem for free. And so the implementation for Map and Select are now provided by default by the code-gen.

So, for the example below:

    [Union]
    public abstract partial class Maybe<A>
    {
        public abstract Maybe<A> Just(A value);
        public abstract Maybe<A> Nothing();
    }

This code is auto-generated:

public static Maybe<B> Map<A, B>(this Maybe<A> ma, System.Func<A, B> f) =>
    ma switch
    {
        Just<A> _v => new Just<B>(f(_v.Value)), 
        Nothing<B> _v => new Nothing<B>(), 
        _ => throw new System.NotSupportedException()
    };

public static Maybe<B> Select<A, B>(this Maybe<A> ma, System.Func<A, B> f) =>
    ma switch
    {
        Just<A> _v => new Just<B>(f(_v.Value)), 
        Nothing<B> _v => new Nothing<B>(), 
        _ => throw new System.NotSupportedException()
    };

This will currently only support single generic argument unions, it will be expanded later to provide a Map function for each generic argument. The other limitation is that if any of the cases have their own generic arguments, then the Map function won't be generated. I expect this to cover a large number of use-cases though.

Any problems please report in the repo Issues.

Paul

Aff and Eff monad updates (breaking changes)

I've done some refactoring of the Aff and Eff monads as I slowly progress toward a v4.0 release of language-ext:

• RunIO has been renamed to Run
• HasCancel has been rationalised to only use non-Eff and non-Aff properties
  • It was a mistake to use them, I should have just used regular properties and then made access via static functions
  • This has now been done and should make it a bit more obvious how to use when building your own runtimes
• Support for MapAsync on the Eff monads
  • That means we don't need both a Aff and Eff property in the Has* traits, as an Eff can be MapAsyncd into an Aff efficiently
  • So traits now should only return something like Eff<RT, FileIO> FileEff (obviously different types based on your trait)
• The wrappers for the .NET System namespace have now been factored out into a new library LanguageExt.Sys
  • The naming wasn't great, and it was clear it was going to get pretty messy as I wrapped more of the System IO operations
  • Also, there's a chance you wouldn't want to use the implementations I have built, so having the main namespace cluttered with IO traits and types that you might not need was a bit ugly.
• The default 'live' runtime has also been factored out
• A new test runtime has been added - for unit testing
  • A new built-in mocked in-memory file-system - this allows unit tests to work with files without having to physically create them on a disk
  • A new built-in mocked in-memory console - this allows unit tests to mock key-presses in a console, as well as the option the look at the in-memory console buffer to assert correct values being written
  • A time provider - allows the system clock to appear to run from any date, or even be frozen in time
  • An in-memory mocked 'System.Environment'

Finally, I'm starting to document the Aff and Eff usage. This will be fleshed out more over the next few weeks.

All deployed to nu-get now.