Parameter-by-parameter continuation using featurizing

[KalelR]

So far the only continuation method using featurizing is the global pool version, based on MCBB. But there's no reason why we can't do a parameter-by-parameter continuation like with the recurrences method. This pull request implements this method, based heavily on the code for the recurrences. There are very few changes needed to do this. The idea is that:

1. You find the attractors and fractions via basins_fractions using AttractorsViaFeaturizing. Then you extract the attractors from this by re-integrating attractors from one initial condition belonging to each cluster you found.
2. Take ics from previous attractors and use them as additional seeds for the subsequent parameter value. This is the biggest difference to the recurrences version. For featurizing, these additional_ics need to be put together with the normally sampled ics and clustered together. What I did was to add an additional_ics keyword to basins_fractions, which receives these ics, extracts their features, and adds these additional features to the normal features Dataset. This was the least disruptive way I found. Note that, to ensure that the ic from each attractor will be characterized as an attractor, it is repeated a min_neighbors number of times.

What I did was:

1. Extend the extract_attractors function for when ics are a sampler function. For this, the sampler function is deepcopied, then one copy is sent to extract features, and another sent to regenerate the attractors.
2. Modify the seeds_from_attractor function for the featurizing
3. Create FeaturizingFindAndMatch and a new continuation function in the continuation_featurizing_findandmatch file.

Note that the changes are small enough that we can unify both methods in the same function, reducing the duplication. I'm opening the pull request to let you know about this and hear your insights. The code design can still be improved for sure!

I have tested this for lorenz and it seems to have worked, but I'd still like to test more become merging.

[codecov-commenter]

Codecov Report

Merging #69 (6519ff0) into main (6a4fd8f) will decrease coverage by 1.95%.
The diff coverage is 31.03%.

@@            Coverage Diff             @@
##             main      #69      +/-   ##
==========================================
- Coverage   68.47%   66.52%   -1.95%     
==========================================
  Files          21       22       +1     
  Lines        1126     1180      +54     
==========================================
+ Hits          771      785      +14     
- Misses        355      395      +40     

Impacted Files	Coverage Δ
...tinuation/continuation_featurizing_findandmatch.jl	0.00% <0.00%> (ø)
src/mapping/attractor_mapping_featurizing.jl	67.53% <90.00%> (+5.23%)	⬆️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[KalelR]

Part of the code in this PR was offering support for extracting attractors in the Featurizing case when ics is a sampler function, not a pre-generated Dataset. The nice part would be saving up memory, since we wouldn't need to keep lots of unnecessary initial conditions in memory.

But I just realized that this can't be done when the feature extraction is parallelized: the initial conditions can't be re-generated from the sampler function if they are originally generated in a threaded loop. The uncertain order of the thread accesses makes the access to the sampler unpredictable, and so impossible to re-generate.

Simple parallelizing is in general much more useful than saving up memory, I am deleting these attempts. Therefore, the continuation needs to be done by giving a Dataset with the pre-generated initial conditions, so that the attractors can be regenerated. This is working fine on the tests I'm doing.

[awage]

We have the extract_attractors backend. The method dispatch on the mapper type.

[KalelR]

I'll change it!

[awage]

In this method you compare de distance between trajectories that spans the attractors. Can we compute the distance between each clusters of points in the feature space? Is it more reliable than matching trajectories?

[KalelR]

So matching by comparing distances between the features, instead of attractors? It's a nice idea. As I understand, this could be done by the user via the distance function, right? Receive the attractor, extract the feature and compute the distance.

One advantage of matching directly by the features is that one wouldn't need to reconstruct the attractors, and so could still use ics as a sampler function. But I think the generality of having the attractors (generated by having ìcsas a dataset) outweights the memory savings of havingics` as a function. What do you think?

[Datseris]

So matching by comparing distances between the features, instead of attractors? It's a nice idea. As

@KalelR that's what I was trying to tell you in-person in Portland... That you convert the clusters into StateSpaceSet and then call the match_attractor_ids! function on the clusters themselves. The function doesn't know what an attractor is; it could be an actual attractor or any StateSpaceSet.

[Datseris]

With this matching process, you completely alleviate the need to re-construct the attractors and to store/find an initial condition to go to each attravtcort. Yet you still keep the slice-by-slice continuation in featurizing that you want.

[awage]

You can match a subset of the cluster to avoid a memory bottleneck.

[Datseris]

point is that, if you reconstruct the attractors, you can still do that (just extract the features again) but also much more (simple example being Euclidean distance).

Yes, I understand this. What I am not sure is whether there is benefit in mixing the two approaches in this way. The benefit of the recurrence approach is the rigorousness that you have a guarantee that you find an attractor, while for featurizing cluster != attractor.

do you guys think that this is not worth it? That if the attractors have been identified by clusterizing the features, then they also should be matched by comparing the features?

I am suggesting that, however I would like first to see results of matching directly on the feature space before converting to attractors. I think this is the best option. But I could be wrong, and what you prpose, to go for feature space to attractor space, then match, and then go back again, could be a better approach. It is much more complicated though both logically and in terms of code. So I would still argue with should first try parameter-by-parameter matching directly on the feature space, simply by considering each cluster / group-of-features as a StateSpaceSet and then literally calling the match_attractor_ids! function.

[Datseris]

p.s.: Sorry for the late reply.

[awage]

Is there a metric to measure how good is the matching between two clusters of features? For example distance correlation would be an example.

[Datseris]

good idea!

[Datseris]

Actually, I'm not sure because this measure is invariant w.r.t. absolute position of the clusters. While for us this is crucial. If one cluster is identical to the other but shifted by 10 in each dimension, these should be totally different clusters / attractors.

[Datseris]

bump @KalelR

[Datseris]

bump @KalelR

[KalelR]

bump @KalelR

Hey, yes! I'll work on this and the other related projects this weekend/next week. Will update this soon :)

[KalelR]

Hey guys, sorry for taking so long to come back to this. I still have some questions I'd like to understand your point of view about before proceeding. I'm asking these questions because I'm thinking of my use-cases, but maybe there are others I haven't considered. To recap, the continuation should:

1. Find the attractors and fractions via basins_fractions using AttractorsViaFeaturizing.
2. Match attractors or clouds of features
3. Repeat 1-2, using ics from previous attractors as additional seeds for the subsequent parameter value.

The matching can be done in 3 ways:

1. Comparing the clouds of features
2. Comparing the attractors directly. This requires that initial condition with a distinct label to be integrated and considered as an attractor. This is the default procedure we've always done, and is indeed done by default on basins_fractions when ics is a pre-generated dataset. In this sense, the continuation function already has access to the attractors, no modification needed.
3. Compare features from the attractors (a specific case of 2). One feature per attractor, not a cloud.

My main question is: what is the point of doing the matching 1 (via clouds of features)? The arguments as I see are

1. No need to re-construct the attractors. True, but basins_fractions already does this for pre-generated ics. For sampler() ics this would be an advantage. But I don't see the real gain just from this.
2. Simplicity. The clustering is done on features, so the matching also should. This is the strongest point in my view. But why should it done for the cloud of features then, instead of a single features (like in matching 3)?
3. Featurizing does not guarantee that each cloud of feature corresponds to an attractor. This is true, but then what is the point of doing continuation if one does not trust the single-parameter algorithm to start with? If we are doing attractor continuation, we should first make sure that Featurizing is finding real attractors anyway. Would there be another use case in which the user does not trust the Featurizing algorithm? Also, in deterministic systems, decent features should convergence when on the same attractor. So clouds of features, in the sense of features centered around an average but with some deviation, are undesirable, a result of short integration times or bad features. Why would we choose to use the clouds then, instead of the centroid or some single feature from the cloud?

Further, say we choose to the cloud of features. As Alex said, using all of them might use too be much memory, or be too slow. Then we would need to select a few. How do we this in an objective way?

[Datseris]

Hey,

sorry for taking long to reply! So, here is how I see things:

1. Comparing the clouds of features

3. Compare features from the attractors (a specific case of 2). One feature per attractor, not a cloud.

Heh, but here is where I see things differently. This is one way to get "one feature per attractor". But another one is to get the centroid of the features. Which is a natural usage of the sets_distace function: it has Centroid out of the box. No need to find attractor, re-eveolve initial condition on the attractor, and then estimate the "features of the attractor". The centroid should be doing good enough of a job by itself and should be close to the "features of the attractor".

That is why, all the way from the start, I was arguing in favor of matching directly on the cloud of points. Maybe i wasn't making myself clear, but thankfully you reached the same conclusion.

So here is how I see it going:

1. continuation happens slice by slice. Hence, we need a new continuation method. Something like FeaturizeGroupParameterByParameter (perhaps a better name will come up).
2. The source code of this new continuation is very similar, almost identical, to RecurrencesFindAndMatch. The only difference is that instead of calling the setsofsets_disance function with attractors, we call it with clouds of features. Both attractors and clouds of features are StateSpaceSets.

That's it. You have to write very little source code with this approach.

(Hence, I would argue, if you go with this approach, please start a new pull request).

---

What remains to be discussed is if we believe there is any gain in the functionality of this PR, that "extracts" attractors from the grouping_featurizing method. I am not sure about that. At the moment, what does the function extract_attractors do when an AttractorsViaFeaturizing is given as an input?

[Datseris]

Then we would need to select a few. How do we this in an objective way?

Using the sample function. We already do this in the clustering method by subsampling features. Here you just need to make sure to sample per cluster. So each cluster provides 10% of its features.

[Datseris]

bump @KalelR

[Datseris]

bump @KalelR

[Datseris]

(also probably better to start a clean PR)

[Datseris]

See AttractorsSeedContinueMatch global continuation.