What benchmark tasks shall we add to mofdscribe?

[kjappelbaum]

An important aspect we want to promote with mofdscribe is to make it easier to compare models developed for reticular chemistry (common task framework, see Donoho). Even though there are clearly issues with benchmarks (the community tends to overfit them) we still think that they would be very useful.

Currently, we have some benchmark tasks implemented in mofdscribe but we would like to revise and extend this. For this we would welcome any feedback.

In the current concept of benchmark tasks, each benchmark describes:

• a set of structures along with labels
• a splitting strategy

Cory Simon (@SimonEnsemble) created an overview of existing datasets: https://github.com/SimonEnsemble/porous-material-AI-gym

[kjappelbaum]

Another thing I have been thinking a lot about is how a "blind test" could look like. That is, a challenge where the test set is not public (and we hence can avoid adaptive overfitting or other issues). I feel it would be great if this would not come from one of the main academic MOF groups (as they would all want to participate) but is organized by some external entity (CSD? @ml-evs or industry)

[ml-evs]

I only have 1 week left, but happy to raise this up the flag pole and see if there is interest at CCDC?

[kjappelbaum]

I'd be interested - but if it is awkward to raise this point, no worries ;) And sorry for spamming you all the time ;)

[ml-evs]

No problem! It's not awkward, but everyone is away atm so it might have to wait until my final wrap-up meetings. At the very least, I can introduce you to the people who might be interested.

[kjappelbaum]

@FMcil do you think we can add anything interesting from PrIsMa? I think we do not have that many data points yet (<2000?). Do we have some other process-model outputs? Would also nice to expose the RSM materials as dataset, but I do not know what labels we have.

[FMcil]

Yeah ~2000 datapoints but each has many labels (~46 labels for each set of process conditions and source-sink combination) from the entire platform (process model, techno-economic model, life-cycle assessment). The data available is maybe still WIP I believe. I agree it would be nice to also expose the RSM with PrISMa labels for the purposes of benchmarking, especially since you're making it easy to calculate a huge number of features!

Also, there may be some interesting ML based process model benchmarking that could be done. The field really needs it but these
surrogate models tend to use isotherms as features and this may makes the label/feature distinction in mofdscribe a bit weird. We have lots of data from our DAC detailed model which could be a fun playground for this.

[kjappelbaum]

@arosen93 do you have any wishes for what benchmark tasks we should include?

[Andrew-S-Rosen]

No particular wishes at the moment but will give this some thought!