Format Migration

[baconpaul]

Hello!

First, thanks for writing both tao::PEGTL and tao::json. I'm a maintainer on the surge synthesizer projects and we already use PEGTL in our synth to do easy patch searching (by generating SQL queries onto sqlite) and I am now working on a new project which is using tao::json.

And tao::json is amazing. What a clever and elegant design. Thanks.

There is one question I am grappling with though. We are using json in our new project for a couple of things but my question arises in using it for patch storage. We serialize our objects to a JSON document and that is the state of our synth. But because it is a user authored document, these JSON documents will be long lived. People will have a reasonable expectation that a version of the software in 5 years could open a document authored today.

The solution to this is well worn. Put a streaming version number in your JSON. Cool. If I was doing this with jsoncpp I would have patch (my top level container) have a "streamingVersion" value on it and read it and then pass that all the way down my call stack as I parsed out subordinate types. Fine.

But I don't quite have that context here. Because I'm specializing traits to stream my objects the contained types don't know document object of which they are a member.

A simple pseudocode example is imagine I have

struct Sample { std::filesystem::path p };
struct Zone { std::shared_ptr<Sample> sample; int startPoint{0}; int key{60}; };
struct Patch { std::vector<std::shared_ptr<Zone>> zones; }

i've coded it so i specialize assign and to on sample in the obvious way (assign basically puts p.u8string() onto it and from does a load of the path it just found is one way you could do it).

But if in the future we add another property to sample with a default value, or heck even have an array of paths, I will still want to be able to parse the old document which only had one path. I guess I could do this by poking at the document and seeing if v.at() throws exceptions, but ideally the Sample parse would know if it is in the context of being subordinate to a Patch parse and then get that information.

So I guess my question is: If I have a 'top level document' with a property and I want that property available to the ::to traits specialized method of each sub object when that first one has initiated a parse, where do I put it? What are some patterns I could use and how should I think about this problem with the library here?

Thank you so much

[ColinH]

First, thank you for your kind feedback! Regarding your question, this sounds a little more complicated than what you can do with the currently supplied features (the binding traits contain provisions for multiple versions, but perhaps not with the granularity that you need, just a general "try this for everything and if it doesn't work try the next version"). Although I do wonder, is it necessary for e.g. the Sample to know in which version of a Patch it is occurring, or would it be sufficient for it to manage its own versioning? And do you already have some source code for this?

[ColinH]

We have half a solution for how to pass additional arguments to the to() or as() traits functions, half in the sense that the framework is there but we haven't fully committed to it (yet) and most traits specialisations are not compatible with it:

Next to as() and to() The class basic_value also has as_with() and to_with() which forward arbitrary many additional arguments to the traits' as() and to(). If you can get this working by extending a couple of traits specialisations to forward, but otherwise ignore, these additional arguments then please let us know, though I can't promise that we will incorporate anything at this point.

[baconpaul]

Oh interesting

I ended up making something a bit different work which is based on the insight that if I am unpacking a parent object with a version I can establish the graph as I go and propagate version downwards through the objects I am deserializing in many cases by or indeed having each object participate in a version of its own and a parent link tyoe tbing but I’m still not sure it is fully satisfactory.

Im currently using your library for our entire multi threaded communication by the way and it is a joy to use and screamingly fast

[baconpaul]

Hi!

Just re-bouncing this question from a year and a half ago

The way I ended up having to do this was somewhat unsatisfactory; when I start a top level parse I set a bit of global state that I'm doing that and also what my parse version is. This makes my conversion non-thread-safe in a rather worrying way.

What I would love to do and just spent a few hours on is trying to make it so my public_base has a member I can set and then copy that when I do my associated nested to calls and that didn't work.

So i guess I'm still sort of left wanting to thread some sort of state down some of my parses and not others. I was actually wondering if the consumer could propagate this state for me somehow onto my basic values, but haven't experimented with that. But just wanted to re-up this in case any thinking has evolved on your end. Thanks!

[baconpaul]

Ahh nothing like re-asking on GitHub to figure out your problem. Sigh.

I found a way to do this basically by doing a parse, a partial extract, a decoration of my basic_value public base recursively, and then a to. Sorry for the noise!