vds format: the content should be a full JSON Lines file

[aborruso]

Hi,
if I have this CSV

field1,field2,field3
Andy,47,

and set field2 as #, and save it as jsonl I have

{"field1": "Andy", "field2": 47, "field3": ""}

field2 is a number.

If I export it as vds I have that field2 is not a number.

{"field1": "Andy", "field2": "47", "field3": ""}

What about to create a vds output that is fully compliant with field type chosen?

[frosencrantz]

I'm hitting a similar issue: #885

Also, I have some additional thoughts about the new vds format:

1. It would be useful if the base format of vds was a standard type, so it would be possible to work with other tools such as mlr, jq or even older versions of Visidata. I would think JSON is a good base format, and the vds format is already partly using that.
2. I typically run the same queries, where the columns are stable, but the data (rows,cells) are different every time. It would be useful if we could optionally specify column metadata in a separate file rather than bundled with the data.
3. More open-ended idea is maybe vds could hold more Visidata metadata you want to save when working with some data, like the playback commands, options, input_history, etc.

[saulpw]

Thanks for filing #885, @frosencrantz, that's a legitimate issue.

For (1), .vds is almost entirely JSONL; you can strip out the metadata lines with sed if you want; this is a by-design but off-label usage of the internal format. In fact if you have a single-sheet .vds you can load it into VisiData v2.0 or 2.1 with -f jsonl, which since 2.0 skips lines that begin with #, so you can turn a .vdj cmdlog into a hashbang script. So I completely agree with your (1), it was a design goal from the outset, and I did the best job that I could after taking the various factors into consideration. If the format doesn't meet your implied higher standard, you can posit a concrete use case as a feature request.

For (2), that's a reasonable idea, I can see creating a command to apply an empty .vds (with only metadata and no data) onto any other kind of sheet. It's a feature request, please file it as such in our standard process and support its development and your wish might just come true.

And for (3), yup. I had that thought too. Lots of options here. How about we all play with it for a few days or weeks and see how it works and what concrete use cases we find ourselves wanting? Feel free (you or anyone) to post ideas in this discussion; the more concrete and well-thought-through the better.

[frosencrantz]

For item 1, one of the issues was generating the metadata header. I was thinking the same thing about using sed to add/remove the leading #, but why the bother of working with sed, when it would be easier to use the unprocessed file with a regular JSON editing tool directly. I'm a happy user of sed, but I use it a lot less since I started using vd. I feel like I've subscribed to the belief your tool is able to handle and share common standard data formats, and I feel like adding a specialized non-standard format breaks some sort of promise.

The main things I would like to change externally would be:

• Column width to hide certain columns by default or shape them to fit the window
• Reorder columns

Likely more if the vds format provided a way to expand/unexpand container columns or reference parts of a container. Or add expression columns (python/regex).

The extra #'s aren't a blocker, but they seem like a nuisance. I'm unclear on what the constraints are that require the format to need the comment. The main constraint sounds like vd 2.0 will ignore jsonl lines starting with '#'. Though there are likely more constraints.

I'll share an anecdote about the make program, that feels relevant:

Soon after make was released, one of its first users complained that
the requirement that each command line begin with a tab was suboptimal,
and that some alternative approach should be used. As the story goes,
the author was receptive to this criticism, but explained that sadly it
was now too late to change the way things worked, because too many
people were used to current behavior. The punchline, of course, was
that at this point make had perhaps dozens of users.

I think VisiData has many users so incompatibility is bad. But I feel like plain json/jsonl would be readable by earlier versions of Visidata. But not having had to implement this feature I'm missing some of that base understanding. Maybe it is a mix of both json&jsonl

2. Feature requested 😃: http://github.com/saulpw/visidata/issues/892

3. Will have to think about this some more. Options would be helpful here, like for coloring, date format, column separators. If you are willing to do Make a package #2. It seems like playback commands would also be very helpful, to add expression columns, if not supported by the vds format. I use vd files mostly for reshaping a table (hiding&reordering cols, adding expressions, unfurling, etc).

Also, I would like to say I'm very excited about this new feature. The feedback is meant to be constructive or informative.

[saulpw]

That is interesting, @aborruso. I specifically chose to stringify everything as a kind of "what you see is what you save", and let the column types convert them to their typed values when they're reloaded. It's also necessary to stringify date values since there is no JSON date type. For example, if you load benchmark.csv and type 'date' as a `date' and then save as a .vds file, what would you want the saved string to be?

1. the original date string
2. a full date-time string with maximal precision (probably even over-specified)
3. the date string with the displayed precision

And then, we need to apply the same rule to float. Should 4.0 look like `3.9999997' in the save format, or like '4.014824' if that was the original value?

I chose (3), and this is why even int values (which I agree could always be actual JSON ints) are stringified.

Also, I am curious why you would care about the internal format and want it to be any certain way. Are you wanting to send the output into another tool, as @frosencrantz does? Or is it just a sense of tidiness?

[aborruso]

Hi @saulpw VisiData is a really great idea, and it's your idea.

When I set a CSV field as inteteger in a vds file, I obtain that if I open it again I will view it in the same way I have saved it: an integer field.

A strategy could be to save in JSONL format in the best way is it possible to using JSON format. If a field is an integer, save it as JSON integer.

But if I have this kind of input json

{ "name": "DEF", "a": 1, "date": "2021-01-02", "time": "06:00" }
{ "name": "ABC", "a": 1, "date": "2021-01-02", "time": "05:40" }

and I open it in VD, it read the a field as integer. If then I save it in vds I have the a field as string

{"name": "DEF", "a": "1", "date": "2021-01-02", "time": "06:00"}
{"name": "ABC", "a": "1", "date": "2021-01-02", "time": "05:40"}

The vds file format is a usual a great idea of yours. It can stay that way and I don't see anything critical about it.
But it's a text format and it could be interesting to use it some ETL pipeline, that use also other tools. And only for this reason I would prefer that if I save an integer, I will have a JSON integer.

Thank you for all

[geekscrapy]

I'd very much prefer that the data is stored as it's raw form and the typing and formatting is applied on load. You never know when you might need the extra precision. I would consider a vds format to be as lossless as possible, basically a carbon copy of what you had open, just without all the command history (although, that might also be useful!)

So basically stored as D/M/Y as per your example

[saulpw]

Ah, so if you had a CSV with a column of ints (as text, because all CSV data is text), then even if you typed the column as int, the JSON would store the int value as text? It seems that this is the opposite of what @aborruso suggested in the initial request.

[geekscrapy]

Yea I guess so.

Tbh, I see the vds format as one as its own, it stores what it needs to, for its own purposes and shouldn't try and accommodate other formats (if it does though that's a happy coincidence). There is a robust export and saving mechanism with vd and it can save a ton of formats which are only a command line away. If you need json, go ahead and open the file in vd and save as json. If the vds format tries to be everything, it will become something that's not json, and not fully useful to vd (or there are lots of gotchas and edge cases like ints as strs or similar), adding cool things to it in the future may be restricted due to trying to stick to jsonl, which is odd, as we have a json exporter...

I get that it'd be convenient for other tools but they will probably still have vds import woes however the vds format is laid out. For example, at the moment, vds is using # which is not jsonl and may break other tools on import. Trying to make it compatible as jsonl will not only lure other tool into thinking it is (but actually it's not), but it may also restrict any major changes to the vds format in the future.

As far as I'm concerned vds could be a proprietary binary format, which is done for speed, and vd awesomeness. To allow it to be used in other tools, the code is opensource (unlike lots of other binary formats) and it's not as if there aren't many ways to export to a format you need from within vd. And I'm sure that if needs be, there'll be a vds to X converter library if needs be.

Sorry for the long post!

[frosencrantz]

Having had more time to mull this over. I think the vds format works really great for the original use case that caused it to be created. It is really great that you can run it like a script.

It might be asking too much to ask for the JSON format to provide the fidelity of the original data. I think that could be helped by #885 .

[saulpw]

I decided to make it save the typed values, because containers can't be stringified and then reconstituted.

We'll keep the metadata "comment" lines for now though. Thanks everyone for chiming in.