Saving in VisiData

[anjakefala]

We would like to hear your feedback for how you want saving to work in VisiData.

This is a document which hopes to capture all of the relevant considerations and cases, and my understanding of the current behaviour.

The Important Players

• When saving, VisiData queries a user for a path to save to.
• The path will either be a filename (of the structure {filename}.{ext}) or a directory (/path/to/foldername/).

• There is also an options.save_filetype which is the fileformat to be saved in.
  • By default it is set to '' (or at least, I think that is the plan Revert default VisiData behaviour to choose saver from extension #1265).

Behaviour

commit-sheet (z Ctrl+S)

• VisiData saves directly to the source file, in the form that it was loaded in. (i.e. if the source file is a SQLite database, VisiData will save to that filename with save_sqlite.)
• options.save_filetype does not affect this behaviour.

save-sheet (Ctrl+S)

• An input path representing a single file ({filename}.{ext}) is expected.
• By default, VisiData will look for a save_ext to save to in the fileformat ext. If this savefunc is not available, VisiData will fail. (Discussed here: [save] Saving a file without a saver saves to tsv by default. #1072)
• If options.save_filetype is set, VisiData will save to that filetype preferentially over using save_ext: (Discussed here: Need some way to save different type than file extension #1242)
• If saving a sheet, with no pre-existing source, the default .{ext} for the user save prompt will be options.save_filetype or 'vds') (Proposed by @geekscrapy)

save-sheets (g Ctrl+S)

• VisiData will accept either a /path/to/foldername or a {filename}.{ext}.
• If {filename}.{ext} is provided, the which file format decision behaviour is similar to Ctrl+S, except that the save_format chosen must support saving multiple sheets to a single file. (for e.g. save_xlsx.) If it does not, it will fail.
• If /path/to/filename is provided, then each sheet will be saved as an individual file to that directory. The chosen format is options.save_filetype. If by default, options.save_filetype is '', then we will need a fallback (Perhaps 'vds'?) or a fail followed by a prompt asking users to set options.save_filetype on the OptionsSheet.

Also relevant to saving

• In all of these cases, if options.confirm_overwrite is set, and {filename}.{ext} already exists, the user most be queried before the data is overwritten.
  • It would probably be ideal if options.confirm_overwrite prompt included the ultimate savefunc being used.
• -b -o should behave with the same decision tree as if the user was using Ctrl+S within VisiData.

User Preferences based on my understanding

• To not be caught off-guard by the save format being used, when saving over an existing file. ([save] Saving a file without a saver saves to tsv by default. #1072)
• For the .ext to take priority, if the user has not been fiddling with anything, and the savefunc for it exists (basically, this has been the default behaviour up until now).
• To be able to override saving using save_ext (Need some way to save different type than file extension #1242).
• Users would probably be happy for there to be a reasonable fallback (e.g. .vds), as long as an existing file is not being over-written in a format that they are not expecting.

[anjakefala]

Writing this up made me realise #1265 breaks g Ctrl+S to a directory. Breaks it in the sense, that its default behaviour is to Not Work without an options.save_filetype.
So, it seems like we have two distinct concepts: save_filetype and save_fallback.
Where save_fallback never overrides, it falls to it if there is no other filetype information and save_filetype overrides all other considerations in the decision tree.

[ajkerrigan]

Thanks for writing this up @anjakefala, there's so much more going on here than I usually hit 😅 ! A couple initial thoughts:

• I love defaulting to .vds when VisiData can't determine a better option. It can handle multiple sheets and keeps column attributes, so if I rethink the target format later nothing is lost :). If that's a surprise, it's a friendly one!
• Good point about commit-sheet not having to figure out filetypes. That command seems to call commit() and putChanges() rather than any of the save_* methods which makes sense 👍 .
• I'm trying to get g Ctrl+S to Not Work but with or without the change from Revert default VisiData behaviour to choose saver from extension #1265 it keeps working... I may not be trying the right (wrong?) combination of commands / sheets / options though.

[reagle]

I've a question about saving.

1. I load a two sheet notebook "responses-grades-cda.xlsx".
2. g ctrl+s
3. it gives me a blank line -- why not default to name of file without xlsx extension?
4. I type in file name without extension + vds: "responses-grades-cda.vds".
5. I get an error: "AttributeError: 'X1sxIndexSheet' object has no attribute 'join'"

[anjakefala]

These seem like bugs, so I opened a new issue here for them: #1720