Add `--format` argument to `nickel query` command #2015

suimong · 2024-07-29T09:46:59Z

Closes #1976

This PR adds the ability for nickel query to export the result to JSON, YAML & TOML, similar to that of nickel export. The output schema looks like:

{
  doc | Nullable String,
  "optional" | Bool,
  not_exported | Bool,
  priority | String,   # variants include: "default", "neutral", "force", and string representation of rational number ("1", "5/2", etc.)
  type | Nullable String,
  contracts | Array String,
  sub_fields | Nullable (Array String),
  value | String | optional,
}

During implementation, I found a few areas that ask for "designing":

whether to use Nullable T or T | optional for some fields in the output schema
how to serialize the priority field
the logic that dictates the presence of the value field

suimong

@yannham do you think it's ok that in this PR, the --format flag ignores all customzation flags (--doc, --default etc.) and just exports all metadata fields and value? I tried to make them compatible but after some experimentation I'm inclined to keep the two features independent. The main reason is that by default --format exports all metadata fields which I think is desirable, but a few of them does not have a corresponding flag (namely optional, not_exported, priority), so customization for json export would be half baked and inconsistent with the current markdown output. That is, if we leave the markdown rendering code untouched.

yannham · 2024-08-01T14:27:37Z

I think it's reasonable. Those options are useful to minimize the markdown output when one is only interested in one particular field, but It's probably less useful for a JSON export, which can be transformed afterwards by another tool (like...Nickel 👿 ?) or the end consumer if needed. I don't remember clap well enough to know if we can encode that constraint directly. I suppose we can (such that the CLI prints an error automatically if you specify both --doc and --format). In any case, as long as it's documented properly in the argument description and validated in one way or another, it's fine by me 👍

suimong · 2024-08-01T14:48:55Z

Cool! Will get the PR ready for review soon.

yannham · 2024-08-08T16:37:45Z

cli/src/query.rs

+ let query_res = program.query().map(|field| QueryResult(field));
+ if let Ok(res) = query_res {
+ self.export(res, format).report_with_program(program)?
+ } else {


Nitpick: instead of duplicationg "No metadata found for this field", could we define found outside of the if let Some(format) = self.format {...} else {...} and just set found to true in theif (or false in the else, depending on the initial value of found), and move the if !found below also after the whole if-then-else?

yannham · 2024-08-08T16:56:30Z

core/src/serialize.rs

+ #[default]
+ Json,


I wonder if we shouldn't rather have an additional Markdown variant, which is the default (and should be the documented default in the CLI as well). I think it's good practice that the normal behavior (without --format) can be recovered in a more precise way with an argument - typically, nickel export is the same as nickel export --format json, etc. - so that the default behavior is just that, a default value.

On the other hand, it makes the logic of when we should allow --doc, etc. more complex. So I'm not entirely sure. Maybe we can start as it is now, but then I would remove the Default impl of ExportFormatCommon, because it doesn't really have a default value.

yannham · 2024-08-08T16:59:24Z

core/src/term/mod.rs

+impl Serialize for MergePriority {
+ fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
+ where
+ S: Serializer,
+ {
+ match self {
+ Self::Bottom => serializer.serialize_unit_variant("priority", 0, "default"),
+ Self::Neutral => serializer.serialize_unit_variant("priority", 3, "neutral"),
+ Self::Numeral(n) => serializer.serialize_newtype_variant("priority", 2, "numeral", &n),
+ Self::Top => serializer.serialize_unit_variant("priority", 3, "force"),
+ }
+ }
+}


This looks pretty mechanic - can't serde derive it automatically with #[derive(Serialiaze)] on MergePriority ?

yannham · 2024-08-08T16:59:35Z

core/src/serialize.rs

@@ -47,6 +46,25 @@ impl fmt::Display for ExportFormat {
 }
 }

+/// Available common export formats.
+#[derive(Copy, Clone, Eq, PartialEq, Debug, Default, clap::ValueEnum)]
+pub enum ExportFormatCommon {


ExportFormatCommon isn't very telling, IMHO (how it is different than ExportFormat or other formats described elsewhere in the codebase?). If there's no good general way to describe it, maybe something more descriptive such as MetadataExportFormat is less confusing?

core/src/repl/query_print.rs

suimong · 2024-08-09T06:10:34Z

@yannham Thanks for the review! I'm very sorry that I forgot to push one last commit before marking this ready for review.

Additionally, here are a few questions on which I'd like to have your opinion:

whether to use Nullable T or T | optional for some fields in the output schema
how to serialize the priority field. Currently everything is serialized to a string, including the Numeral variant. An alternative would be to separate the Numeral variant from Bottom and Top, serializing the Numeral variant to json number.
the logic that dictates the presence of the value field.

github-actions · 2024-08-09T06:21:05Z

Bencher

Report	Fri, August 9, 2024 at 06:21:04 UTC
Project	nickel
Branch	2015/merge
Testbed	ubuntu-latest

⚠️ WARNING: The following Measure does not have a Threshold. Without a Threshold, no Alerts will ever be generated!
Latency (latency)
Click here to create a new Threshold
For more information, see the Threshold documentation.
To only post results if a Threshold exists, set the --ci-only-thresholds CLI flag.

Click to view all benchmark results

Benchmark	Latency	Latency Results nanoseconds (ns)
fibonacci 10	➖ (view plot)	489,740.00
pidigits 100	➖ (view plot)	3,193,800.00
product 30	➖ (view plot)	787,510.00
scalar 10	➖ (view plot)	1,471,300.00
sum 30	➖ (view plot)	786,270.00

Bencher - Continuous Benchmarking
View Public Perf Page
Docs | Repo | Chat | Help

Ben Yang added 3 commits July 29, 2024 16:40

WIP.

62093c9

WIP on interface

228b1d3

WIP

a579dd3

suimong commented Aug 1, 2024

View reviewed changes

WIP

da75096

suimong marked this pull request as ready for review August 2, 2024 11:06

suimong and others added 2 commits August 2, 2024 19:07

Merge branch 'master' into cli_query_export

c1bcb34

WIP.

1527f87

yannham reviewed Aug 8, 2024

View reviewed changes

core/src/repl/query_print.rs Show resolved Hide resolved

Ben Yang added 4 commits August 9, 2024 09:52

Merge branch 'master' into cli_query_export

5773d84

Fix CI; improve impl Serialize for MergePriority

da8eeba

remove Default impl for ExportFormatCommon

c23c85b

change naming.

9cf566f

Merge branch 'master' into cli_query_export

e4722c5

yannham self-assigned this Sep 5, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add `--format` argument to `nickel query` command #2015

Add `--format` argument to `nickel query` command #2015

suimong commented Jul 29, 2024 •

edited

Loading

suimong left a comment •

edited

Loading

yannham commented Aug 1, 2024

suimong commented Aug 1, 2024

yannham Aug 8, 2024

yannham Aug 8, 2024

yannham Aug 8, 2024

yannham Aug 8, 2024

suimong commented Aug 9, 2024 •

edited

Loading

github-actions bot commented Aug 9, 2024

Add --format argument to nickel query command #2015

Are you sure you want to change the base?

Add --format argument to nickel query command #2015

Conversation

suimong commented Jul 29, 2024 • edited Loading

suimong left a comment • edited Loading

Choose a reason for hiding this comment

yannham commented Aug 1, 2024

suimong commented Aug 1, 2024

yannham Aug 8, 2024

Choose a reason for hiding this comment

yannham Aug 8, 2024

Choose a reason for hiding this comment

yannham Aug 8, 2024

Choose a reason for hiding this comment

yannham Aug 8, 2024

Choose a reason for hiding this comment

suimong commented Aug 9, 2024 • edited Loading

github-actions bot commented Aug 9, 2024

Bencher

Add `--format` argument to `nickel query` command #2015

Add `--format` argument to `nickel query` command #2015

suimong commented Jul 29, 2024 •

edited

Loading

suimong left a comment •

edited

Loading

suimong commented Aug 9, 2024 •

edited

Loading