From 5089feb3f20a3a7687023091d6b230afc0bb7dae Mon Sep 17 00:00:00 2001 From: Phil Shafer Date: Mon, 17 May 2021 13:55:02 -0400 Subject: [PATCH 1/4] add documentation for csv encoder --- libxo/Makefile.am | 1 + 1 file changed, 1 insertion(+) diff --git a/libxo/Makefile.am b/libxo/Makefile.am index a4d94aca..9f17527e 100644 --- a/libxo/Makefile.am +++ b/libxo/Makefile.am @@ -74,6 +74,7 @@ man5_files = \ xo_format.5 man7_files = \ + libxo-csv.7 \ xo_options.7 man_MANS = ${man3_files} ${man5_files} ${man7_files} From 2999538cd63edc8737501dd5abe6df47e3ddd646 Mon Sep 17 00:00:00 2001 From: Phil Shafer Date: Mon, 17 May 2021 13:55:25 -0400 Subject: [PATCH 2/4] add options docs to libxo(3) --- libxo/libxo.3 | 91 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 91 insertions(+) diff --git a/libxo/libxo.3 b/libxo/libxo.3 index f0796203..95c00598 100644 --- a/libxo/libxo.3 +++ b/libxo/libxo.3 @@ -86,6 +86,95 @@ suited for terminal output and HTML is suited for display in a web browser (see .Xr xohtml 1 ). .Pp +.Nm libxo +uses command line options to trigger rendering behavior. +The following options are recognised: +.Pp +.Bl -tag -width "--libxo" +.It +\-\^\-libxo +.It +\-\^\-libxo= +.It +\-\^\-libxo: +.El +.Pp +Options is a comma-separated list of tokens that correspond to output +styles, flags, or features: +.Pp +.Bl -tag -width "12345678" +.It Sy "Token Action" +.It Dv dtrt +Enable "Do The Right Thing" mode +.It Dv html +Emit HTML output +.It Dv indent=xx +Set the indentation level +.It Dv info +Add info attributes (HTML) +.It Dv json +Emit JSON output +.It Dv keys +Emit the key attribute for keys (XML) +.It Dv log-gettext +Log (via stderr) each +.Xr gettext 3 +string lookup +.It Dv log-syslog +Log (via stderr) each syslog message (via +.Xr xo_syslog 3 ) +.It Dv no-humanize +Ignore the {h:} modifier (TEXT, HTML) +.It Dv no-locale +Do not initialize the locale setting +.It Dv no-retain +Prevent retaining formatting information +.It Dv no-top +Do not emit a top set of braces (JSON) +.It Dv not-first +Pretend the 1st output item was not 1st (JSON) +.It Dv pretty +Emit pretty-printed output +.It Dv retain +Force retaining formatting information +.It Dv text +Emit TEXT output +.It Dv underscores +Replace XML-friendly "-"s with JSON friendly "_"s e +.It Dv units +Add the 'units' (XML) or 'data-units (HTML) attribute +.It Dv warn +Emit warnings when libxo detects bad calls +.It Dv warn-xml +Emit warnings in XML +.It Dv xml +Emit XML output +.It Dv xpath +Add XPath expressions (HTML) +.El +.Pp +The +.Dq brief-options +are single letter commands, designed for those with +too little patience to use real tokens. +No comma separator is used. +.Bl -column "i" +.It Sy "Token Action" +.It "H " "Enable HTML output (XO_STYLE_HTML)" +.It "I " "Enable info output (XOF_INFO)" +.It "i " "Indent by " +.It "J " "Enable JSON output (XO_STYLE_JSON)" +.It "P " "Enable pretty-printed output (XOF_PRETTY)" +.It "T " "Enable text output (XO_STYLE_TEXT)" +.It "W " "Enable warnings (XOF_WARN)" +.It "X " "Enable XML output (XO_STYLE_XML)" +.It "x " "Enable XPath data (XOF_XPATH)" +.El +.Pp +Refer to +.Xr xo_options 7 +for additional option information. +.Pp The .Nm library allows an application to generate text, XML, JSON, @@ -291,6 +380,7 @@ Instructs to use an alternative set of low-level output functions. .El .Sh SEE ALSO +.Xr libxo-csv 7, .Xr xo 1 , .Xr xolint 1 , .Xr xo_attr 3 , @@ -303,6 +393,7 @@ to use an alternative set of low-level output functions. .Xr xo_no_setlocale 3 , .Xr xo_open_container 3 , .Xr xo_open_list 3 , +.Xr xo_options 7, .Xr xo_parse_args 3 , .Xr xo_set_allocator 3 , .Xr xo_set_flags 3 , From e7a83d28375ac85c84fc32504b847f18c7488a18 Mon Sep 17 00:00:00 2001 From: Phil Shafer Date: Mon, 17 May 2021 13:55:34 -0400 Subject: [PATCH 3/4] add options docs to libxo(3) --- libxo/xo_parse_args.3 | 90 ++----------------------------------------- 1 file changed, 4 insertions(+), 86 deletions(-) diff --git a/libxo/xo_parse_args.3 b/libxo/xo_parse_args.3 index 543ffec9..e631af63 100644 --- a/libxo/xo_parse_args.3 +++ b/libxo/xo_parse_args.3 @@ -24,7 +24,9 @@ .Sh DESCRIPTION The .Fn xo_parse_args -function is used to process command-line arguments. +function is used to process command-line arguments, which are +described in +.Xr xo_options 7 . .Nm libxo specific options are processed and removed @@ -42,91 +44,6 @@ Following the call to .Fn xo_parse_args , the application can process the remaining arguments in a normal manner. .Pp -.Nm libxo -uses command line options to trigger rendering behavior. -The following options are recognised: -.Pp -.Bl -tag -width "--libxo" -.It -\-\^\-libxo -.It -\-\^\-libxo= -.It -\-\^\-libxo: -.El -.Pp -Options is a comma-separated list of tokens that correspond to output -styles, flags, or features: -.Pp -.Bl -tag -width "12345678" -.It Sy "Token Action" -.It Dv dtrt -Enable "Do The Right Thing" mode -.It Dv html -Emit HTML output -.It Dv indent=xx -Set the indentation level -.It Dv info -Add info attributes (HTML) -.It Dv json -Emit JSON output -.It Dv keys -Emit the key attribute for keys (XML) -.It Dv log-gettext -Log (via stderr) each -.Xr gettext 3 -string lookup -.It Dv log-syslog -Log (via stderr) each syslog message (via -.Xr xo_syslog 3 ) -.It Dv no-humanize -Ignore the {h:} modifier (TEXT, HTML) -.It Dv no-locale -Do not initialize the locale setting -.It Dv no-retain -Prevent retaining formatting information -.It Dv no-top -Do not emit a top set of braces (JSON) -.It Dv not-first -Pretend the 1st output item was not 1st (JSON) -.It Dv pretty -Emit pretty-printed output -.It Dv retain -Force retaining formatting information -.It Dv text -Emit TEXT output -.It Dv underscores -Replace XML-friendly "-"s with JSON friendly "_"s e -.It Dv units -Add the 'units' (XML) or 'data-units (HTML) attribute -.It Dv warn -Emit warnings when libxo detects bad calls -.It Dv warn-xml -Emit warnings in XML -.It Dv xml -Emit XML output -.It Dv xpath -Add XPath expressions (HTML) -.El -.Pp -The -.Dq brief-options -are single letter commands, designed for those with -too little patience to use real tokens. -No comma separator is used. -.Bl -column "i" -.It Sy "Token Action" -.It "H " "Enable HTML output (XO_STYLE_HTML)" -.It "I " "Enable info output (XOF_INFO)" -.It "i " "Indent by " -.It "J " "Enable JSON output (XO_STYLE_JSON)" -.It "P " "Enable pretty-printed output (XOF_PRETTY)" -.It "T " "Enable text output (XO_STYLE_TEXT)" -.It "W " "Enable warnings (XOF_WARN)" -.It "X " "Enable XML output (XO_STYLE_XML)" -.It "x " "Enable XPath data (XOF_XPATH)" -.El -.Pp The .Fn xo_set_program function sets name of the program as reported by @@ -149,6 +66,7 @@ must be maintained by the caller. .Pp .Sh SEE ALSO .Xr xo_emit 3 , +.Xr xo_options 7, .Xr libxo 3 .Sh HISTORY The From 41423083bc4c9f36ce49bdf7eb28fb7357d106a2 Mon Sep 17 00:00:00 2001 From: Phil Shafer Date: Mon, 17 May 2021 13:55:43 -0400 Subject: [PATCH 4/4] initial version --- libxo/libxo-csv.7 | 274 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 274 insertions(+) create mode 100644 libxo/libxo-csv.7 diff --git a/libxo/libxo-csv.7 b/libxo/libxo-csv.7 new file mode 100644 index 00000000..6e043820 --- /dev/null +++ b/libxo/libxo-csv.7 @@ -0,0 +1,274 @@ +.\" # +.\" # Copyright (c) 2021, Juniper Networks, Inc. +.\" # All rights reserved. +.\" # This SOFTWARE is licensed under the LICENSE provided in the +.\" # ../Copyright file. By downloading, installing, copying, or +.\" # using the SOFTWARE, you agree to be bound by the terms of that +.\" # LICENSE. +.\" # Phil Shafer, May 2021 +.\" +.Dd May 13, 2021 +.Dt LIBXO-CSV 7 +.Os +.Sh NAME +.Nm --libxo encoder=csv[+options] +.Nd a CVS encoder for libxo\-based commands +.Sh DESCRIPTION +The +.Nm libxo +library supports a "pluggable" encoder mechanism, and ships with an +encoder for CSV (comma separated values) files. The encoder allows +paths and fields to be selected out of the output contents: +.Bd -literal -offset indent + % df --libxo @csv + name,total-blocks,used-blocks,available-blocks,used-percent,mounted-on + zroot/ROOT/default,3825984331,29376725,3796607605,1,/ + devfs,1,1,0,100,/dev + zroot/usr/home,3808301289,11693684,3796607605,0,/usr/home + zroot/var/audit,3796607806,201,3796607605,0,/var/audit + ... + % df --libxo @csv+leafs=name.used-percent + name,used-percent + zroot/ROOT/default,1 + devfs,100 + zroot/usr/home,0 + zroot/var/audit,0 + ... + % df --libxo @csv+leafs=available-blocks+no-header / + 3796607605 +.Ed + contains software to generate four "built-in" +formats: text, XML, JSON, and HTML. +These formats are common and useful, but there are other common and +useful formats that users will want, and including them all in the +libxo software would be difficult and cumbersome. +.Pp +To allow support for additional encodings, libxo includes a +"pluggable" extension mechanism for dynamically loading new encoders. +.Nm libxo -based +applications can automatically use any installed encoder. +.Pp +Use the "encoder=XXX" option to access encoders. The following +example uses the "cbor" encoder, saving the output into a file: +.Bd -literal -offset indent + df --libxo encoder=cbor > df-output.cbor +.Ed +.Pp +Encoders can support specific options that can be accessed by +following the encoder name with a colon (':') or a plus sign ('+') and +one of more options, separated by the same character: +.Bd -literal -offset indent + df --libxo encoder=csv+path=filesystem+leaf=name+no-header + df --libxo encoder=csv:path=filesystem:leaf=name:no-header +.Ed +.Pp +These examples instructs libxo to load the "csv" encoder and pass the +following options: +.Bd -literal -offset indent + path=filesystem + leaf=name + no-header +.Ed +.Pp +Each of these option is interpreted by the encoder, and all such +options names and semantics are specific to the particular encoder. +Refer to the intended encoder for documentation on its options. +.Pp +The string "@" can be used in place of the string "encoder=". +.Bd -literal -offset indent + df --libxo @csv:no-header +.Ed +.Sh The CSV (Comma Separated Values) Encoder +.Nm libxo +ships with a custom encoder for "CSV" files, a common format for +comma separated values. The output of the CSV encoder can be loaded +directly into spreadsheets or similar applications. +.Pp +A standard for CSV files is provided in RFC 4180, but since the +format predates that standard by decades, there are many minor +differences in CSV file consumers and their expectations. The CSV +encoder has a number of options to tailor output to those +expectations. +.Pp +Consider the following XML: +.Bd -literal -offset indent + % list-items --libxo xml,pretty + + + + GRO-000-415 + gum + 1412 + 54 + 10 + + + HRD-000-212 + rope + 85 + 4 + 2 + + + HRD-000-517 + ladder + 0 + 2 + 1 + + + +.Ed +.Pp +This output is a list of `instances` (named "item"), each containing a +set of `leafs` ("sku", "name", etc). +.Pp +The CSV encoder will emit the leaf values in this output as `fields` +inside a CSV `record`, which is a line containing a set of +comma-separated values: +.Bd -literal -offset indent + % list-items --libxo encoder=csv + sku,name,sold,in-stock,on-order + GRO-000-415,gum,1412,54,10 + HRD-000-212,rope,85,4,2 + HRD-000-517,ladder,0,2,1 +.Ed +.Pp +Be aware that since the CSV encoder looks for data instances, when +used with +.Nm xo , +the `--instance` option will be needed: +.Bd -literal -offset indent + % xo --libxo encoder=csv --instance foo 'The {:product} is {:status}\n' stereo "in route" + product,status + stereo,in route +.Ed +.Sh The "path" Option +By default, the CSV encoder will attempt to emit any list instance +generated by the application. +In some cases, this may be unacceptable, and a specific list may be +desired. +.Pp +Use the "path" option to limit the processing of output to a specific +hierarchy. The path should be one or more names of containers or +lists. +.Pp +For example, if the "list-items" application generates other lists, +the user can give "path=top/data/item" as a path: +.Bd -literal -offset indent + % list-items --libxo encoder=csv:path=top/data/item + sku,name,sold,in-stock,on-order + GRO-000-415,gum,1412,54,10 + HRD-000-212,rope,85,4,2 + HRD-000-517,ladder,0,2,1 +.Ed +.Pp +Paths are "relative", meaning they need not be a complete set +of names to the list. This means that "path=item" may be sufficient +for the above example. +.Sh The "leafs" Option +The CSV encoding requires that all lines of output have the same +number of fields with the same order. In contrast, XML and JSON allow +any order (though libxo forces key leafs to appear before other +leafs). +.Pp +To maintain a consistent set of fields inside the CSV file, the same +set of leafs must be selected from each list item. By default, the +CSV encoder records the set of leafs that appear in the first list +instance it processes, and extract only those leafs from future +instances. If the first instance is missing a leaf that is desired by +the consumer, the "leaf" option can be used to ensure that an empty +value is recorded for instances that lack a particular leaf. +.Pp +The "leafs" option can also be used to exclude leafs, limiting the +output to only those leafs provided. +.Pp +In addition, the order of the output fields follows the order in which +the leafs are listed. "leafs=one.two" and "leafs=two.one" give +distinct output. +.Pp +So the "leafs" option can be used to expand, limit, and order the set +of leafs. +.Pp +The value of the leafs option should be one or more leaf names, +separated by a period ("."): +.Bd -literal -offset indent + % list-items --libxo encoder=csv:leafs=sku.on-order + sku,on-order + GRO-000-415,10 + HRD-000-212,2 + HRD-000-517,1 + % list-items -libxo encoder=csv:leafs=on-order.sku + on-order,sku + 10,GRO-000-415 + 2,HRD-000-212 + 1,HRD-000-517 +.Ed +.Pp +Note that since libxo uses terminology from YANG (:RFC:`7950`), the +data modeling language for NETCONF (:RFC:`6241`), which uses "leafs" +as the plural form of "leaf". libxo follows that convention. +.Sh The "no-header" Option +CSV files typical begin with a line that defines the fields included +in that file, in an attempt to make the contents self-defining: +.Bd -literal -offset indent + sku,name,sold,in-stock,on-order + GRO-000-415,gum,1412,54,10 + HRD-000-212,rope,85,4,2 + HRD-000-517,ladder,0,2,1 +.Ed +.Pp +There is no reliable mechanism for determining whether this header +line is included, so the consumer must make an assumption. +.Pp +The csv encoder defaults to producing the header line, but the +"no-header" option can be included to avoid the header line. +.Sh The "no-quotes" Option +RFC 4180 specifies that fields containing spaces should be quoted, but +many CSV consumers do not handle quotes. The "no-quotes" option +instruct the CSV encoder to avoid the use of quotes. +.Sh The "dos" Option +RFC 4180 defines the end-of-line marker as a carriage return +followed by a newline. This "CRLF" convention dates from the distant +past, but its use was anchored in the 1980s by the `DOS` operating +system. +.Pp +The CSV encoder defaults to using the standard Unix end-of-line +marker, a simple newline. Use the "dos" option to use the `CRLF` +convention. +.Sh Option Handling +The handling of command-line options is complex, since there are three +hierarchies in use, but the rules are: +.Bl -bullet +.It +commas separate +.Nm libxo +options +.Bd -literal -ofset indent + \-\-libxo json,pretty,warn +.Ed +.It +pluses separate encoder options +.Bd -literal -ofset indent +\-\-libxo @csv+dos+no-header,warn +.Ed +.It +periods separate tag names +.Bd -literal -ofset indent +\-\-libxo @csv+leafs=name.used.free+dos,warn +.El +.Sh SEE ALSO +.Xr libxo 3 , +.Xr xo_options 7 +.Sh HISTORY +The +.Nm libxo +library first appeared in +.Fx 11.0 . +The CSV encoder first appeared in +.Fx 13.0 . +.Sh AUTHORS +.Nm libxo +was written by +.An Phil Shafer Aq Mt phil@freebsd.org . +