From 2ffb3b0433bc19d8bdcb5501eb56806d87c72531 Mon Sep 17 00:00:00 2001 From: Sharon Lifshitz Date: Wed, 25 Sep 2019 11:03:21 +0300 Subject: [PATCH 1/4] [DOC] Frames GS: Minor rephrasing & cosmetic fixes in the Overview [IG-12272 IG-12092] --- getting-started/frames.ipynb | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/getting-started/frames.ipynb b/getting-started/frames.ipynb index d83b396c..9fbd69c7 100644 --- a/getting-started/frames.ipynb +++ b/getting-started/frames.ipynb @@ -21,7 +21,7 @@ "\n", "## Overview\n", "\n", - "[V3IO Frames](https://github.com/v3io/frames) (**\"Frames\"**) is a multi-model open-source data-access library, developed by Iguazio, which provides a unified high-performance DataFrame API for loading, storing, and accessing data in the data store of the Iguazio Data Science Platform (**\"the platform\"**).\n", + "[V3IO Frames](https://github.com/v3io/frames) (**\"Frames\"**) is a multi-model open-source data-access library, developed by Iguazio, which provides a unified high-performance DataFrame API for working with data in the data store of the Iguazio Data Science Platform (**\"the platform\"**).\n", "Frames currently supports the NoSQL (key/value), stream, and time-series (TSDB) data models via its `\"kv\"`, `\"stream\"`, and `\"tsdb\"` backends.\n", "\n", "To use Frames, you first need to import the **v3io_frames** library and create and initialize a client object — an instance of the`Client` class.
\n", @@ -33,7 +33,6 @@ "- `write` — write one or more DataFrames to the backend.\n", "- `execute` — execute a command on the backend. Each backend may support multiple commands.\n", "\n", - "\n", "For a detailed description of the Frames API, see the [Frames documentation](https://github.com/v3io/frames/blob/development/README.md).
\n", "For more help and usage details, use the internal API help — `.?` in Jupyter Notebook or `print(..__doc__)`.
\n", "For example, the following command returns information about the read operation for a client object named `client`:\n", From 2db4ad9abf4b1c89066d6ae5fe89d7541a6a4ab3 Mon Sep 17 00:00:00 2001 From: Sharon Lifshitz Date: Wed, 25 Sep 2019 14:55:42 +0300 Subject: [PATCH 2/4] [DOC] Frames GS: replace "access token" terminology with "access key" [IG-12272 IG-12092] --- getting-started/frames.ipynb | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/getting-started/frames.ipynb b/getting-started/frames.ipynb index 9fbd69c7..3858979f 100644 --- a/getting-started/frames.ipynb +++ b/getting-started/frames.ipynb @@ -54,9 +54,9 @@ "\n", "> **Note:**\n", "> - The client constructor's `container` parameter is set to `\"users\"` for accessing data in the platform's \"users\" data container.\n", - "> - Because no authentication credentials are passed to the constructor, Frames will use the access token that's assigned to the `V3IO_ACCESS_KEY` environment variable.\n", - "> The platform's Jupyter Notebook service defines this variable automatically and initializes it to an access token for the running user of the service.\n", - "> You can pass different credentials by using the constructor's `token` parameter (platform access token) or `user` and `password` parameters (platform username and password)." + "> - Because no authentication credentials are passed to the constructor, Frames will use the access key that's assigned to the `V3IO_ACCESS_KEY` environment variable.\n", + "> The platform's Jupyter Notebook service defines this variable automatically and initializes it to a valid access key for the running user of the service.\n", + "> You can pass different credentials by using the constructor's `token` parameter (platform access key) or `user` and `password` parameters (platform username and password)." ] }, { From 86e5e2467be8539386d7dc8ba09ccd190301f9da Mon Sep 17 00:00:00 2001 From: Sharon Lifshitz Date: Wed, 25 Sep 2019 16:02:11 +0300 Subject: [PATCH 3/4] [DOC] Frames GS: Cosmetics (don't quote backend names) + minor Stream overview editing [IG-12272 IG-12092] --- getting-started/frames.ipynb | 52 +++++++++++++++++++----------------- 1 file changed, 27 insertions(+), 25 deletions(-) diff --git a/getting-started/frames.ipynb b/getting-started/frames.ipynb index 3858979f..7cf72d51 100644 --- a/getting-started/frames.ipynb +++ b/getting-started/frames.ipynb @@ -8,9 +8,9 @@ "\n", "- [Overview](#frames-overview)\n", "- [Initialization](#frames-init)\n", - "- [Working with NoSQL Tables (\"kv\" Backend)](#frames-kv)\n", - "- [Working with Time-Series Databases (\"tsdb\" Backend)](#frames-tsdb)\n", - "- [Working with Streams (\"stream\" Backend)](#frames-stream)\n", + "- [Working with NoSQL Tables (kv Backend)](#frames-kv)\n", + "- [Working with Time-Series Databases (tsdb Backend)](#frames-tsdb)\n", + "- [Working with Streams (stream Backend)](#frames-stream)\n", "- [Cleanup](#frames-cleanup)" ] }, @@ -22,7 +22,7 @@ "## Overview\n", "\n", "[V3IO Frames](https://github.com/v3io/frames) (**\"Frames\"**) is a multi-model open-source data-access library, developed by Iguazio, which provides a unified high-performance DataFrame API for working with data in the data store of the Iguazio Data Science Platform (**\"the platform\"**).\n", - "Frames currently supports the NoSQL (key/value), stream, and time-series (TSDB) data models via its `\"kv\"`, `\"stream\"`, and `\"tsdb\"` backends.\n", + "Frames currently supports the NoSQL (key/value), stream, and time-series (TSDB) data models via its `kv`, `stream`, and `tsdb` backends.\n", "\n", "To use Frames, you first need to import the **v3io_frames** library and create and initialize a client object — an instance of the`Client` class.
\n", "The `Client` class features the following object methods for supporting basic data operations:\n", @@ -78,9 +78,9 @@ "metadata": {}, "source": [ "\n", - "## Working with NoSQL Tables (\"kv\" Backend)\n", + "## Working with NoSQL Tables (kv Backend)\n", "\n", - "This section demonstrates how to use the `\"kv\"` Frames backend to write and read NoSQL data in the platform.\n", + "This section demonstrates how to use the `kv` Frames backend to write and read NoSQL data in the platform.\n", "\n", "- [Initialization](#frames-kv-init)\n", "- [Load Data from Amazon S3](frames-kv-load-data-s3)\n", @@ -311,9 +311,9 @@ "\n", "### Write to a NoSQL Table\n", "\n", - "Use the `write` method of the Frames client with the `\"kv\"` backend to write the data that was read in the previous step to a NoSQL table.
\n", + "Use the `write` method of the Frames client with the `kv` backend to write the data that was read in the previous step to a NoSQL table.
\n", "The mandatory `table` parameter specifies the relative table path within the data container that was configured for the Frames client (see the [main initialization](#frames-init) step).\n", - "In the following example, the relative table path is set by using the `table` variable that was defined in the [\"kv\" backend initialization](#frames-kv-init) step.
\n", + "In the following example, the relative table path is set by using the `table` variable that was defined in the [kv backend initialization](#frames-kv-init) step.
\n", "The `dfs` parameter can be set either to a single DataFrame (as done in the following example) or to multiple DataFrames — either as a DataFrame iterator or as a list of DataFrames." ] }, @@ -337,7 +337,7 @@ "\n", "> **Note:** To query a table in a platform data container, the table path in the `from` section of the SQL query should be of the format `v3io..\"/path/to/table\"`.\n", "> See [Presto Data Paths](https://www.iguazio.com/docs/tutorials/latest-release/getting-started/fundamentals/#data-paths-presto) in the platform documentation.\n", - "> In the following example, the path is set by using the `sql_table_path` variable that was defined in the [\"kv\" backend initialization](#frames-kv-init) step.\n", + "> In the following example, the path is set by using the `sql_table_path` variable that was defined in the [kv backend initialization](#frames-kv-init) step.\n", "> Unless you changed the code, this variable translates to `v3io.users.\"/examples/bank\"`; for example, `v3io.users.\"iguazio/examples/bank\"` for user \"iguazio\"." ] }, @@ -2330,7 +2330,7 @@ "\n", "### Read from the Table Using the Frames API\n", "\n", - "Use the `read` method of the Frames client with the `\"kv\"` backend to read data from your NoSQL table.
\n", + "Use the `read` method of the Frames client with the `kv` backend to read data from your NoSQL table.
\n", "The `read` method can return a DataFrame or a DataFrame iterator (a stream), as demonstrated in the following examples.\n", "\n", "- [Read Using a Single DataFrame](#frames-kv-read-frames-api-single-df)\n", @@ -2664,7 +2664,7 @@ "\n", "### Delete the NoSQL Table\n", "\n", - "Use the `delete` method of the Frames client with the `\"kv\"` backend to delete the NoSQL table that was used in the previous steps." + "Use the `delete` method of the Frames client with the `kv` backend to delete the NoSQL table that was used in the previous steps." ] }, { @@ -2682,9 +2682,9 @@ "metadata": {}, "source": [ "\n", - "## Working with Time-Series Databases (\"tsdb\" Backend)\n", + "## Working with Time-Series Databases (tsdb Backend)\n", "\n", - "This section demonstrates how to use the `\"tsdb\"` Frames backend to create a time-series database (TSDB) table in the platform, ingest data into the table, and read from the table (i.e., submit TSDB queries).\n", + "This section demonstrates how to use the `tsdb` Frames backend to create a time-series database (TSDB) table in the platform, ingest data into the table, and read from the table (i.e., submit TSDB queries).\n", "\n", "- [Initialization](#frames-tsdb-init)\n", "- [Create a TSDB Table](#frames-tsdb-create)\n", @@ -2721,9 +2721,9 @@ "\n", "### Create a TSDB Table\n", "\n", - "Use the `create` method of the Frames client with the `\"tsdb\"` backend to create a new TSDB table.
\n", + "Use the `create` method of the Frames client with the `tsdb` backend to create a new TSDB table.
\n", "The mandatory `table` parameter specifies the relative table path within the data container that was configured for the Frames client (see the [main initialization](#frames-init) step).\n", - "In the following example, the relative table path is set by using the `tsdb_table` variable that was defined in the [\"tsdb\" backend initialization](#frames-tsdb-init) step.
\n", + "In the following example, the relative table path is set by using the `tsdb_table` variable that was defined in the [tsdb backend initialization](#frames-tsdb-init) step.
\n", "You can optionally use the `attrs` parameter to provide additional arguments.\n", "For example, you can set the `rate` argument to the TSDB’s metric-samples ingestion rate (`\"[0-9]+/[smh]\"`; for example, `1/s`); the rate should be calculated according to the slowest expected ingestion rate." ] @@ -2745,7 +2745,7 @@ "\n", "### Write to the TSDB Table\n", "\n", - "Use the `write` method of the Frames client with the `\"tsdb\"` backend to ingest data from a pandas DataFrame into your TSDB table.
\n", + "Use the `write` method of the Frames client with the `tsdb` backend to ingest data from a pandas DataFrame into your TSDB table.
\n", "The primary-key attribute of platform TSDB tables (i.e., the DataFrame index column) must hold the sample time of the data (displayed as `time` in read outputs).
\n", "In addition, TSDB table items (rows) can optionally have sub-index columns (attributes) that are called labels.\n", "You can add labels to TSDB table items in one of two ways; you can also combine these methods:\n", @@ -2819,7 +2819,7 @@ "\n", "### Read from the TSDB Table\n", "\n", - "Use the `read` method of the Frames client with the `\"tsdb\"` backend to read data from your TSDB table (i.e., query the database)." + "Use the `read` method of the Frames client with the `tsdb` backend to read data from your TSDB table (i.e., query the database)." ] }, { @@ -2920,7 +2920,7 @@ "\n", "### Delete the TSDB Table\n", "\n", - "Use the `delete` method of the Frames client with the `\"tsdb\"` backend to delete the TSDB table that was used in the previous steps." + "Use the `delete` method of the Frames client with the `tsdb` backend to delete the TSDB table that was used in the previous steps." ] }, { @@ -2937,9 +2937,11 @@ "metadata": {}, "source": [ "\n", - "## Working with Streams (\"stream\" Backend)\n", + "## Working with Streams (stream Backend)\n", "\n", "The platform supports streams that have an AWS Kinesis-like API. For more information, see the [platform documentation](https://www.iguazio.com/docs/concepts/latest-release/streams/).\n", + "
\n", + "This section demonstrates how to use the `streams` Frames backend to work with streams in the platform.\n", "\n", "- [Initialization](#frames-stream-init)\n", "- [Create a Stream](#frames-stream-create)\n", @@ -2978,9 +2980,9 @@ "\n", "### Create a Stream\n", "\n", - "Use the `create` method of the Frames client with the `\"stream\"` backend to create a new data stream.
\n", + "Use the `create` method of the Frames client with the `stream` backend to create a new data stream.
\n", "The mandatory `table` parameter specifies the relative stream path within the data container that was configured for the Frames client (see the [main initialization](#frames-init) step).\n", - "In the following example, the relative stream path is set by using the `strm` variable that was defined in the [\"stream\" backend initialization](#frames-stream-init) step.
\n", + "In the following example, the relative stream path is set by using the `strm` variable that was defined in the [stream backend initialization](#frames-stream-init) step.
\n", "You can optionally use the `attrs` parameter to provide additional arguments.\n", "For example, you can set the `shards` argument to the number of shards in the stream, or you can set the `retention_hours` argument to the stream's retention period in hours." ] @@ -3015,7 +3017,7 @@ "\n", "#### Use the Write Method to Perform a Batch Update\n", "\n", - "Use the `write` method of the Frames client with the `\"stream\"` backend to ingest multiple records into your stream (batch update), as demonstrated in the following example.
\n", + "Use the `write` method of the Frames client with the `stream` backend to ingest multiple records into your stream (batch update), as demonstrated in the following example.
\n", "The `dfs` parameter can be set either to a single DataFrame (as done in the following example) or to multiple DataFrames — either as a DataFrame iterator or as a list of DataFrames." ] }, @@ -3044,7 +3046,7 @@ "\n", "#### Use the Execute Method's Put Command to Update a Single Record\n", "\n", - "Use the `put` command of the `execute` method of the Frames client with the `\"stream\"` backend to add a single record to a stream.
\n", + "Use the `put` command of the `execute` method of the Frames client with the `stream` backend to add a single record to a stream.
\n", "Use the `args` parameter of the `put` command to provide the necessary information:\n", "set the mandatory `data` argument to the ingested record data.\n", "You can optionally set the `clientinfo` argument to additional metadata and the `partition` argument to to a partition key; records with the same partition key are assigned to the same shard." @@ -3105,7 +3107,7 @@ "\n", "### Read from the Stream\n", "\n", - "Use the `read` method of the Frames client with the `\"stream\"` backend to read data from your stream.
\n", + "Use the `read` method of the Frames client with the `stream` backend to read data from your stream.
\n", "The mandatory `seek` parameter specifies the seek method, which determines the location within the target stream shard from which to read; some methods require setting additional parameters:\n", "\n", "- `\"earliest\"` — start from the earliest point in the shard; (no additional parameters).\n", @@ -3156,7 +3158,7 @@ "\n", "### Delete the Stream\n", "\n", - "Use the `delete` method of the Frames client with the `\"stream\"` backend to delete the TSDB table that was used in the previous steps." + "Use the `delete` method of the Frames client with the `stream` backend to delete the TSDB table that was used in the previous steps." ] }, { From d4e58bda4ad8ab5e648469b59cc86df22e0c740e Mon Sep 17 00:00:00 2001 From: Sharon Lifshitz Date: Wed, 25 Sep 2019 18:53:37 +0300 Subject: [PATCH 4/4] [DOC] Frames GS: Edit the TSDB create method attrs doc [IG-12272 IG-12092] --- getting-started/frames.ipynb | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/getting-started/frames.ipynb b/getting-started/frames.ipynb index 7cf72d51..5cc24667 100644 --- a/getting-started/frames.ipynb +++ b/getting-started/frames.ipynb @@ -2724,8 +2724,10 @@ "Use the `create` method of the Frames client with the `tsdb` backend to create a new TSDB table.
\n", "The mandatory `table` parameter specifies the relative table path within the data container that was configured for the Frames client (see the [main initialization](#frames-init) step).\n", "In the following example, the relative table path is set by using the `tsdb_table` variable that was defined in the [tsdb backend initialization](#frames-tsdb-init) step.
\n", - "You can optionally use the `attrs` parameter to provide additional arguments.\n", - "For example, you can set the `rate` argument to the TSDB’s metric-samples ingestion rate (`\"[0-9]+/[smh]\"`; for example, `1/s`); the rate should be calculated according to the slowest expected ingestion rate." + "The `attrs` parameter is used to set additional arguments.\n", + "You must set the `rate` argument to the ingestion rate of the TSDB metric-samples, as `\"[0-9]+/[smh]\"` (where `s` = seconds, `m` = minutes, and `h` = hours); for example, `1/s` (one sample per minute).\n", + "The rate should be calculated according to the slowest expected ingestion rate.\n", + "You can also set additional optional arguments, such as `aggregates` or `aggregation-granularity`." ] }, {