Skip to content

Commit

Permalink
Merge branch 'development' of github.com:v3io/tutorials
Browse files Browse the repository at this point in the history
  • Loading branch information
@Sharon-iguazio
Sharon-iguazio committed Sep 25, 2019
2 parents 4567a6d + df69f1e commit 5711a0d
Showing 1 changed file with 35 additions and 32 deletions.
67 changes: 35 additions & 32 deletions getting-started/frames.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,9 @@
"\n",
"- [Overview](#frames-overview)\n",
"- [Initialization](#frames-init)\n",
"- [Working with NoSQL Tables (\"kv\" Backend)](#frames-kv)\n",
"- [Working with Time-Series Databases (\"tsdb\" Backend)](#frames-tsdb)\n",
"- [Working with Streams (\"stream\" Backend)](#frames-stream)\n",
"- [Working with NoSQL Tables (kv Backend)](#frames-kv)\n",
"- [Working with Time-Series Databases (tsdb Backend)](#frames-tsdb)\n",
"- [Working with Streams (stream Backend)](#frames-stream)\n",
"- [Cleanup](#frames-cleanup)"
]
},
Expand All @@ -21,8 +21,8 @@
"<a id=\"frames-overview\"></a>\n",
"## Overview\n",
"\n",
"[V3IO Frames](https://github.com/v3io/frames) (**\"Frames\"**) is a multi-model open-source data-access library, developed by Iguazio, which provides a unified high-performance DataFrame API for loading, storing, and accessing data in the data store of the Iguazio Data Science Platform (**\"the platform\"**).\n",
"Frames currently supports the NoSQL (key/value), stream, and time-series (TSDB) data models via its `\"kv\"`, `\"stream\"`, and `\"tsdb\"` backends.\n",
"[V3IO Frames](https://github.com/v3io/frames) (**\"Frames\"**) is a multi-model open-source data-access library, developed by Iguazio, which provides a unified high-performance DataFrame API for working with data in the data store of the Iguazio Data Science Platform (**\"the platform\"**).\n",
"Frames currently supports the NoSQL (key/value), stream, and time-series (TSDB) data models via its `kv`, `stream`, and `tsdb` backends.\n",
"\n",
"To use Frames, you first need to import the **v3io_frames** library and create and initialize a client object &mdash; an instance of the`Client` class.<br>\n",
"The `Client` class features the following object methods for supporting basic data operations:\n",
Expand All @@ -33,7 +33,6 @@
"- `write` &mdash; write one or more DataFrames to the backend.\n",
"- `execute` &mdash; execute a command on the backend. Each backend may support multiple commands.\n",
"\n",
"\n",
"For a detailed description of the Frames API, see the [Frames documentation](https://github.com/v3io/frames/blob/development/README.md).<br>\n",
"For more help and usage details, use the internal API help &mdash; `<client object>.<command>?` in Jupyter Notebook or `print(<client object>.<command>.__doc__)`.<br>\n",
"For example, the following command returns information about the read operation for a client object named `client`:\n",
Expand All @@ -55,9 +54,9 @@
"\n",
"> **Note:**\n",
"> - The client constructor's `container` parameter is set to `\"users\"` for accessing data in the platform's \"users\" data container.\n",
"> - Because no authentication credentials are passed to the constructor, Frames will use the access token that's assigned to the `V3IO_ACCESS_KEY` environment variable.\n",
"> The platform's Jupyter Notebook service defines this variable automatically and initializes it to an access token for the running user of the service.\n",
"> You can pass different credentials by using the constructor's `token` parameter (platform access token) or `user` and `password` parameters (platform username and password)."
"> - Because no authentication credentials are passed to the constructor, Frames will use the access key that's assigned to the `V3IO_ACCESS_KEY` environment variable.\n",
"> The platform's Jupyter Notebook service defines this variable automatically and initializes it to a valid access key for the running user of the service.\n",
"> You can pass different credentials by using the constructor's `token` parameter (platform access key) or `user` and `password` parameters (platform username and password)."
]
},
{
Expand All @@ -79,9 +78,9 @@
"metadata": {},
"source": [
"<a id='frames-kv'></a>\n",
"## Working with NoSQL Tables (\"kv\" Backend)\n",
"## Working with NoSQL Tables (kv Backend)\n",
"\n",
"This section demonstrates how to use the `\"kv\"` Frames backend to write and read NoSQL data in the platform.\n",
"This section demonstrates how to use the `kv` Frames backend to write and read NoSQL data in the platform.\n",
"\n",
"- [Initialization](#frames-kv-init)\n",
"- [Load Data from Amazon S3](frames-kv-load-data-s3)\n",
Expand Down Expand Up @@ -312,9 +311,9 @@
"<a id=\"frames-kv-write\"></a>\n",
"### Write to a NoSQL Table\n",
"\n",
"Use the `write` method of the Frames client with the `\"kv\"` backend to write the data that was read in the previous step to a NoSQL table.<br>\n",
"Use the `write` method of the Frames client with the `kv` backend to write the data that was read in the previous step to a NoSQL table.<br>\n",
"The mandatory `table` parameter specifies the relative table path within the data container that was configured for the Frames client (see the [main initialization](#frames-init) step).\n",
"In the following example, the relative table path is set by using the `table` variable that was defined in the [\"kv\" backend initialization](#frames-kv-init) step.<br>\n",
"In the following example, the relative table path is set by using the `table` variable that was defined in the [kv backend initialization](#frames-kv-init) step.<br>\n",
"The `dfs` parameter can be set either to a single DataFrame (as done in the following example) or to multiple DataFrames &mdash; either as a DataFrame iterator or as a list of DataFrames."
]
},
Expand All @@ -338,7 +337,7 @@
"\n",
"> **Note:** To query a table in a platform data container, the table path in the `from` section of the SQL query should be of the format `v3io.<container name>.\"/path/to/table\"`.\n",
"> See [Presto Data Paths](https://www.iguazio.com/docs/tutorials/latest-release/getting-started/fundamentals/#data-paths-presto) in the platform documentation.\n",
"> In the following example, the path is set by using the `sql_table_path` variable that was defined in the [\"kv\" backend initialization](#frames-kv-init) step.\n",
"> In the following example, the path is set by using the `sql_table_path` variable that was defined in the [kv backend initialization](#frames-kv-init) step.\n",
"> Unless you changed the code, this variable translates to `v3io.users.\"<running user>/examples/bank\"`; for example, `v3io.users.\"iguazio/examples/bank\"` for user \"iguazio\"."
]
},
Expand Down Expand Up @@ -2331,7 +2330,7 @@
"<a id=\"frames-kv-read-frames-api\"></a>\n",
"### Read from the Table Using the Frames API\n",
"\n",
"Use the `read` method of the Frames client with the `\"kv\"` backend to read data from your NoSQL table.<br>\n",
"Use the `read` method of the Frames client with the `kv` backend to read data from your NoSQL table.<br>\n",
"The `read` method can return a DataFrame or a DataFrame iterator (a stream), as demonstrated in the following examples.\n",
"\n",
"- [Read Using a Single DataFrame](#frames-kv-read-frames-api-single-df)\n",
Expand Down Expand Up @@ -2665,7 +2664,7 @@
"<a id=\"frames-kv-delete\"></a>\n",
"### Delete the NoSQL Table\n",
"\n",
"Use the `delete` method of the Frames client with the `\"kv\"` backend to delete the NoSQL table that was used in the previous steps."
"Use the `delete` method of the Frames client with the `kv` backend to delete the NoSQL table that was used in the previous steps."
]
},
{
Expand All @@ -2683,9 +2682,9 @@
"metadata": {},
"source": [
"<a id='frames-tsdb'></a>\n",
"## Working with Time-Series Databases (\"tsdb\" Backend)\n",
"## Working with Time-Series Databases (tsdb Backend)\n",
"\n",
"This section demonstrates how to use the `\"tsdb\"` Frames backend to create a time-series database (TSDB) table in the platform, ingest data into the table, and read from the table (i.e., submit TSDB queries).\n",
"This section demonstrates how to use the `tsdb` Frames backend to create a time-series database (TSDB) table in the platform, ingest data into the table, and read from the table (i.e., submit TSDB queries).\n",
"\n",
"- [Initialization](#frames-tsdb-init)\n",
"- [Create a TSDB Table](#frames-tsdb-create)\n",
Expand Down Expand Up @@ -2722,11 +2721,13 @@
"<a id=\"frames-tsdb-create\"></a>\n",
"### Create a TSDB Table\n",
"\n",
"Use the `create` method of the Frames client with the `\"tsdb\"` backend to create a new TSDB table.<br>\n",
"Use the `create` method of the Frames client with the `tsdb` backend to create a new TSDB table.<br>\n",
"The mandatory `table` parameter specifies the relative table path within the data container that was configured for the Frames client (see the [main initialization](#frames-init) step).\n",
"In the following example, the relative table path is set by using the `tsdb_table` variable that was defined in the [\"tsdb\" backend initialization](#frames-tsdb-init) step.<br>\n",
"You can optionally use the `attrs` parameter to provide additional arguments.\n",
"For example, you can set the `rate` argument to the TSDB’s metric-samples ingestion rate (`\"[0-9]+/[smh]\"`; for example, `1/s`); the rate should be calculated according to the slowest expected ingestion rate."
"In the following example, the relative table path is set by using the `tsdb_table` variable that was defined in the [tsdb backend initialization](#frames-tsdb-init) step.<br>\n",
"The `attrs` parameter is used to set additional arguments.\n",
"You must set the `rate` argument to the ingestion rate of the TSDB metric-samples, as `\"[0-9]+/[smh]\"` (where `s` = seconds, `m` = minutes, and `h` = hours); for example, `1/s` (one sample per minute).\n",
"The rate should be calculated according to the slowest expected ingestion rate.\n",
"You can also set additional optional arguments, such as `aggregates` or `aggregation-granularity`."
]
},
{
Expand All @@ -2746,7 +2747,7 @@
"<a id=\"frames-tsdb-write\"></a>\n",
"### Write to the TSDB Table\n",
"\n",
"Use the `write` method of the Frames client with the `\"tsdb\"` backend to ingest data from a pandas DataFrame into your TSDB table.<br>\n",
"Use the `write` method of the Frames client with the `tsdb` backend to ingest data from a pandas DataFrame into your TSDB table.<br>\n",
"The primary-key attribute of platform TSDB tables (i.e., the DataFrame index column) must hold the sample time of the data (displayed as `time` in read outputs).<br>\n",
"In addition, TSDB table items (rows) can optionally have sub-index columns (attributes) that are called labels.\n",
"You can add labels to TSDB table items in one of two ways; you can also combine these methods:\n",
Expand Down Expand Up @@ -2820,7 +2821,7 @@
"<a id=\"frames-tsdb-read\"></a>\n",
"### Read from the TSDB Table\n",
"\n",
"Use the `read` method of the Frames client with the `\"tsdb\"` backend to read data from your TSDB table (i.e., query the database)."
"Use the `read` method of the Frames client with the `tsdb` backend to read data from your TSDB table (i.e., query the database)."
]
},
{
Expand Down Expand Up @@ -2921,7 +2922,7 @@
"<a id=\"frames-tsdb-delete\"></a>\n",
"### Delete the TSDB Table\n",
"\n",
"Use the `delete` method of the Frames client with the `\"tsdb\"` backend to delete the TSDB table that was used in the previous steps."
"Use the `delete` method of the Frames client with the `tsdb` backend to delete the TSDB table that was used in the previous steps."
]
},
{
Expand All @@ -2938,9 +2939,11 @@
"metadata": {},
"source": [
"<a id='frames-stream'></a>\n",
"## Working with Streams (\"stream\" Backend)\n",
"## Working with Streams (stream Backend)\n",
"\n",
"The platform supports streams that have an AWS Kinesis-like API. For more information, see the [platform documentation](https://www.iguazio.com/docs/concepts/latest-release/streams/).\n",
"<br>\n",
"This section demonstrates how to use the `streams` Frames backend to work with streams in the platform.\n",
"\n",
"- [Initialization](#frames-stream-init)\n",
"- [Create a Stream](#frames-stream-create)\n",
Expand Down Expand Up @@ -2979,9 +2982,9 @@
"<a id=\"frames-stream-create\"></a>\n",
"### Create a Stream\n",
"\n",
"Use the `create` method of the Frames client with the `\"stream\"` backend to create a new data stream.<br>\n",
"Use the `create` method of the Frames client with the `stream` backend to create a new data stream.<br>\n",
"The mandatory `table` parameter specifies the relative stream path within the data container that was configured for the Frames client (see the [main initialization](#frames-init) step).\n",
"In the following example, the relative stream path is set by using the `strm` variable that was defined in the [\"stream\" backend initialization](#frames-stream-init) step.<br>\n",
"In the following example, the relative stream path is set by using the `strm` variable that was defined in the [stream backend initialization](#frames-stream-init) step.<br>\n",
"You can optionally use the `attrs` parameter to provide additional arguments.\n",
"For example, you can set the `shards` argument to the number of shards in the stream, or you can set the `retention_hours` argument to the stream's retention period in hours."
]
Expand Down Expand Up @@ -3016,7 +3019,7 @@
"<a id=\"frames-stream-write-batch-update\"></a>\n",
"#### Use the Write Method to Perform a Batch Update\n",
"\n",
"Use the `write` method of the Frames client with the `\"stream\"` backend to ingest multiple records into your stream (batch update), as demonstrated in the following example.<br>\n",
"Use the `write` method of the Frames client with the `stream` backend to ingest multiple records into your stream (batch update), as demonstrated in the following example.<br>\n",
"The `dfs` parameter can be set either to a single DataFrame (as done in the following example) or to multiple DataFrames &mdash; either as a DataFrame iterator or as a list of DataFrames."
]
},
Expand Down Expand Up @@ -3045,7 +3048,7 @@
"<a id=\"frames-stream-execute-put\"></a>\n",
"#### Use the Execute Method's Put Command to Update a Single Record\n",
"\n",
"Use the `put` command of the `execute` method of the Frames client with the `\"stream\"` backend to add a single record to a stream.<br>\n",
"Use the `put` command of the `execute` method of the Frames client with the `stream` backend to add a single record to a stream.<br>\n",
"Use the `args` parameter of the `put` command to provide the necessary information:\n",
"set the mandatory `data` argument to the ingested record data.\n",
"You can optionally set the `clientinfo` argument to additional metadata and the `partition` argument to to a partition key; records with the same partition key are assigned to the same shard."
Expand Down Expand Up @@ -3106,7 +3109,7 @@
"<a id=\"frames-stream-read\"></a>\n",
"### Read from the Stream\n",
"\n",
"Use the `read` method of the Frames client with the `\"stream\"` backend to read data from your stream.<br>\n",
"Use the `read` method of the Frames client with the `stream` backend to read data from your stream.<br>\n",
"The mandatory `seek` parameter specifies the seek method, which determines the location within the target stream shard from which to read; some methods require setting additional parameters:\n",
"\n",
"- `\"earliest\"` &mdash; start from the earliest point in the shard; (no additional parameters).\n",
Expand Down Expand Up @@ -3157,7 +3160,7 @@
"<a id=\"frames-tsdb-stream\"></a>\n",
"### Delete the Stream\n",
"\n",
"Use the `delete` method of the Frames client with the `\"stream\"` backend to delete the TSDB table that was used in the previous steps."
"Use the `delete` method of the Frames client with the `stream` backend to delete the TSDB table that was used in the previous steps."
]
},
{
Expand Down

0 comments on commit 5711a0d

Please sign in to comment.