Skip to content

Commit

Permalink
updating documentation with the new fields for each processor
Browse files Browse the repository at this point in the history
  • Loading branch information
anmunoz committed Feb 6, 2019
1 parent 3c4ae49 commit ace42b7
Show file tree
Hide file tree
Showing 15 changed files with 28 additions and 1,255 deletions.
1,229 changes: 0 additions & 1,229 deletions docs/apiary/apiary.apib

This file was deleted.

Binary file modified docs/images/controller-postgresql.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
File renamed without changes
File renamed without changes
Binary file modified docs/images/processor-mongo.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified docs/images/processor-mysql.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified docs/images/processor-postgresql.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 1 addition & 1 deletion docs/installation_and_administration_guide/draco_gui.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ user interface usually running at 8080 port (http://localhost:8080/nifi) and ide

The next image provides you the location of many components of Draco. Please put special
attention to the template button, play button and processor component, you will use them later.
![Draco-gui](../images/cygnus-toolbar-components.png)
![Draco-gui](../images/draco-toolbar-components.png)
There are no permissions configured by default, so anyone is able to view and modify the dataflow.
For information on securing the system, see the System Administrator’s Guide.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -29,12 +29,12 @@ Where:
|Name|Default Value|Allowable Values|Description|
|--- |--- |--- |--- |
|Base Path|contentListener| |Base path for incoming connections. This property has to match with the notify attribute of the subscription made in ORION. Expression Language: true (will be evaluated using variable registry only)|
|Listening Port| | |The port to listen on for incoming connections. This value needs to be included in the subscription. Expression Language: true (will be evaluated using variable registry only)|
|Max Data to Receive per Second| | |The maximum amount of data to receive per second; this allows the bandwidth to be throttled to a specified data rate; if not specified, the data rate is not throttled|
|SSL Context Service| | | Controller Service API, RestrictedSSLContextServiceImplementation, StandardRestrictedSSLContextService|
|Listening Port|no | |The port to listen on for incoming connections. This value needs to be included in the subscription. Expression Language: true (will be evaluated using variable registry only)|
|Max Data to Receive per Second|no | |The maximum amount of data to receive per second; this allows the bandwidth to be throttled to a specified data rate; if not specified, the data rate is not throttled|
|SSL Context Service|no | | Controller Service API, RestrictedSSLContextServiceImplementation, StandardRestrictedSSLContextService|
|Authorized DN Pattern|.*| |A Regular Expression to apply against the Distinguished Name of incoming connections. If the Pattern does not match the DN, the connection will be refused. |
|Max Unconfirmed Flowfile Time|60 secs| |The maximum amount of time to wait for a FlowFile to be confirmed before it is removed from the cache |
|HTTP Headers to receive as Attributes (Regex)| | | Specifies the Regular Expression that determines the names of HTTP Headers that should be passed along as FlowFile attributes. You can capture all the headers by setting to .* otherwise you have to include at least Fiware-service, Fiware-ServicePath and Optionally X-Auth-Token|
|HTTP Headers to receive as Attributes (Regex)|no | | Specifies the Regular Expression that determines the names of HTTP Headers that should be passed along as FlowFile attributes. You can capture all the headers by setting to .* otherwise you have to include at least Fiware-service, Fiware-ServicePath and Optionally X-Auth-Token|
|Return Code|200| |The HTTP return code returned after every HTTP call. |

### NGSIToMySQL Processor
Expand All @@ -44,15 +44,15 @@ Where:

|Name|Default Value|Allowable Values|Description|
|--- |--- |--- |--- |
|JDBC Connection Pool| | |Controller service for connecting to a specific database engine|
|JDBC Connection Pool|no | |Controller service for connecting to a specific database engine|
|NGSI version| v2| |List of supported versions of NGSI (v2 and ld), currently only support v2|
|Data Model| db-by-entity | |The data model for creating the tables when an event have been received you can choose between: db-by-service-path or db-by-entity, default value is db-by-service-path|
|Attribute persistence| row | row, column | The mode of storing the data inside of the table|
|Default Service|test| |In case you dont set the Fiware-Service header in the context broker, this value will be used as Fiware-Service |
|Default Service path|/path| |In case you dont set the Fiware-ServicePath header in the context broker, this value will be used as Fiware-ServicePath |
|Enable encoding| true | true, false | True applies the new encoding, false applies the old encoding.|
|Enable lowercase|true| true, false |True for creating the Schema and Tables name with lowercase.|
|Batch size| | |The preferred number of FlowFiles to put to the database in a single transaction|
|Batch size|10 | |The preferred number of FlowFiles to put to the database in a single transaction|
|Rollback on failure |false | true, false|Specify how to handle error. By default (false), if an error occurs while processing a FlowFile, the FlowFile will be routed to 'failure' or 'retry' relationship based on error type, and processor can continue with next FlowFile. Instead, you may want to rollback currently processed FlowFiles and stop further processing immediately. In that case, you can do so by enabling this 'Rollback On Failure' property. If enabled, failed FlowFiles will stay in the input relationship without penalizing it and being processed repeatedly until it gets processed successfully or removed by other means. It is important to set adequate 'Yield Duration' to avoid retrying too frequently.|


Expand All @@ -64,15 +64,15 @@ The NGSIToPostgreSQL processor takes the FLowFile generated by the ListenHTTP pr

|Name|Default Value|Allowable Values|Description|
|--- |--- |--- |--- |
|JDBC Connection Pool| | |Controller service for connecting to a specific database engine|
|JDBC Connection Pool|no | |Controller service for connecting to a specific database engine|
|NGSI version| v2| |List of supported versions of NGSI (v2 and ld), currently only support v2|
|Data Model| db-by-entity | |The data model for creating the tables when an event have been received you can choose between: db-by-service-path or db-by-entity, default value is db-by-service-path|
|Attribute persistence| row | row, column | The mode of storing the data inside of the table|
|Default Service|test| |In case you dont set the Fiware-Service header in the context broker, this value will be used as Fiware-Service
|Default Service path|/path| |In case you dont set the Fiware-ServicePath header in the context broker, this value will be used as Fiware-ServicePath |
|Enable encoding| true | true, false | True applies the new encoding, false applies the old encoding.|
|Enable lowercase|true| true, false |True for creating the Schema and Tables name with lowercase.|
|Batch size| | |The preferred number of FlowFiles to put to the database in a single transaction|
|Batch size|10 | |The preferred number of FlowFiles to put to the database in a single transaction|
|Rollback on failure |false | true, false|Specify how to handle error. By default (false), if an error occurs while processing a FlowFile, the FlowFile will be routed to 'failure' or 'retry' relationship based on error type, and processor can continue with next FlowFile. Instead, you may want to rollback currently processed FlowFiles and stop further processing immediately. In that case, you can do so by enabling this 'Rollback On Failure' property. If enabled, failed FlowFiles will stay in the input relationship without penalizing it and being processed repeatedly until it gets processed successfully or removed by other means. It is important to set adequate 'Yield Duration' to avoid retrying too frequently.|

### NGSIToMongo Processor
Expand All @@ -96,6 +96,5 @@ The NGSIToPostgreSQL processor takes the FLowFile generated by the ListenHTTP pr
| Data Expiration | 0 | | Collections will be removed if older than the value specified in seconds. The reference of time is the one stored in the `recvTime` property. Set to 0 if not wanting this policy. |
| Collections Size | 0 | | The oldest data (according to insertion time) will be removed if the size of the data collection gets bigger than the value specified in bytes. Notice that the size-based truncation policy takes precedence over the time-based one. Set to 0 if not wanting this policy. Minimum value (different than 0) is 4096 bytes. |
| Max Documents | 0 | | The oldest data (according to insertion time) will be removed if the number of documents in the data collections goes beyond the specified value. Set to 0 if not wanting this policy. |
| Write Concern| ACKNOWLEDGED | |The write concern to use|

Once you have your processors configured, you need to connect the source (Listen HTTP processor) and the sinks (NGSIToMySQL, NGSIToPostgreSQL, NGSIToMongo ). In the next section we will explain how to establish and configure this connection.
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ You can access each template by dragging the Template button to the canvas in th
the template that you need, all the processors and controllers with a basic configuration
will be put and displayed on the Draco canvas.

![Draco-template1](../images/cygnus-template1.png)
![Draco-template1](../images/draco-template1.png)



Expand Down
7 changes: 3 additions & 4 deletions docs/processors_catalogue/ngsi_mongo_sink.md
Original file line number Diff line number Diff line change
Expand Up @@ -290,20 +290,19 @@ If `data_model=dm-by-entity` and `attr_persistence=column` then `NGSIToMongo` wi

|Name|Default Value|Allowable Values|Description|
|---|---|---|---|
| **Mongo URI** | no | localhost:27017 | FQDN/IP:port where the MongoDB server runs (standalone case) or comma-separated list of FQDN/IP:port pairs where the MongoDB replica set members run. |
| Mongo username | no | <i>empty</i> | If empty, no authentication is done. |
| Mongo password | no | <i>empty</i> | If empty, no authentication is done. |
| **Mongo URI** | no | | "MongoURI, typically of the form: mongodb://host1[:port1][,host2[:port2],...]". Also you can add user and pass, example: mongodb://user:password@host1:port |
| **NGSI version**| v2| |list of supported version of NGSI (v2 and ld), currently only support v2|
| **Data Model**| db-by-entity | |The Data model for creating the tables when an event have been received you can choose between: db-by-service-path or db-by-entity, default value is db-by-service-path|
| **Attribute persistence**| row | row, column | The mode of storing the data inside of the table allowable values are row and column|
| Default Service|test| |In case you dont set the Fiware-Service header in the context broker, this value will be used as Fiware-Service
| Default Service path|/path| |In case you dont set the Fiware-ServicePath header in the context broker, this value will be used as Fiware-ServicePath |
| Enable encoding| true | true, false | true applies the new encoding, false applies the old encoding.|
| Enable lowercase|true| true, false |true for creating the Schema and Tables name with lowercase.|
| Database prefix | sth_ | ||
| Collection prefix | sth_ | | `system.` is not accepted. |
| Data Expiration | 0 | | Collections will be removed if older than the value specified in seconds. The reference of time is the one stored in the `recvTime` property. Set to 0 if not wanting this policy. |
| Collections Size | 0 | | The oldest data (according to insertion time) will be removed if the size of the data collection gets bigger than the value specified in bytes. Notice that the size-based truncation policy takes precedence over the time-based one. Set to 0 if not wanting this policy. Minimum value (different than 0) is 4096 bytes. |
| Max Documents | 0 | | The oldest data (according to insertion time) will be removed if the number of documents in the data collections goes beyond the specified value. Set to 0 if not wanting this policy. |
| Write Concern| ACKNOWLEDGED | |The write concern to use|

A configuration example could be:

Expand Down
6 changes: 4 additions & 2 deletions docs/processors_catalogue/ngsi_mysql_sink.md
Original file line number Diff line number Diff line change
Expand Up @@ -197,13 +197,15 @@ If `attr_persistence=colum` then `NGSIToMySQL` will persist the data within the

|Name|Default Value|Allowable Values|Description|
|--- |--- |--- |--- |
|**JDBC Connection Pool**| | |Controller service for connecting to a specific database engine|
|**JDBC Connection Pool**|no | |Controller service for connecting to a specific database engine|
|**NGSI version**| v2| |list of supported version of NGSI (v2 and ld), currently only support v2|
|**Data Model**| db-by-entity | |The Data model for creating the tables when an event have been received you can choose between: db-by-service-path or db-by-entity, default value is db-by-service-path|
|**Attribute persistence**| row | row, column | The mode of storing the data inside of the table allowable values are row and column|
| Default Service|test| |In case you dont set the Fiware-Service header in the context broker, this value will be used as Fiware-Service
| Default Service path|/path| |In case you dont set the Fiware-ServicePath header in the context broker, this value will be used as Fiware-ServicePath |
|Enable encoding| true | true, false | true applies the new encoding, false applies the old encoding.|
|Enable lowercase|true| true, false |true for creating the Schema and Tables name with lowercase.|
|**Batch size**| | |The preferred number of FlowFiles to put to the database in a single transaction|
|**Batch size**|10 | |The preferred number of FlowFiles to put to the database in a single transaction|
|**Rollback on failure**|false | true, false|Specify how to handle error. By default (false), if an error occurs while processing a FlowFile, the FlowFile will be routed to 'failure' or 'retry' relationship based on error type, and processor can continue with next FlowFile. Instead, you may want to rollback currently processed FlowFiles and stop further processing immediately. In that case, you can do so by enabling this 'Rollback On Failure' property. If enabled, failed FlowFiles will stay in the input relationship without penalizing it and being processed repeatedly until it gets processed successfully or removed by other means. It is important to set adequate 'Yield Duration' to avoid retrying too frequently.|

A configuration example could be:
Expand Down
6 changes: 4 additions & 2 deletions docs/processors_catalogue/ngsi_postgresql_sink.md
Original file line number Diff line number Diff line change
Expand Up @@ -221,13 +221,15 @@ Coming soon.

|Name|Default Value|Allowable Values|Description|
|--- |--- |--- |--- |
|**JDBC Connection Pool**| | |Controller service for connecting to a specific database engine|
|**JDBC Connection Pool**|no | |Controller service for connecting to a specific database engine|
|**NGSI version**| v2| |list of supported version of NGSI (v2 and ld), currently only support v2|
|**Data Model**| db-by-entity | |The Data model for creating the tables when an event have been received you can choose between: db-by-service-path or db-by-entity, default value is db-by-service-path|
|**Attribute persistence**| row | row, column | The mode of storing the data inside of the table allowable values are row and column|
| Default Service|test| |In case you dont set the Fiware-Service header in the context broker, this value will be used as Fiware-Service
| Default Service path|/path| |In case you dont set the Fiware-ServicePath header in the context broker, this value will be used as Fiware-ServicePath |
|Enable encoding| true | true, false | true applies the new encoding, false applies the old encoding.|
|Enable lowercase|true| true, false |true for creating the Schema and Tables name with lowercase.|
|**Batch size**| | |The preferred number of FlowFiles to put to the database in a single transaction|
|**Batch size**|10 | |The preferred number of FlowFiles to put to the database in a single transaction|
|**Rollback on failure** |false | true, false|Specify how to handle error. By default (false), if an error occurs while processing a FlowFile, the FlowFile will be routed to 'failure' or 'retry' relationship based on error type, and processor can continue with next FlowFile. Instead, you may want to rollback currently processed FlowFiles and stop further processing immediately. In that case, you can do so by enabling this 'Rollback On Failure' property. If enabled, failed FlowFiles will stay in the input relationship without penalizing it and being processed repeatedly until it gets processed successfully or removed by other means. It is important to set adequate 'Yield Duration' to avoid retrying too frequently.|

A configuration example could be:
Expand Down
4 changes: 2 additions & 2 deletions docs/processors_catalogue/ngsi_rest_handler.md
Original file line number Diff line number Diff line change
Expand Up @@ -115,12 +115,12 @@ ngsi-event-2={
|Name|Default Value|Allowable Values|Description|
|--- |--- |--- |--- |
|**Base Path**|contentListener| |Base path for incoming connectionsSupports, this has to match with teh notify attribute of the subscription made in ORION Expression Language: true (will be evaluated using variable registry only)|
|**Listening Port**| | |The Port to listen on for incoming connectionsSupports, also need to be including in the subscription, Expression Language: true (will be evaluated using variable registry only)|
|**Listening Port**|no | |The Port to listen on for incoming connectionsSupports, also need to be including in the subscription, Expression Language: true (will be evaluated using variable registry only)|
|Max Data to Receive per Second| | |The maximum amount of data to receive per second; this allows the bandwidth to be throttled to a specified data rate; if not specified, the data rate is not throttled|
|SSL Context Service| |Controller Service API: RestrictedSSLContextServiceImplementation: StandardRestrictedSSLContextService|The Controller Service to use in order to obtain an SSL Context|
|Authorized DN Pattern| |.*| |A Regular Expression to apply against the Distinguished Name of incoming connections. If the Pattern does not match the DN, the connection will be refused.|
|Max Unconfirmed Flowfile Time|60 secs| |The maximum amount of time to wait for a FlowFile to be confirmed before it is removed from the cache|
|**HTTP Headers to receive as Attributes (Regex)**| | |Specifies the Regular Expression that determines the names of HTTP Headers that should be passed along as FlowFile attributes. You have to include at least Fiware-service, Fiware-Service-Path and Optionally X-Auth-Token|
|**HTTP Headers to receive as Attributes (Regex)**|no | |Specifies the Regular Expression that determines the names of HTTP Headers that should be passed along as FlowFile attributes. You have to include at least Fiware-service, Fiware-Service-Path and Optionally X-Auth-Token|
|Return Code|200| |The HTTP return code returned after every HTTP call|

A configuration example could be:
Expand Down
10 changes: 5 additions & 5 deletions docs/quick_start_guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,8 @@ Obviously, you will need docker and docker-compose installed and running in you
### <a name="section2.1"></a>Building from sources
(1) Start by cloning the `fiware-Draco` repository:

git clone https://github.com/ging/fiware-Draco.git
cd fiware-Draco
git clone https://github.com/ging/fiware-draco.git
cd fiware-draco

Change directory:

Expand All @@ -39,7 +39,7 @@ If you want, you can open another terminal and check the available images at you
```
sudo docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
Draco latest 6a9e16550c82 10 seconds ago 462.1 MB
draco latest 6a9e16550c82 10 seconds ago 462.1 MB
mysql latest 273a1eca2d3a 2 weeks ago 194.6 MB
```

Expand All @@ -52,7 +52,7 @@ First, go to your browser and open Draco using this URL http://localhost:9090/ni

The next image provides you the location of many components of Draco. Please put special
attention to the template button, play button and processor component, you will use them later.
![Draco-gui](./images/cygnus-toolbar-components.png)
![Draco-gui](./images/draco-toolbar-components.png)

Now go to the Components toolbar which is placed in the upper section of the NiFi GUI,
find the template icon and drag and drop it inside the Draco user space.
Expand All @@ -65,7 +65,7 @@ the second processor called NGSIToMySQL is in charge to get the NGSIv2 events an
persist that data into the MySQL database. The third is an optional processor for storing
the logs of the received events.

![Draco-template1](./images/cygnus-template1.png)
![Draco-template1](./images/draco-template1.png)

Before starting the processors, you need to set your MySQL password and enable the DBCConnectionPool controller.
For doing that please follow the instructions:
Expand Down

0 comments on commit ace42b7

Please sign in to comment.