Skip to content

Commit

Permalink
Sprint 36 master (#13)
Browse files Browse the repository at this point in the history
* Create Get New And Updated Files Polling Trigger
* Create Upsert File Action
* Create Create Folder Action
* Create Get File Action
* Create Delete File Action
  • Loading branch information
Olha Virolainen authored Mar 26, 2020
1 parent 49c634e commit 03e49c7
Show file tree
Hide file tree
Showing 33 changed files with 1,906 additions and 79 deletions.
4 changes: 2 additions & 2 deletions .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,8 @@ jobs:
paths:
- node_modules
- run:
name: Running Mocha Unit&Integration Tests
command: npm test && npm run integration-test
name: Running Mocha Unit
command: npm test
workflows:
version: 2
build_and_test:
Expand Down
8 changes: 6 additions & 2 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
# 0.0.1 (March 2, 2020)
# 1.0.0 (March 26, 2020)

* Project Init
* Create Get New And Updated Files Polling Trigger
* Create Upsert File Action
* Create Create Folder Action
* Create Get File Action
* Create Delete File Action
85 changes: 80 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,13 @@
* [Requirements](#requirements)
* [Environment variables](#environment-variables)
* [Credentials](#credentials)

* [Triggers](#triggers)

* [Get New And Updated Files Polling](#get-new-and-updated-files-polling)
* [Actions](#actions)

* [Get File](#get-file)
* [Upsert File](#upsert-file)
* [Delete File](#delete-file)
* [Create Folder](#create-folder)
* [Known Limitations](#known-limitations)
* [License](#license)

Expand All @@ -25,23 +27,96 @@ This is the component for working with Microsoft OneDrive storage service on [el
The component provides ability to connect to Microsoft OneDrive storage service.

### Completeness Matrix
![image]()
![image](https://user-images.githubusercontent.com/16806832/77531578-d58d1280-6e9b-11ea-8802-4ed8e492f081.png)

[Completeness Matrix]()
[Completeness Matrix](https://docs.google.com/spreadsheets/d/1xXDb039POOWOKE7Iamfuz5si7Y7bX1l8mJSuDb4Gums/edit#gid=0)

## Requirements

#### Environment variables
Name|Mandatory|Description|Values|
|----|---------|-----------|------|
|`OAUTH_CLIENT_ID`| true | Microsoft Graph Application OAuth2 Client ID | Can be found in your application page on [https://portal.azure.com](https://portal.azure.com) |
|`OAUTH_CLIENT_SECRET`| true | Microsoft Graph Application OAuth2 Client Secret | Can be found in your application page on [https://portal.azure.com](https://portal.azure.com) |
|`LOG_LEVEL`| false | Controls logger level | `trace`, `debug`, `info`, `warning`, `error` |
|`ATTACHMENT_MAX_SIZE`| false | For `elastic.io` attachments configuration. Maximal possible attachment size in bytes. By default set to 1000000 and according to platform limitations CAN'T be bigger than that. | Up to `1000000` bytes|

## Credentials
To create new credentials you need to authorize in Microsoft system using OAuth2 protocol.

## Triggers
### Get New And Updated Files Polling
Triggers to get all new and updated files since last polling. Polling is provided by `lastModifiedDateTime` file's property.

#### List of Expected Config fields
* **Drive Identity** - OneDrive instance to work with. Selects by owner
* **Folder path** - Dropdown list with folder path where new and updated path should be polled
* **Emit Behaviour** - Options are: default is `Emit Individually` emits each object in separate message, `Fetch All` emits all objects in one message
* **Start Time** - Start datetime of polling. Default min date:-271821-04-20T00:00:00.000Z
* **End Time** - End datetime of polling. Default max date: +275760-09-13T00:00:00.000Z
* **Size Of Polling Page** - Indicates the size of pages to be fetched. Defaults to 1000
* **Expand Children** - checkbox for polling files from child folders
* **Enable File Attachments** - checkbox for attaching files content to action response

## Actions
### Get File
Action to get item from OneDrive by provided path in selected disc.

#### Input fields description
* **Drive Identity** - OneDrive instance to work with. Selects by owner
* **Enable File Attachments** - checkbox for attaching files content to action response
#### Metadata fields description
* **Path** - Full path to item to create or replace
#### Input example:
```
{
"path": "base_folder/inner_folder/file.any"
}
```

### Upsert File
Action upserts (create or replace) with first file from attachment by provided path in Microsoft One Drive
#### Input fields description
* **Drive Identity** - OneDrive instance to work with. Selects by owner
#### Metadata fields description
* **Path** - Full path to item to create or replace
#### Input example:
```
{
"path": "base_folder/inner_folder/file.any"
}
```

### Delete File
Action to delete item from OneDrive by provided path in selected disc.
Returns filename if file was deleted and empty message if already wasn't exist
#### Input fields description
* **Drive Identity** - OneDrive instance to work with. Selects by owner
#### Metadata fields description
* **Path** - Full path to item to delete
#### Input example:
```
{
"path": "base_folder/inner_folder/file.any"
}
```

### Create Folder
Create new folder in provided `path`. If `path` not exist component will fail.
#### Input fields description
* **Drive Identity** - OneDrive instance to work with. Selects by owner
* **Conflict Behaviour** - behaviour in case folder already exists. Default: `Fail`. Options: `Fail`, `Replace`, `Rename`.
1. `Fail` - fails if folder with same name already exists under provided `path`
2. `Rename` - rename folder if folder with same name already exists under provided `path`. Examples: `exists` -> `exists_1`, `exists_1` -> `exists_1_1`
3. `Replace` - replace folder if folder with same name already exists under provided `path`. Note: files inside folder will not be replaced, but last modified date of folder will be updated
#### Metadata fields description
* **Path** - Path to to folder where new folder will be created. Use empty string or `/` for root
* **Name** - Name of new folder

## Known Limitations

1. Maximal possible size for an attachment is 10 MB.
2. Attachments mechanism does not work with [Local Agent Installation](https://support.elastic.io/support/solutions/articles/14000076461-announcing-the-local-agent-)

## License

Expand Down
180 changes: 173 additions & 7 deletions component.json
Original file line number Diff line number Diff line change
Expand Up @@ -30,32 +30,198 @@
}
},
"triggers": {
"getNewAndUpdatedObjectsPolling": {
"main": "./lib/triggers/pollingTrigger",
"title": "Get New And Updated Files Polling",
"description": "Polling trigger",
"type": "polling",
"fields": {
"driveId": {
"label": "Drive Identity",
"required": true,
"viewClass": "SelectView",
"model": "getDisks",
"prompt": "Select a disk"
},
"itemId": {
"label": "Folder path",
"required": false,
"viewClass": "SelectView",
"model": "getFolderPaths",
"prompt": "Select a folder path"
},
"emitBehaviour": {
"label": "Emit Behaviour",
"viewClass": "SelectView",
"required": false,
"prompt": "Default Emit Individually",
"model": {
"emitIndividually": "Emit Individually",
"fetchAll": "Fetch All"
}
},
"startTime": {
"label": "Start Time",
"viewClass": "TextFieldView",
"required": false,
"note": "Indicates the beginning time to start polling from, default 1970-01-01T00:00:00.000Z",
"placeholder": "1970-01-01T00:00:00.000Z"
},
"endTime": {
"label": "End Time",
"viewClass": "TextFieldView",
"required": false,
"note": "If provided, do not fetch records modified after this time (defaults to never)",
"placeholder": "3000-01-01T00:00:00.000Z"
},
"pageSize": {
"label": "Size Of Polling Page",
"viewClass": "TextFieldView",
"required": false,
"placeholder": "Default: 1000"
},
"expandChildren": {
"label": "Expand Children",
"viewClass": "CheckBoxView"
},
"attachFile": {
"label": "Enable File Attachments",
"viewClass": "CheckBoxView"
}
}
}
},
"actions": {
"readFile": {
"title": "Dummy Action",
"description": "Dummy action to test credentials",
"main": "./lib/actions/dummyAction.js",
"lookupFile": {
"title": "Get File",
"description": "Action to get file by provided path",
"main": "./lib/actions/getFile.js",
"fields": {
"driveId": {
"label": "Drive Identity",
"required": true,
"viewClass": "SelectView",
"model": "getDisks",
"prompt": "Select a disk"
},
"attachFile": {
"label": "Enable File Attachments",
"viewClass": "CheckBoxView"
}
},
"metadata": {
"in": {
"type": "object",
"properties": {
"data": {
"path": {
"type": "string",
"required": false
"required": true
}
}
},
"out":"./lib/schemas/file.out.json"
}
},
"deleteItem": {
"title": "Delete File",
"description": "Action to delete file by provided path",
"main": "./lib/actions/deleteItem.js",
"fields": {
"driveId": {
"label": "Drive Identity",
"required": true,
"viewClass": "SelectView",
"model": "getDisks",
"prompt": "Select a disk"
}
},
"metadata": {
"in": {
"type": "object",
"properties": {
"path": {
"type": "string",
"required": true
}
}
},
"out": {
"type": "object",
"properties": {
"data": {
"path": {
"type": "string",
"required": false
}
}
}
}
},
"upsertFile": {
"title": "Upsert File",
"description": "Action upserts (create or replace) file by path",
"main": "./lib/actions/upsertFile.js",
"fields": {
"driveId": {
"label": "Drive Identity",
"required": true,
"viewClass": "SelectView",
"model": "getDisks",
"prompt": "Select a disk"
}
},
"metadata": {
"in": {
"type": "object",
"properties": {
"path": {
"type": "string",
"required": true
}
}
},
"out":"./lib/schemas/file.out.json"
}
},
"createFolder": {
"title": "Create Folder",
"description": "Action creates a new folder by provided path",
"main": "./lib/actions/createFolder.js",
"fields": {
"driveId": {
"label": "Drive Identity",
"required": true,
"viewClass": "SelectView",
"model": "getDisks",
"prompt": "Select a disk"
},
"conflictBehaviour" : {
"label": "Conflict Behaviour",
"required": false,
"viewClass": "SelectView",
"model" : {
"fail" : "Fail",
"replace": "Replace",
"rename": "Rename"
},
"prompt": "What to do if folder already exist? Default: Fail"
}
},
"metadata": {
"in": {
"type": "object",
"properties": {
"name": {
"type": "string",
"required": true
},
"path": {
"type": "string",
"required": true
}
}
},
"out":"./lib/schemas/file.out.json"
}
}
}
}
22 changes: 22 additions & 0 deletions lib/actions/createFolder.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
const { messages } = require('elasticio-node');
const { Client } = require('../client');

exports.getDisks = require('../utils/metadataDrivesProcessor').getDisks;

exports.process = async function (msg, cfg) {
this.logger.info('Create folder action started');
this.logger.trace('Input message: %j', msg);
this.logger.trace('Input configuration: %j', cfg);
const { path, name } = msg.body;
const client = new Client(this.logger, cfg);
const id = await client.isExist(path);
if (id === false || id === null || id === undefined) {
this.logger.error('Cant create folder in path: %s. Path is not exist in drive: %s', path, cfg.driveId);
throw new Error(`Error: Path: "${path}" is not exist in drive: ${cfg.driveId}`);
}
const body = await client.createFolder(id, name);
this.logger.trace('Successfully create folder in path: %s', path);
this.logger.trace('Emitting body: %j', body);
this.logger.info('Create folder finished');
return messages.newMessageWithBody({ result: body });
};
25 changes: 25 additions & 0 deletions lib/actions/deleteItem.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
const { messages } = require('elasticio-node');

const { Client } = require('../client');
const { OneDriveDelete } = require('../utils/deleteUtil');

exports.process = async function (msg, cfg) {
this.logger.trace('Input message: %j', msg);
this.logger.trace('Input configuration: %j', cfg);

const client = new Client(this.logger, cfg);
const deleteUtils = new OneDriveDelete(this.logger, client);

try {
await deleteUtils.deleteObject(msg.body.path);
return messages.newMessageWithBody({ path: msg.body.path });
} catch (err) {
if (err.message.includes('itemNotFound')) {
return messages.newEmptyMessage();
}
throw err;
}
};


exports.getDisks = require('../utils/metadataDrivesProcessor').getDisks;
7 changes: 0 additions & 7 deletions lib/actions/dummyAction.js

This file was deleted.

Loading

0 comments on commit 03e49c7

Please sign in to comment.