| Features | Description |
|---|---|
| Pipeline Structure | |
| Overview | This is the overview structure of your yaml file. |
| Stages | This is the definition and usage of a stage. |
| Tasks | This is the definition and usage of a task. |
| Functions | |
Read Type Functions |
|
| http-request | This function is used to call http request urls. |
| http-request-dynamic | This function is used to call http request urls with several dynamic params. |
Filter Type Function |
|
| xpath | This function is used for webscrapping tasks to do html/xml data filtering. |
| json-path | This function is used for api-requesting tasks to do json data filtering. |
| sql (filter) | This function is used for tabular data filtering. |
Transform Type Function |
|
| decompress | This function is used for decompressing gzip data and return the origin decompressed content. |
| flatten-lists-to-dataframe | This function is used for flattening json-path return json lists to dataframe columns. |
| json-array-to-dataframe | This function is used for turning your json-array rows into a dataframe. |
| rename-columns | This function is used for renaming your dataframe columns. |
| sampling | This function is used for taking samples from dataframes. |
| split-dataframe-rows | This function is used for turning your dataframe rows into list of dataframes. |
| string-injecting | This function is used for replacing certain substring inside the other string. |
| transform-to-dataframe | This function is used for turning your dataframe rows into list of dataframes. |
| sub-selection | This function is used for doing sub-selection of your original upstream task input. |
Merge Type Function |
|
| sql (merge) | This function is used for tabular data merging. |
Concat Type Function |
|
| concat | This function is used for the last task in a stage for concating several dataframes' data into a big dataframe. |