diff --git a/apps/docs/versioned_docs/version-0.6.0/dev/01-intro.mdx b/apps/docs/versioned_docs/version-0.6.0/dev/01-intro.mdx new file mode 100644 index 000000000..cdb9efe14 --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/dev/01-intro.mdx @@ -0,0 +1,64 @@ +--- +sidebar_position: 1 +--- + +# Introduction for Jayvee Developers + +import MascotImageUrl from '@site/static/img/mascots/mascot3.png'; + +
+
+ +
+ + +

"On the tracks of Jayvee's internals"

+ +
+ +## How to contribute + +In order to contribute to the Jayvee project, please have a look at our [contribution guide](https://github.com/jvalue/jayvee/blob/main/CONTRIBUTING.md). Before planning a contribution, please read the [design principles](./05-design-principles.md) and consider if your changes fit the vision expressed there. + +The overall project is licensed under the `AGPL-3.0-only` license and is compliant to the [REUSE Specification](https://reuse.software/spec/) by the [Free Software Foundation Europe](https://fsfe.org/). +Because of this, contributions are required to adhere to the license and follow that specification. +More details on this topic can be found [here](./02-dev-processes/03-licensing-and-copyright.md). + +And last but not least, please read and follow our [code of conduct](https://github.com/jvalue/jayvee/blob/main/CODE_OF_CONDUCT.md). + +## Project overview + +The Jayvee repository is located [here](https://github.com/jvalue/jayvee) on GitHub. +It uses an [Nx mono-repository](https://nx.dev/) setup and contains all related projects, most notably the Jayvee language server and interpreter. +Please have a look at the [architecture overview](./03-architecture-overview.md) for more details. + +## Prerequisites + +Node.js and npm are required for development. +It is recommended to use their LTS version to avoid any potential compatibility issues. +Also, refer to this [quick start guide](https://github.com/jvalue/jayvee#development-quickstart) for developers. + +In order to run the Jayvee VS Code extension during development, VS Code is required as IDE. +Using VS Code also allows installing the [Langium VS Code extension](https://marketplace.visualstudio.com/items?itemName=langium.langium-vscode) for better IDE support when working with Langium grammar files. + +## Resources for getting started + +### Langium + +- [**Langium documentation**](https://langium.org/docs/) +- Practical examples using Langium: + - [Langium examples](https://github.com/langium/langium/tree/main/examples): Official example languages by Langium + - [Langium's grammar language](https://github.com/langium/langium/tree/main/packages/langium): The implementation of Langium's own grammar language + - [Language server for SQL](https://github.com/langium/langium-sql): An implementation of a language server for SQL + - [MiniLogo DSL](https://github.com/langium/langium-minilogo): A domain-specific language for drawing pictures + - [Lox language](https://github.com/langium/langium-lox): An implementation of the Lox scripting language (also see the "Crafting Interpreters" book below) + - [SimpleUI DSL](https://github.com/TypeFox/langium-ui-framework): A domain-specific language for generating user interfaces + - [STPA-DSL](https://github.com/kieler/stpa): A domain-specific language for System-Theoretic Process Analysis +- [Langium playground](https://langium.org/playground/): A tool for testing Langium grammars and visualizing resulting ASTs +- [Typefox blog](https://www.typefox.io/blog/): A blog by the company behind Langium, mostly posting Langium-related content. + +### Building compilers / interpreters / DSLs + +- [Crafting Interpreters](https://craftinginterpreters.com/contents.html): A book by Robert Nystrom on implementing an interpreter for the Lox scripting language +- [DSL Engineering](https://voelter.de/dslbook/markusvoelter-dslengineering-1.0.pdf): A book by Markus Voelter on designing, implementing and using domain-specific languages +- [Awesome Compilers](https://github.com/aalhour/awesome-compilers#readme): A list featuring many resources on compilers and interpreters diff --git a/apps/docs/versioned_docs/version-0.6.0/dev/01-intro.mdx.license b/apps/docs/versioned_docs/version-0.6.0/dev/01-intro.mdx.license new file mode 100644 index 000000000..17c5d2bad --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/dev/01-intro.mdx.license @@ -0,0 +1,3 @@ +SPDX-FileCopyrightText: 2023 Friedrich-Alexander-Universitat Erlangen-Nurnberg + +SPDX-License-Identifier: AGPL-3.0-only diff --git a/apps/docs/versioned_docs/version-0.6.0/dev/02-dev-processes/01-rfc-process.md b/apps/docs/versioned_docs/version-0.6.0/dev/02-dev-processes/01-rfc-process.md new file mode 100644 index 000000000..074440850 --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/dev/02-dev-processes/01-rfc-process.md @@ -0,0 +1,13 @@ +--- +title: Language Design Process (RFCs) +sidebar_position: 1 +--- + +We use RFCs to discuss changes to the language before implementing them. You can have a look at all closed (accepted / rejected) RFCs [here](https://github.com/jvalue/jayvee/pulls?q=is%3Apr+is%3Aclosed+RFC+), and all RFCs under discussion [here](https://github.com/jvalue/jayvee/pulls?q=is%3Apr+is%3Aopen+RFC). + +If you want to contribute an RFC please follow these steps: +1. Make a copy of the **template** at `rfc/0000-rfc-template.md` and place it into the `rfc` folder. +2. Create a draft for the RFC on a new branch. Follow the `TODOs` in template to do so. +3. Open a pull request with prefix `RFC ` in the title. +4. Address the reviews. Consider opening a new PR if major things need to be addressed and the discussion log becomes too confusing. +5. Once accepted, create an issue with the necessary steps to implement the RFC. \ No newline at end of file diff --git a/apps/docs/versioned_docs/version-0.6.0/dev/02-dev-processes/01-rfc-process.md.license b/apps/docs/versioned_docs/version-0.6.0/dev/02-dev-processes/01-rfc-process.md.license new file mode 100644 index 000000000..17c5d2bad --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/dev/02-dev-processes/01-rfc-process.md.license @@ -0,0 +1,3 @@ +SPDX-FileCopyrightText: 2023 Friedrich-Alexander-Universitat Erlangen-Nurnberg + +SPDX-License-Identifier: AGPL-3.0-only diff --git a/apps/docs/versioned_docs/version-0.6.0/dev/02-dev-processes/02-debug-vs-code-extension.md b/apps/docs/versioned_docs/version-0.6.0/dev/02-dev-processes/02-debug-vs-code-extension.md new file mode 100644 index 000000000..5fbcf07cd --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/dev/02-dev-processes/02-debug-vs-code-extension.md @@ -0,0 +1,25 @@ +--- +title: Debugging via the VS Code extension +sidebar_position: 2 +--- + +The VS Code extension of Jayvee can be used to interactively debug the language server. +During development, when using VS Code as IDE, another instance of VS Code can be opened which has the most recent, locally built VS Code extension loaded. +This can be achieved using the launch configuration `Run extension` the `Run and Debug` side-menu of VS Code or by pressing the `F5` key if that launch configuration is already selected there. + +Note that there is no file watching mechanism involved. +So in order to reflect changes to the source code, the additional VS Code instance has to be **closed and reopened** as described above. + +## How to attach a debugger + +1. Use the launch configuration `Run extension` to open the additional VS Code instance with the extension loaded +2. Use the launch configuration `Attach to Language Server` to attach the debugger + +Any set breakpoints should now be marked as active and pause the execution once they are reached. + +## How to view logs of the language server + +In the additional VS Code instance, it is possible to view the logs of the language server. +They might also be helpful for debugging since any uncaught errors are logged with their stack trace by the Langium framework. + +To view the logs, open the bottom panel called `Output` and select `Jayvee` in the dropdown menu. diff --git a/apps/docs/versioned_docs/version-0.6.0/dev/02-dev-processes/02-debug-vs-code-extension.md.license b/apps/docs/versioned_docs/version-0.6.0/dev/02-dev-processes/02-debug-vs-code-extension.md.license new file mode 100644 index 000000000..17c5d2bad --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/dev/02-dev-processes/02-debug-vs-code-extension.md.license @@ -0,0 +1,3 @@ +SPDX-FileCopyrightText: 2023 Friedrich-Alexander-Universitat Erlangen-Nurnberg + +SPDX-License-Identifier: AGPL-3.0-only diff --git a/apps/docs/versioned_docs/version-0.6.0/dev/02-dev-processes/03-licensing-and-copyright.md b/apps/docs/versioned_docs/version-0.6.0/dev/02-dev-processes/03-licensing-and-copyright.md new file mode 100644 index 000000000..201e21ca7 --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/dev/02-dev-processes/03-licensing-and-copyright.md @@ -0,0 +1,69 @@ +--- +title: Licensing and copyright +sidebar_position: 3 +--- + +The [Jayvee repository](https://github.com/jvalue/jayvee) is REUSE compliant, meaning that it fulfills the +[REUSE Specification](https://reuse.software/spec/) by the [Free Software Foundation Europe](https://fsfe.org/) (FSFE). +This makes clear under which license each project file is licensed under and who owns the copyright, not only for +humans but also for machines. + +This is done by explicitly adding copyright and licensing information to each file of the project. This is achieved +by either using a comment header or a separate `*.license` file in case comments are not possible. + +See more information. + +## What license is used in this project and who is the copyright holder? + +The entire project is licensed under [AGPL-3.0-only](https://spdx.org/licenses/AGPL-3.0-only.html), the +license file can be found [here](https://github.com/jvalue/jayvee/blob/main/LICENSES/AGPL-3.0-only.txt). +The copyright holder is the [Friedrich-Alexander-Universität Erlangen-Nürnberg](https://www.fau.eu/). + +## How to submit a REUSE compliant contribution? + +In case you want to contribute to the project, you will need to ensure that all of your contributed files are REUSE +compliant. In order to achieve this, you need to add a key-value pair for both copyright and licensing information +following this schema: + +``` +SPDX-FileCopyrightText: 2023 Friedrich-Alexander-Universitat Erlangen-Nurnberg + +SPDX-License-Identifier: AGPL-3.0-only +``` + +In case a file allows comments, use single-line comments to add the copyright and licensing information at the top +of the file. Otherwise, create a corresponding `*.license` file with the text above as its content. You can have a +look at some existing project files to get an impression on it is done in practice. + +For files with common file extensions, you can use the [reuse CLI tool](https://github.com/fsfe/reuse-tool) to add +licensing and copyright information automatically. + +For more details, you can have a look at the [Getting Started tutorial](https://reuse.software/tutorial/) on the REUSE +website. + +## How to validate REUSE compliance? + +When you make a contribution and open a new pull request, the CI checks whether your contribution is REUSE compliant +using the [reuse CLI tool](https://github.com/fsfe/reuse-tool). + +In order to validate REUSE compliance in your local development environment, you have to install the +[reuse CLI tool](https://github.com/fsfe/reuse-tool) and run the following command in the projects' root folder: + +```bash +reuse lint +``` + +You can also set up a pre-commit hook, so the above command is run before each commit. +See [here](https://reuse.readthedocs.io/en/latest/readme.html#run-as-pre-commit-hook) for details on how to set it up. + +## How to hide `*.license` files in IDEs + +During development, the file explorer of your IDE may be cluttered due to the numerous `*.license` files in the +project. Luckily, most IDEs allow hiding certain files, e.g. by specifying a pattern to exclude them from the +explorer. + +Below, you can find instructions on how to hide `*.license` files in commonly used IDEs: +- **Visual Studio Code**: Add `**/*.license` to the +[`files.exclude` setting](https://code.visualstudio.com/docs/getstarted/userinterface#_explorer) +- **WebStorm**: Add `*.license` to the +[excluded files setting](https://www.jetbrains.com/help/webstorm/configuring-project-structure.html#exclude-by-pattern) diff --git a/apps/docs/versioned_docs/version-0.6.0/dev/02-dev-processes/03-licensing-and-copyright.md.license b/apps/docs/versioned_docs/version-0.6.0/dev/02-dev-processes/03-licensing-and-copyright.md.license new file mode 100644 index 000000000..17c5d2bad --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/dev/02-dev-processes/03-licensing-and-copyright.md.license @@ -0,0 +1,3 @@ +SPDX-FileCopyrightText: 2023 Friedrich-Alexander-Universitat Erlangen-Nurnberg + +SPDX-License-Identifier: AGPL-3.0-only diff --git a/apps/docs/versioned_docs/version-0.6.0/dev/02-dev-processes/04-release-jayvee-version.md b/apps/docs/versioned_docs/version-0.6.0/dev/02-dev-processes/04-release-jayvee-version.md new file mode 100644 index 000000000..374f3468b --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/dev/02-dev-processes/04-release-jayvee-version.md @@ -0,0 +1,22 @@ +--- +title: Releasing a new Jayvee version +sidebar_position: 4 +--- + +## Version Numbers + +In this early stage of the project we do not yet follow [semantic versioning](https://semver.org/) since we expect the introduction of breaking changes frequently. +To indicate that, we only release alpha versions where the `version` is incremented with every release. +- For the npm packages, we use the version `0.0.`. +- For the GitHub releases, we use the git tag `v0.0.-alpha`. + +## Jayvee Release Procedure + +For releasing a new version of Jayvee, you need to complete the following steps: + +1. Increment the version in the `package.json` file. +2. Run `npm i` to update the `package-lock.json`. +3. Run `npx nx run docs:version-snapshot` to generate a snapshot of the docs for the previous version. +4. If you are on a feature or dev branch, merge into main. +5. Create a GitHub release on the main branch. Attach a changelog. +6. The CI/CD will deal with the rest. diff --git a/apps/docs/versioned_docs/version-0.6.0/dev/02-dev-processes/04-release-jayvee-version.md.license b/apps/docs/versioned_docs/version-0.6.0/dev/02-dev-processes/04-release-jayvee-version.md.license new file mode 100644 index 000000000..17c5d2bad --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/dev/02-dev-processes/04-release-jayvee-version.md.license @@ -0,0 +1,3 @@ +SPDX-FileCopyrightText: 2023 Friedrich-Alexander-Universitat Erlangen-Nurnberg + +SPDX-License-Identifier: AGPL-3.0-only diff --git a/apps/docs/versioned_docs/version-0.6.0/dev/02-dev-processes/_category_.json b/apps/docs/versioned_docs/version-0.6.0/dev/02-dev-processes/_category_.json new file mode 100644 index 000000000..c8f512450 --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/dev/02-dev-processes/_category_.json @@ -0,0 +1,8 @@ +{ + "label": "Development Process", + "position": 2, + "link": { + "type": "generated-index", + "description": "Here you can find general processes around developing Jayvee." + } +} diff --git a/apps/docs/versioned_docs/version-0.6.0/dev/02-dev-processes/_category_.json.license b/apps/docs/versioned_docs/version-0.6.0/dev/02-dev-processes/_category_.json.license new file mode 100644 index 000000000..17c5d2bad --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/dev/02-dev-processes/_category_.json.license @@ -0,0 +1,3 @@ +SPDX-FileCopyrightText: 2023 Friedrich-Alexander-Universitat Erlangen-Nurnberg + +SPDX-License-Identifier: AGPL-3.0-only diff --git a/apps/docs/versioned_docs/version-0.6.0/dev/03-architecture-overview.md b/apps/docs/versioned_docs/version-0.6.0/dev/03-architecture-overview.md new file mode 100644 index 000000000..b609a6135 --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/dev/03-architecture-overview.md @@ -0,0 +1,51 @@ +--- +title: Architecture overview +sidebar_position: 4 +--- + +Jayvee has a clear separation between the language itself and its execution. +This guide gives an overview of the overall architecture. + +## Language Server + +On the pure language side, the central project is the [language server](https://github.com/jvalue/jayvee/tree/main/libs/language-server) which is heavily built upon the [Langium framework](https://langium.org/). +It contains the syntax definition (i.e. the grammar) and is capable of performing static semantic analysis on models, so invalid models can be rejected and errors are reported to the user. +It uses the [Language Server Protocol](https://microsoft.github.io/language-server-protocol/) (LSP) for communicating with IDEs in order to provide common features such as diagnostics, auto completion and much more. + +**Note:** The [Langium framework](https://langium.org/) generate TypeScript files for the abstract syntax tree (AST), based on the grammar specification. +The following locations might be especially helpful to understand the grammar and its AST: + +- The Langium grammar files (see [here](https://github.com/jvalue/jayvee/tree/main/libs/language-server/src/grammar) or locally at `libs/language-server/src/grammar`; with `.langium` file ending). These files define the **syntax of the language**. +- The generated TypeScript AST files (execute `npm run generate` to generate them at `libs/language-server/src/lib/ast/generated` in your local repository). These files include **TypeScript interfaces for AST nodes** (e.g., `BlockDefinition`) and **guard methods** (e.g., `isBlockDefinition`). + They reflect the input of the grammar files regarding naming. +- The remaining source files of the language server implement the language server protocol (LSP) and the additional validations beyond the syntax of Jayvee. + +## Interpreter + +The Jayvee [interpreter](https://github.com/jvalue/jayvee/tree/main/apps/interpreter) on the other hand is capable of running Jayvee models. +Therefore, it uses the language server as a library to perform lexing, parsing and semantic analysis on given models. +In case no errors were found during these phases, it executes the model by interpreting it. +This means that models are not compiled to any other language, like a compiler does, but rather directly executed on the fly without any code generation involved. +The interpreter comes with a command-line interface (CLI) for users, so they are able to execute Jayvee models easily. + +The following diagram visualizes the architecture described so far: + +```mermaid +graph LR +model[Jayvee Model] --> lexical(Lexical Analysis) +subgraph Jayvee Interpreter + subgraph Jayvee Language Server + subgraph Langium Framework + lexical --> syntax(Syntax Analysis) + end + syntax --> semantic(Semantic Analysis) + end + semantic --> interpretation(Interpretation) +end +``` + +## Jayvee Extensions + +Lastly, there are Jayvee extensions for adding additional features to Jayvee. +See [here](./04-guides/06-jayvee-extensions.md) for more details. +It is worth pointing out that extensions also follow the separation between language and execution described above. diff --git a/apps/docs/versioned_docs/version-0.6.0/dev/03-architecture-overview.md.license b/apps/docs/versioned_docs/version-0.6.0/dev/03-architecture-overview.md.license new file mode 100644 index 000000000..17c5d2bad --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/dev/03-architecture-overview.md.license @@ -0,0 +1,3 @@ +SPDX-FileCopyrightText: 2023 Friedrich-Alexander-Universitat Erlangen-Nurnberg + +SPDX-License-Identifier: AGPL-3.0-only diff --git a/apps/docs/versioned_docs/version-0.6.0/dev/04-guides/01-jayvee-grammar.md b/apps/docs/versioned_docs/version-0.6.0/dev/04-guides/01-jayvee-grammar.md new file mode 100644 index 000000000..356fb09de --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/dev/04-guides/01-jayvee-grammar.md @@ -0,0 +1,34 @@ +--- +title: The Jayvee grammar +sidebar_position: 1 +--- + +The grammar of Jayvee describes the syntax and structure of the language. +It is located in the language server project. +The grammar itself is written in [Langium's own grammar language](https://langium.org/docs/grammar-language/) which is similar to [Xtext](https://www.eclipse.org/Xtext/) grammars. +Such grammar files are easily identifiable via their `.langium` file extension. + +The grammar is used to generate TypeScript interfaces that represent the Abstract Syntax Tree (AST) and different, semantically equivalent files to define syntax highlighting for different applications. +For instance, a [TextMate](https://macromates.com/manual/en/language_grammars) file is generated for the syntax highlighting in the Jayvee VS Code extension whereas a [Monarch](https://microsoft.github.io/monaco-editor/monarch.html) file is generated for the syntax highlighting in the Monaco editor. + +For further information on the grammar language of Langium, visit the corresponding [Langium documentation](https://langium.org/docs/grammar-language/). + +## Working with the grammar + +To run the code generation, either use `npm run generate` for solely the code generation or `npm run build` for an entire build. The code generation also generates further code, like the standard library. + +Whenever the grammar is changed and the code generation is run during development, it is advisory to **close and reopen the IDE**, so the changes are noticed and the file indexing is updated. + +## How to rename AST nodes + +Renaming AST nodes is not as straight forward as one might initially assume. +When renaming individual rules in the grammar, just the generated TypeScript code in `ast.ts` will reflect the renaming, but not the rest of the codebase. + +The following steps explain how to rename an AST node, so the change is reflected in the entire codebase. As an example, an AST node called `A` is supposed to be renamed to `B`: + +- Open the `ast.ts` file in the language server project +- Locate the `A` interface / type and the `isA` typeguard +- Use the rename feature by the IDE to perform the renaming from `A` to `B` and from `isA` to `isB` +- Open the grammar and locate the `A` rule +- Use the rename feature by the Langium VS Code extension to perform the renaming from `A` to `B` +- Run `npm run generate` diff --git a/apps/docs/versioned_docs/version-0.6.0/dev/04-guides/01-jayvee-grammar.md.license b/apps/docs/versioned_docs/version-0.6.0/dev/04-guides/01-jayvee-grammar.md.license new file mode 100644 index 000000000..17c5d2bad --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/dev/04-guides/01-jayvee-grammar.md.license @@ -0,0 +1,3 @@ +SPDX-FileCopyrightText: 2023 Friedrich-Alexander-Universitat Erlangen-Nurnberg + +SPDX-License-Identifier: AGPL-3.0-only diff --git a/apps/docs/versioned_docs/version-0.6.0/dev/04-guides/02-working-with-the-ast.md b/apps/docs/versioned_docs/version-0.6.0/dev/04-guides/02-working-with-the-ast.md new file mode 100644 index 000000000..bb5a300a3 --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/dev/04-guides/02-working-with-the-ast.md @@ -0,0 +1,150 @@ +--- +title: Working with the AST +sidebar_position: 2 +--- + +The nodes of the Abstract Syntax Tree (AST) consist of types and interfaces generated from the language grammar. +See [here](./01-jayvee-grammar.md) for more information on that topic. + +The following sections provide practical guides and tips for working with nodes of the AST. + +## Dealing with potentially incomplete AST nodes + +According to the generated interfaces, properties of AST nodes are never `undefined`. +In practice however, this is not always the case. + +For example, consider the language server being confronted with an incomplete Jayvee model with syntax errors. +In such cases, properties of AST nodes are in fact `undefined`, despite their interface definition. +In the interpreter however, after lexical and syntactic analysis succeeded, it can be assumed that all AST nodes are complete (i.e. they have no undefined properties) and even that all references are resolved. + +In order to avoid accessing properties of AST nodes that are potentially undefined, it is recommended to access them via the [`?.` operator](https://www.typescriptlang.org/docs/handbook/release-notes/typescript-3-7.html#optional-chaining) rather than the regular `.` operator. +Note that this might result in a conflict with the ESLint rule [`@typescript-eslint/no-unnecessary-condition`](https://typescript-eslint.io/rules/no-unnecessary-condition/), so it has to be disabled manually for such cases. + +For disabling the rule in an entire file, place this comment below the copyright and licensing header: + +```typescript +/* eslint-disable @typescript-eslint/no-unnecessary-condition */ +``` + +For just a single line, place this comment above that particular line: + +```typescript +// eslint-disable-next-line @typescript-eslint/no-unnecessary-condition +``` + +
+ +Full example + +Consider an exemplary AST node `A` with a property `x` of type `string`. To access that property safely: + +```typescript +import { A } from './ast' + +const astNode: A; + +// eslint-disable-next-line @typescript-eslint/no-unnecessary-condition +const property: string | undefined = astNode?.x; +``` + +
+ +## Usage of `assertUnreachable` + +Most times, it is beneficial to make case distinctions exhaustive, especially when working with AST nodes, properties with a [union literal type](https://www.typescriptlang.org/docs/handbook/2/everyday-types.html#literal-types) or enums. +Exhaustiveness in this context means, that the TypeScript compiler is supposed to yield an error if a case distinction does not cover all possibilities. + +Langium offers a function called `assertUnreachable` which is capable of enforcing exhaustiveness and producing compiler errors in case of violations. See the following examples to get an idea on how to use it in practice: + +
+ +Example for an exhaustive switch statement on a union literal type + +```typescript +import { assertUnreachable } from 'langium'; + +const operator: '+' | '-'; + +switch(operator) { + case '+': { + // ... + break; + } + case '-': { + // ... + break; + } + default: { + // To ensure the switch being exhaustive on `operator`: + assertUnreachable(operator); + } +} +``` + +
+ +
+ +Example for an exhaustive if-elseif-else cascade using typeguards + +Consider the exemplary AST nodes `A`, `B` and `C` and that `A = B | C`: + +```typescript +import { assertUnreachable } from 'langium'; +import { A, B, isB, C, isC } from './ast' + +const astNode: A; + +if (isB(astNode)) { + // `astNode` has type `B` here +} else if (isC(astNode)) { + // `astNode` has type `C` here +} else { + // To ensure the if-elseif-else cascade being exhaustive on `astNode`: + assertUnreachable(astNode); +} +``` + +
+ +## Usage of `assert` for expressing runtime expectations + +During development, it may occur that a certain condition is expected to **be always true** at runtime. +For example, an AST node being of a certain type or a property being defined. +The type system of TypeScript is not always able to infer such facts, so developers may have to express these expectations explicitly in their code. +Examples are [type assertions](https://www.typescriptlang.org/docs/handbook/advanced-types.html) or the [non-null assertion operator](https://www.typescriptlang.org/docs/handbook/release-notes/typescript-2-0.html#non-null-assertion-operator). +Their usage may be problematic in case the condition does not always hold, e.g. due to a bug in the program or a wrong expectation by the programmer. +In such cases, it is hard to locate the origin and debug the program because such operations are erased in the compiled JavaScript code. + +To avoid these issues, it is recommended to express such expectations as boolean expressions that are actually validated at runtime. +This can be easily achieved by using the `assert` function. +It evaluates a given boolean expression and throws an `AssertionError` in case it evaluates to `false`. +After calling `assert`, the type system of TypeScript assumes the condition to be `true` and afterwards narrows types accordingly. + +Here is an example of how to use it in practice: + +```typescript +// We use the `assert` function from the `assert` library, not `node:assert` (to preserve browser compatibility) + +// eslint-disable-next-line unicorn/prefer-node-protocol +import { strict as assert } from 'assert'; + +import { A, B, isB } from './ast'; + +const astNode: A; +assert(isB(astNode)); +// Here `astNode` has type `B` + +const referenced = astNode?.reference?.ref; +assert(referenced !== undefined); +// Here `referred` is not `undefined` +``` + +## AST wrapper classes + +The generated interfaces for AST nodes in `ast.ts` are only meant to represent the AST structurally, they don't define any behavior. +Also, in case of syntactic sugar, there may be different kinds of AST nodes representing the same semantic language concept. + +To cope with this problem, there is the concept of an `AstNodeWrapper`. +An AST node wrapper is capable of wrapping AST nodes that represent the same semantic language concept and adding behavior to them via custom methods. +To get an impression on how this can be done in practice, please have a look at the [`PipeWrapper` class](https://github.com/jvalue/jayvee/blob/main/libs/language-server/src/lib/ast/wrappers/pipe-wrapper.ts) and the [`AstNodeWrapper` interface](https://github.com/jvalue/jayvee/blob/main/libs/language-server/src/lib/ast/wrappers/ast-node-wrapper.ts). diff --git a/apps/docs/versioned_docs/version-0.6.0/dev/04-guides/02-working-with-the-ast.md.license b/apps/docs/versioned_docs/version-0.6.0/dev/04-guides/02-working-with-the-ast.md.license new file mode 100644 index 000000000..17c5d2bad --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/dev/04-guides/02-working-with-the-ast.md.license @@ -0,0 +1,3 @@ +SPDX-FileCopyrightText: 2023 Friedrich-Alexander-Universitat Erlangen-Nurnberg + +SPDX-License-Identifier: AGPL-3.0-only diff --git a/apps/docs/versioned_docs/version-0.6.0/dev/04-guides/03-validation-and-diagnostics.md b/apps/docs/versioned_docs/version-0.6.0/dev/04-guides/03-validation-and-diagnostics.md new file mode 100644 index 000000000..328d57285 --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/dev/04-guides/03-validation-and-diagnostics.md @@ -0,0 +1,64 @@ +--- +title: Validation and diagnostics +sidebar_position: 3 +--- + +Validation in the context of Jayvee can be understood as semantic analysis of Jayvee models. +The purpose is to uncover errors in models besides lexing, parsing and linking errors that the Langium framework already detects out of the box. +In case such errors are found, the language server makes the IDE display a diagnostic. +This means that the location of the error is underlined and an error message is shown. + +For example, each pipeline is expected to contain at least one starting block that requires no input. +Therefore, the language server analyzes every pipeline in a Jayvee model and checks for the presence of such a starting block. +In case no such block is found, the IDE underlines the pipeline definition in a red color and displays an error message. + +## How to implement validation + +### For arbitrary AST nodes + +In the file [`validation-registry.ts`](https://github.com/jvalue/jayvee/blob/main/libs/language-server/src/lib/validation/validation-registry.ts) of the language server, there is a registry containing validation functions for various AST nodes. +A validation function provided for a certain kind of AST node is called for every occurrence of such a node in a Jayvee model. +E.g. when registering a validation function for `PipelineDefinition` nodes, that function gets called once for each pipeline. + +A validation function always operates on a single, concrete AST node. +Besides the AST node, a `ValidationContext` object is passed to the function for reporting diagnostics via its `accept` method. + +The overall validation for a specific kind of AST node is usually subdivided into smaller checks that are called sequentially. +This potentially allows aborting the validation early, i.e. before all checks were run. +Aborting early may be useful if errors were reported during previous checks. +This can be determined via the `ValidationContext` which keeps track whether any errors occurred so far. + +A practical example where this plays a role are blocks: +When the type of a block is unknown to the language server, it makes no sense to validate inputs / outputs and properties of that block. +So, in case an error was reported during the check of the block type, the validation of that block is discontinued at that point. + +### For properties of blocks and constraints + +Validating property values of blocks / constraints works a bit differently because their individual validation is already incorporated into the overall validation framework. + +In general, the validation logic for a property values is located in the meta information of the block type / constraint type. +There, properties are defined using the `PropertySpecification` interface. +In order to add validation logic to a certain property, a validation function needs to be added to that property. + +For more complex validations that need to take multiple property values into account, it is also possible to provide a more general validation function which operates on the entire `PropertyBody` AST node. + +See [`text-range-selector-meta-inf.ts`](https://github.com/jvalue/jayvee/blob/main/libs/extensions/std/lang/src/text-range-selector-meta-inf.ts) for an example which includes both cases described above. + +## Reporting diagnostics + +During validation, diagnostics can be reported by calling the `accept` method of the given `ValidationContext` object. + +The call requires the severity of the diagnostic (`error`, `warning`, `info` or `hint`), the error message and its location. +The location is described by a `DiagnosticInfo` object which contains the affected AST node and optionally some further restrictions. +For more details, have a look at the [`DiagnosticInfo` interface](https://github.com/langium/langium/blob/main/packages/langium/src/validation/validation-registry.ts) by Langium. + +## Common issues + +When working on validations, a diagnostic with the following message may unexpectedly occur: `An error occurred during validation: ...` + +Such a diagnostic is generated by the Langium framework if an error object is thrown within the validation. +In most cases, the reason is the access of an `undefined` AST node property or an `AssertionError`. +For more details on such errors and how to avoid them, have a look at the documentation on [how to work with the AST](./02-working-with-the-ast.md). + +One way locate the origin of such errors is to debug the Jayvee VS Code extension, see [here](../02-dev-processes/02-debug-vs-code-extension.md) for more details. +Another possibility is to debug the interpreter and set appropriate breakpoints in the language server codebase. diff --git a/apps/docs/versioned_docs/version-0.6.0/dev/04-guides/03-validation-and-diagnostics.md.license b/apps/docs/versioned_docs/version-0.6.0/dev/04-guides/03-validation-and-diagnostics.md.license new file mode 100644 index 000000000..17c5d2bad --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/dev/04-guides/03-validation-and-diagnostics.md.license @@ -0,0 +1,3 @@ +SPDX-FileCopyrightText: 2023 Friedrich-Alexander-Universitat Erlangen-Nurnberg + +SPDX-License-Identifier: AGPL-3.0-only diff --git a/apps/docs/versioned_docs/version-0.6.0/dev/04-guides/04-expressions-and-operators.md b/apps/docs/versioned_docs/version-0.6.0/dev/04-guides/04-expressions-and-operators.md new file mode 100644 index 000000000..d201864e1 --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/dev/04-guides/04-expressions-and-operators.md @@ -0,0 +1,151 @@ +--- +title: Expressions and operators +sidebar_position: 4 +--- + +Jayvee supports arbitrarily nested expressions and offers a variety of different operators. +Such expressions are similar to those used in programming languages, and they are intended to be read and understood intuitively. + +The following sections explain the definition of expressions in the language grammar, their type inference mechanism, and how the evaluation of expressions works. +As a small practical example, you may also have a look at the [arithmetics example by Langium](https://github.com/langium/langium/tree/main/examples/arithmetics) which has a heavy focus on mathematical expressions and functions. + +## Expressions in the grammar + +Expressions in the Jayvee grammar are defined in [`expression.langium`](https://github.com/jvalue/jayvee/blob/main/libs/language-server/src/grammar/expression.langium) and consist of operators (unary / binary / ternary) and literals. +Unary operators only have a single operand (e.g. the `not` operator), binary operators require two operands (e.g. the `*` operator) and ternary operators require three operands. + +The grammar is written in a way that literals end up in the leaves of the resulting AST and the nodes above represent the operators. + +As an example, have a look at the following AST structure of the expression `(-3 + 5) * 7`: + +```mermaid +classDiagram + +class `:BinaryOperator` { + operator = '*' +} + +%% Uses spaces in the name so it is unique +class ` :BinaryOperator` { + operator = '+' +} + +class `:UnaryOperator` { + operator = '-' +} + +%% Uses spaces in the name so it is unique +class ` :NumericLiteral` { + value = 3 +} + +%% Uses spaces in the name so it is unique +class ` :NumericLiteral` { + value = 5 +} + +class `:NumericLiteral` { + value = 7 +} + +`:BinaryOperator` --> ` :BinaryOperator` : leftOperand +`:BinaryOperator` --> `:NumericLiteral` : rightOperand +` :BinaryOperator` --> `:UnaryOperator` : leftOperand +`:UnaryOperator` --> ` :NumericLiteral` : operand +` :BinaryOperator` --> ` :NumericLiteral` : rightOperand +``` + +The AST is constructed in a way that ensures an unambiguous evaluation order when using depth-first search. +This implies that the AST structure already reflects the precedence and associativity of the operators, and that parentheses do not need to be explicitly represented. + +For more details on how such a grammar for expressions can be realized, see the [official Langium documentation](https://langium.org/docs/grammar-language/#tree-rewriting-actions), [this blog post](https://www.typefox.io/blog/parsing-expressions-with-xtext) and the following two sections which discuss the precedence and associativity of operators. + +### Precedence of operators + +The grammar implicitly defines the precedence of operators. +The operator precedence defines the order in which operators are evaluated within an expression. +For example, multiplication has a higher precedence than addition, which leads to multiplications being performed before additions. +In order to manually override such precedence conventions, parentheses can be used in expressions. + +The diagram below shows a more extensive precedence hierarchy, focused on common arithmetic operators. +Note that the hierarchy is arranged in ascending order, according to the operator precedence. +Such an order is similar to how operators in the actual Jayvee grammar are arranged: + +```mermaid +graph TD + subgraph BinaryOperators + add/sub(Addition Operator / Subtraction operator) --> mul/div(Multiplication Operator / Division Operator) + mul/div --> exp/root(Exponential Operator / Root Operator) + end + subgraph UnaryOperators + exp/root --> plus/minus(Plus Operator / Minus Operator) + end + plus/minus --> literal(Numeric Literal) +``` + +In the Jayvee grammar, every level of precedence is represented by a single grammar rule. +Within each such rule, the grammar rule which defines the operators with the next higher precedence is referenced. +E.g. the `AdditiveExpression` rule refers to the `MultiplicativeExpression` rule because multiplicative operators are supposed to have the next higher precedence. + +In order to alter the precedence of operators, their hierarchy of grammar rules has to be adjusted accordingly in [`expression.langium`](https://github.com/jvalue/jayvee/blob/main/libs/language-server/src/grammar/expression.langium). + +### Associativity of operators + +The associativity of operators defines the order in which operators with the same precedence are evaluated when they appear in succession without parentheses. +Operators may be either **left-associative**, **right-associative** or **non-associative**. +For example, depending on the associativity of the binary `+` operator, the expression `a + b + c` has different semantics: + +- Left-associative: `(a + b) + c` (evaluation from left to right) +- Right-associative: `a + (b + c)` (evaluation from right to left) +- Non-associative: _syntax error_, the operator cannot be chained + +The associativity of operators is also defined in the Jayvee grammar, more specifically by the structure of the grammar rules that define operators. +For details on how to encode the different kinds of associativity in grammar rules, have a look at the "Associativity" section near the end of [this blog post](https://www.typefox.io/blog/parsing-expressions-with-xtext). +The patterns described in the linked article can be found in [`expression.langium`](https://github.com/jvalue/jayvee/blob/main/libs/language-server/src/grammar/expression.langium), so each operator grammar rule encodes its own associativity. + +## Typing of expressions + +Jayvee has a mechanism for inferring and validating the type of a given expression. +This is achieved using a recursive algorithm which performs depth-first search on a given expression: + +The base case for the algorithm are literals, i.e. the tree leaves of an expression. +Depending on the type of literal, their type can be inferred trivially (in case of value literals) or otherwise from the context where the expression is located. + +For operators, the types of their operands are first inferred via recursion, and then it is checked whether they are supported by the operator. +Next, given the operand types, the resulting type is computed. +Such behavior is defined in a _type computer class_ located [here](https://github.com/jvalue/jayvee/tree/main/libs/language-server/src/lib/ast/expressions/type-computers). +Additionally, in [`operator-registry.ts`](https://github.com/jvalue/jayvee/blob/main/libs/language-server/src/lib/ast/expressions/operator-registry.ts), a type computer is registered for each kind of operator. + +In case the algorithm fails to infer a type, e.g. due to unsupported operand types or unresolved references in literals, the resulting type is `undefined`. +In order to report diagnostics in such cases, a `ValidationContext` object can be supplied when calling the type inference. + +## Evaluation of expressions + +The evaluation has the purpose of computing a single value out of an expression. +For a successful evaluation, it is a **precondition** that **a type could successfully be inferred** from the respective expression. +Also, the **value for each free variable** in that expression (i.e. literals resembling placeholders for values) needs to be **known beforehand**. +Therefore, an `ExecutionContext` object is passed to the evaluation which holds values for free variables in the current context. + +### Algorithm + +The algorithm for evaluating expressions is also based on a recursive depth-first search, similar to how the type inference works: + +Again, literals are the base case. A value literal trivially evaluates to its own value. +Other literals that resemble free variables evaluate to their value provided by the given `ExecutionContext` object. + +Regarding operators, their operands first are evaluated recursively. +Next, using the concrete operand values, the operator computes its resulting value. +This is defined in operator evaluator classes located [here](https://github.com/jvalue/jayvee/tree/main/libs/language-server/src/lib/ast/expressions/evaluators). +The classes are registered in [operator-registry.ts](https://github.com/jvalue/jayvee/blob/main/libs/language-server/src/lib/ast/expressions/operator-registry.ts) for every operator, similar to the previously mentioned type computers. + +The result of an evaluation may be `undefined` in case any errors occurred. +If the preconditions were all met, such errors are most likely arithmetic errors (like division by zero). +It is possible to provide a `ValidationContext` object to the evaluation for reporting such errors as diagnostics. + +### Evaluation strategies + +There are different evaluation strategies to choose from. +They affect the way, expressions are evaluated and thus have an impact on their semantics: + +- `exhaustive`: A full depth-first search is performed, all parts of an expression are evaluated. +- `lazy`: An evaluation with the least effort is performed. Logical operators use [short circuit semantics](https://en.wikipedia.org/wiki/Short-circuit_evaluation) and binary operators don't evaluate their right operand if the left one evaluated to `undefined`. diff --git a/apps/docs/versioned_docs/version-0.6.0/dev/04-guides/04-expressions-and-operators.md.license b/apps/docs/versioned_docs/version-0.6.0/dev/04-guides/04-expressions-and-operators.md.license new file mode 100644 index 000000000..17c5d2bad --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/dev/04-guides/04-expressions-and-operators.md.license @@ -0,0 +1,3 @@ +SPDX-FileCopyrightText: 2023 Friedrich-Alexander-Universitat Erlangen-Nurnberg + +SPDX-License-Identifier: AGPL-3.0-only diff --git a/apps/docs/versioned_docs/version-0.6.0/dev/04-guides/05-standard-library.md b/apps/docs/versioned_docs/version-0.6.0/dev/04-guides/05-standard-library.md new file mode 100644 index 000000000..2da1646f0 --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/dev/04-guides/05-standard-library.md @@ -0,0 +1,48 @@ +--- +title: Working with the Standard Library +sidebar_position: 5 +--- + +Jayvee ships with its own standard library on board, including the most often used _value types_, transformations, and so on. +The standard library itself is written in `.jv` files [here](https://github.com/jvalue/jayvee/tree/main/libs/language-server/src/stdlib/). + +## Standard Library Contents + +The following elements are part of the standard library: + +## Builtin Contents + +The implementations of built-in contents are not expressed in Jayvee itself but on the TypeScript layer. Examples: + +- **Builtin value types**: These _value types_ are the base for defining user-defined _value types_ in Jayvee, e.g., `text`, `integer`, `decimal`, `boolean`. +- **Builtin io types**: These _io types_ are used to describe in inputs and outputs of _block types_, e.g., `Sheet`, `File`. +- **Builtin block types**: These _block types_ are the very basic building _blocks_ in Jayvee, e.g., `HttpExtractor`, `SqliteLoader`. +- **Builtin constraint types**: These _constraint types_ are constraints with custom logic, e.g., `LengthConstraint`, `RegexConstraint`. + +Builtin definitions are usually generated and added to the standard library from the internal representations of the concepts. + +### User-defined Contents + +The implementations of user-defined contents are expressed in Jayvee itself. Examples: + +- **User-defined value types**: These _value types_ are based on built-in or other user-defined _value types_. Their definition is expressed natively in Jayvee, e.g., `Percent`. +- **User-defined block types**: These _block types_ are based on built-in or other user-defined _block types_. Their definition is expressed natively in Jayvee. + +We use `jv` files to add user-defined _value types_ to the standard library (see below). + +## Extending the Standard Library + +Just add `jv` files to the directory [here](https://github.com/jvalue/jayvee/tree/main/libs/language-server/src/stdlib/). It is crawled hierarchically, meaning that you can also organize files in folders. + +## Implementation + +### 1. Code generation + +We use code generation to transform these `.jv` files into TypeScript files that the language server can used. The [generation script](https://github.com/jvalue/jayvee/tree/main/tools/scripts/language-server/generate-stdlib.mjs) is run via `npm run generate` next to the AST generation. + +### 2. Builtin libraries + +The solution we chose to implement the standard library mechanism is close to the [built-in library tutorial](https://langium.org/guides/builtin-library/) by Langium. The following components are of interest: + +- [JayveeWorkspaceManager](https://github.com/jvalue/jayvee/tree/main/libs/language-server/src/lib/builtin-library/jayvee-workspace-manager.ts) in the `language-server` that registers all libraries with the langium framework. +- [StandardLibraryFileSystemProvider](https://github.com/jvalue/jayvee/tree/main/apps/vs-code-extension/src/standard-library-file-system-provider.ts) in the `vs-code-extension` that registers all libraries with the vscode plugin framework. diff --git a/apps/docs/versioned_docs/version-0.6.0/dev/04-guides/05-standard-library.md.license b/apps/docs/versioned_docs/version-0.6.0/dev/04-guides/05-standard-library.md.license new file mode 100644 index 000000000..17c5d2bad --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/dev/04-guides/05-standard-library.md.license @@ -0,0 +1,3 @@ +SPDX-FileCopyrightText: 2023 Friedrich-Alexander-Universitat Erlangen-Nurnberg + +SPDX-License-Identifier: AGPL-3.0-only diff --git a/apps/docs/versioned_docs/version-0.6.0/dev/04-guides/06-jayvee-extensions.md b/apps/docs/versioned_docs/version-0.6.0/dev/04-guides/06-jayvee-extensions.md new file mode 100644 index 000000000..e10d24313 --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/dev/04-guides/06-jayvee-extensions.md @@ -0,0 +1,190 @@ +--- +title: Jayvee Extensions +sidebar_position: 6 +--- + +## Concepts + +### Jayvee extension + +A Jayvee extension is used to add new block types to the language without having to modify the actual grammar of the language. Such a Jayvee extension usually consists of two parts: a language extension and an execution extension which are described in the upcoming two sections. + +Jayvee extensions that shall be used by default are bundled into the so-called [standard extension](https://github.com/jvalue/jayvee/tree/main/libs/extensions/std). That way, the [language server](https://github.com/jvalue/jayvee/tree/main/libs/language-server) and the [interpreter](https://github.com/jvalue/jayvee/tree/main/apps/interpreter) are able to load them out of the box. + +### Language extension + +A language extension defines meta information of block types which are required by the +[language server](https://github.com/jvalue/jayvee/tree/main/libs/language-server). +Such meta information describes properties of +block types such as their names, input / output types and their properties. + +Note that language extensions don't define any behavior. Instead, this is done by the corresponding execution extension. + +### Execution extension + +An execution extension defines behavior for block types. They build on the meta information from the corresponding +language extension, e.g. input / output types of the block need to match the signature of the execution method and +properties are accessed by their specified name. + +Execution extensions are only required by the [interpreter](https://github.com/jvalue/jayvee/tree/main/apps/interpreter) and not necessarily by the [language server](https://github.com/jvalue/jayvee/tree/main/libs/language-server) as they solely define behavior. + +## Recipes + +### Add a new Jayvee execution extension + +#### 1. Generate an execution libraries + +```bash +npx nx g @nx/node:library --name="extensions//exec" +``` + +#### 2. Create extension classes + +In `libs/extensions//exec/src/extension.ts`: + +```typescript +import { + BlockExecutorClass, + JayveeExecExtension, +} from '@jvalue/jayvee-execution'; + +export class MyExecExtension extends JayveeExecExtension { + getBlockExecutors(): BlockExecutorClass[] { + return []; + } +} +``` + +#### 3. Export extension classes + +In `libs/extensions//exec/src/index.ts`: + +```typescript +export * from './extension'; +``` + +#### 4. Register new extension classes in the standard extension + +In `libs/extensions/std/exec/src/extension.ts`: + +```typescript +// ... + +import { MyExecExtension } from '@jvalue/jayvee-extensions//exec'; + +export class StdExecExtension extends JayveeExecExtension { + private readonly wrappedExtensions: JayveeExecExtension[] = [ + // ... + // Register your execution extension here: + new MyExecExtension(), + // ... + ]; + + // ... +} +``` + +### Add a new block type in a Jayvee extension + +#### 1. Create a built-in block type + +Define the syntax of the new _block type_ in the [language server's built-in _block types_](https://github.com/jvalue/jayvee/tree/main/libs/language-server/src/stdlib/builtin-blocktypes). + +The following example defines a block type `MyExtractor` with a text property called `url` and a property `retries` with a default value: + +```jayvee +builtin blocktype MyExtractor { + input default oftype None; + output default oftype Sheet; + + property url oftype text; + property retries oftype interger: 10; +} +``` + +The new block type will be automatically registered on the language server startup. + +#### 2. Add custom validation logic (if required) + +If the block type and/or its properties requires custom validation logic, you can implement it in the [language server's block type specific checks](https://github.com/jvalue/jayvee/tree/main/libs/language-server/src/lib/validation/checks/blocktype-specific). + +#### 3. Implement `BlockExecutor` + +The following example implements an executor for the previously defined block type `MyExtractor`. + +The `execute` method defines the behavior when a block is executed. Its signature matches the input and output types defined in `MyExtractor.jv` file. + +In `libs/extensions//exec/src/lib/my-extractor-executor.ts`: + +```typescript +import * as R from '@jvalue/jayvee-execution'; +import { + AbstractBlockExecutor, + BlockExecutorClass, + ExecutionContext, + Sheet, + implementsStatic, +} from '@jvalue/jayvee-execution'; +import { IOType, PrimitiveValuetypes } from '@jvalue/jayvee-language-server'; + +@implementsStatic() +export class MyExtractorExecutor + extends AbstractBlockExecutor +{ + // Needs to match the type in meta information: + public static readonly type = 'MyExtractor'; + + public readonly inputType = IOType.NONE; + public readonly outputType = IOType.SHEET; + + async doExecute( + input: None, + context: ExecutionContext, + ): Promise> { + // Accessing property values by their name: + const url = context.getPropertyValue( + 'url', + PrimitiveValuetypes.Text, + ); + const limit = context.getPropertyValue( + 'limit', + PrimitiveValuetypes.Integer, + ); + + // ... + + if (error) { + return R.err(...); + } + + return R.ok(...); + } +} +``` + +> **Info** +> The interface `BlockExecutor` is used as an API for block executors. The abstract class `AbstractBlockExecutor` gives some further functionality for free, e.g., debug logging. + +> **Warning** +> The generic types of `AbstractBlockExecutor` need to match the input and output types of the corresponding _block type_ definition. + +#### 4. Register the new `BlockExecutor` in the execution extension + +In `libs/extensions//exec/src/extension.ts`: + +```typescript +// ... + +import { MyExtractorExecutor } from './lib/my-extractor-executor'; + +export class MyExecExtension extends JayveeExecExtension { + getBlockExecutors(): BlockExecutorClass[] { + return [ + // ... + // Register your block executor here: + MyExtractorExecutor, + // ... + ]; + } +} +``` diff --git a/apps/docs/versioned_docs/version-0.6.0/dev/04-guides/06-jayvee-extensions.md.license b/apps/docs/versioned_docs/version-0.6.0/dev/04-guides/06-jayvee-extensions.md.license new file mode 100644 index 000000000..17c5d2bad --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/dev/04-guides/06-jayvee-extensions.md.license @@ -0,0 +1,3 @@ +SPDX-FileCopyrightText: 2023 Friedrich-Alexander-Universitat Erlangen-Nurnberg + +SPDX-License-Identifier: AGPL-3.0-only diff --git a/apps/docs/versioned_docs/version-0.6.0/dev/04-guides/_category_.json b/apps/docs/versioned_docs/version-0.6.0/dev/04-guides/_category_.json new file mode 100644 index 000000000..80f6fce8f --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/dev/04-guides/_category_.json @@ -0,0 +1,8 @@ +{ + "label": "Guides", + "position": 4, + "link": { + "type": "generated-index", + "description": "Here you can find guides that will help you developing certain aspects of Jayvee." + } +} diff --git a/apps/docs/versioned_docs/version-0.6.0/dev/04-guides/_category_.json.license b/apps/docs/versioned_docs/version-0.6.0/dev/04-guides/_category_.json.license new file mode 100644 index 000000000..17c5d2bad --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/dev/04-guides/_category_.json.license @@ -0,0 +1,3 @@ +SPDX-FileCopyrightText: 2023 Friedrich-Alexander-Universitat Erlangen-Nurnberg + +SPDX-License-Identifier: AGPL-3.0-only diff --git a/apps/docs/versioned_docs/version-0.6.0/dev/05-design-principles.md b/apps/docs/versioned_docs/version-0.6.0/dev/05-design-principles.md new file mode 100644 index 000000000..8b15dd8c3 --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/dev/05-design-principles.md @@ -0,0 +1,24 @@ +--- +title: Design Principles +sidebar_position: 5 +--- + +When deciding on new features for the domain-specific language itself, we try to adhere to the following high level guidelines. Of course, none of these statements is set in stone and every decision is a tradeoff. + +## Jayvee Manifesto +_Inspired by the [Agile Manifesto](https://agilemanifesto.org/)._ + +We are uncovering better ways of _modeling data pipelines by providing a domain-specific language for data engineering and making it easy for everyone to participate in it_. + +Through this work we have come to value: + +1. **Describing a goal state** over how to get there. +2. **Explicit modeling** over hidden magic. +3. **Composition** over inheritance. +4. **Flat structures** over deep nesting. + +That is, while there is value in the items on the right, we value the items on the left more. + +Through this work, we also have come to explore: + +5. **Libraries** over language features. \ No newline at end of file diff --git a/apps/docs/versioned_docs/version-0.6.0/dev/05-design-principles.md.license b/apps/docs/versioned_docs/version-0.6.0/dev/05-design-principles.md.license new file mode 100644 index 000000000..17c5d2bad --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/dev/05-design-principles.md.license @@ -0,0 +1,3 @@ +SPDX-FileCopyrightText: 2023 Friedrich-Alexander-Universitat Erlangen-Nurnberg + +SPDX-License-Identifier: AGPL-3.0-only diff --git a/apps/docs/versioned_docs/version-0.6.0/dev/12-jayvee-testing.md b/apps/docs/versioned_docs/version-0.6.0/dev/12-jayvee-testing.md new file mode 100644 index 000000000..47b086edf --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/dev/12-jayvee-testing.md @@ -0,0 +1,259 @@ +--- +title: Writing tests for Jayvee +--- + +In order to ensure that Jayvee works as intended and to catch breaking changes, we have implemented the following components for regression testing: + +- Testing utils: utils to create Langium Typescript objects from \*.jv test assets (see [here](#testing-utils)) as well as mocks for execution testing (see [here](#testing-utils-1)) +- [Grammar tests](#grammar-tests): test the grammar parsing and validation +- [Execution tests](#execution-tests): test the execution of blocks + +## Conventions + +All of the existing tests follow these conventions: + +1. The `.spec.ts` file is located next to the `.ts` file itself. +2. The `*.jv` assets are located inside a `test/assets/` folder. + Take a look at one of the exisiting tests for more details. + +## Grammar tests + +These kind of tests are mainly located inside the [language-server](https://github.com/jvalue/jayvee/tree/main/libs/language-server) as well as the language parts of each extension (for example [std/lang](https://github.com/jvalue/jayvee/tree/main/libs/extensions/std/lang)). + +### Testing utils + +The testing utils are located inside the `language-server` in a dedicated [test folder](https://github.com/jvalue/jayvee/tree/main/libs/language-server/src/test). +These utils can be imported using `@jvalue/jayvee-language-server/test` and contain the following parts: + +[**langium-utils.ts**](https://github.com/jvalue/jayvee/blob/main/libs/language-server/src/test/langium-utils.ts): +This utils file contains two functions: + +- `parseHelper` to simplify parsing the input (content of a \*.jv file) and returning the corresponding `LangiumDocument`, and +- `validationHelper` parse and validate the created document. + They are kept in a separate file due to being copied from the Langium repository and thus subject to a different code license and copyright. + +[**utils.ts**](https://github.com/jvalue/jayvee/blob/main/libs/language-server/src/test/utils.ts): +This file contains custom testing utility utils functions, like `readJvTestAssetHelper` for reading jv test assets. +Example: + +```ts +import path from 'node:path'; + +import { createJayveeServices } from '@jvalue/jayvee-language-server'; +import { + ParseHelperOptions, + expectNoParserAndLexerErrors, + parseHelper, + readJvTestAssetHelper, +} from '@jvalue/jayvee-language-server/test'; +import { AstNode, LangiumDocument } from 'langium'; + +describe('My example test', () => { + let parse: ( + input: string, + options?: ParseHelperOptions, + ) => Promise>; + + const readJvTestAsset = readJvTestAssetHelper( + __dirname, + '../../test/assets/', // localized path to test assets folder + ); + + beforeAll(() => { + // [...] register extensions etc + const services = createJayveeServices(NodeFileSystem).Jayvee; // Or retrieve them if services already exist + // Parse function for Jayvee (without validation) + parse = parseHelper(services); + }); + + // [...] + + it('My dummy test', () => { + const text = readJvTestAsset('/.jv'); + + const document = await parse(text); + expectNoParserAndLexerErrors(document); + // Rest of test + }); +}); +``` + +If you want to simply validate the test assets, simply replace `parseHelper` with `validationHelper` (and adjust the types). +You can find detailed documentation of all the utility functions directly in the code. + +[**extension/**](https://github.com/jvalue/jayvee/tree/main/libs/language-server/src/test/extension): +This folder contains a Jayvee extension for testing. +If there are certain blocks required for testing a certain feature, they can be defined here. +One such example is the already defined `TestProperty` block which has a multitude of different properties, each with a different type. +This block is used for testing properties and property-assignments. +The extension provides loader and extractor blocks for all IOTypes without any properties. +These blocks are automatically generated at runtime with the following naming scheme: +`Test${ioType}${io === 'input' ? 'Loader' : 'Extractor'}` (Example: `TestFileExtractor`). +This allows for easy (grammar) testing of non loader/extractor blocks: + +```jv +pipeline Pipeline { + + TestExtractor -> BlockUnderTest -> TestLoader; + + block BlockUnderTest oftype CellWriter { + at: range A1:A3; + write: ['values', 'to', 'write']; + } + + block TestExtractor oftype TestSheetExtractor { } + block TestLoader oftype TestSheetLoader { } +} +``` + +### Existing tests + +Currently there are already tests for the following parts: + +- Language-server validation checks (located [here](https://github.com/jvalue/jayvee/tree/main/libs/language-server/src/lib/validation)) +- Language-server constraint validation (located [here](https://github.com/jvalue/jayvee/tree/main/libs/language-server/src/lib/constraint)) +- Custom block (property) validation of the three existing extensions (std extension located [here](https://github.com/jvalue/jayvee/blob/main/libs/extensions/std/lang/src)) +- Grammar validation tests for all official full examples from the [/example](https://github.com/jvalue/jayvee/tree/main/example) folder (located [here](https://github.com/jvalue/jayvee/blob/main/libs/extensions/std/lang/src/example-validation.spec.ts)) +- Grammar validation tests for all block examples of the std extension (located [here](https://github.com/jvalue/jayvee/blob/main/libs/extensions/std/lang/src/meta-inf-example-validation.spec.ts)) + +## Execution tests + +These kind of tests are mainly located inside the [interpreter](https://github.com/jvalue/jayvee/tree/main/libs/language-server), the [interpreter-lib](https://github.com/jvalue/jayvee/tree/main/libs/interpreter-lib), the [execution lib](https://github.com/jvalue/jayvee/tree/main/libs/execution) as well as the execution parts of each extension (for example [std/exec](https://github.com/jvalue/jayvee/tree/main/libs/extensions/std/exec)). + +### Testing utils + +The testing utils for execution tests are spread between the extensions, with the interfaces and base utils located inside the [execution lib](https://github.com/jvalue/jayvee/tree/main/libs/execution). +They can be imported using `@jvalue/jayvee-extensions/rdbms/test`, `@jvalue/jayvee-extensions/std/test` and `@jvalue/jayvee-execution/test`. + +[**block-executor-mocks.ts**](https://github.com/jvalue/jayvee/blob/main/libs/execution/test/block-executor-mock.ts): +`BlockExecutorMock` interface for defining mocks for `AbstractBlockExecutor`. Generally only loader and executor blocks require mocks, because they interact with "the outside world" (i.e. `HttpExtractor` making http calls). +Due to how vastly different each `BlockExecutor` can be, this interface is very simple, containing only a `setup(...args: unknown[])` and a `restore()` method. See below for existing implementations. + +[**rdbms/exec/test**](https://github.com/jvalue/jayvee/tree/main/libs/extensions/rdbms/exec/test): +Contains the implementation of `BlockExecutorMock` for `PostgresLoaderExecutor` and `SQLiteLoaderExecutor`. +Both of these executors are mocked using `vi.mock` to mock the corresponding libraries (`pg` and `sqlite3`) +**Usage:** + +```ts +import { + PostgresLoaderExecutorMock, + SQLiteLoaderExecutorMock, +} from '@jvalue/jayvee-extensions/rdbms/test'; + +// Global mocking of external library at the top of test file required, +// even though the mocking is encapsulated in helper classes +vi.mock('pg', () => { + const mClient = { + connect: vi.fn(), + query: vi.fn(), + end: vi.fn(), + }; + return { + default: { + Client: vi.fn(() => mClient), + }, + }; +}); +vi.mock('sqlite3', () => { + const mockDB = { + close: vi.fn(), + run: vi.fn(), + }; + return { + default: { + Database: vi.fn(() => mockDB), + }, + }; +}); + +describe('Dummy describe', () => { + // [...] + + let postgresLoaderMock: PostgresLoaderExecutorMock; + let sqliteLoaderMock: SQLiteLoaderExecutorMock; + + beforeAll(() => { + postgresLoaderMock = new PostgresLoaderExecutorMock(); + sqliteLoaderMock = new SQLiteLoaderExecutorMock(); + }); + + afterEach(() => { + postgresLoaderMock.restore(); + sqliteLoaderMock.restore(); + }); + + it('Dummy test', async () => { + // Prepare mocks + postgresLoaderMock.setup(); + sqliteLoaderMock.setup(); + + // [...] execute test + + expect(postgresLoaderMock.pgClient.connect).toBeCalledTimes(1); + expect(postgresLoaderMock.pgClient.query).toBeCalledTimes(1); + expect(postgresLoaderMock.pgClient.end).toBeCalledTimes(1); + expect(sqliteLoaderMock.sqliteClient.run).toBeCalledTimes(1); + expect(sqliteLoaderMock.sqliteClient.close).toBeCalledTimes(1); + }); +}); +``` + +[**std/exec/test/mocks**](https://github.com/jvalue/jayvee/tree/main/libs/extensions/std/exec/test): +Contains the implementation of `BlockExecutorMock` for `HttpExtractorExecutorMock`. +This implementation uses [nock](https://www.npmjs.com/package/nock) for mocking HTTP(S) responses. +The `setup` method is further specified requiring one parameter `registerMocks: () => Array`, which returns all used `nock.Scope` (i.e. the return value of `nock('')`), see usage below: +**Usage:** + +```ts +import path from 'node:path'; + +import { HttpExtractorExecutorMock } from '@jvalue/jayvee-extensions/std/test'; + +describe('Dummy describe', () => { + // [...] + + let httpExtractorMock: HttpExtractorExecutorMock; + + beforeAll(() => { + httpExtractorMock = new HttpExtractorExecutorMock(); + }); + + afterEach(() => { + httpExtractorMock.restore(); + }); + + it('should have no errors when executing gtfs-static-and-rt.jv example', async () => { + // Prepare mocks + httpExtractorMock.setup(() => { + return [ + nock('') + .get('') + .replyWithFile( + 200, + path.resolve(__dirname, '../test/assets/file1.zip'), + { + 'Content-Type': 'application/octet-stream', + }, + ), + nock('') + .get('') + .replyWithFile(200, path.resolve(__dirname, '../test/assets/file2'), { + 'Content-Type': 'application/octet-stream', + }) + .get('') + .reply(200, { content: 'My dummy json reply.' }), + ]; + }); + + // [...] execute test + + expect(httpExtractorMock.nockScopes.every((scope) => scope.isDone())); + }); +}); +``` + +### Existing tests + +Currently there are already tests for the following parts: + +- Smoke test for official examples (located [here](https://github.com/jvalue/jayvee/blob/main/apps/interpreter/src/examples-smoke-test.spec.ts)) diff --git a/apps/docs/versioned_docs/version-0.6.0/dev/12-jayvee-testing.md.license b/apps/docs/versioned_docs/version-0.6.0/dev/12-jayvee-testing.md.license new file mode 100644 index 000000000..17c5d2bad --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/dev/12-jayvee-testing.md.license @@ -0,0 +1,3 @@ +SPDX-FileCopyrightText: 2023 Friedrich-Alexander-Universitat Erlangen-Nurnberg + +SPDX-License-Identifier: AGPL-3.0-only diff --git a/apps/docs/versioned_docs/version-0.6.0/user/block-types/ArchiveInterpreter.md b/apps/docs/versioned_docs/version-0.6.0/user/block-types/ArchiveInterpreter.md new file mode 100644 index 000000000..2a622e276 --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/block-types/ArchiveInterpreter.md @@ -0,0 +1,33 @@ +--- +title: ArchiveInterpreter +--- + + + +Input type: `File` + +Output type: `FileSystem` + +## Description + +Interprets a `File` as an archive file and converts it to a `FileSystem`. The archive file root is considered the root of the `FileSystem`. + +## Example 1 + +```jayvee + block ZipArchiveInterpreter oftype ArchiveInterpreter { + archiveType: "zip"; + } +``` + +Interprets a `File` as a ZIP-archive and creates a `FileSystem` of its extracted contents. + +## Properties + +### `archiveType` + +Type `text` + +#### Description + +The archive type to be interpreted, e.g., "zip" or "gz". diff --git a/apps/docs/versioned_docs/version-0.6.0/user/block-types/ArchiveInterpreter.md.license b/apps/docs/versioned_docs/version-0.6.0/user/block-types/ArchiveInterpreter.md.license new file mode 100644 index 000000000..42737858e --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/block-types/ArchiveInterpreter.md.license @@ -0,0 +1,3 @@ +SPDX-FileCopyrightText: 2023 Friedrich-Alexander-Universitat Erlangen-Nurnberg + +SPDX-License-Identifier: AGPL-3.0-only \ No newline at end of file diff --git a/apps/docs/versioned_docs/version-0.6.0/user/block-types/CSVFileLoader.md b/apps/docs/versioned_docs/version-0.6.0/user/block-types/CSVFileLoader.md new file mode 100644 index 000000000..2923b9382 --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/block-types/CSVFileLoader.md @@ -0,0 +1,61 @@ +--- +title: CSVFileLoader +--- + + + +Input type: `Table` + +Output type: `None` + +## Description + +Loads a `Table` into a CSV file sink. + +## Example 1 + +```jayvee + block CarsLoader oftype CSVFileLoader { + file: "./cars.csv"; + } +``` + +A SQLite file `cars.csv` is created in the working directory. + +## Properties + +### `file` + +Type `text` + +#### Description + +The path to a CSV file that will be created if it does not exist. + IF THE FILE ALREADY EXISTS, IT WILL BE OVERWRITTEN + The usual file extension is `.csv`. + +### `delimiter` + +Type `text` + +Default: `","` + +### `enclosing` + +Type `text` + +Default: `""` + +#### Description + +The enclosing character that may be used for values in the CSV file. + +### `enclosingEscape` + +Type `text` + +Default: `""` + +#### Description + +The character to escape enclosing characters in values. diff --git a/apps/docs/versioned_docs/version-0.6.0/user/block-types/CSVFileLoader.md.license b/apps/docs/versioned_docs/version-0.6.0/user/block-types/CSVFileLoader.md.license new file mode 100644 index 000000000..42737858e --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/block-types/CSVFileLoader.md.license @@ -0,0 +1,3 @@ +SPDX-FileCopyrightText: 2023 Friedrich-Alexander-Universitat Erlangen-Nurnberg + +SPDX-License-Identifier: AGPL-3.0-only \ No newline at end of file diff --git a/apps/docs/versioned_docs/version-0.6.0/user/block-types/CSVInterpreter.md b/apps/docs/versioned_docs/version-0.6.0/user/block-types/CSVInterpreter.md new file mode 100644 index 000000000..9242e0459 --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/block-types/CSVInterpreter.md @@ -0,0 +1,55 @@ +--- +title: CSVInterpreter +--- + + + +Input type: `TextFile` + +Output type: `Sheet` + +## Description + +Interprets an input file as a csv-file containing string-values delimited by `delimiter` and outputs a `Sheet`. + +## Example 1 + +```jayvee + block AgencyCSVInterpreter oftype CSVInterpreter { + delimiter: ";"; + } +``` + +Interprets an input file as a csv-file containing string-values delimited by `;` and outputs `Sheet`. + +## Properties + +### `delimiter` + +Type `text` + +Default: `","` + +#### Description + +The delimiter for values in the CSV file. + +### `enclosing` + +Type `text` + +Default: `""` + +#### Description + +The enclosing character that may be used for values in the CSV file. + +### `enclosingEscape` + +Type `text` + +Default: `""` + +#### Description + +The character to escape enclosing characters in values. diff --git a/apps/docs/versioned_docs/version-0.6.0/user/block-types/CSVInterpreter.md.license b/apps/docs/versioned_docs/version-0.6.0/user/block-types/CSVInterpreter.md.license new file mode 100644 index 000000000..42737858e --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/block-types/CSVInterpreter.md.license @@ -0,0 +1,3 @@ +SPDX-FileCopyrightText: 2023 Friedrich-Alexander-Universitat Erlangen-Nurnberg + +SPDX-License-Identifier: AGPL-3.0-only \ No newline at end of file diff --git a/apps/docs/versioned_docs/version-0.6.0/user/block-types/CellRangeSelector.md b/apps/docs/versioned_docs/version-0.6.0/user/block-types/CellRangeSelector.md new file mode 100644 index 000000000..1e98ad8ec --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/block-types/CellRangeSelector.md @@ -0,0 +1,33 @@ +--- +title: CellRangeSelector +--- + + + +Input type: `Sheet` + +Output type: `Sheet` + +## Description + +Selects a subset of a `Sheet` to produce a new `Sheet`. + +## Example 1 + +```jayvee + block CarsCoreDataSelector oftype CellRangeSelector { + select: range A1:E*; + } +``` + +Selects the cells in the given range and produces a new `Sheet` containing only the selected cells. + +## Properties + +### `select` + +Type `CellRange` + +#### Description + +The cell range to select. See . diff --git a/apps/docs/versioned_docs/version-0.6.0/user/block-types/CellRangeSelector.md.license b/apps/docs/versioned_docs/version-0.6.0/user/block-types/CellRangeSelector.md.license new file mode 100644 index 000000000..42737858e --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/block-types/CellRangeSelector.md.license @@ -0,0 +1,3 @@ +SPDX-FileCopyrightText: 2023 Friedrich-Alexander-Universitat Erlangen-Nurnberg + +SPDX-License-Identifier: AGPL-3.0-only \ No newline at end of file diff --git a/apps/docs/versioned_docs/version-0.6.0/user/block-types/CellWriter.md b/apps/docs/versioned_docs/version-0.6.0/user/block-types/CellWriter.md new file mode 100644 index 000000000..0997804d6 --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/block-types/CellWriter.md @@ -0,0 +1,53 @@ +--- +title: CellWriter +--- + + + +Input type: `Sheet` + +Output type: `Sheet` + +## Description + +Writes textual values into cells of a `Sheet`. The number of text values needs to match the number of cells to write into. + +## Example 1 + +```jayvee + block NameHeaderWriter oftype CellWriter { + at: cell A1; + write: ["Name"]; + } +``` + +Write the value "Name" into cell `A1`. + +## Example 2 + +```jayvee + block HeaderSequenceWriter oftype CellWriter { + at: range A1:A2; + write: ["Name", "Age"]; + } +``` + +Write the values "Name", "Age" into cells `A1` and `A2`. + +## Properties + +### `write` + +Type `Collection` + +#### Description + +The values to write. + +### `at` + +Type `CellRange` + +#### Description + +The cells to write into. See . diff --git a/apps/docs/versioned_docs/version-0.6.0/user/block-types/CellWriter.md.license b/apps/docs/versioned_docs/version-0.6.0/user/block-types/CellWriter.md.license new file mode 100644 index 000000000..42737858e --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/block-types/CellWriter.md.license @@ -0,0 +1,3 @@ +SPDX-FileCopyrightText: 2023 Friedrich-Alexander-Universitat Erlangen-Nurnberg + +SPDX-License-Identifier: AGPL-3.0-only \ No newline at end of file diff --git a/apps/docs/versioned_docs/version-0.6.0/user/block-types/ColumnDeleter.md b/apps/docs/versioned_docs/version-0.6.0/user/block-types/ColumnDeleter.md new file mode 100644 index 000000000..20a04a340 --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/block-types/ColumnDeleter.md @@ -0,0 +1,33 @@ +--- +title: ColumnDeleter +--- + + + +Input type: `Sheet` + +Output type: `Sheet` + +## Description + +Deletes columns from a `Sheet`. Column IDs of subsequent columns will be shifted accordingly, so there will be no gaps. + +## Example 1 + +```jayvee + block MpgColumnDeleter oftype ColumnDeleter { + delete: [column B]; + } +``` + +Deletes column B (i.e. the second column). + +## Properties + +### `delete` + +Type `Collection` + +#### Description + +The columns to delete. Has to be a full column. See . diff --git a/apps/docs/versioned_docs/version-0.6.0/user/block-types/ColumnDeleter.md.license b/apps/docs/versioned_docs/version-0.6.0/user/block-types/ColumnDeleter.md.license new file mode 100644 index 000000000..42737858e --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/block-types/ColumnDeleter.md.license @@ -0,0 +1,3 @@ +SPDX-FileCopyrightText: 2023 Friedrich-Alexander-Universitat Erlangen-Nurnberg + +SPDX-License-Identifier: AGPL-3.0-only \ No newline at end of file diff --git a/apps/docs/versioned_docs/version-0.6.0/user/block-types/FilePicker.md b/apps/docs/versioned_docs/version-0.6.0/user/block-types/FilePicker.md new file mode 100644 index 000000000..9fd0d5ad4 --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/block-types/FilePicker.md @@ -0,0 +1,33 @@ +--- +title: FilePicker +--- + + + +Input type: `FileSystem` + +Output type: `File` + +## Description + +Selects one `File` from a `FileSystem` based on its relative path to the root of the `FileSystem`. If no file matches the relative path, no output is created and the execution of the pipeline is aborted. + +## Example 1 + +```jayvee + block AgencyFilePicker oftype FilePicker { + path: "./agency.txt"; + } +``` + +Tries to pick the file `agency.txt` from the root of the provided `FileSystem`. If `agency.txt` exists it is passed on as `File`, if it does not exist the execution of the pipeline is aborted. + +## Properties + +### `path` + +Type `text` + +#### Description + +The path of the file to select, relative to the root of the provided `FileSystem`. diff --git a/apps/docs/versioned_docs/version-0.6.0/user/block-types/FilePicker.md.license b/apps/docs/versioned_docs/version-0.6.0/user/block-types/FilePicker.md.license new file mode 100644 index 000000000..42737858e --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/block-types/FilePicker.md.license @@ -0,0 +1,3 @@ +SPDX-FileCopyrightText: 2023 Friedrich-Alexander-Universitat Erlangen-Nurnberg + +SPDX-License-Identifier: AGPL-3.0-only \ No newline at end of file diff --git a/apps/docs/versioned_docs/version-0.6.0/user/block-types/GtfsRTInterpreter.md b/apps/docs/versioned_docs/version-0.6.0/user/block-types/GtfsRTInterpreter.md new file mode 100644 index 000000000..2f55e64aa --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/block-types/GtfsRTInterpreter.md @@ -0,0 +1,80 @@ +--- +title: GtfsRTInterpreter +--- + + + +Input type: `File` + +Output type: `Sheet` + +## Description + +Interprets an protobuf file (binary) of type `File` by decoding the file according to `gtfs-realtime.proto`. Outputs the extracted entity defined by `entity` as a `Sheet` + +## Example 1 + +```jayvee + block GtfsRTTripUpdateInterpreter oftype GtfsRTInterpreter{ + entity: "trip_update"; + } +``` + +A file is interpretet as an GTFS-RT file, which contains TripUpdate. + +## Properties + +### `entity` + +Type `text` + +#### Description + +Entity to process from GTFS-RT-feed (`trip_update`, `alert` or `vehicle`). + + We currently support following Output-Sheets, each are an equivalent to the flattened Element Index defined in (just required fields are included): + + Entity TripUpdate: + ``` + [ + 'header.gtfs_realtime_version', + 'header.timestamp', + 'header.incrementality', + 'entity.id', + 'entity.trip_update.trip.trip_id', + 'entity.trip_update.trip.route_id', + 'entity.trip_update.stop_time_update.stop_sequence', + 'entity.trip_update.stop_time_update.stop_id', + 'entity.trip_update.stop_time_update.arrival.time', + 'entity.trip_update.stop_time_update.departure.time', + ]; + + ``` + Entity VehiclePosition: + ``` + [ + 'header.gtfs_realtime_version', + 'header.timestamp', + 'header.incrementality', + 'entity.id', + 'entity.vehicle_position.vehicle_descriptor.id', + 'entity.vehicle_position.trip.trip_id', + 'entity.vehicle_position.trip.route_id', + 'entity.vehicle_position.position.latitude', + 'entity.vehicle_position.position.longitude', + 'entity.vehicle_position.timestamp', + ]; + ``` + + Entity Alert: + ``` + [ + 'header.gtfs_realtime_version', + 'header.timestamp', + 'header.incrementality', + 'entity.id', + 'entity.alert.informed_entity.route_id', + 'entity.alert.header_text', + 'entity.alert.description_text', + ]; + ``` diff --git a/apps/docs/versioned_docs/version-0.6.0/user/block-types/GtfsRTInterpreter.md.license b/apps/docs/versioned_docs/version-0.6.0/user/block-types/GtfsRTInterpreter.md.license new file mode 100644 index 000000000..42737858e --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/block-types/GtfsRTInterpreter.md.license @@ -0,0 +1,3 @@ +SPDX-FileCopyrightText: 2023 Friedrich-Alexander-Universitat Erlangen-Nurnberg + +SPDX-License-Identifier: AGPL-3.0-only \ No newline at end of file diff --git a/apps/docs/versioned_docs/version-0.6.0/user/block-types/HttpExtractor.md b/apps/docs/versioned_docs/version-0.6.0/user/block-types/HttpExtractor.md new file mode 100644 index 000000000..523e4eb39 --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/block-types/HttpExtractor.md @@ -0,0 +1,73 @@ +--- +title: HttpExtractor +--- + + + +Input type: `None` + +Output type: `File` + +## Description + +Extracts a `File` from the web. + +## Example 1 + +```jayvee + block CarsFileExtractor oftype HttpExtractor { + url: "tinyurl.com/4ub9spwz"; + } +``` + +Fetches a file from the given URL. + +## Properties + +### `url` + +Type `text` + +#### Description + +The URL to the file in the web to extract. + +### `retries` + +Type `integer` + +Default: `0` + +#### Description + +Configures how many retries should be executed after a failure fetching the data. + +### `retryBackoffMilliseconds` + +Type `integer` + +Default: `1000` + +#### Description + +Configures the wait time in milliseconds before executing a retry. + +### `retryBackoffStrategy` + +Type `text` + +Default: `"exponential"` + +#### Description + +Configures the wait strategy before executing a retry. Can have values "exponential" or "linear". + +### `followRedirects` + +Type `boolean` + +Default: `true` + +#### Description + +Indicates, whether to follow redirects on get requests. If `false`, redirects are not followed. Default `true` diff --git a/apps/docs/versioned_docs/version-0.6.0/user/block-types/HttpExtractor.md.license b/apps/docs/versioned_docs/version-0.6.0/user/block-types/HttpExtractor.md.license new file mode 100644 index 000000000..42737858e --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/block-types/HttpExtractor.md.license @@ -0,0 +1,3 @@ +SPDX-FileCopyrightText: 2023 Friedrich-Alexander-Universitat Erlangen-Nurnberg + +SPDX-License-Identifier: AGPL-3.0-only \ No newline at end of file diff --git a/apps/docs/versioned_docs/version-0.6.0/user/block-types/LocalFileExtractor.md b/apps/docs/versioned_docs/version-0.6.0/user/block-types/LocalFileExtractor.md new file mode 100644 index 000000000..24dcd7c7e --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/block-types/LocalFileExtractor.md @@ -0,0 +1,33 @@ +--- +title: LocalFileExtractor +--- + + + +Input type: `None` + +Output type: `File` + +## Description + +Extracts a `File` from the local file system. + +## Example 1 + +```jayvee + block CarsFileExtractor oftype LocalFileExtractor { + filePath: "cars.csv"; + } +``` + +Extracts a file from the given path on the local file system. + +## Properties + +### `filePath` + +Type `text` + +#### Description + +The path to the file in the local file system to extract. Path can not traverse up the directory tree. diff --git a/apps/docs/versioned_docs/version-0.6.0/user/block-types/LocalFileExtractor.md.license b/apps/docs/versioned_docs/version-0.6.0/user/block-types/LocalFileExtractor.md.license new file mode 100644 index 000000000..42737858e --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/block-types/LocalFileExtractor.md.license @@ -0,0 +1,3 @@ +SPDX-FileCopyrightText: 2023 Friedrich-Alexander-Universitat Erlangen-Nurnberg + +SPDX-License-Identifier: AGPL-3.0-only \ No newline at end of file diff --git a/apps/docs/versioned_docs/version-0.6.0/user/block-types/PostgresLoader.md b/apps/docs/versioned_docs/version-0.6.0/user/block-types/PostgresLoader.md new file mode 100644 index 000000000..91782ab5b --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/block-types/PostgresLoader.md @@ -0,0 +1,78 @@ +--- +title: PostgresLoader +--- + + + +Input type: `Table` + +Output type: `None` + +## Description + +Loads a `Table` into a PostgreSQL database sink. + +## Example 1 + +```jayvee + block CarsLoader oftype PostgresLoader { + host: "localhost"; + port: 5432; + username: "postgres"; + password: "postgres"; + database: "CarsDB"; + table: "Cars"; + } +``` + +A local Postgres instance is filled with table data about cars. + +## Properties + +### `host` + +Type `text` + +#### Description + +The hostname or IP address of the Postgres database. + +### `port` + +Type `integer` + +#### Description + +The port of the Postgres database. + +### `username` + +Type `text` + +#### Description + +The username to login to the Postgres database. + +### `password` + +Type `text` + +#### Description + +The password to login to the Postgres database. + +### `database` + +Type `text` + +#### Description + +The database to use. + +### `table` + +Type `text` + +#### Description + +The name of the table to write into. diff --git a/apps/docs/versioned_docs/version-0.6.0/user/block-types/PostgresLoader.md.license b/apps/docs/versioned_docs/version-0.6.0/user/block-types/PostgresLoader.md.license new file mode 100644 index 000000000..42737858e --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/block-types/PostgresLoader.md.license @@ -0,0 +1,3 @@ +SPDX-FileCopyrightText: 2023 Friedrich-Alexander-Universitat Erlangen-Nurnberg + +SPDX-License-Identifier: AGPL-3.0-only \ No newline at end of file diff --git a/apps/docs/versioned_docs/version-0.6.0/user/block-types/RowDeleter.md b/apps/docs/versioned_docs/version-0.6.0/user/block-types/RowDeleter.md new file mode 100644 index 000000000..920ab1555 --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/block-types/RowDeleter.md @@ -0,0 +1,33 @@ +--- +title: RowDeleter +--- + + + +Input type: `Sheet` + +Output type: `Sheet` + +## Description + +Deletes one or more rows from a `Sheet`. Row IDs of subsequent rows will be shifted accordingly, so there will be no gaps. + +## Example 1 + +```jayvee + block SecondRowDeleter oftype RowDeleter { + delete: [row 2]; + } +``` + +Deletes row 2 (i.e. the second row). + +## Properties + +### `delete` + +Type `Collection` + +#### Description + +The rows to delete. See . diff --git a/apps/docs/versioned_docs/version-0.6.0/user/block-types/RowDeleter.md.license b/apps/docs/versioned_docs/version-0.6.0/user/block-types/RowDeleter.md.license new file mode 100644 index 000000000..42737858e --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/block-types/RowDeleter.md.license @@ -0,0 +1,3 @@ +SPDX-FileCopyrightText: 2023 Friedrich-Alexander-Universitat Erlangen-Nurnberg + +SPDX-License-Identifier: AGPL-3.0-only \ No newline at end of file diff --git a/apps/docs/versioned_docs/version-0.6.0/user/block-types/SQLiteLoader.md b/apps/docs/versioned_docs/version-0.6.0/user/block-types/SQLiteLoader.md new file mode 100644 index 000000000..d48e6f0ff --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/block-types/SQLiteLoader.md @@ -0,0 +1,52 @@ +--- +title: SQLiteLoader +--- + + + +Input type: `Table` + +Output type: `None` + +## Description + +Loads a `Table` into a SQLite database sink. + +## Example 1 + +```jayvee + block CarsLoader oftype SQLiteLoader { + table: "cars"; + file: "./cars.db"; + } +``` + +A SQLite file `cars.db` is created in the working directory. Incoming data is written to the table `cars`. + +## Properties + +### `table` + +Type `text` + +#### Description + +The name of the table to write into. + +### `file` + +Type `text` + +#### Description + +The path to a SQLite file that will be created if it does not exist. Usual file extensions are `.sqlite` and `.db`. + +### `dropTable` + +Type `boolean` + +Default: `true` + +#### Description + +Indicates, whether to drop the table before loading data into it. If `false`, data is appended to the table instead of dropping it. diff --git a/apps/docs/versioned_docs/version-0.6.0/user/block-types/SQLiteLoader.md.license b/apps/docs/versioned_docs/version-0.6.0/user/block-types/SQLiteLoader.md.license new file mode 100644 index 000000000..42737858e --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/block-types/SQLiteLoader.md.license @@ -0,0 +1,3 @@ +SPDX-FileCopyrightText: 2023 Friedrich-Alexander-Universitat Erlangen-Nurnberg + +SPDX-License-Identifier: AGPL-3.0-only \ No newline at end of file diff --git a/apps/docs/versioned_docs/version-0.6.0/user/block-types/SheetPicker.md b/apps/docs/versioned_docs/version-0.6.0/user/block-types/SheetPicker.md new file mode 100644 index 000000000..26710170b --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/block-types/SheetPicker.md @@ -0,0 +1,33 @@ +--- +title: SheetPicker +--- + + + +Input type: `Workbook` + +Output type: `Sheet` + +## Description + +Selects one `Sheet` from a `Workbook` based on its `sheetName`. If no sheet matches the name, no output is created and the execution of the pipeline is aborted. + +## Example 1 + +```jayvee + block AgencySheetPicker oftype SheetPicker { + sheetName: "AgencyNames"; + } +``` + +Tries to pick the sheet `AgencyNames` from the provided `Workbook`. If `AgencyNames` exists it is passed on as `Sheet`, if it does not exist the execution of the pipeline is aborted. + +## Properties + +### `sheetName` + +Type `text` + +#### Description + +The name of the sheet to select. diff --git a/apps/docs/versioned_docs/version-0.6.0/user/block-types/SheetPicker.md.license b/apps/docs/versioned_docs/version-0.6.0/user/block-types/SheetPicker.md.license new file mode 100644 index 000000000..42737858e --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/block-types/SheetPicker.md.license @@ -0,0 +1,3 @@ +SPDX-FileCopyrightText: 2023 Friedrich-Alexander-Universitat Erlangen-Nurnberg + +SPDX-License-Identifier: AGPL-3.0-only \ No newline at end of file diff --git a/apps/docs/versioned_docs/version-0.6.0/user/block-types/TableInterpreter.md b/apps/docs/versioned_docs/version-0.6.0/user/block-types/TableInterpreter.md new file mode 100644 index 000000000..35970d776 --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/block-types/TableInterpreter.md @@ -0,0 +1,63 @@ +--- +title: TableInterpreter +--- + + + +Input type: `Sheet` + +Output type: `Table` + +## Description + +Interprets a `Sheet` as a `Table`. In case a header row is present in the sheet, its names can be matched with the provided column names. Otherwise, the provided column names are assigned in order. + +## Example 1 + +```jayvee + block CarsTableInterpreter oftype TableInterpreter { + header: true; + columns: [ + "name" oftype text, + "mpg" oftype decimal, + "cyl" oftype integer, + ]; + } +``` + +Interprets a `Sheet` about cars with a topmost header row and interprets it as a `Table` by assigning a primitive value type to each column. The column names are matched to the header, so the order of the type assignments does not matter. + +## Example 2 + +```jayvee + block CarsTableInterpreter oftype TableInterpreter { + header: false; + columns: [ + "name" oftype text, + "mpg" oftype decimal, + "cyl" oftype integer, + ]; + } +``` + +Interprets a `Sheet` about cars without a topmost header row and interprets it as a `Table` by sequentially assigning a name and a primitive value type to each column of the sheet. Note that the order of columns matters here. The first column (column `A`) will be named "name", the second column (column `B`) will be named "mpg" etc. + +## Properties + +### `header` + +Type `boolean` + +Default: `true` + +#### Description + +Whether the first row should be interpreted as header row. + +### `columns` + +Type `Collection` + +#### Description + +Collection of value type assignments. Uses column names (potentially matched with the header or by sequence depending on the `header` property) to assign a primitive value type to each column. diff --git a/apps/docs/versioned_docs/version-0.6.0/user/block-types/TableInterpreter.md.license b/apps/docs/versioned_docs/version-0.6.0/user/block-types/TableInterpreter.md.license new file mode 100644 index 000000000..42737858e --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/block-types/TableInterpreter.md.license @@ -0,0 +1,3 @@ +SPDX-FileCopyrightText: 2023 Friedrich-Alexander-Universitat Erlangen-Nurnberg + +SPDX-License-Identifier: AGPL-3.0-only \ No newline at end of file diff --git a/apps/docs/versioned_docs/version-0.6.0/user/block-types/TableTransformer.md b/apps/docs/versioned_docs/version-0.6.0/user/block-types/TableTransformer.md new file mode 100644 index 000000000..fd7393006 --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/block-types/TableTransformer.md @@ -0,0 +1,79 @@ +--- +title: TableTransformer +--- + + + +Input type: `Table` + +Output type: `Table` + +## Description + +Applies a transform on each value of a column. The input port type of the used transform has to match the type of the input column. + +## Example 1 + +```jayvee + + transform CelsiusToFahrenheit { + from Celsius oftype decimal; + to Fahrenheit oftype decimal; + + Fahrenheit: (Celsius * 9/5) + 32; + } + + block CelsiusToFahrenheitTransformer oftype TableTransformer { + inputColumns: ['temperature']; + outputColumn: 'temperature'; + uses: CelsiusToFahrenheit; + } +``` + +Given a column "temperature" with temperature values in Celsius, it overwrites the column with computed values in Fahrenheit by using the `CelsiusToFahrenheit` transform. The transform itself is defined elsewhere in the model. + +## Example 2 + +```jayvee + + transform CelsiusToFahrenheit { + from Celsius oftype decimal; + to Fahrenheit oftype decimal; + + Fahrenheit: (Celsius * 9/5) + 32; + } + + block CelsiusToFahrenheitTransformer oftype TableTransformer { + inputColumns: ['temperatureCelsius']; + outputColumn: 'temperatureFahrenheit'; + uses: CelsiusToFahrenheit; + } +``` + +Given a column "temperatureCelsius" with temperature values in Celsius, it adds a new column "temperatureFahrenheit" with computed values in Fahrenheit by using the `CelsiusToFahrenheit` transform. The transform itself is defined elsewhere in the model. + +## Properties + +### `inputColumns` + +Type `Collection` + +#### Description + +The names of the input columns. The columns have to be present in the table and match with the transform's input port types. + +### `outputColumn` + +Type `text` + +#### Description + +The name of the output column. Overwrites the column if it already exists, or otherwise creates a new one. + +### `uses` + +Type `Transform` + +#### Description + +Reference to the transform that is applied to the column. diff --git a/apps/docs/versioned_docs/version-0.6.0/user/block-types/TableTransformer.md.license b/apps/docs/versioned_docs/version-0.6.0/user/block-types/TableTransformer.md.license new file mode 100644 index 000000000..42737858e --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/block-types/TableTransformer.md.license @@ -0,0 +1,3 @@ +SPDX-FileCopyrightText: 2023 Friedrich-Alexander-Universitat Erlangen-Nurnberg + +SPDX-License-Identifier: AGPL-3.0-only \ No newline at end of file diff --git a/apps/docs/versioned_docs/version-0.6.0/user/block-types/TextFileInterpreter.md b/apps/docs/versioned_docs/version-0.6.0/user/block-types/TextFileInterpreter.md new file mode 100644 index 000000000..0fefdcc24 --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/block-types/TextFileInterpreter.md @@ -0,0 +1,35 @@ +--- +title: TextFileInterpreter +--- + + + +Input type: `File` + +Output type: `TextFile` + +## Description + +Interprets a `File` as a `TextFile`. + +## Properties + +### `encoding` + +Type `text` + +Default: `"utf-8"` + +#### Description + +The encoding used for decoding the file contents. + +### `lineBreak` + +Type `Regex` + +Default: `{}` + +#### Description + +The regex for identifying line breaks. diff --git a/apps/docs/versioned_docs/version-0.6.0/user/block-types/TextFileInterpreter.md.license b/apps/docs/versioned_docs/version-0.6.0/user/block-types/TextFileInterpreter.md.license new file mode 100644 index 000000000..42737858e --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/block-types/TextFileInterpreter.md.license @@ -0,0 +1,3 @@ +SPDX-FileCopyrightText: 2023 Friedrich-Alexander-Universitat Erlangen-Nurnberg + +SPDX-License-Identifier: AGPL-3.0-only \ No newline at end of file diff --git a/apps/docs/versioned_docs/version-0.6.0/user/block-types/TextLineDeleter.md b/apps/docs/versioned_docs/version-0.6.0/user/block-types/TextLineDeleter.md new file mode 100644 index 000000000..b36a46c48 --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/block-types/TextLineDeleter.md @@ -0,0 +1,23 @@ +--- +title: TextLineDeleter +--- + + + +Input type: `TextFile` + +Output type: `TextFile` + +## Description + +Deletes individual lines from a `TextFile`. + +## Properties + +### `lines` + +Type `Collection` + +#### Description + +The line numbers to delete. diff --git a/apps/docs/versioned_docs/version-0.6.0/user/block-types/TextLineDeleter.md.license b/apps/docs/versioned_docs/version-0.6.0/user/block-types/TextLineDeleter.md.license new file mode 100644 index 000000000..42737858e --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/block-types/TextLineDeleter.md.license @@ -0,0 +1,3 @@ +SPDX-FileCopyrightText: 2023 Friedrich-Alexander-Universitat Erlangen-Nurnberg + +SPDX-License-Identifier: AGPL-3.0-only \ No newline at end of file diff --git a/apps/docs/versioned_docs/version-0.6.0/user/block-types/TextRangeSelector.md b/apps/docs/versioned_docs/version-0.6.0/user/block-types/TextRangeSelector.md new file mode 100644 index 000000000..75f5e68e4 --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/block-types/TextRangeSelector.md @@ -0,0 +1,36 @@ +--- +title: TextRangeSelector +--- + + + +Input type: `TextFile` + +Output type: `TextFile` + +## Description + +Selects a range of lines from a `TextFile`. + +## Properties + +### `lineFrom` + +Type `integer` + +Default: `1` + +#### Description + +Inclusive beginning line number for the selection. + +### `lineTo` + +Type `integer` + +Default: `9007199254740991` + +#### Description + +Inclusive ending line number for the selection. + The default value is the biggest usable integer. diff --git a/apps/docs/versioned_docs/version-0.6.0/user/block-types/TextRangeSelector.md.license b/apps/docs/versioned_docs/version-0.6.0/user/block-types/TextRangeSelector.md.license new file mode 100644 index 000000000..42737858e --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/block-types/TextRangeSelector.md.license @@ -0,0 +1,3 @@ +SPDX-FileCopyrightText: 2023 Friedrich-Alexander-Universitat Erlangen-Nurnberg + +SPDX-License-Identifier: AGPL-3.0-only \ No newline at end of file diff --git a/apps/docs/versioned_docs/version-0.6.0/user/block-types/XLSXInterpreter.md b/apps/docs/versioned_docs/version-0.6.0/user/block-types/XLSXInterpreter.md new file mode 100644 index 000000000..3dd9a3c91 --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/block-types/XLSXInterpreter.md @@ -0,0 +1,23 @@ +--- +title: XLSXInterpreter +--- + + + +Input type: `File` + +Output type: `Workbook` + +## Description + +Interprets an input file as a XLSX-file and outputs a `Workbook` containing `Sheet`s. + +## Example 1 + +```jayvee + block AgencyXLSXInterpreter oftype XLSXInterpreter { } +``` + +Interprets an input file as a XLSX-file and outputs a `Workbook` containing `Sheet`s. + +## Properties diff --git a/apps/docs/versioned_docs/version-0.6.0/user/block-types/XLSXInterpreter.md.license b/apps/docs/versioned_docs/version-0.6.0/user/block-types/XLSXInterpreter.md.license new file mode 100644 index 000000000..42737858e --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/block-types/XLSXInterpreter.md.license @@ -0,0 +1,3 @@ +SPDX-FileCopyrightText: 2023 Friedrich-Alexander-Universitat Erlangen-Nurnberg + +SPDX-License-Identifier: AGPL-3.0-only \ No newline at end of file diff --git a/apps/docs/versioned_docs/version-0.6.0/user/block-types/_category_.json b/apps/docs/versioned_docs/version-0.6.0/user/block-types/_category_.json new file mode 100644 index 000000000..63f8d0660 --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/block-types/_category_.json @@ -0,0 +1,8 @@ +{ + "label": "Block Types", + "position": 3, + "link": { + "type": "generated-index", + "description": "These block types are shipped with Jayvee and are available right out of the box." + } +} diff --git a/apps/docs/versioned_docs/version-0.6.0/user/block-types/_category_.json.license b/apps/docs/versioned_docs/version-0.6.0/user/block-types/_category_.json.license new file mode 100644 index 000000000..17c5d2bad --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/block-types/_category_.json.license @@ -0,0 +1,3 @@ +SPDX-FileCopyrightText: 2023 Friedrich-Alexander-Universitat Erlangen-Nurnberg + +SPDX-License-Identifier: AGPL-3.0-only diff --git a/apps/docs/versioned_docs/version-0.6.0/user/cell-range.md b/apps/docs/versioned_docs/version-0.6.0/user/cell-range.md new file mode 100644 index 000000000..ecc8d8abc --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/cell-range.md @@ -0,0 +1,243 @@ +--- +sidebar_position: 9 +--- + +# Cell Range + +Cell ranges can be used to select a subset of cells in a 2D data structure, such as a `Sheet`. The syntax for cell ranges follows conventions from spreadsheet-software. Rows are referenced by one-based indices and columns by capitalized characters, ordered alphabetically. + + + + + + + + + + + + + + + + + + + + + + + + + + +
ABC
1Cell A1Cell B1Cell C1
2Cell A2Cell B2Cell C2
3Cell A3Cell B3Cell C3
+ +Cell ranges can be expressed as whole rows using the **`row`** keyword, whole columns using the **`column`** keyword and custom ranges using the **`range`** keyword. + +## Selecting Custom Ranges + +Using the **`range`** keyword, custom ranges can be selected. Ranges must define a start cell and end cell with the syntax `range :`. All cells between these cells are part of the range as if a user had selected the start cell in a spreadsheet-software and dragged the mouse until the end cell. For example `range A1:B2` is a range over four cells, from cell `A1` to `B2`. + +Instead of a specific character or integer, the placeholder `*` denotes the last available cell in the row or column. For example: `A*` is the last cell in column `A` and `*2` is the last cell in row `2`. + +### Examples +The following `CellRangeSelector` block will select the four cells in the top left corner of a `Sheet`: + +
+ +```jayvee + block ExampleDataSelector oftype CellRangeSelector { + select: range A1:B2; + } +``` + + + + + + + + + + + + + + + + + + + + + + + + + + +
ABC
1Cell A1Cell B1Cell C1
2Cell A2Cell B2Cell C2
3Cell A3Cell B3Cell C3
+
+ +The following `CellRangeSelector` block will select cells from the first to the last cell in row 2 in a `Sheet`: + +
+ +```jayvee + block ExampleDataSelector oftype CellRangeSelector { + select: range A2:*2; + } +``` + + + + + + + + + + + + + + + + + + + + + + + + + + +
ABC
1Cell A1Cell B1Cell C1
2Cell A2Cell B2Cell C2
3Cell A3Cell B3Cell C3
+
+ +The following `CellRangeSelector` block will select cells from the top-left most cell to the last cell in column B in a `Sheet`: + +
+ +```jayvee + block ExampleDataSelector oftype CellRangeSelector { + select: range A1:B*; + } +``` + + + + + + + + + + + + + + + + + + + + + + + + + + +
ABC
1Cell A1Cell B1Cell C1
2Cell A2Cell B2Cell C2
3Cell A3Cell B3Cell C3
+
+ +## Selecting Rows + +Using the **`row`** keyword, individual rows can be selected. For example, `row 2` will select the second row in a `Sheet`. By adding multiple rows to a `Collection`, multiple rows can be selected. + +### Examples +The following `RowDeleter` block will delete the first two rows of a `Sheet`: + +
+ +```jayvee + block ExampleRowDeleter oftype RowDeleter { + delete: [row 1, row 2]; + } +``` + + + + + + + + + + + + + + + + + + + + + + + + + + +
ABC
1Cell A1Cell B1Cell C1
2Cell A2Cell B2Cell C2
3Cell A3Cell B3Cell C3
+
+ +## Selecting Columns + +Using the **`column`** keyword, individual columns can be selected. For example, `column 2` will select the second column in a `Sheet`. By adding multiple columns to a `Collection`, multiple columns can be selected. + +### Examples +The following `ColumnDeleter` block will delete the first two columns of a `Sheet`: + +
+ +```jayvee + block ExampleColumnDeleter oftype ColumnDeleter { + delete: [column A, column B]; + } +``` + + + + + + + + + + + + + + + + + + + + + + + + + + +
ABC
1Cell A1Cell B1Cell C1
2Cell A2Cell B2Cell C2
3Cell A3Cell B3Cell C3
+
\ No newline at end of file diff --git a/apps/docs/versioned_docs/version-0.6.0/user/cell-range.md.license b/apps/docs/versioned_docs/version-0.6.0/user/cell-range.md.license new file mode 100644 index 000000000..db3db822d --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/cell-range.md.license @@ -0,0 +1,3 @@ +SPDX-FileCopyrightText: 2024 Friedrich-Alexander-Universitat Erlangen-Nurnberg + +SPDX-License-Identifier: AGPL-3.0-only diff --git a/apps/docs/versioned_docs/version-0.6.0/user/composite-block-types.md b/apps/docs/versioned_docs/version-0.6.0/user/composite-block-types.md new file mode 100644 index 000000000..e0e9df159 --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/composite-block-types.md @@ -0,0 +1,118 @@ +--- +sidebar_position: 4 +--- + +# Composite Block Types + +_Composite block types_ are a way to create new _block types_ in Jayvee by combining the functionality of existing _blocks_ and _pipes_. By relying on _composite block types_ instead of implementing more _built-in block types_ in a language interpreter, Jayvee supports easy extension by users. + +_Composite block types_ define: + +- with the `property` keyword: properties with a name and [value type](<./core-concepts.md#value types>), optionally a default value +- with the `input` keyword: one input with a name and _io type_ (that can be `None`) +- with the `output` keyword: one output with a name and _io type_ (that can be `None`) +- one _pipeline_ definition, starting from the input (using its name) and ending in the output (again using its name) +- all _blocks_ that are used in the _pipeline_ definition (either _blocks_ of _built-in_ or _composite block types_) + +## Example + +As an example, the common use-case of extracting a CSV file from a web server using HTTP. With _built-in block types_, a _pipeline_ would start with a `HttpExtractor` source that downloads a file from the internet and outputs a binary file. This file must be interpreted as text (using a `TextFileInterpreter`) and finally as `Sheet` (using a `CSVInterpreter`). + +### Implementation with built-in block types + +```mermaid +flowchart LR + A[HttpExtractor] --> B(TextFileInterpreter) + B --> C(CSVInterpreter) + C --> D(TableInterpreter) + D --> E[SQLiteSink] +``` + +A _pipeline_ with _blocks_ using _built-in block types_ is very verbose: + +```jayvee +pipeline CarsPipeline { + CarsExtractor + -> CarsTextFileInterpreter + -> CarsCSVInterpreter + -> CarsTableInterpreter + -> CarsLoader; + + block CarsExtractor oftype HttpExtractor { + url: "https://example.com/cars.csv"; + } + + block CarsTextFileInterpreter oftype TextFileInterpreter { } + + block CarsCSVInterpreter oftype CSVInterpreter { + enclosing: '"'; + } + // ... further block definitions +} +``` + +### Refactoring using composite block types + +The common use-case of downloading a CSV file using HTTP can be refactored into a _composite block type_. Note that we define all properties of the _built-in blocks_ that are used as properties of the new `CSVExtractor` _block type_ (but add fallback values). If some internal configuration is always the same, we could also not expose it as a property of the new _block type_. + +```jayvee +// Define a new composite block type named CSVExtractor outside of the pipeline +composite blocktype CSVExtractor { + // Properties of the CSVExtractor, some with default values + property url oftype text; + property delimiter oftype text: ','; + property enclosing oftype text: ''; + property enclosingEscape oftype text: ''; + + // Input and outputs + input inputName oftype None; + output outputName oftype Sheet; + + // Pipeline definition from input, over blocks defined later, to output + inputName + ->FileExtractor + ->FileTextInterpreter + ->FileCSVInterpreter + ->outputName; + + // Block definitions using values from properties by name + block FileExtractor oftype HttpExtractor { url: url; } + block FileTextInterpreter oftype TextFileInterpreter {} + + block FileCSVInterpreter oftype CSVInterpreter { + delimiter: delimiter; + enclosing: enclosing; + enclosingEscape: enclosingEscape; + } +} +``` + +With the new `CSVExtractor` _composite block type_, the _pipeline_ now looks like this. + +```mermaid +flowchart LR + CSVExtractor --> D(TableInterpreter) + D --> E[SQLiteSink] + + subgraph CSVExtractor + A[HttpExtractor] --> B(TextFileInterpreter) + B --> C(CSVInterpreter) +end +``` + +If the `CSVExtractor` is available in the scope of the `CarsPipeline` from before (e.g., by defining it above the _pipeline_), it can then be used to shorten the actual _pipeline_ code. + +```jayvee +pipeline CarsPipeline { + // HttpExtractor, TextFileInterpreter and CSVInterpreter have been replaced by CSVExtractor + CarsExtractor + -> CarsTableInterpreter + -> CarsLoader; + + block CarsExtractor oftype CSVExtractor { + url: "https://example.com/cars.csv"; + } + + // ... further block definitions +} +``` diff --git a/apps/docs/versioned_docs/version-0.6.0/user/composite-block-types.md.license b/apps/docs/versioned_docs/version-0.6.0/user/composite-block-types.md.license new file mode 100644 index 000000000..17c5d2bad --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/composite-block-types.md.license @@ -0,0 +1,3 @@ +SPDX-FileCopyrightText: 2023 Friedrich-Alexander-Universitat Erlangen-Nurnberg + +SPDX-License-Identifier: AGPL-3.0-only diff --git a/apps/docs/versioned_docs/version-0.6.0/user/constraint-types/AllowlistConstraint.md b/apps/docs/versioned_docs/version-0.6.0/user/constraint-types/AllowlistConstraint.md new file mode 100644 index 000000000..a8b7f0260 --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/constraint-types/AllowlistConstraint.md @@ -0,0 +1,27 @@ +--- +title: AllowlistConstraint +--- + + + +Compatible value type: text + +## Description + +Limits the values to a defined a set of allowed values. Only values in the list are valid. + +## Example 1 + +```jayvee + publish constraint TimeUnitString oftype AllowlistConstraint { + allowlist: ["ms", "s", "min", "h", "d", "m", "y"]; + } +``` + +Only allows the common abbreviations for millisecond, second, minute, etc.. + +## Properties + +### `allowlist` + +Type `Collection` diff --git a/apps/docs/versioned_docs/version-0.6.0/user/constraint-types/AllowlistConstraint.md.license b/apps/docs/versioned_docs/version-0.6.0/user/constraint-types/AllowlistConstraint.md.license new file mode 100644 index 000000000..42737858e --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/constraint-types/AllowlistConstraint.md.license @@ -0,0 +1,3 @@ +SPDX-FileCopyrightText: 2023 Friedrich-Alexander-Universitat Erlangen-Nurnberg + +SPDX-License-Identifier: AGPL-3.0-only \ No newline at end of file diff --git a/apps/docs/versioned_docs/version-0.6.0/user/constraint-types/DenylistConstraint.md b/apps/docs/versioned_docs/version-0.6.0/user/constraint-types/DenylistConstraint.md new file mode 100644 index 000000000..5399c8652 --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/constraint-types/DenylistConstraint.md @@ -0,0 +1,27 @@ +--- +title: DenylistConstraint +--- + + + +Compatible value type: text + +## Description + +Defines a set of forbidden values. All values in the list are considered invalid. + +## Example 1 + +```jayvee + publish constraint NoPrimaryColors oftype DenylistConstraint { + denylist: ["red", "blue", "yellow"]; + } +``` + +Denies all primary colors. + +## Properties + +### `denylist` + +Type `Collection` diff --git a/apps/docs/versioned_docs/version-0.6.0/user/constraint-types/DenylistConstraint.md.license b/apps/docs/versioned_docs/version-0.6.0/user/constraint-types/DenylistConstraint.md.license new file mode 100644 index 000000000..42737858e --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/constraint-types/DenylistConstraint.md.license @@ -0,0 +1,3 @@ +SPDX-FileCopyrightText: 2023 Friedrich-Alexander-Universitat Erlangen-Nurnberg + +SPDX-License-Identifier: AGPL-3.0-only \ No newline at end of file diff --git a/apps/docs/versioned_docs/version-0.6.0/user/constraint-types/LengthConstraint.md b/apps/docs/versioned_docs/version-0.6.0/user/constraint-types/LengthConstraint.md new file mode 100644 index 000000000..02f19039e --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/constraint-types/LengthConstraint.md @@ -0,0 +1,37 @@ +--- +title: LengthConstraint +--- + + + +Compatible value type: text + +## Description + +Limits the length of a string with an upper and/or lower boundary. + Only values with a length within the given range are valid. + +## Example 1 + +```jayvee + publish constraint ShortAnswerConstraint oftype LengthConstraint { + minLength: 0; + maxLength: 20; + } +``` + +A text publish constraint with 0 to 20 characters. + +## Properties + +### `minLength` + +Type `integer` + +Default: `0` + +### `maxLength` + +Type `integer` + +Default: `9007199254740991` diff --git a/apps/docs/versioned_docs/version-0.6.0/user/constraint-types/LengthConstraint.md.license b/apps/docs/versioned_docs/version-0.6.0/user/constraint-types/LengthConstraint.md.license new file mode 100644 index 000000000..42737858e --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/constraint-types/LengthConstraint.md.license @@ -0,0 +1,3 @@ +SPDX-FileCopyrightText: 2023 Friedrich-Alexander-Universitat Erlangen-Nurnberg + +SPDX-License-Identifier: AGPL-3.0-only \ No newline at end of file diff --git a/apps/docs/versioned_docs/version-0.6.0/user/constraint-types/RangeConstraint.md b/apps/docs/versioned_docs/version-0.6.0/user/constraint-types/RangeConstraint.md new file mode 100644 index 000000000..6dadb9f58 --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/constraint-types/RangeConstraint.md @@ -0,0 +1,60 @@ +--- +title: RangeConstraint +--- + + + +Compatible value type: decimal + +## Description + +Limits the range of a number value with an upper and/or lower boundary which can be inclusive or exclusive. Only values within the given range are considered valid. + +## Example 1 + +```jayvee + publish constraint HundredScale oftype RangeConstraint { + lowerBound: 1; + upperBound: 100; + } +``` + +A scale between (and including) 1 and 100. + +## Example 2 + +```jayvee + publish constraint HundredScale oftype RangeConstraint { + lowerBound: 1; + lowerBoundInclusive: false; + upperBound: 100; + } +``` + +A scale for numbers strictly larger than 1 and less or equal to 100. + +## Properties + +### `lowerBound` + +Type `decimal` + +Default: `-9007199254740991` + +### `lowerBoundInclusive` + +Type `boolean` + +Default: `true` + +### `upperBound` + +Type `decimal` + +Default: `9007199254740991` + +### `upperBoundInclusive` + +Type `boolean` + +Default: `true` diff --git a/apps/docs/versioned_docs/version-0.6.0/user/constraint-types/RangeConstraint.md.license b/apps/docs/versioned_docs/version-0.6.0/user/constraint-types/RangeConstraint.md.license new file mode 100644 index 000000000..42737858e --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/constraint-types/RangeConstraint.md.license @@ -0,0 +1,3 @@ +SPDX-FileCopyrightText: 2023 Friedrich-Alexander-Universitat Erlangen-Nurnberg + +SPDX-License-Identifier: AGPL-3.0-only \ No newline at end of file diff --git a/apps/docs/versioned_docs/version-0.6.0/user/constraint-types/RegexConstraint.md b/apps/docs/versioned_docs/version-0.6.0/user/constraint-types/RegexConstraint.md new file mode 100644 index 000000000..75a886608 --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/constraint-types/RegexConstraint.md @@ -0,0 +1,28 @@ +--- +title: RegexConstraint +--- + + + +Compatible value type: text + +## Description + +Limits the values complying with a regex. + Only values that comply with the regex are considered valid. + +## Example 1 + +```jayvee + publish constraint IPv4Format oftype RegexConstraint { + regex: /^((25[0-5]|(2[0-4]|1\d|[1-9]|)\d)\.?\b){4}$/; + } +``` + +Text that complies with the IPv4 address format. + +## Properties + +### `regex` + +Type `Regex` diff --git a/apps/docs/versioned_docs/version-0.6.0/user/constraint-types/RegexConstraint.md.license b/apps/docs/versioned_docs/version-0.6.0/user/constraint-types/RegexConstraint.md.license new file mode 100644 index 000000000..42737858e --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/constraint-types/RegexConstraint.md.license @@ -0,0 +1,3 @@ +SPDX-FileCopyrightText: 2023 Friedrich-Alexander-Universitat Erlangen-Nurnberg + +SPDX-License-Identifier: AGPL-3.0-only \ No newline at end of file diff --git a/apps/docs/versioned_docs/version-0.6.0/user/constraint-types/_category_.json b/apps/docs/versioned_docs/version-0.6.0/user/constraint-types/_category_.json new file mode 100644 index 000000000..6cb00cc8c --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/constraint-types/_category_.json @@ -0,0 +1,8 @@ +{ + "label": "Constraint Types", + "position": 6, + "link": { + "type": "generated-index", + "description": "These constraint types are shipped with Jayvee and are available right out of the box." + } +} diff --git a/apps/docs/versioned_docs/version-0.6.0/user/constraint-types/_category_.json.license b/apps/docs/versioned_docs/version-0.6.0/user/constraint-types/_category_.json.license new file mode 100644 index 000000000..17c5d2bad --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/constraint-types/_category_.json.license @@ -0,0 +1,3 @@ +SPDX-FileCopyrightText: 2023 Friedrich-Alexander-Universitat Erlangen-Nurnberg + +SPDX-License-Identifier: AGPL-3.0-only diff --git a/apps/docs/versioned_docs/version-0.6.0/user/core-concepts.md b/apps/docs/versioned_docs/version-0.6.0/user/core-concepts.md new file mode 100644 index 000000000..87cd67693 --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/core-concepts.md @@ -0,0 +1,133 @@ +--- +sidebar_position: 2 +--- + +# Core Concepts + +The core concepts of Jayvee are _pipelines_, _blocks_, and _value types_. + +## Pipelines + +A _pipeline_ is a sequence of different computing steps, the _blocks_. +The default output of a _block_ becomes the default input of the next _block_, building a chain of computing steps. +In the scope of a _pipeline_, you can connect these _blocks_ via the _pipe_ syntax: + +```jayvee +pipeline CarsPipeline { + // Assumption: blocks "GasReserveHttpExtractor", "GasReserveCSVInterpreter", "GasReserveTableInterpreter", and "GasReserveLoader" are defined + + GasReserveHttpExtractor + -> GasReserveTextFileInterpreter + -> GasReserveCSVInterpreter + -> GasReserveTableInterpreter + -> GasReserveLoader; +} +``` + +## Blocks + +A _block_ is a processing step within a _pipeline_. +It can have a default input and a default output. +We differentiate the following types of _blocks_: + +- _Extractor blocks_ do not have a default input but only a default output. They model a **data source**. +- _Transformator blocks_ have a default input and a default output. They model a **transformation**. +- _Loader blocks_ do have a default input but nor a default output. They model a **data sink**. + +The general structure of a _pipeline_ consisting of different blocks is the following: + +```mermaid +flowchart LR + A[ExtractorBlock] --> B(TransformatorBlock) + B --> C(TransformatorBlock) + C --> D(LoaderBlock) +``` + +The common syntax of _blocks_ is at its core a key-value map to provide configuration to the _block_. +The availability of property keys and their respective _value types_ is determined by the type of the _block_, called _block type_ - indicated by the identifier after the keyword `oftype`: + +```jayvee +block GasReserveHttpExtractor oftype HttpExtractor { + // key: value + url: "https://www.bundesnetzagentur.de/_tools/SVG/js2/_functions/csv_export.html?view=renderCSV&id=1089590"; +} +``` + +In the example above, the `url` property of type `text` is defined by the corresponding `HttpExtractor` _block type_. + +_Blocks_ can be either defined as part of the language, called _built-in_ or defined as composition of existing _blocks_ by users in Jayvee, called _composite block types_. See the documentation for [_composite block types_](./composite-block-types.md). + +## Value types + +A _value type_ is the definition of a data type of the processed data. +Some _blocks_ use _value types_ to define logic (like filtering or assessing the data type in a data sink). +We differentiate the following kinds of _value types_: + +- _Built-in value types_ come with the basic version of Jayvee. See [built-in value types](./value-types/built-in-value-types). +- _Primitive value types_ can be defined by the user to model domain-specific data types and represent a single value. + _Constraints_ can be added to a _primitive value types_. + See [primitive value types](./value-types/primitive-value-types). +- _Compound value types_: UPCOMING. + +```jayvee +valuetype GasFillLevel oftype integer { + constraints: [ GasFillLevelRange ]; +} + +constraint GasFillLevelRange on decimal: + value >= 0 and value <= 100; +``` + +## Transforms + +_Transforms_ are used to transform data from one _value type_ to a different one. For more details, see [transforms](./transforms.md). + +```jayvee +transform CelsiusToKelvin { + from tempCelsius oftype decimal; + to tempKelvin oftype decimal; + + tempKelvin: tempCelsius + 273.15; +} +``` + +## Publishing / using model elements + +If you want to use a model element in a different file other than the one you define it, you need to _publish_ and _use_ it. + +1. Publish the element to make it usable in other files. + +```jayvee +// Either publish right away when defining an element +publish constraint GasFillLevelRange on decimal: + value >= 0 and value <= 100; + +// Or define first and publish separately +constraint GasFillLevelRange on decimal: + value >= 0 and value <= 100; + +publish GasFillLevelRange; +publish GasFillLevelRange as PercentGasFillLevel; // Alias for renaming the published element +``` + +2. Use the element in another file + +```jayvee +// Define from where you want to take elements + +// Wildcard "*" makes all published elements of the file available +use * from './relative/path/to/file.jv'; + +// Named use only makes the named elements available +use { GasFillLevelRange } './relative/path/to/file.jv'; +use { GasFillLevelRange as FillLevelRange } './relative/path/to/file.jv'; // Alias locally renames elements + + +// Then just use them as if they were defined on root level + +valuetype GasFillLevel oftype integer { + constraints: [ GasFillLevelRange ]; // GasFillLevelRange is defined in another file +} +``` + +Currently, only root-level elements can be published, so elements defined within a pipeline cannot be used in other files. diff --git a/apps/docs/versioned_docs/version-0.6.0/user/core-concepts.md.license b/apps/docs/versioned_docs/version-0.6.0/user/core-concepts.md.license new file mode 100644 index 000000000..17c5d2bad --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/core-concepts.md.license @@ -0,0 +1,3 @@ +SPDX-FileCopyrightText: 2023 Friedrich-Alexander-Universitat Erlangen-Nurnberg + +SPDX-License-Identifier: AGPL-3.0-only diff --git a/apps/docs/versioned_docs/version-0.6.0/user/examples/README.mdx b/apps/docs/versioned_docs/version-0.6.0/user/examples/README.mdx new file mode 100644 index 000000000..3b8b94487 --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/examples/README.mdx @@ -0,0 +1,15 @@ +--- +sidebar_position: 10 +--- + +# Jayvee Examples + +Examples of Jayvee models. +Copy them to your local file system and execute them with the `jv` command on your command line (see [usage](/docs/user/intro#usage)). +You can [find all examples on Github](https://github.com/jvalue/jayvee/tree/main/example). + +```mdx-code-block +import DocCardList from '@theme/DocCardList'; + + +``` diff --git a/apps/docs/versioned_docs/version-0.6.0/user/examples/README.mdx.license b/apps/docs/versioned_docs/version-0.6.0/user/examples/README.mdx.license new file mode 100644 index 000000000..17c5d2bad --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/examples/README.mdx.license @@ -0,0 +1,3 @@ +SPDX-FileCopyrightText: 2023 Friedrich-Alexander-Universitat Erlangen-Nurnberg + +SPDX-License-Identifier: AGPL-3.0-only diff --git a/apps/docs/versioned_docs/version-0.6.0/user/examples/cars.md b/apps/docs/versioned_docs/version-0.6.0/user/examples/cars.md new file mode 100644 index 000000000..4bc741b6a --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/examples/cars.md @@ -0,0 +1,113 @@ +--- +title: cars +--- + +```jayvee +// SPDX-FileCopyrightText: 2023 Friedrich-Alexander-Universitat Erlangen-Nurnberg +// +// SPDX-License-Identifier: AGPL-3.0-only + +// Example 1: Cars +// Learning goals: +// - Understand the core concepts pipeline, block, and pipe +// - Understand the general structure of a pipeline + +// 1. This Jayvee model describes a pipeline +// from a CSV file in the web +// to a SQLite file sink. +pipeline CarsPipeline { + + // 2. We describe the structure of the pipeline, + // usually at the top of the pipeline. + // by connecting blocks via pipes. + + // 3. Syntax of a pipe + // connecting the block CarsExtractor + // with the block CarsTextFileInterpreter. + CarsExtractor + -> CarsTextFileInterpreter; + + // 4. The output of the preceding block is hereby used + // as input for the succeeding block. + + // 5. Pipes can be further chained, + // leading to an overview of the pipeline. + CarsTextFileInterpreter + -> CarsCSVInterpreter + -> NameHeaderWriter + -> CarsTableInterpreter + -> CarsLoader; + + + // 6. Below the pipes, we usually define the blocks + // that are connected by the pipes. + + // 7. Blocks instantiate a block type by using the oftype keyword. + // The block type defines the available properties that the block + // can use to specify the intended behavior of the block + block CarsExtractor oftype HttpExtractor { + + // 8. Properties are assigned to concrete values. + // Here, we specify the URL where the file shall be downloaded from. + url: "https://gist.githubusercontent.com/noamross/e5d3e859aa0c794be10b/raw/b999fb4425b54c63cab088c0ce2c0d6ce961a563/cars.csv"; + } + + // 9. The HttpExtractor requires no input and produces a binary file as output. + // This file has to be interpreted, e.g., as text file. + block CarsTextFileInterpreter oftype TextFileInterpreter { } + + // 10. Next, we interpret the text file as sheet. + // A sheet only contains text cells and is useful for manipulating the shape of data before assigning more strict value types to cells. + block CarsCSVInterpreter oftype CSVInterpreter { + enclosing: '"'; + } + + // 11. We can write into cells of a sheet using the CellWriter block type. + block NameHeaderWriter oftype CellWriter { + // 12. We utilize a syntax similar to spreadsheet programs. + // Cell ranges can be described using the keywords "cell", "row", "column", or "range" that indicate which + // cells are selected for the write action. + at: cell A1; + + // 13. For each cell we selected with the "at" property above, + // we can specify what value shall be written into the cell. + write: [ + "name" + ]; + } + + // 14. As a next step, we interpret the sheet as a table by adding structure. + // We define a value type per column that specifies the data type of the column. + // Rows that include values that are not valid according to the their value types are dropped automatically. + block CarsTableInterpreter oftype TableInterpreter { + header: true; + columns: [ + "name" oftype text, + "mpg" oftype decimal, + "cyl" oftype integer, + "disp" oftype decimal, + "hp" oftype integer, + "drat" oftype decimal, + "wt" oftype decimal, + "qsec" oftype decimal, + "vs" oftype integer, + "am" oftype integer, + "gear" oftype integer, + "carb" oftype integer + ]; + } + + // 15. As a last step, we load the table into a sink, + // here into a sqlite file. + // The structural information of the table is used + // to generate the correct table. + block CarsLoader oftype SQLiteLoader { + table: "Cars"; + file: "./cars.sqlite"; + } + + // 16. Congratulations! + // You can now use the sink for your data analysis, app, + // or whatever you want to do with the cleaned data. +} +``` \ No newline at end of file diff --git a/apps/docs/versioned_docs/version-0.6.0/user/examples/cars.md.license b/apps/docs/versioned_docs/version-0.6.0/user/examples/cars.md.license new file mode 100644 index 000000000..42737858e --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/examples/cars.md.license @@ -0,0 +1,3 @@ +SPDX-FileCopyrightText: 2023 Friedrich-Alexander-Universitat Erlangen-Nurnberg + +SPDX-License-Identifier: AGPL-3.0-only \ No newline at end of file diff --git a/apps/docs/versioned_docs/version-0.6.0/user/examples/electric-vehicles.md b/apps/docs/versioned_docs/version-0.6.0/user/examples/electric-vehicles.md new file mode 100644 index 000000000..173f77000 --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/examples/electric-vehicles.md @@ -0,0 +1,139 @@ +--- +title: electric-vehicles +--- + +```jayvee +// SPDX-FileCopyrightText: 2023 Friedrich-Alexander-Universitat Erlangen-Nurnberg +// +// SPDX-License-Identifier: AGPL-3.0-only + +// Example 2: Electric Vehicles +// Learning goals: +// - Understand further core concepts transforms and value types +// - Understand how to construct a pipeline with multiple sinks +// - Understand the use of runtime parameters + +// 0. We can use elements defined in other files with the "use" syntax. +// In this case, we use the value type UsStateCode when later specifying the table column value types. +use { UsStateCode } from './state-codes.jv'; + + +// 1. This Jayvee model describes a pipeline +// from a CSV file in the web +// to a SQLite file and a PostgreSQL db sink. +pipeline ElectricVehiclesPipeline { + // See here for meta-data of the data source + // https://catalog.data.gov/dataset/electric-vehicle-population-data/resource/fa51be35-691f-45d2-9f3e-535877965e69 + + // 2. At the top of a pipeline, we describe the + // structure of the pipeline. The first part until + // the ElectricRangeTransformer is a linear sequence + // of blocks. From there we can see a split into two + // parallel sequences that load the data in to two + // different sinks. + ElectricVehiclesHttpExtractor + -> ElectricVehiclesTextFileInterpreter + -> ElectricVehiclesCSVInterpreter + -> ElectricVehiclesTableInterpreter + -> ElectricRangeTransformer; + + ElectricRangeTransformer + -> ElectricVehiclesSQLiteLoader; + + ElectricRangeTransformer + -> ElectricVehiclesPostgresLoader; + + // 3. After the pipeline structure, we define the blocks used. + block ElectricVehiclesHttpExtractor oftype HttpExtractor { + url: "https://data.wa.gov/api/views/f6w7-q2d2/rows.csv?accessType=DOWNLOAD"; + } + + block ElectricVehiclesTextFileInterpreter oftype TextFileInterpreter { } + + block ElectricVehiclesCSVInterpreter oftype CSVInterpreter { } + + block ElectricVehiclesTableInterpreter oftype TableInterpreter { + header: true; + columns: [ + // 4. Here, a user-deifned value type is used to describe this column. + // The capital letter indicates that the value type is not built-in + // by convention. The value type itself is defined further below. + "VIN (1-10)" oftype VehicleIdentificationNumber10, + "County" oftype text, + "City" oftype text, + "State" oftype UsStateCode, // We can just use the element as if it was defined in this file. + "Postal Code" oftype text, + "Model Year" oftype integer, + "Make" oftype text, + "Model" oftype text, + "Electric Vehicle Type" oftype text, + "Clean Alternative Fuel Vehicle (CAFV) Eligibility" oftype text, + "Electric Range" oftype integer, + "Base MSRP" oftype integer, + "Legislative District" oftype text, + "DOL Vehicle ID" oftype integer, + "Vehicle Location" oftype text, + "Electric Utility" oftype text, + "2020 Census Tract" oftype text, + ]; + } + + // 5. This block describes the application of a transform function + // taking a column as input and adding another computed column. + // The applied transform function is defined below and referenced + // by the "use" property. + block ElectricRangeTransformer oftype TableTransformer { + inputColumns: [ + "Electric Range" + ]; + outputColumn: "Electric Range (km)"; + uses: MilesToKilometers; + } + + // 6. Here, we define a transform function, taking parameters + // as input ("from" keyword), and producing an output ("to" keyword). + // Inputs and outputs have to be further described by a value type. + transform MilesToKilometers { + from miles oftype decimal; + to kilometers oftype integer; + + // 7. In order to express what the transform function does, + // we assign an expression to the output. Values from the input and output of the transform can be referred to by name. + kilometers: round (miles * 1.609344); + } + + block ElectricVehiclesSQLiteLoader oftype SQLiteLoader { + table: "ElectricVehiclePopulationData"; + file: "./electric-vehicles.sqlite"; + } + + block ElectricVehiclesPostgresLoader oftype PostgresLoader { + // 8. The requires keyword allows us to define runtime parameters. + // These values have to be provided as environment variables when interpreting the Jayvee model. + host: requires DB_HOST; + port: requires DB_PORT; + username: requires DB_USERNAME; + password: requires DB_PASSWORD; + database: requires DB_DATABASE; + table: "ElectricVehiclePopulationData"; + } +} + +// 9. Below the pipeline, we model user-define value types. +// We give them a speaking name and provide a base value type +// that this value type builts on. User-defined value types always place additional constraints on existing value types. +valuetype VehicleIdentificationNumber10 oftype text { + // 10. Value types can be further refined by providing constraints. + constraints: [ + OnlyCapitalLettersAndDigits, + ExactlyTenCharacters, + ]; +} + +// 11. This constraint works on text value types and requires values +// to match a given regular expression in order to be valid. +constraint OnlyCapitalLettersAndDigits on text: value matches /^[A-Z0-9]*$/; + +constraint ExactlyTenCharacters on text: value.length == 10; + +``` \ No newline at end of file diff --git a/apps/docs/versioned_docs/version-0.6.0/user/examples/electric-vehicles.md.license b/apps/docs/versioned_docs/version-0.6.0/user/examples/electric-vehicles.md.license new file mode 100644 index 000000000..42737858e --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/examples/electric-vehicles.md.license @@ -0,0 +1,3 @@ +SPDX-FileCopyrightText: 2023 Friedrich-Alexander-Universitat Erlangen-Nurnberg + +SPDX-License-Identifier: AGPL-3.0-only \ No newline at end of file diff --git a/apps/docs/versioned_docs/version-0.6.0/user/examples/gtfs-rt.md b/apps/docs/versioned_docs/version-0.6.0/user/examples/gtfs-rt.md new file mode 100644 index 000000000..7a213e153 --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/examples/gtfs-rt.md @@ -0,0 +1,133 @@ +--- +title: gtfs-rt +--- + +```jayvee +// SPDX-FileCopyrightText: 2023 Friedrich-Alexander-Universitat Erlangen-Nurnberg +// +// SPDX-License-Identifier: AGPL-3.0-only + +// Example 3: GTFS Realtime Data +// Learning goals: +// - Understand the construction of a csv file with multiple tables +// - Understand how to work with live data + +// 1. This Jayvee model describes a pipeline +// from a GTFS RT data source in the web +// to a SQLite file with multiple tables. +pipeline GtfsRTSimplePipeline { + + // 2. As you can see here, we have three independent + // sequences of pipes in this pipeline. + GTFSRTTripUpdateFeedExtractor + ->GtfsRTTripUpdateInterpreter + ->TripUpdateTableInterpreter + ->TripUpdateLoader; + + GTFSRTVehiclePositionFeedExtractor + ->GtfsRTVehiclePositionInterpreter + ->VehiclePositionTableInterpreter + ->VehicleLoader; + + GTFSRTAlertFeedExtractor + ->GtfsRTAlertInterpreter + ->AlertTableInterpreter + ->AlertLoader; + + // 3. We define a series of HttpExtractors that each pull data + // from an HTTP endpoint + block GTFSRTTripUpdateFeedExtractor oftype HttpExtractor { + url: "https://proxy.transport.data.gouv.fr/resource/bibus-brest-gtfs-rt-trip-update"; + } + + block GTFSRTVehiclePositionFeedExtractor oftype HttpExtractor { + url: "https://proxy.transport.data.gouv.fr/resource/bibus-brest-gtfs-rt-vehicle-position"; + } + + block GTFSRTAlertFeedExtractor oftype HttpExtractor { + url: "https://proxy.transport.data.gouv.fr/resource/bibus-brest-gtfs-rt-alerts"; + } + + // 4. In the next step, we use the domain-specific GtfsRTInterpreter + // to interpret the fetched files as sheets + block GtfsRTTripUpdateInterpreter oftype GtfsRTInterpreter { + entity: "trip_update"; + } + + block GtfsRTAlertInterpreter oftype GtfsRTInterpreter { + entity: "alert"; + } + + block GtfsRTVehiclePositionInterpreter oftype GtfsRTInterpreter { + entity: "vehicle"; + } + + // 5. Next, we interpret the sheets as tables + block TripUpdateTableInterpreter oftype TableInterpreter { + header: true; + columns: [ + "header.gtfs_realtime_version" oftype text, + "header.timestamp" oftype text, + "header.incrementality" oftype text, + "entity.id" oftype text, + "entity.trip_update.trip.trip_id" oftype text, + "entity.trip_update.trip.route_id" oftype text, + "entity.trip_update.stop_time_update.stop_sequence" oftype text, + "entity.trip_update.stop_time_update.stop_id" oftype text, + "entity.trip_update.stop_time_update.arrival.time" oftype text, + "entity.trip_update.stop_time_update.departure.time" oftype text, + ]; + } + + block VehiclePositionTableInterpreter oftype TableInterpreter { + header: true; + columns: [ + "header.gtfs_realtime_version" oftype text, + "header.timestamp" oftype text, + "header.incrementality" oftype text, + "entity.id" oftype text, + "entity.vehicle_position.vehicle_descriptor.id" oftype text, + "entity.vehicle_position.trip.trip_id" oftype text, + "entity.vehicle_position.trip.route_id" oftype text, + "entity.vehicle_position.position.latitude" oftype text, + "entity.vehicle_position.position.longitude" oftype text, + "entity.vehicle_position.timestamp" oftype text + ]; + } + + block AlertTableInterpreter oftype TableInterpreter { + header: true; + columns: [ + 'header.gtfs_realtime_version' oftype text, + 'header.timestamp' oftype text, + 'header.incrementality' oftype text, + 'entity.id' oftype text, + 'entity.alert.informed_entity.route_id' oftype text, + 'entity.alert.header_text' oftype text, + 'entity.alert.description_text' oftype text, + ]; + } + + // 6. Last, we load the tables into the same SQLite file. + // Each loader has to define a different table name. + // For working with live data, we use the property "dropTable: false" + // to append data instead of deleting the previous data. + block TripUpdateLoader oftype SQLiteLoader { + table: "gtfs-rt-trip_update"; + file: "./gtfs.sqlite"; + dropTable: false; + } + + block VehicleLoader oftype SQLiteLoader { + table: "gtfs-rt-vehicle_position"; + file: "./gtfs.sqlite"; + dropTable: false; + } + + block AlertLoader oftype SQLiteLoader { + table: "gtfs-rt-alert"; + file: "./gtfs.sqlite"; + dropTable: false; + } +} +``` \ No newline at end of file diff --git a/apps/docs/versioned_docs/version-0.6.0/user/examples/gtfs-rt.md.license b/apps/docs/versioned_docs/version-0.6.0/user/examples/gtfs-rt.md.license new file mode 100644 index 000000000..42737858e --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/examples/gtfs-rt.md.license @@ -0,0 +1,3 @@ +SPDX-FileCopyrightText: 2023 Friedrich-Alexander-Universitat Erlangen-Nurnberg + +SPDX-License-Identifier: AGPL-3.0-only \ No newline at end of file diff --git a/apps/docs/versioned_docs/version-0.6.0/user/examples/gtfs-static.md b/apps/docs/versioned_docs/version-0.6.0/user/examples/gtfs-static.md new file mode 100644 index 000000000..b8188dced --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/examples/gtfs-static.md @@ -0,0 +1,131 @@ +--- +title: gtfs-static +--- + +```jayvee +// SPDX-FileCopyrightText: 2023 Friedrich-Alexander-Universitat Erlangen-Nurnberg +// +// SPDX-License-Identifier: AGPL-3.0-only + +// Example 4: GTFS Static Data +// Learning goals: +// - Understand how to work with file systems + +// 1. This Jayvee model describes a pipeline +// from a zip file in the GTFS format in the web +// to a joint SQLite file with multiple tables. +pipeline GtfsPipeline { + + // 2. The origin for multiple pipe sequences is a zip + // file. Each file in this zip is further processed + // by its own sequence of blocks and pipes. + GTFSSampleFeedExtractor + -> AgencyInterpreter + -> AgencyLoader; + + GTFSSampleFeedExtractor + -> CalendarDatesInterpreter + -> CalendarDatesLoader; + + GTFSSampleFeedExtractor + -> CalendarInterpreter + -> CalendarLoader; + + GTFSSampleFeedExtractor + -> FareAttributesInterpreter + -> FareAttributesLoader; + + GTFSSampleFeedExtractor + -> FareRulesInterpreter + -> FareRulesLoader; + + GTFSSampleFeedExtractor + -> FrequenciesInterpreter + -> FrequenciesLoader; + + GTFSSampleFeedExtractor + -> RoutesInterpreter + -> RoutesLoader; + + GTFSSampleFeedExtractor + -> ShapesInterpreter + -> ShapesLoader; + + GTFSSampleFeedExtractor + -> StopTimesInterpreter + -> StopTimesLoader; + + GTFSSampleFeedExtractor + -> StopsInterpreter + -> StopsLoader; + + GTFSSampleFeedExtractor + -> TripsInterpreter + -> TripsLoader; + + // 3. As a first step, we download the zip file and interpret it. + block GTFSSampleFeedExtractor oftype GTFSExtractor { + url: "https://developers.google.com/static/transit/gtfs/examples/sample-feed.zip"; + } + + // 4. Next, interpret the zip files contents according to the different elements + // from the GTFS standard. + block AgencyInterpreter oftype GTFSAgencyInterpreter { } + block CalendarDatesInterpreter oftype GTFSCalendarDatesInterpreter { } + block CalendarInterpreter oftype GTFSCalendarInterpreter { } + block FareAttributesInterpreter oftype GTFSFareAttributesInterpreter { } + block FareRulesInterpreter oftype GTFSFareRulesInterpreter { } + block FrequenciesInterpreter oftype GTFSFrequenciesInterpreter { } + block RoutesInterpreter oftype GTFSRoutesInterpreter { } + block ShapesInterpreter oftype GTFSShapesInterpreter { } + block StopTimesInterpreter oftype GTFSStopTimesInterpreter { } + block StopsInterpreter oftype GTFSStopsInterpreter { } + block TripsInterpreter oftype GTFSTripsInterpreter { } + + // 5. Finally, write the interpreted tables into a SQLite database + block AgencyLoader oftype SQLiteLoader { + table: "agency"; + file: "./gtfs.sqlite"; + } + block CalendarDatesLoader oftype SQLiteLoader { + table: "calendar_dates"; + file: "./gtfs.sqlite"; + } + block CalendarLoader oftype SQLiteLoader { + table: "calendar"; + file: "./gtfs.sqlite"; + } + block FareAttributesLoader oftype SQLiteLoader { + table: "fare_attributes"; + file: "./gtfs.sqlite"; + } + block FareRulesLoader oftype SQLiteLoader { + table: "fare_rules"; + file: "./gtfs.sqlite"; + } + block FrequenciesLoader oftype SQLiteLoader { + table: "frequencies"; + file: "./gtfs.sqlite"; + } + block RoutesLoader oftype SQLiteLoader { + table: "routes"; + file: "./gtfs.sqlite"; + } + block ShapesLoader oftype SQLiteLoader { + table: "shapes"; + file: "./gtfs.sqlite"; + } + block StopTimesLoader oftype SQLiteLoader { + table: "stop_times"; + file: "./gtfs.sqlite"; + } + block StopsLoader oftype SQLiteLoader { + table: "stops"; + file: "./gtfs.sqlite"; + } + block TripsLoader oftype SQLiteLoader { + table: "trips"; + file: "./gtfs.sqlite"; + } +} +``` \ No newline at end of file diff --git a/apps/docs/versioned_docs/version-0.6.0/user/examples/gtfs-static.md.license b/apps/docs/versioned_docs/version-0.6.0/user/examples/gtfs-static.md.license new file mode 100644 index 000000000..42737858e --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/examples/gtfs-static.md.license @@ -0,0 +1,3 @@ +SPDX-FileCopyrightText: 2023 Friedrich-Alexander-Universitat Erlangen-Nurnberg + +SPDX-License-Identifier: AGPL-3.0-only \ No newline at end of file diff --git a/apps/docs/versioned_docs/version-0.6.0/user/examples/state-codes.md b/apps/docs/versioned_docs/version-0.6.0/user/examples/state-codes.md new file mode 100644 index 000000000..2569f1fb4 --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/examples/state-codes.md @@ -0,0 +1,85 @@ +--- +title: state-codes +--- + +```jayvee +// SPDX-FileCopyrightText: 2023 Friedrich-Alexander-Universitat Erlangen-Nurnberg +// +// SPDX-License-Identifier: AGPL-3.0-only + +// This file belongs to example 2: Electric Vehicles +// Learning goals: +// - Understand how to publish elements to use them in other files + +// 1. The publish keyword can be used directly when defining an element. +publish valuetype UsStateCode oftype text { + constraints: [ + UsStateCodeAllowlist, + ]; +} + +// 2. The publish keyword can be used after a definition as well (see below). +// When using this delayed publish syntax, the published element can also be renamed. Doing so will make it available using the changed name whenever it is imported with the `use` syntax in another file. +constraint UsStateCodeAllowlist on text: value in [ + "AL", + "AK", + "AZ", + "AR", + "AS", + "CA", + "CO", + "CT", + "DE", + "DC", + "FL", + "GA", + "GU", + "HI", + "ID", + "IL", + "IN", + "IA", + "KS", + "KY", + "LA", + "ME", + "MD", + "MA", + "MI", + "MN", + "MS", + "MO", + "MT", + "NE", + "NV", + "NH", + "NJ", + "NM", + "NY", + "NC", + "ND", + "MP", + "OH", + "OK", + "OR", + "PA", + "PR", + "RI", + "SC", + "SD", + "TN", + "TX", + "TT", + "UT", + "VT", + "VA", + "VI", + "WA", + "WV", + "WI", + "WY", +]; + +publish UsStateCodeAllowlist as UsStateCodeConstraint; + +``` \ No newline at end of file diff --git a/apps/docs/versioned_docs/version-0.6.0/user/examples/state-codes.md.license b/apps/docs/versioned_docs/version-0.6.0/user/examples/state-codes.md.license new file mode 100644 index 000000000..42737858e --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/examples/state-codes.md.license @@ -0,0 +1,3 @@ +SPDX-FileCopyrightText: 2023 Friedrich-Alexander-Universitat Erlangen-Nurnberg + +SPDX-License-Identifier: AGPL-3.0-only \ No newline at end of file diff --git a/apps/docs/versioned_docs/version-0.6.0/user/examples/workbooks-xlsx.md b/apps/docs/versioned_docs/version-0.6.0/user/examples/workbooks-xlsx.md new file mode 100644 index 000000000..a37764c42 --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/examples/workbooks-xlsx.md @@ -0,0 +1,103 @@ +--- +title: workbooks-xlsx +--- + +```jayvee +// SPDX-FileCopyrightText: 2023 Friedrich-Alexander-Universitat Erlangen-Nurnberg +// +// SPDX-License-Identifier: AGPL-3.0-only + +// Example 1: LightTrapping +// Learning goals: +// - Understand how to work with XLSX files and workbooks + +// 1. This Jayvee model describes a pipeline +// from a XLSX file with multiple Sheets in the web +// to a SQLite file sink. +pipeline LightTrappingSiliconSolarCellsPipeline { + // 2. We directly get the xlsx file from the web via the HttpExtractor + // The data is provided under CC BY-SA 4.0 + // Saive, Rebecca (2023). Data supporting the publication: + // Light trapping in thin silicon solar cells: a review on fundamentals and technologies. + // 4TU.ResearchData. Dataset. https://doi.org/10.4121/14554815.v1 + block LightTrappingSiliconSolarCellsExtractor oftype HttpExtractor { + url: "https://figshare.com/ndownloader/files/27923598"; + } + + // 3. The incoming file is interpreted as a XLSX file and transformed into a Workbook + // Workbooks contain at least 1 Sheet. Every sheet has a unique name. + block LightTrappingSiliconSolarCellsTextXLSXInterpreter oftype XLSXInterpreter { } + + // 4.1 Here, we pick one sheet with the name 'RefractiveIndexSi GaAs' from the Workbook to use within our pipeline. + // The output type from SheetPicker is Sheet, which was already introduced in the cars example + block LightTrappingSiliconSolarCellsSheetpicker oftype SheetPicker { + sheetName: 'RefractiveIndexSi GaAs'; + } + + block NameHeaderWriter oftype CellWriter { + at: range F1:L1; + write: [ + "F", + "G", + "nm", + "wl", + "n2", + "k2", + "alpha (cm-1)2" + ]; + } + + block LightTrappingSiliconSolarCellsTableInterpreter oftype TableInterpreter { + header: true; + columns: [ + "Wavelength" oftype integer, + "Wavelength (µm)" oftype decimal, + "n" oftype decimal, + "k" oftype text, + "alpha (cm-1)" oftype text, + "nm" oftype decimal, + "n2" oftype text, + "k2" oftype decimal, + "alpha (cm-1)2" oftype decimal + ]; + } + + block LightTrappingSiliconSolarCellsLoader oftype SQLiteLoader { + table: "LightTrappingSiliconSolarCells"; + file: "./LightTrappingSiliconSolarCells.sqlite"; + } + + // 4.2 Here, we pick another sheet named 'Wavelength thickness trapping' from the Workbook + block SecondLightTrappingSiliconSolarCellsSheetpicker oftype SheetPicker { + sheetName: 'Wavelength thickness trapping'; + } + + block SecondLightTrappingSiliconSolarCellsTableInterpreter oftype TableInterpreter { + header: true; + columns: [ + "n" oftype decimal, + "Wavelength (µm)" oftype decimal, + ]; + } + + block SecondLightTrappingSiliconSolarCellsLoader oftype SQLiteLoader { + + table: "SecondLightTrappingSiliconSolarCells"; + file: "./LightTrappingSiliconSolarCells.sqlite"; + } + + LightTrappingSiliconSolarCellsExtractor + -> LightTrappingSiliconSolarCellsTextXLSXInterpreter + -> LightTrappingSiliconSolarCellsSheetpicker + -> NameHeaderWriter + -> LightTrappingSiliconSolarCellsTableInterpreter + -> LightTrappingSiliconSolarCellsLoader; + + // 5. Once the XLSX file is interpreted, we can split the pipeline and + // work separately on the different sheets from our input file + LightTrappingSiliconSolarCellsTextXLSXInterpreter + -> SecondLightTrappingSiliconSolarCellsSheetpicker + -> SecondLightTrappingSiliconSolarCellsTableInterpreter + -> SecondLightTrappingSiliconSolarCellsLoader; +} +``` \ No newline at end of file diff --git a/apps/docs/versioned_docs/version-0.6.0/user/examples/workbooks-xlsx.md.license b/apps/docs/versioned_docs/version-0.6.0/user/examples/workbooks-xlsx.md.license new file mode 100644 index 000000000..42737858e --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/examples/workbooks-xlsx.md.license @@ -0,0 +1,3 @@ +SPDX-FileCopyrightText: 2023 Friedrich-Alexander-Universitat Erlangen-Nurnberg + +SPDX-License-Identifier: AGPL-3.0-only \ No newline at end of file diff --git a/apps/docs/versioned_docs/version-0.6.0/user/expressions.md b/apps/docs/versioned_docs/version-0.6.0/user/expressions.md new file mode 100644 index 000000000..a2cceb49c --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/expressions.md @@ -0,0 +1,87 @@ +--- +sidebar_position: 7 +--- + +# Expressions + +Expressions in Jayvee are arbitrarily nested statements. They consist of: +- literals (e.g., numbers `5` or strings `"Example"`) +- variables (e.g., declared by `from` properties in [Transforms](./transforms.md)) +- operators (e.g., `*` or `sqrt`) + +Expressions get evaluated at runtime by the interpreter to a [built-in _value type_](./value-types/built-in-value-types). + +### Example + +The following expression is evaluated to the `integer` `10`: `(2 + 3) * 2` + +The following expression is evaluated to the `boolean` `true`: `"Example" == "Example"` + +The following expression is evaluated to the `text` `I love Datypus`: `"I love platypuses" replace /platypuses/ with "Datypus"` + +### List of Operators + +#### Arithmetics (binary operators) +- `+` for addition, e.g., `5 + 3` evaluates to `8` +- `-` for subtraction, e.g., `5 - 3` evaluates to `2` +- `*` for multiplication, e.g., `5 * 3` evaluates to `15` +- `/` for division, e.g., `6 / 3` evaluates to `2` +- `%` for modulo, e.g., `5 % 3` evaluates to `2` +- `pow` for power, e.g., `2 pow 3` evaluates to `8` +- `root` for root, e.g., `27 root 3` evaluates to `3` + +#### Arithmetics (unary operators) +- `+` for positive signing, e.g., `+5` evaluates to `5` +- `-` for negative signing, e.g., `-5` evaluates to `-5` +- `sqrt` for square root, e.g., `sqrt 9` evaluates to `3` +- `foor` for flooring a number, e.g., `floor 5.3` evaluates to `5` +- `ceil` for ceiling a number, e.g., `floor 5.3` evaluates to `6` +- `round` for rounding a number, e.g., `floor 5.3` evaluates to `5` + +#### Relational (binary operators) +- `<` for smaller, e.g., `3 < 3` evaluates to `false` +- `<=` for smaller or equal, e.g., `3 <= 3` evaluates to `true` +- `>` for greater, e.g., `3 > 3` evaluates to `false` +- `>=` for greater or equal, e.g., `3 >= 3` evaluates to `true` +- `==` for equal, e.g., `3 == 3` evaluates to `true` +- `!=` for not equal, e.g., `3 != 3` evaluates to `false` + +#### Logical (binary operators) +- `and` for a logical and (both need to be true to evaluate to true) +- `or` for a logical or (at least left or right needs to be true to evaluate to true) +- `xor` for a logical xor (either left or right needs to be true to evaluate to true) + +#### Logical (unary operators) +- `not` for logical negation, `not true` evaluates to `false` + +#### Others (binary operators) +- `matches` for a regex match, e.g., `"A07" matches /^[A-Z0-9]*$/` evaluates to `true` +- `in` for inclusion in an array, e.g., `"a" in ["a", "b", "c"]` evaluates to `true` + +#### Text manipulation (unary operators) +- `lowercase` converts all alphabetic characters in a text to lowercase +- `uppercase` converts all alphabetic characters in a text to uppercase + +#### Text manipulation (ternary operators) +- `replace [...] with [...]` replaces regex matches in a text with a string + +### Operator Details + +#### `in` Operator + +The `in` operator checks whether a value is included in a collection of values. For example: + +```jayvee +4.5 in [3, 6.5] // evaluates to false +3 in [3.0, 6.5] // evaluates to true +"a" in ["a", "b", "c"] // evaluates to true +``` + +The operator supports `text`, `integer` and `decimal` values as operands. The compatibility of left and right operand types follows these rules: +- For the `in` operator we have a type for the needle (left operand) and a type for the elements in the haystack (right operand). +- There is an automated type conversion as long as it is lossless and clearly defined (integer to decimal as of now). +- We allow any combination of operands that has either: (i) An automated type conversion from needle type (left operand) to the type of the elements in the haystack (right operand), or (ii) the other way around. + + +### Further reading +For a deeper documentation of how expressions and operators work internally, refer to the [developer docs](../dev/04-guides/04-expressions-and-operators.md). diff --git a/apps/docs/versioned_docs/version-0.6.0/user/expressions.md.license b/apps/docs/versioned_docs/version-0.6.0/user/expressions.md.license new file mode 100644 index 000000000..17c5d2bad --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/expressions.md.license @@ -0,0 +1,3 @@ +SPDX-FileCopyrightText: 2023 Friedrich-Alexander-Universitat Erlangen-Nurnberg + +SPDX-License-Identifier: AGPL-3.0-only diff --git a/apps/docs/versioned_docs/version-0.6.0/user/intro/_category_.json b/apps/docs/versioned_docs/version-0.6.0/user/intro/_category_.json new file mode 100644 index 000000000..9da536e2f --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/intro/_category_.json @@ -0,0 +1,8 @@ +{ + "label": "Introduction to Jayvee", + "position": 1, + "link": { + "type": "generated-index", + "description": "All the essential information to get started with Jayvee." + } +} diff --git a/apps/docs/versioned_docs/version-0.6.0/user/intro/_category_.json.license b/apps/docs/versioned_docs/version-0.6.0/user/intro/_category_.json.license new file mode 100644 index 000000000..17c5d2bad --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/intro/_category_.json.license @@ -0,0 +1,3 @@ +SPDX-FileCopyrightText: 2023 Friedrich-Alexander-Universitat Erlangen-Nurnberg + +SPDX-License-Identifier: AGPL-3.0-only diff --git a/apps/docs/versioned_docs/version-0.6.0/user/intro/intro.mdx b/apps/docs/versioned_docs/version-0.6.0/user/intro/intro.mdx new file mode 100644 index 000000000..7cd5a4010 --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/intro/intro.mdx @@ -0,0 +1,108 @@ +--- +sidebar_position: 1 +title: Getting Started +--- + +# Introduction to Jayvee + +import MascotImageUrl from '@site/static/img/mascots/mascot2.png'; + +
+
+ +
+ + +

"Making data engineering easy, reliable, and safe"

+ +
+ +Jayvee is a domain-specific language (DSL) for automated processing of data pipelines. +The Jayvee interpreter allows executing such data pipelines on local machines. +Data engineers can use Jayvee and its interpreter to clean and preprocess data for later activities like data science or machine learning. + +## Installation + +Install the interpreter via `npm`. You will need a **nodejs version >= 17.0.0**. + +```bash +npm install -g @jvalue/jayvee-interpreter +``` + +You can install a specific version using the `@`-syntax, e.g., version `0.0.17`: + +```bash +npm install -g @jvalue/jayvee-interpreter@0.0.17 +``` + +## Update + +Details about how to update Jayvee and the VSCode extension can be found [here](./update.md). + +## Usage + +### Show help + +```console +jv -h +``` + +### Run a `.jv` file + +```console +jv +``` + +Run with **additional debug output**: + +```console +jv -d +``` + +With **runtime parameters**: + +```console +jv -e = -e = ... +``` + +### Debug a `.jv` file + +Print debugging is further configured by the parameters `--debug-granularity` and `--debug-target`. + +```console +jv -d -dg peek +``` + +The value of the parameter `--debug-granularity` (short `-dg`) can have the following values: + +- `peek` to log a short summary, including a small subset of data +- `exhaustive` to log a summary, including the full data +- `minimal` to log a summary, including no additional data (default). + To see logs, debugging has to be enabled using the `-d` flag. + +```console +jv -d --debug-granularity peek +``` + +The parameter `--debug-target` (short `-dt`) allows to specify which blocks should be logged for debugging. Separate block names by comma if multiple blocks are targeted. All blocks are logged if the parameter is omitted. + +```console +jv -d --debug-granularity peek --debug-target MyExtractorBlock,MySinkBlock +``` + +## Examples + +You can find multiple examples with inline explanations [here](../examples/README.mdx). You can copy them to your local file system and execute them with the `jv` command on your command line (see [usage](#usage)). + +## VSCode Plugin + +To set up Jayvee locally in VS Code, you need to install the latest Jayvee VS Code extension. +To install the most recent extension, go to our [latest release](https://github.com/jvalue/jayvee/releases/latest) +and download the `jayvee.vsix` file from the release assets. +Next, go to [this page](https://code.visualstudio.com/docs/editor/extension-marketplace#_install-from-a-vsix) and +follow the instructions for installing the downloaded extension. + +## Troubleshooting + +1. Error `structuredClone is not defined` + - Please make sure you use node version 17+. diff --git a/apps/docs/versioned_docs/version-0.6.0/user/intro/intro.mdx.license b/apps/docs/versioned_docs/version-0.6.0/user/intro/intro.mdx.license new file mode 100644 index 000000000..17c5d2bad --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/intro/intro.mdx.license @@ -0,0 +1,3 @@ +SPDX-FileCopyrightText: 2023 Friedrich-Alexander-Universitat Erlangen-Nurnberg + +SPDX-License-Identifier: AGPL-3.0-only diff --git a/apps/docs/versioned_docs/version-0.6.0/user/intro/update.md b/apps/docs/versioned_docs/version-0.6.0/user/intro/update.md new file mode 100644 index 000000000..3dd746c9e --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/intro/update.md @@ -0,0 +1,52 @@ +--- +sidebar_position: 2 +title: Update Jayvee +--- + +# How to update Jayvee + +Jayvee is consistently getting updates. To ensure you use the most recent version, please regularly update the interpreter and VSCode extension. + +### Update Jayvee + Extension + +To update Jayvee, you need to: + +1. Update the interpreter by reinstalling it using npm: + +```bash +npm install -g @jvalue/jayvee-interpreter +``` + +Note: Updating to a specific version works using the `@`-syntax, e.g., version `0.0.17`: + +```bash +npm install -g @jvalue/jayvee-interpreter@0.0.17 +``` + +2. Update your VSCode Extension: + +- Go [here](https://github.com/jvalue/jayvee/releases/latest) to find the latest(or a specific) version of the VSCode Extenstion. + +- Then, download the latest `jayvee.vsix` file. + +- Finally, to install the extension using the CLI, paste the code below into your command line: + +```bash +code --install-extension jayvee.vsix +``` + +If you'd rather use the manual installation, follow this [link](https://code.visualstudio.com/docs/editor/extension-marketplace#_install-from-a-vsix) for the official VSCode documentation. + +### Version Check + +To verify wether the wanted version of Jayvee and VSCode extension where installed successfully, you can run in your command line: + +For **Jayvee**: + +```bash +jv -V +``` + +For the **VSCode extension**: + +Go to the extensions menu, and look for `Jayvee`. The version is then displayed on the information page of the extension. diff --git a/apps/docs/versioned_docs/version-0.6.0/user/intro/update.md.license b/apps/docs/versioned_docs/version-0.6.0/user/intro/update.md.license new file mode 100644 index 000000000..17c5d2bad --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/intro/update.md.license @@ -0,0 +1,3 @@ +SPDX-FileCopyrightText: 2023 Friedrich-Alexander-Universitat Erlangen-Nurnberg + +SPDX-License-Identifier: AGPL-3.0-only diff --git a/apps/docs/versioned_docs/version-0.6.0/user/runtime-parameters.md b/apps/docs/versioned_docs/version-0.6.0/user/runtime-parameters.md new file mode 100644 index 000000000..c68f8ce36 --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/runtime-parameters.md @@ -0,0 +1,27 @@ +--- +sidebar_position: 10 +--- + +# Runtime Parameters + +Property values in Jayvee can be assigned to _values_ or left open for later configuration via _runtime parameters_. + +## Syntax + +_Runtime parameters_ are indicated by the `requires` keyword, followed by the identifier of the parameter. Example + +```jayvee +block CarsLoader oftype SQLiteLoader { + table: "Cars"; + file: requires CARS_SQLITE_FILE; +} +``` + +## CLI + +The interpreter CLI has to define all existing runtime parameters for execution. +Use the CLI flag `-e` to to define them as key-value pairs of identifier and value. + +```console +jv -e = -e = ... +``` diff --git a/apps/docs/versioned_docs/version-0.6.0/user/runtime-parameters.md.license b/apps/docs/versioned_docs/version-0.6.0/user/runtime-parameters.md.license new file mode 100644 index 000000000..17c5d2bad --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/runtime-parameters.md.license @@ -0,0 +1,3 @@ +SPDX-FileCopyrightText: 2023 Friedrich-Alexander-Universitat Erlangen-Nurnberg + +SPDX-License-Identifier: AGPL-3.0-only diff --git a/apps/docs/versioned_docs/version-0.6.0/user/transforms.md b/apps/docs/versioned_docs/version-0.6.0/user/transforms.md new file mode 100644 index 000000000..468a22cb9 --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/transforms.md @@ -0,0 +1,77 @@ +--- +sidebar_position: 8 +--- + +# Transforms + +_Transforms_ are a concept in Jayvee to define the transformation of individual values. +They are similar to functions in programming languages, i.e. they perform computations on some input values and produce output values. _Transforms_ work by mapping input values to outputs using [expressions](./expressions.md). + +:::info Important + +Up to version `0.0.16`, we only supported a single input for transformers! + +::: + +:::info Important + +In its current state, Jayvee only supports a arbitrary numbers of inputs and a single output for transforms. +For the future, it is planned to support arbitrary numbers for outputs as well. + +::: + + +## Syntax + +The general syntax of _transforms_ looks like this: + +```jayvee +transform { + from oftype ; + to oftype ; + + : ; +} +``` + +The `transform` keyword is used to define a _transform_ and give it a name. +The curly braces denote the body of the _transform_. + +The body first contains the definitions of input and output ports. +Input ports are defined using the `from` keyword whereas output ports use the `to` keyword. +Next, they are given a name and, after the `oftype` keyword, typed with a _value type_. + +Below, there needs to be an output assignment for each output port. +The output assignment defines how a particular output value is computed. +They consist of the name of an output port, followed by a `:`. +Next, an [expression](./expressions.md) specifies how the output value shall be computed. +Names of input ports can be used in such an expression to refer to input values. + +### Example + +The following transform converts temperature values from degree Celsius to Kelvin: + +```jayvee +transform CelsiusToKelvin { + from tempCelsius oftype decimal; + to tempKelvin oftype decimal; + + tempKelvin: tempCelsius + 273.15; +} +``` + +The following _transform_ converts a text based status into a boolean value, `true` if the text is `Active`, `false` for any other value: + +```jayvee +transform StatusToBoolean { + from statusText oftype text; + to statusBoolean oftype boolean; + + statusBoolean: statusText == "Active"; +} +``` + +## Applying transforms to table columns + +Transforms can be applied to columns of a table. +Please refer to the documentation of the [`TableTransformer` block type](./block-types/TableTransformer.md) to find out how. diff --git a/apps/docs/versioned_docs/version-0.6.0/user/transforms.md.license b/apps/docs/versioned_docs/version-0.6.0/user/transforms.md.license new file mode 100644 index 000000000..17c5d2bad --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/transforms.md.license @@ -0,0 +1,3 @@ +SPDX-FileCopyrightText: 2023 Friedrich-Alexander-Universitat Erlangen-Nurnberg + +SPDX-License-Identifier: AGPL-3.0-only diff --git a/apps/docs/versioned_docs/version-0.6.0/user/value-types/_category_.json b/apps/docs/versioned_docs/version-0.6.0/user/value-types/_category_.json new file mode 100644 index 000000000..37375108e --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/value-types/_category_.json @@ -0,0 +1,8 @@ +{ + "label": "Value Types", + "position": 5, + "link": { + "type": "generated-index", + "description": "Jayvee supports these different kinds of value types." + } +} diff --git a/apps/docs/versioned_docs/version-0.6.0/user/value-types/_category_.json.license b/apps/docs/versioned_docs/version-0.6.0/user/value-types/_category_.json.license new file mode 100644 index 000000000..17c5d2bad --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/value-types/_category_.json.license @@ -0,0 +1,3 @@ +SPDX-FileCopyrightText: 2023 Friedrich-Alexander-Universitat Erlangen-Nurnberg + +SPDX-License-Identifier: AGPL-3.0-only diff --git a/apps/docs/versioned_docs/version-0.6.0/user/value-types/built-in-value-types.md b/apps/docs/versioned_docs/version-0.6.0/user/value-types/built-in-value-types.md new file mode 100644 index 000000000..5369b529e --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/value-types/built-in-value-types.md @@ -0,0 +1,98 @@ +--- +title: Built-in Value Types +--- + + + +# Description + +For an introduction to _value types_, see the [core concepts](../core-concepts). +_Built-in value types_ come with the basic version of Jayvee. +They are the basis for more restricted [_primitive value types_](./primitive-value-types) +that fullfil [_constraints_](./primitive-value-types#constraints). + +# Available built-in value types + +## boolean + +### Description + +A boolean value. +Examples: true, false + +### Example 1 + +```jayvee +block ExampleTableInterpreter oftype TableInterpreter { + header: true; + columns: [ + "columnName" oftype boolean + ]; +} +``` + +A block of type `TableInterpreter` that + interprets data in the column `columnName` as `boolean`. + +## decimal + +### Description + +A decimal value. +Example: 3.14 + +### Example 1 + +```jayvee +block ExampleTableInterpreter oftype TableInterpreter { + header: true; + columns: [ + "columnName" oftype decimal + ]; +} +``` + +A block of type `TableInterpreter` that + interprets data in the column `columnName` as `decimal`. + +## integer + +### Description + +An integer value. +Example: 3 + +### Example 1 + +```jayvee +block ExampleTableInterpreter oftype TableInterpreter { + header: true; + columns: [ + "columnName" oftype integer + ]; +} +``` + +A block of type `TableInterpreter` that + interprets data in the column `columnName` as `integer`. + +## text + +### Description + +A text value. +Example: "Hello World" + +### Example 1 + +```jayvee +block ExampleTableInterpreter oftype TableInterpreter { + header: true; + columns: [ + "columnName" oftype text + ]; +} +``` + +A block of type `TableInterpreter` that + interprets data in the column `columnName` as `text`. diff --git a/apps/docs/versioned_docs/version-0.6.0/user/value-types/built-in-value-types.md.license b/apps/docs/versioned_docs/version-0.6.0/user/value-types/built-in-value-types.md.license new file mode 100644 index 000000000..42737858e --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/value-types/built-in-value-types.md.license @@ -0,0 +1,3 @@ +SPDX-FileCopyrightText: 2023 Friedrich-Alexander-Universitat Erlangen-Nurnberg + +SPDX-License-Identifier: AGPL-3.0-only \ No newline at end of file diff --git a/apps/docs/versioned_docs/version-0.6.0/user/value-types/primitive-value-types.md b/apps/docs/versioned_docs/version-0.6.0/user/value-types/primitive-value-types.md new file mode 100644 index 000000000..4fac4b448 --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/value-types/primitive-value-types.md @@ -0,0 +1,47 @@ +--- +sidebar_position: 2 +--- + +# Primitive Value Types + +_Primitive value types_ are based on _built-in value types_ and use a collection of _constraints_ to restrict the range of valid values. +Such _constraints_ are implicitly connected via a logical `AND` relation. +Note that the _constraints_ need to be applicable to the base-type of the _value type_ - indicated by the identifier after the keyword `oftype`: + +```jayvee +valuetype GasFillLevel oftype integer { + constraints: [ GasFillLevelRange ]; +} +``` + +## Constraints + +_Constraints_ for _value types_ declare the validity criteria that each concrete value is checked against. + +### Syntax 1: Expression syntax + +The syntax of expression-based _constraints_ uses an expression that evaluates to `true` or `false` for the given `value`. The type of the values the expression is working in is indicated ofter the keyword `on`: + +```jayvee +constraint GasFillLevelRange on decimal: + value >= 0 and value <= 100; +``` + +Refer to the [expression documentation](../expressions.md) for further reading on expressions. + +### Syntax 2: Block-like syntax + +The syntax of _constraints_ is similar to the syntax of _blocks_. +The availability of property keys and their respective _value types_ is determined by the type of the _constraint_ - indicated by the identifier after the keyword `oftype`: + +```jayvee +constraint GasFillLevelRange oftype RangeConstraint { + lowerBound: 0; + lowerBoundInclusive: true; + upperBound: 100; + upperBoundInclusive: true; +} +``` + +Note that the type of _constraint_ also determines its applicability to _value types_. +For instance, a `RangeConstraint` can only be applied to the numerical types `integer` and `decimal`. diff --git a/apps/docs/versioned_docs/version-0.6.0/user/value-types/primitive-value-types.md.license b/apps/docs/versioned_docs/version-0.6.0/user/value-types/primitive-value-types.md.license new file mode 100644 index 000000000..17c5d2bad --- /dev/null +++ b/apps/docs/versioned_docs/version-0.6.0/user/value-types/primitive-value-types.md.license @@ -0,0 +1,3 @@ +SPDX-FileCopyrightText: 2023 Friedrich-Alexander-Universitat Erlangen-Nurnberg + +SPDX-License-Identifier: AGPL-3.0-only diff --git a/apps/docs/versioned_sidebars/version-0.6.0-sidebars.json b/apps/docs/versioned_sidebars/version-0.6.0-sidebars.json new file mode 100644 index 000000000..217726ac9 --- /dev/null +++ b/apps/docs/versioned_sidebars/version-0.6.0-sidebars.json @@ -0,0 +1,14 @@ +{ + "userDocsSidebar": [ + { + "type": "autogenerated", + "dirName": "user" + } + ], + "devDocsSidebar": [ + { + "type": "autogenerated", + "dirName": "dev" + } + ] +} diff --git a/apps/docs/versioned_sidebars/version-0.6.0-sidebars.json.license b/apps/docs/versioned_sidebars/version-0.6.0-sidebars.json.license new file mode 100644 index 000000000..42737858e --- /dev/null +++ b/apps/docs/versioned_sidebars/version-0.6.0-sidebars.json.license @@ -0,0 +1,3 @@ +SPDX-FileCopyrightText: 2023 Friedrich-Alexander-Universitat Erlangen-Nurnberg + +SPDX-License-Identifier: AGPL-3.0-only \ No newline at end of file diff --git a/apps/docs/versions.json b/apps/docs/versions.json index 4407a2477..e69502b55 100644 --- a/apps/docs/versions.json +++ b/apps/docs/versions.json @@ -1,4 +1,5 @@ [ + "0.6.0", "0.5.0", "0.4.0", "0.3.0", diff --git a/package-lock.json b/package-lock.json index 56f9535bc..0022b027a 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "jayvee", - "version": "0.5.0", + "version": "0.6.0", "lockfileVersion": 2, "requires": true, "packages": { "": { "name": "jayvee", - "version": "0.5.0", + "version": "0.6.0", "dependencies": { "@docusaurus/core": "2.4.1", "@docusaurus/preset-classic": "2.4.1", diff --git a/package.json b/package.json index e45396f37..08af2be90 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "jayvee", - "version": "0.5.0", + "version": "0.6.0", "scripts": { "nx": "nx", "format": "nx format:write",