A dataspace management system

Project Onboarding Documentation

Table of Contents

1. Introduction
2. Executing ODIN
   • Docker
   • Creating the environment
     • Requirements
     • Backend
     • Intents modules
     • Frontend
3. Project Structure
   • Backend
     • Database division
     • Architecture
     • Code style
   • Frontend
     • Architecture
     • Dependencies
4. Demo
5. People
6. Related links
7. License

Introduction

Welcome to ODIN! This documentation is designed to help new developers quickly get started with the project and understand its architecture, dependencies, and configuration.

Executing ODIN

Docker

The easiest way to execute ODIN is via the Docker environment provided in the repository. To do so, make sure that Docker is installed in your machine and running. Once this is done, go to the root folder of the repository and run the following command:

docker-compose build

This will build the containers and the environment. Once it finishes, to launch ODIN run the next line, also in the root folder:

 docker-compose up

After some seconds all the services will have deployed and will be ready to be used.

Note: to use the intent anticipation/generation module it is necessary to populate a GraphDB instance with an initial set of triples. To do so, go to the deployed GraphDB URL (by default, localhost:7200), select import on the top left and then upload RDF files. Upload the following file: KnowledgeBase.nt

Creating the environment

This section details how to set up ODIN outside of the Docker environment, which requires downloading all the necessary dependencies and running the corresponding services.

Requirements

Ensure that you have the following prerequisites installed:

• Node.js (version >=20.0.0)
• NPM
• Yarn You can install it using npm install -g yarn (or on a macOS install it using Homebrew using brew install yarn)
• Quasar You can install it using npm install -g @quasar/cli
• Gradle (version >=8.5)
• Java (version 11)
• Python (version >=3.12)
• Hadoop: only for Windows. Create a folder, e.g. C:/Work/Hadoop/bin, and place there the files hadoop.dl and winutils.exe from the following link. Finally, add the path (e.g. C:/Work/Hadoop/bin) to the path environment variable.

Once everything is installed, clone the repository:

git clone https://github.com/dtim-upc/ODIN.git
cd ODIN

Backend

1. In the root folder execute the following command to compile all the different modules and run their respective tests. For Windows:

   gradlew build

   For Unix:

   ./gradle build

2. In the same directory, run the following command to launch the backend API. For Windows:

   gradlew bootRun

   For Unix:

   ./gradle bootRun

Intents modules

The intent-generation functionalities are separated into two different modules, which can be found in the backend folder. Both are built with Python and, as such, can not be natively executed by ODIN. Instead, they offers APIs to which ODIN connects to and sends requests.

1. Go to ODIN/Modules/IntentSpecification2WorkflowGenerator. First install all the required libraries with the following command:

   pip install -r requirements.txt    

   Then, launch the API with the following line

   flask --app api\api_main.py run    

2. Go to ODIN/Modules/IntentAnticipation. First install all the required libraries with the following command:

   pip install -r requirements.txt    

   Then, launch the APIs (there are two) with the following line

   python .\start_apis.py  

Frontend

1. Open in the terminal the ODIN/frontend folder.

2. Execute npm install.

3. Then, execute yarn install (on macOS it is possible you need to run yarn install --ignore-engines).

4. Finally, execute quasar dev (or on a macOS do it from the node_modules directory using node_modules/@quasar/app-vite/bin/quasar dev. This will open your browser with the URL http://localhost:9000/#/projects.

Note: that you must have Quasar CLI as it's mentioned in the Prerequisites section. If there's an error like Global Quasar CLI • ⚠️ Error Unknown command "dev", it's because you are not in the correct path, or you don't have Quasar CLI installed.

Project Structure

The project structure is primarily divided into two code sections (backend and frontend) and a third section for documentation, where the UML diagram is located. To access the UML diagram, VisualParadigm is required.

The code sections consist of two IntelliJ projects. Please note that access to IntelliJ Ultimate is available with a UPC email account.

Backend

ODIN's backend is developed in Java and utilizes the Spring framework. As the backend for ODIN, its primary role is to orchestrate calls to various Nextia libraries within the department. Each Nextia library serves a distinct functionality, and ODIN also manages application persistence.

ODIN's persistence is bifurcated into two components: graph persistence, currently using Jena, and relational database persistence using ORM with the embedded H2 database.

The different Nextia modules are presented next:

• NextiaCore: Contains domain class definitions for the system and serves as a cross-cutting library. It lacks business logic, focusing solely on Plain Old Java Object (POJO) classes.

• NextiaDataLayer: Manages an intermediate persistence layer for accessing data from uploaded datasets.

• NextiaBS: Extracts schemas from datasets loaded into ODIN and generates wrappers, which are SQL queries demonstrating how to access the original dataset's data.

• NextiaJD: Join discovery. Responsible for automatically discovering attribute alignments from datasets.

• NextiaDI: Handles data integration. Given two schemas and alignments, it integrates them and generates the corresponding integrated graph.

Internally within ODIN, there are NextiaStore and NextiaGraphy. The former manages persistence, while the latter generates a string representation for visually rendering graphs generated by NextiaBS in the frontend.

The diagram described above would look like this:

graph TD;

subgraph ODIN
  NextiaStore[fa:fa-database <a href='https://github.com/dtim-upc/ODIN/tree/main/api/src/main/java/edu/upc/essi/dtim/odin/NextiaStore' >NextiaStore</a>]
  NextiaGraphy[fa:fa-paint-brush <a href='https://github.com/dtim-upc/ODIN/tree/main/api/src/main/java/edu/upc/essi/dtim/odin/nextiaInterfaces/NextiaGraphy' >NextiaGraphy</a>]
end

subgraph MODULES
  NextiaBS[fa:fa-file-code <a href='https://github.com/dtim-upc/NextiaBS' >NextiaBS</a>]
  NextiaJD[fa:fa-copy <a href='https://github.com/dtim-upc/NextiaJD2' >NextiaJD</a>]
  NextiaDI[fa:fa-object-group <a href='https://github.com/dtim-upc/NextiaDI' >NextiaDI</a>]
  NextiaQR[fa:fa-search <a href='https://github.com/dtim-upc/NextiaQR' >NextiaQR</a>]
end

subgraph TRANSVERSAL
  NextiaCore[fa:fa-book <a href='https://github.com/dtim-upc/NextiaCore' >NextiaCore</a>]
  NextiaDataLayer[fa:fa-universal-access <a href='https://github.com/dtim-upc/NextiaDataLayer' >NextiaDataLayer</a>]
end

ODIN-->NextiaCore;
ODIN-->NextiaDataLayer-->NextiaCore;
ODIN-->NextiaBS-->NextiaCore;
ODIN-->NextiaJD-->NextiaCore;
ODIN-->NextiaDI-->NextiaCore;
ODIN-->NextiaQR-->NextiaCore;
NextiaJD-->NextiaDataLayer;
NextiaQR-->NextiaDataLayer;

Loading

Summary modules structure

Transversal Modules

• NextiaCore: Contains domain class definitions and acts as a cross-cutting library.
• NextiaDataLayer: Manages an intermediate persistence layer to access data from loaded datasets.

Nextia Modules

• NextiaBS: Extracts schemas from loaded datasets in ODIN and generates wrappers.
• NextiaJD: Join discovery. Automatically discovers attribute alignments.
• NextiaDI: Handles data integration. Integrates two schemas and generates the corresponding integrated graph.
• NextiaQR: (Optional) Performs query-based searches on datasets.

Internal NextiaStore and NextiaGraphy

• NextiaStore: Manages persistence.
• NextiaGraphy: Generates a string representation for rendering graphs in the frontend.

Database Division

NextiaStore is divided into two types of databases:

1. Relational Database (H2):

   • This is used to store relational data as could be some part of a project, datasets, user information...
   • If you must change something, make the necessary changes in the orm.xml in the api/src/main/resources/META-INF/orm.xml path. This file contains the mapping one-to-one with our domain objects persisted (ORM syntax).

2. Graph Database (Jena):

   • This one is used to store the schemas of our graphs. This graphs now are implemented under an interface as Model Jena's object.

A usage example of the two databases are available in getProjectById(String projectId) or also saveProject(Project project) where we must get from the RelationalDB some information and the other related with its schemas from the GraphDB.

Both types are structured by the same pattern (see NextiaStore folder): Factory, Interface, Implementation. See the examples to know how to use them.

UML Diagram

The UML diagram provides an overview of the project structure. The key sections are described below:

Architecture

To see the UML class diagram, download Visual Paradigm Community Edition. It will ask you an email by the end of the installation or the first time you open the program just to send you the key activation code.

Then open the UML conceptual ODIN.vpp

It should look like this:

Simple interaction with a Nextia module

sequenceDiagram
    participant Frontend
    participant Controller
    participant Service
    participant Interface
    participant InterfaceImpl
    participant Nextia

    Frontend->>Controller: petition
    Controller->>Service: ask service
    Service->>Interface: request template
    Interface->>InterfaceImpl: d
    InterfaceImpl->>Nextia: info
    Nextia-->>InterfaceImpl: answer
    InterfaceImpl-->>Interface: answer
    Interface-->>Service: html & javascript
    Service-->>Controller: iframe ready
    Controller->>Service: set mermaid data on iframe
    Service->>Service: render mermaid
    Controller-->>Frontend: answer

Loading

Code Style

Overview

This project follows a set of coding standards and practices to ensure consistency, readability, and maintainability of the codebase. Adhering to these guidelines will help streamline collaboration and make the codebase more accessible to developers.

Branching Strategy

We follow a feature-based branching strategy, where each new feature or task is developed on a dedicated branch. Once a feature is complete and tested, it is merged into the main branch. Branches should be named descriptively, reflecting the nature of the feature or task they represent.

Example git graph:

gitGraph:
commit "Ashish"
branch newbranch
checkout newbranch
commit id:"1111"
commit tag:"test"
checkout main
commit type: HIGHLIGHT
commit
merge newbranch
commit
branch b2
commit

Loading

After a successful merge, the feature branch should be deleted to keep the repository clean and avoid unnecessary noise.

Code Organization

Main Branch

The main branch must always contain a working version of the project. Developers should avoid pushing directly to the main branch and instead use feature branches for development.

Naming Conventions

• CamelCase: Variable and method names should use CamelCase. For example: myVariable, calculateTotalAmount().

• Class Naming:

  • Interfaces: Named as ConcreteNameInterface.java (e.g., GraphStoreInterface.java).
  • Implementations: Named as ConcreteNameTechnologyUsedImpl (e.g., GraphStoreJenaImpl.java). Controller and Service Naming: Controllers should be named with a descriptive term followed by "Controller" (e.g., IntegrationController). The corresponding service should be named with a similar descriptive term followed by "Service" (e.g., IntegrationService).

• Communication between Components: Controllers only communicate with their associated services. Services can interact with other services but should not communicate directly with controllers. Interfaces are used for communication with other modules.

Dependency Management

• Interfaces for Communication: Use interfaces to define contracts for communication between modules. This enhances modularity and allows for easier maintenance and testing.

Frontend

Architecture

The frontend is organized by components: forms, tables, etc.

There's an special form: FormNewRepository.vue. It has a static part that represents de global repository information and then the dynamic part that charges from the backend all the jsons files located in the backend in this path api/src/main/resources/frontend-schemas/RepositoryForms. By using this json structures it builds the hierarchy part of the repository specialitation.

Component Overview:

1. Global Repository Information:

There is a static part of the form that represents global repository information. This part likely includes fields and input elements for information that is applicable to the entire repository.

2. Dynamic Part - Repository Specialization:

The dynamic part of the form is responsible for loading data from the backend. Specifically, it loads JSON files located in the backend at the path: api/src/main/resources/frontend-schemas/RepositoryForms.

2.1. Dynamic Part Details:

• JSON Files in Backend:

In the backend, there is a directory path specified as api/src/main/resources/frontend-schemas/RepositoryForms. This path contains JSON files.

• Purpose of JSON Files:

These JSON files likely contain schema or structure information for various forms related to repository specialization.

• Runtime Loading:

At runtime, the FormNewRepository.vue component fetches and loads these JSON files from the backend.

• Building Hierarchy:

The loaded JSON structures are used to dynamically construct or build a hierarchy within the form. This hierarchy is likely related to the specialization of the repository and is represented in the form.

Dependencies

Dependencies are located in the package.json file.

Demo

2023-11-29.10-35-53.mp4

People

These researchers have significantly contributed to shaping ODIN into a more robust and feature-rich platform.

• Javier Flores
• Cristina Gómez
• Sergi Nadal
• Oscar Romero
• Marc Maynou
• Juan Emilio Olivan
• Víctor Asenjo

Related links

License

This project is licensed under the GNU General Public License v3.0.