PIX Architecture

[iharsuvorau]

Hi @JonasBerx, @ira-halenok, @orlenyslp, @marlondumas.

The Friday's discussion about the architecture is captured at this page, https://github.com/AutomatedProcessImprovement/process-improvement-explorer/wiki/Architecture-Discussion-1

Let's keep it going here (Discussions section of the repository), there (Wiki for longer storage), and during the future meetings.

[JonasBerx]

Maybe a little specific already;
What techstack will we use?

For UI, i suppose we stick with React.
The "backend" system will be written in Python or are we going to use something new/faster?

Are we going to use NoSQL db's for disk storage?

I think its good that we define those asap so we have an idea what the techstack will be for the prototype. As we aim to complete that before end of June.

Also for work division, i think it is good we could clear out who is available and when so tasks can be assigned accordingly.

These few days will be spent on setting up the requirements and dev environment and we can aim to start writing code by Wednesday/Friday?

[iharsuvorau]

What techstack will we use?

The "backend" system will be written in Python or are we going to use something new/faster?

I guess, as soon as we agree on HTTP API endpoints and provided functionality by it, it's better we don't micromanage each service leaving fun for developers to try new things and use their favourite tools.

However, in general, we use Python for the back-end and data analysis. We optimise only when something is slow and is a problem for us. As the slowest parts in our case are anyway related to our core services that gonna run from minutes to hours, the PIX Backend and utility services performance wouldn't be the bottleneck.

For UI, i suppose we stick with React.

Agree. No reason to change it.

Are we going to use NoSQL db's for disk storage?

I'd split this question into two: what do we use for (1) file storage and (2) metadata storage?

For file storage, it's so much easier just to use a disk and the file system. It's so trivial to read and write to disk that developers often overlook the file system and always default to DBMS, but FS a very well designed piece of software. Providing redundancy is easy (just rsync everything onto another disk). Using SHA-256 hashes of files' content for file names would ensure we don't store same files twice.

For other data, e.g., assets metadata, users, projects, it might be a database. Which one? The simplest db would be SQLite. You don't need to run a separate container for it, meaning, deployment and operations are easier (but I'd explore how it handles concurrent connections). We're also not going to serve the whole world with our database. Because SQLite is portable and lightweight, it's a good choice for microservices unless you need something specific from Postgres or Mongo (or FerretDB).

However, the database choice is not a critical question, especially, when this database is internal to a service and other services don't have access to it. So, I'd guess, it's okay to leave it for a developer to decide. Even storing metadata in a file on disk would work for quite some time for our services. It would also serve as a good first implementation for a service, because it would allow to focus instead on designing better APIs than spending time on more complex tools and their deployment.

What are you proposing for data storage?

---

I'm in the office on Tue and Wed, if you too, we can discuss details in person. I'm on vacation from Jun 15 till Aug 4. But I'm in Estonia, so I'm available in case there's something urgent.

[iharsuvorau]

@JonasBerx @marlondumas This is the first approach to user stories, https://github.com/AutomatedProcessImprovement/process-improvement-explorer/wiki/User-stories. Please, take a look. Could anyone please fill in a user story for Optimus because I have a vague idea of what it does and how (input, output, results representation).

Acceptance criteria often include implementation details but I thought it might be helpful for a developer to understand what is expected.

Column mapping support for an event log is questionable, especially, for the first approach.

Implementing support for users brings a big set of functionality that can be delayed for now because it's not the core service.

[iharsuvorau]

@JonasBerx @marlondumas These are the high-level design principles we discussed with Jonas. I hope it can make onboarding of new developers easier and also unify our approach and technologies. If I'm missing something or you think it's unnecessary, please, let me know.