New operators, lists loaded by CSV and lots of fix

One last one for the road !
As the end of the year is near, here's our 30th release for 2024, packed with usefulness

New operators

• Start with / end with : makes it easy to check the beginning or the end of a string against another value or list. Very useful for getting the country out of the IBAN, the phone operator suffix or the email domain name
• Is multiple of : you can now check with ease if a number is round, and set the thresholds.

Load lists by CSV

No more need to enter values manually to update lists : you can now upload CSV up to 9999 values for your lists. And if you need to update those, you can download them

Case manager

We're still chasing the clicks on the road to efficiency in the case manager

• If a case holds a single decision, it's now displayed by default

Test run

You can now stop an existing test run at any time.

Fixes :

• Unclickable part in the decision viewer
• Access to analytics for all roles
• Adding a description on an object of the data model works again

Documentation 📖

Schematics of Marble technical and functional architecture added to the repo: see the installation/deployment.md page

v0.29.1

What's Changed

• fix: version typo in docker-compose.yaml by @balzdur in #63

Full Changelog: v0.29.0...v0.29.1

Test run, new date operator and case manager efficiency

Right in time for the holidays, it's one of our largest release yet!

Test run

Modifying a scenario or a rule safely is challenging : you'll want to check the impact on detection volumes and efficiency before pushing to production.
With our test run feature, you can know modify an existing scenario, and run the new version in parallel during a set period.

Marble provides you in realtime the difference between both version, both in scenario outcome and rule by rule.
You are now able to check for abnormalities, wrong checks and volume mismatch before releasing a new version of a scenario in production. And you can keep the test run synthesis as an audit proof for the rationale behind every modification.

The test run allows you to compare a scenario version to a live one and define a maximum duration for the run.
Available for SaaS or under self-hosted licencing

Case manager efficiency

With our latest phase of case manager redesign, we're now reducing context switching with :

• Alert prioritization : a default activated toggle displays only the rules generating a hit for the case in each decisions
• Direct to decision display : the decision tab is now set as the default display when opening a case
• Foldable case history : the case timeline is accessible through a button on the right side of the screen and can be folded to save space for investigations
• Trigger object display : the trigger object is now easily accessible at the right of each linked decision
• Pivot accessibility : the pivot value has been moved to the top of each decision trigger object, making it easy to reach and copy in your own back-office.
  Those improvements should significantly reduce the number of clicks needed to overview a case and start the investigation.

We've also added the possibility to upload several files at once to a case.

Hour / Day / Month / Year operator

The builder now allows you to get an extract of a timestamp to use within rules. Based on a timestamp, you can extract the hour, the day of the week (1 = monday), the day of the month, the month of the year or the year.
It's now really easy to check if an operation happened in the middle of the night, or to integrate specific checks during weekends.
You'll need to set the timezone of your instance in the new setting / scenario tab.

Self-hosting

API timeout delays can now be configured

⚠️ API breaking change

We unfortunately had to make a breaking change to the decisions listing API. The GET /decisions endpoint no longer returns the count nested object, the start_index and the end_index of the returned items. Instead, the endpoint now returns a has_next_page boolean field that lets you know if a next pagination page is available. See our documentation page for the up-to-date details.
We absolutely hate to make breaking changes to our public API, but this was one case where the implementation proved fundamentally not scalable, on a feature that we expect was not widely used by API, and where we preferred to make the breaking change now rather than later.

Bugfixes and others

• data model tables, fields, links can now only be created in lower case alphanumeric and "_", and must start with a letter. Backward compatibility is maintained for older fields and tables, but this will avoid errors in the future
• The timeout durations for the API endpoints can now be configured, for self-hosted users of Marble. See the .env.example file for examples (use the BATCH_TIMEOUT_SECOND environment variable)
• Fix a bug where a "required" field in a table could not be added once there were objects ingested into the table. "Requiredness" is now enforced at ingestion time, not in the table itself.

---

We're not done for this year, stay tuned !

Scenario home page, null value handling in rules, UI revamp finished

Features

This week

• we added a brand new home page for every scenario. This is now the entry point to find everything related to the scenario's set up: scheduled and manual batch executions, workflows, versions... with links to the relevant documentation. In the long term, this will also make it easier for us to add more features and configuration at the scenario level.
• we added translation support for the arabic language. The app will also automatically adapt its layout (right to left) when used with the arabic locale. As before, the locale can be selected at the top-left (well, top-right in arabic) corner, next to the logout button, by clicking on the "Marble" logo. Please note that some of the translations have been added automatically by Copilot translation, so feel free to point out any corrections to us (for instance by proposing a pull request). Many thanks to @Morghim for the contribution.
• we now allow to gracefully handle NULL values (and empty string) during the rule execution, by using the "is empty"/"is not empty" operators ! See the screenshots below, documentation will be added shortly for this feature here

Improvements

We also finished a UI revamp that had been in the works for a few weeks. The case manager and scenario page now have a refreshed look, to make some features more easily discoverable (e.g. rule snoozing from the case manager), and also to make place on the screen for future features we want to add.

Bugfixes

• Batch executions of a scenario now properly make all decisions with the scenario version defined on the batch, instead of using the currently live version
• Better line breaking on long rule names
• PG_PASSWORD etc environment variables are no longer required if PG_CONNECTION_STRING is passed
• thresholds for decisions are now capped at -10000 / + 10000, and the limit is enforced in the builder

Some illustrations

New scenario home page

Revamped case manager views

Null value handling in rules

Creating the rule

Decisions with the rule

0.27.0

This week's release contains more bugfixes and small improvements

We are working on the next big feature, coming soon !

App

• layout improvements
• better (formatted) display of timestamps in rule executions
• no longer possible to create a field called "id" in the data model which was causing errors (the name is reserved)

API

• It is now possible to configure a separate database for ingested data (separate from the main db containing organizations, scenarios, decisions, cases...). This is mostly useful for our managed service, but may be helpful for some self-hosted deployments at scale.
• Fix a bug where a scheduled execution that found no objects on which to create decisions, remained stuck in "processing" status
• You can now pass the "allowssl" parameter for the DB connection (through the PG_SSL_MODE env variable). The default value if not passed is now set to prefered (though we recommend require in production).
• Finally, you can also configure the DB connection by using the PG_CONNECTION_STRING env variable (instead of passing PG_USER, PG_PASSWORD etc separately). This can be useful as it gives you more granular control on the options passed.

UI improvements and bugfixes

This week's release is lightweight, it mostly includes improvements on some UI pages and some bug fixes (which hopefully you had not noticed).
Also contains some performance improvements if you are creating large volumes of decisions.

Stay tuned for more the next big feature in a couple of weeks !

0.25.0

Helm chart is available !

Marble now publishes a helm chart, to help with your cloud deployment.
You can find more information here.
The helm chart can be found at europe-west1-docker.pkg.dev/marble-infra/marble/marble-application

Nested aggregates

You can now compute aggregates within aggregates, that is to say use aggregate values as filters in another aggregate.
For instance, you could do something like "compute the number of transactions with value higher than the average transaction for that user".

Reworked batch execution mode

We reworked how batch scenario execution works, to make it more reliable.
In particular, any transient errors encountered during a batch execution will now be retried a number of times.

---

Deployment changes

⚠️⚠️ The recommended and supported way to deploy Marble, including background tasks, is now the one that is used in the docker-compose file in this repository and in the Helm chart.
In particular, running the cron jobs for batch execution, webhooks retry is deprecated and will no longer be supported.
The entrypoint --cron-scheduler continues to be supported for backwards compatibility, but using the new --worker entrypoint is recommended.
⚠️⚠️

Bugfixes and improvements

This week's release mostly contains bug fixes and some background work 🪛

Stay tuned for bigger changes next week !

Partial list:

• fix a bug on decision filtering with inbox_id
• fix a bug on french translations on the scenario outcome panel
• fix a bug on the rule name translation when cloning a rule or creating a new rule

Batch exec performance update & batch ingestion by API

EN

This week's release contains the following new features and improvements

• Improvement of the batch execution of scenarios: the computation can now be done on a much larger number of objects without encountering memory errors, and you can view the progress of a batch scenario. More improvements are planned on this soon.
• Batch ingestion by API is now available ! Send your objects by batches of up to 100 (or more if you are self-hosting)

Other improvements and bugfixes

• fixed a bug introduced in the previous release, where scenarios with a trigger condition of the form "constant = constant" no longer properly executed in batch mode
• improvement of the relevance of the liveness check
• removal of the cron job with "--scheduler" entrypoint. Its content has been merged with another cron job. You are invited to remove the corresponding cron job if you are scheduling the jobs yourself (rather than executing the background job "cron-scheduler")

---

FR

La release de la semaine contient les nouvelles fonctionnalités et améliorations suivantes

• Améliorations sur l'exécution en batch de scénarios: le calcul peut maintenant se faire sur un nombre beaucoup plus grand d'objets sans que le job ne soit en erreur, et vous pouvez observer la progression de l'avancement du job.
• L'ingestion par batch est disponible ! Envoyez vos objets à ingérer dans Marble par batchs de jusqu'à 100 éléments (ou plus en self-hosted).

Autres améliorations et bugfix

• correction d'un bug introduit dans la release précédente, où les scénarios avec une trigger condition de la forme "constante = constante" ne s'exécutaient plus en mode batch
• amélioration du liveness check
• suppression du job cron avec l'entrypoint --scheduler (son contenu a été fusionné avec un autre job cron). Vous êtes invités à supprimer le cron job correspondant (si vous avez vous-mêmes schedulé les cron jobs, plutôt que d'exécuter le job "cron-scheduler" en background)

v0.22 : batch processing & display improvements

This release is the beginning of several major improvements on batch process and frontend UI.
Stay alert, more are coming

Front

• Long field & object names are now displayed properly in the builder
• Language selection is retained after logout
• several dependencies updated for weight and secIT reasons

Backend

• Batch processing is now prepared based on trigger conditions, improving speed of execution in most cases.