Skip to content

AIL Framework v5.9 Released – New Features such as dom-hash correlation, improvements and many bug fixes
Compare
Choose a tag to compare
@adulau adulau released this 18 Oct 13:59
· 40 commits to master since this release
v5.9
ee576ca
This commit was signed with the committer’s verified signature.
Terrtia Thirion Aurélien
GPG key ID: 1E1B1F50D84613D0
Learn about vigilant mode.

AIL Project Release v5.9 - 2024-10-18

We are glad to announce the release of AIL v5.9, packed with numerous updates and fixes that enhance the performance and features of the framework. This release focuses on improvements in qrcode handling, dom-hash support, title processing, and crawler operations, among other critical updates.

dom-hash is a structural fingerprint of the HTML’s Document Object Model (DOM) originally developed by CERT.PL.

The fingerprint is calculated by extracting all the tag names (ignoring the content itself as well as attributes of the HTML Page). The tag names are concatenated with a pipe value |, hashed using the SHA-256 algorithm, and truncated to the first 32 characters.

Screenshot from 2024-10-18 15-57-02

Software such as LookyLoo or MISP have implemented the algorithm, the AIL framework now support the dom-hash algorithm to cluster and group similar page structure.

What's New in v5.9

Changes

  • [qrcodes] Daterange search: Sort qrcode by content. [terrtia]
  • [dom-hash] Add dom-hash object to compute dom-hash for domains and crawled items. [terrtia]
  • [CEDetector] Tag domains. [terrtia]
  • [CEDetector] Add detect message functionality. [terrtia]
  • [titles] Setup Titles queues and integrate CEDetector module. [terrtia]
  • [tools] Add reprocess option for Titles + CEDetector. [terrtia]
  • [crawler] Disable unsafe title auto-tagging. [terrtia]
  • [module] General debugging and improvements. [terrtia]
  • [module] Add CEDetector module. [terrtia]
  • [crawler] Tag domain by vanity. [terrtia]
  • [crawler] Crawl list URLs: Filter duplicates. [terrtia]
  • [crawler] Submit free text of URLs to crawl. [terrtia]
  • [onion module] Filter out Onion v2 domains. [terrtia]
  • [show domain] Display title content and fix item tags. [terrtia]
  • [crawler] Update TOR user agent for more efficient crawling. [terrtia]
  • [message image] Show extracted QR codes in messages. [terrtia]
  • [domain lookup] Extract domains from URL input. [terrtia]
  • [api] Rename domain lookup function. [terrtia]

Fixes

  • [module] Fixed a typo in the module. [terrtia]
  • [tag] Tag delete: Fixed update for first/last seen times. [terrtia]
  • [show domain] Fixed the correlation button when correlation count is zero. [terrtia]
  • [crawler] Filter lookup tags for better accuracy. [terrtia]
  • [crawler] Filter lookup for parent + domain daterange. [terrtia]

We hope these updates enhance your experience with the AIL framework. Stay tuned for future updates as we continue to improve and expand AIL’s capabilities.

Funding

MISP-LEA, a collaborative endeavor between Shadowserver and CIRCL, is a 24-month initiative funded by the European Union. The project’s central aim is to establish operational and enduring MISP and AIL instances dedicated specifically to law enforcement agencies. This setup will facilitate a smoother exchange of evidence between law enforcement agencies and improve the onset of collaborative investigations. For this purpose, the system will ingest data from Shadowserver’s ransomware and C2 infrastructure tracking.

Law enforcement agencies willing to discover and leverage the MISP-LEA platform can apply on the misp-lea.org website.

Assets 2
Loading