AI for semantic recognizing of the users actions

A toolkit for automatic labeling of the screen recordings

Input

• A directory with screenrecording of arbitrary sessions on mac OS
• a screenshot

Output

• a video marked with activity

  • Browser
  • Video game
  • Terminal
  • IDE

• source code detection and extraction

• events

• etherium wallets

• autolabeling of the applications on screen

• autolabeling of the UI elements inside of the application windows

Tools

keylogger

Macro player

• ROS interface to macro player

Generic screenshot containing useful data generator

MacOSX application auto labeler

Creates windows, creates random elements of UI, labels them, interacts with them

HTML page auto labeler

Creates random webpages, labels them, scrolls them, interacts with them

Models

• a model trained on the videos where user is installing an opensource software on the developer machine reading the intructions on the open source software website

• semantic text recognition model:

  • https://github.com/open-mmlab/mmocr