Skip to content

AutomEditor is an AI based video editor that helps video bloggers to remove bloopers automatically. It uses multimodal spatio-temporal blooper recognition and localization approaches. The models were trained in keras and integrate feature fusion techniques from face, body gestures (skelethon), emotions progression, and audio features

License

Notifications You must be signed in to change notification settings

toxtli/AutomEditor

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

47 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Alt text

AutomEditor

AutomEditor is an AI based automatic video editing tool that helps video bloggers to remove bloopers automatically. It uses multimodal spatio-temporal blooper recognition and localization approaches. The models were trained in keras and integrate feature fusion techniques from face, body gestures (skelethon), emotions progression, and audio features.

Alt text

Demo

http://www.carlostoxtli.com/AutomEditor/frontend/ (You need to link your own backend)

Alt text

Instructions

Requirements

Install the following software

  • ffmpeg
  • OpenSMILE
  • OpenFace
  • python 2 or 3 Place the OpenSMILE and OpenFace directories at the side of this cloned directory.

Data preprocessing

Run the scripts in the following order

  • ./backend/feature_extraction/extract_audio_files.py
  • ./backend/feature_extraction/generate_audio_feature.py
  • ./backend/feature_extraction/generate_face_feature.py
  • ./backend/feature_extraction/generate_face_visual.py
  • ./backend/feature_extraction/generate_emotion_feature.py
  • ./backend/feature_extraction/generate_body_feature.py
  • ./backend/feature_extraction/generate_body_visual.py Copy all the .pkl files to the ./backend/data/ folder

Training

For training individual models just give the specific model

  • ./backend/experiment/train.py --model emotion_feature For training fusion models just give the comma separated list of models and add the fusion parameter
  • ./backend/experiment/train.py --fusion --model body_fusion,face_fusion,audio_feature,emotion_feature

User interface

Run the web server

  • ./backend/feature_extraction/server.py Expose the frontend directory to the web, for instance:
  • cd ./frontend/
  • sudo python -m SimpleSTTPServer 80 Navigate the URL, for instance: http://localhost

Architecture

Alt text

References

This work is based on this paper: https://arxiv.org/pdf/1805.00625.pdf

Slides: https://docs.google.com/presentation/d/1ds8nGurfxUgnizWwI61eLh8OTJ4l4K486LwPe38I0Eo

Publication in progress ...

About

AutomEditor is an AI based video editor that helps video bloggers to remove bloopers automatically. It uses multimodal spatio-temporal blooper recognition and localization approaches. The models were trained in keras and integrate feature fusion techniques from face, body gestures (skelethon), emotions progression, and audio features

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages