Skip to content

Latest commit

 

History

History
427 lines (282 loc) · 18.1 KB

README.md

File metadata and controls

427 lines (282 loc) · 18.1 KB
Raw

Dark Vision - Discover dark data in videos with IBM Watson and IBM Bluemix OpenWhisk

Build Status Bluemix Deployments

Dark Vision processes videos to discover dark data. By analyzing video frames with IBM Watson Visual Recognition, Dark Vision builds a summary with a set of tags and famous people or building detected in the video. Use this summary to enhance video search and categorization.

In addition to processing videos, Dark Vision can also processes standalone images.

Watch this Youtube video to know more about the app

Dark Vision

Overview

Built using IBM Bluemix, the application uses:

Extracting frames from a video

The user uploads a video or image using the Dark Vision web application, which stores it in a Cloudant DB. Once the video is uploaded, OpenWhisk detects the new video by listening to Cloudant changes (trigger). OpenWhisk then triggers the video extractor action. During its execution, the extractor produces frames (images) and stores them in Cloudant. The frames are then processed using Watson Visual Recognition and the results are stored in the same Cloudant DB. The results can be viewed using Dark Vision web application OR an iOS application.

Object Storage can be used in addition to Cloudant. When doing so, video and image medadata are stored in Cloudant and the media files are stored in Object Storage.

![Architecture](http://g.gravizo.com/g? digraph G { node [fontname = "helvetica"] rankdir=LR /* stores a video / user -> storage / cloudant change sent to openwhisk / storage -> openwhisk / openwhisk triggers the extractor / openwhisk -> extractor / extractor produces image frames / extractor -> frames / frames are stored in cloudant / frames -> storage / styling */ frames [label="Image Frames"] storage [shape=circle style=filled color="%234E96DB" fontcolor=white label="Data Store"] openwhisk [shape=circle style=filled color="%2324B643" fontcolor=white label="OpenWhisk"] } )

Processing frames and standalone images

Whenever a frame is created and uploaded, Cloudant emits a change event and OpenWhisk triggers the analysis. The analysis is persisted with the image.

![Architecture](http://g.gravizo.com/g? digraph G { node [fontname = "helvetica"] /* stores a image / frame -> storage / cloudant change sent to openwhisk / storage -> openwhisk / openwhisk triggers the analysis / openwhisk -> analysis / extractor produces image frames / {rank=same; frame -> storage -> openwhisk -> analysis -> watson [style=invis] } / analysis calls Watson / analysis -> watson / results are stored / analysis -> storage / styling */ frame [label="Image Frame"] analysis [label="Image Analysis"] storage [shape=circle style=filled color="%234E96DB" fontcolor=white label="Data Store"] openwhisk [shape=circle style=filled color="%2324B643" fontcolor=white label="OpenWhisk"] watson [shape=circle style=filled color="%234E96DB" fontcolor=white label="Watson\nVisual\nRecognition"] } )

Application Requirements

  • IBM Bluemix account. Sign up for Bluemix, or use an existing account.
  • Docker Hub account. Sign up for Docker Hub, or use an existing account.
  • XCode 8.0, iOS 10, Swift 3
  • Node.js >= 6.7.0

Deploying Dark Vision in Bluemix

Dark Vision comes with a default toolchain you can use to deploy the solution with few clicks. If you want to deploy it manually, you can skip this section.

Click Deploy to Bluemix to start the Bluemix DevOps wizard:

Deploy to Bluemix

  1. Select the GitHub box.

  2. Decide whether you want to fork/clone the Dark Vision repository.

  3. If you decide to Clone, set a name for your GitHub repository.

  4. Select the Delivery Pipeline box.

  5. Select the region, organization and space where you want to deploy the web application.

⚠️ Dark Vision is made of two main components: the web application to upload media and view results and the OpenWhisk actions to process the media. OpenWhisk in Bluemix is currently only available in the US South region. If you decide to deploy the web application in another region than US South, make sure to create a space with the same name in the US South region too. The OpenWhisk actions will be deployed to this space in the US South region.

  1. Set the name of the Dark Vision web application. Pick a unique name to avoid conflicts.

  2. Optionally set the admin username and password for the application. When set, the application will prompt for this username and password when uploading videos/images, when resetting a video or an image. If the username and password are not defined, any visitor can upload videos/images for processing.

  3. If you already have a Watson Visual Recognition service instance you want to reuse, retrieve its API key from the credentials and set its value in the form. If you leave it empty, the pipeline will create a new service instance automatically.

  4. Click Create.

  5. Select the Delivery Pipeline named darkvision.

  6. Wait for the Deploy job to complete.

  7. Access the Dark Vision app when it's ready and start uploading videos and images!

Preparing the environment

Get the code

  • Clone the app to your local environment from your terminal using the following command:

    git clone https://github.com/IBM-Bluemix/openwhisk-darkvisionapp.git

  • or Download and extract the source code from this archive

Create the Bluemix Services

Note: if you have existing instances of these services, you don't need to create new instances. You can simply reuse the existing ones.

  1. Open the IBM Bluemix console

  2. Create a Cloudant NoSQL DB service instance named cloudant-for-darkvision

  3. Open the Cloudant service dashboard and create a new database named openwhisk-darkvision

  4. Create a Watson Visual Recognition service instance named visualrecognition-for-darkvision

  5. Optionally create a Object Storage service instance named objectstorage-for-darkvision

If configured, media files will be stored in Object Storage instead of Cloudant. A container named openwhisk-darkvision will be automatically created.

Deploy the web interface to upload videos and images

This simple web user interface is used to upload the videos or images and visualize the results of each frame analysis.

  1. Change to the web directory.
cd openwhisk-darkvisionapp/web

  1. If in the previous section you decided to use existing services instead of creating new ones, open manifest.yml and update the Cloudant service name.

  2. If you configured an Object Storage service, make sure to add its name to the list of services in the manifest.yml services section or to uncomment the existing objectstorage-for-darkvision entry.

  3. Push the application to Bluemix:

cf push

Protecting the upload, delete and reset actions (optional)

By default, anyone can upload/delete/reset videos and images. You can restrict access to these actions by defining the environment variables ADMIN_USERNAME and ADMIN_PASSWORD on your application. This can be done in the Bluemix console or with the command line:

cf set-env openwhisk-darkvision ADMIN_USERNAME admin
cf set-env openwhisk-darkvision ADMIN_PASSWORD aNotTooSimplePassword

Build the Frame Extractor Docker image

Extracting frames from a video is achieved with ffmpeg. ffmpeg is not available to an OpenWhisk action written in JavaScript or Swift. Fortunately OpenWhisk allows to write an action as a Docker image and can retrieve this image from Docker Hub.

To build the extractor image, follow these steps:

  1. Change to the processing/extractor directory.

  2. Ensure your Docker environment works and that you have logged in Docker hub.

  3. Run

./buildAndPush.sh youruserid/yourimagename

Note: On some systems this command needs to be run with sudo.

  1. After a while, your image will be available in Docker Hub, ready for OpenWhisk.

Deploy OpenWhisk Actions

  1. Change to the root directory of the checkout.

  2. Copy the file named template-local.env into local.env

cp template-local.env local.env
  1. Get the service credentials for services created above and replace placeholders in local.env with corresponding values (usernames, passwords, urls). These properties will be injected into a package so that all actions can get access to the services.

If you configured an Object Storage service, specify its properties in this file too but uncommenting the placeholder variables.

  1. Make sure to also update the value of DOCKER_EXTRACTOR_NAME with the name of the Docker image you created in the previous section.

  2. Ensure your OpenWhisk command line interface is property configured with:

wsk list

This shows the packages, actions, triggers and rules currently deployed in your OpenWhisk namespace.

  1. Get dependencies used by the deployment script
npm install

⚠️ Node.js >= 6.7.0 is required

  1. Create the action, trigger and rule using the script from the root directory directory:
node deploy.js --install

The script can also be used to --uninstall the OpenWhisk artifacts to --update the artifacts if you change the action code.

That's it! Use the web application to upload images/videos and view the results! You can also view the results using an iOS application as shown further down this README.

Running the web application locally

  1. Change to the web directory

  2. Get dependencies

npm install
  1. Start the application
npm start

Note: To find the Cloudant database (and Object Storage) to connect to when running locally, the application uses the environment variables defined in local.env in previous steps.

  1. Upload videos through the web user interface. Wait for OpenWhisk to process the videos. Look at the results. While OpenWhisk processes videos, the counters at the top of the application will update. These counters call the /api/status endpoint of the web application to retrieve statistics.

iOS application to view the results (Optional)

The iOS application is a client to the API exposed by the web application to view the results of the analysis of videos. It is an optional piece.

To configure the iOS application, you need the URL of the web application deployed before. The web app exposes an API to list all videos and retrieve the results.

  1. Open ios/darkvision.xcworkspace with XCode

  2. Open the file darkvision/darkvision/model/API.swift

  3. Set the value of the constant apiUrl to the application host previously deployed.

  4. Save the file

Running the iOS application in the simulator

  1. Start the application from XCode with iPad Air 2 as the target

  1. Browse uploaded videos

  1. Select a video

Results are made of the faces detected in the picture and of tags returned by Watson. The tags with the highest confidence score are shown. Tap a tag or a face to change the main image to the frame where this tag or face was detected.

Code Structure

OpenWhisk - Deployment script

File Description
deploy.js Helper script to install, uninstall, update the OpenWhisk trigger, actions, rules used by Dark Vision.

OpenWhisk - Change listener

File Description
changelistener.js Processes Cloudant change events and calls the right actions. It controls the processing flow for videos and frames.

OpenWhisk - Frame extraction

The frame extractor runs as a Docker action created with the OpenWhisk Docker SDK:

  • It uses ffmpeg to extract frames from the video.
  • It is written as a nodejs app to benefit from several nodejs helper packages (Cloudant, ffmpeg, imagemagick)
File Description
Dockerfile Docker file to build the extractor image. It pulls ffmpeg into the image together with node. It also runs npm install for both the server and client.
extract.js The core of the frame extractor. It downloads the video stored in Cloudant, uses ffmpeg to extract frames and video metadata, produces a thumbnail for the video. By default it produces around 15 images for a video. This can be changed by modifying the implementation of getFps.
app.js Adapted from the OpenWhisk Docker SDK to call the extract.js node script.

OpenWhisk - Frame analysis

analysis.js holds the JavaScript code to perform the image analysis:

  1. It retrieves the image data from the Cloudant document. The data has been attached by the frame extractor as an attachment named "image.jpg".
  2. It saves the image file locally.
  3. If needed, it resizes the image so that it matches the requirements of the Watson service
  4. It calls Watson
  5. It attachs the results of the analysis to the image and persist it.

The action runs asynchronously.

The code is very similar to the one used in the Vision app.

Web app

The web application allows to upload videos (and images). It shows the video catalog and for each video the extracted frames.

File Description
app.js The web app backend handles the upload of videos/images, and exposes an API to retrieve all videos, their frames, to compute the summary
Angular controllers Controllers for list of videos, individual video and standalone images
Angular services Services to interact with the backend API

Shared code between OpenWhisk actions and web app

These files are used by the web application and the OpenWhisk actions. They are automatically injected in the OpenWhisk actions by the deploy.js script and during the build of the Docker image. These scripts have dependencies on Cloudant, async, pkgcloud which are provided by default in OpenWhisk Node.js actions.

File Description
cloudantstorage.js Implements API on top of Cloudant to create/read/update/delete video/image metadata and to upload files
objectstorage.js Implements the file upload operations on top of Object Storage. Used by cloudantstorage.js when Object Storage is configured.
cloudant-designs.json Design documents used by the API to expose videos and images. They are automatically loaded into the database when the web app starts for the first time.

iOS

The iOS app is an optional part of the Dark Vision sample app. It uses the API exposed by the web application to display the videos in the catalog and their associated tags.

File Description
API.swift Calls the web app API. Update the constant apiUrl to map to the location of your web app.

Contribute

Please create a pull request with your desired changes.

Troubleshooting

OpenWhisk

Polling activations is good start to debug the OpenWhisk action execution. Run

wsk activation poll

and upload a video for analysis.

Web application

Use

cf logs <appname>

to look at the live logs for the web application

License

See License.txt for license information.

Privacy Notice

The web application includes code to track deployments to IBM Bluemix and other Cloud Foundry platforms. The following information is sent to a Deployment Tracker service on each deployment:

  • Application Name (application_name)
  • Space ID (space_id)
  • Application Version (application_version)
  • Application URIs (application_uris)

This data is collected from the VCAP_APPLICATION environment variable in IBM Bluemix and other Cloud Foundry platforms. This data is used by IBM to track metrics around deployments of sample applications to IBM Bluemix to measure the usefulness of our examples, so that we can continuously improve the content we offer to you. Only deployments of sample applications that include code to ping the Deployment Tracker service will be tracked.

Disabling Deployment Tracking

Deployment tracking can be disabled by removing require("cf-deployment-tracker-client").track(); from the beginning of the web/app.js file.