-
-
Notifications
You must be signed in to change notification settings - Fork 130
Home
In Processing 2.0, we've made the switch to a GStreamer-based video library (excising the Apple horror show that was QuickTime for Java). There are many tales to tell of the quirky and unique beast that is GStreamer, and this document seeks to tell them.
The video library relies on the gstreamer-java bindings to talk to the native GStreamer libraries. The 1.x branch of the video library used the gstreamer-java bindings that were compatible only with the 0.10 version of GStreamer, which is no longer developed. Version 2.0 of the video library uses the new gst1-java-core bindings, derived from the original gstreamer-java bindings, but now compatible with the latest stable version of GStreamer (1.8.x or newer).
The source code of the video library includes an Eclipse project so the development can be carried out within that IDE, but other environments, such as IntelliJ, should work as well. Details on how to setup the development environment using Eclipse are provided in this page.
The easies approach to build the library from source and use it from the PDE is to clone this repo inside the sketchbook's library folder inside the video subfolder (otherwise Processing won't pick the library up):
git clone git@github.com:processing/processing-video.git videothen create a filed called local.properties in the root folder with the following content:
processing.dir=<path to local repository with Processing source code>
(this means that you need to checkout the main Processing repo somewhere in your computer).
After that, you should be able to change into the library's folder and simply build the library using ant:
cd video
ant buildwhich will generate the video.jar file inside library. Processing should be able recognize the new build after restarting the PDE.
The GStreamer project provide packages of the latest stable version of the 1.x branch for various platforms, including Mac and Windows. Most Linux distributions include the latest GStreamer libraries by default. These packages can be downloaded from here. This page from GStreamer's documentation includes useful, up-to-date information on how to deploy GStreamer across different platforms. Even though these deployment guidelines recommend to use a system-wide install of GStreamer, the video library bundles the GStreamer libraries for macOS, Linux, and Windows. This makes the library package fairly large, but the advantage is that there is no need to install GStreamer separately.
In order to bundle the GStreamer libraries when creating a distribution package of the video library, a few utility scripts are included in the repo, that should be run before running ant on Windows and Mac separately:
First, install the latest GStreamer Universal runtime package for MacOS on your development machine, which can be downloaded from here. Then, change into the scripts folder and run the pack_macos_libs.sh script:
cd scripts
./pack_macos_libs.sh
This should generate separate macos-x86_64 and macos-aarch64 folders inside library containing all the GStreamer native libraries for Mac (x86_64 and arm64 architectures). The ant build script will include them into the video.zip package when running the dist target.
Distributing full applications and modules (such as Processing libraries) on recent versions of macOS requires the software to be code signed and notarized by the developer. In order to perform these tasks on your own build of the video library, you'll need a $99/yr Apple developer ID. The ant build script will try to execute code signing (of the macOS GStreamer binaries) and notarization targets for you, but you need to do the following first:
- Create a Developer ID Application certificate in the certificates section of your Apple developer account
- Export the environmental variables APPLE_DEV_COMMON_NAME, VIDEO_APPLE_ID, VIDEO_LIB_PASSWORD, and PROVIDER_SHORT_NAME with the following values:
- APPLE_DEV_COMMON_NAME=Common name of subject in the Apple Developer certificate, can get it by running the following command:
security find-identity - VIDEO_APPLE_ID=Apple developer user ID
- VIDEO_LIB_PASSWORD=App password specifically created for video library. It can be created in this Apple ID site, which also requires an Apple Developer ID
- PROVIDER_SHORT_NAME=ASC provider short name, can be retrieved using the following command:
/usr/bin/xcrun altool --list-providers --username $VIDEO_APPLE_ID --password $VIDEO_LIB_PASSWORD>
Once these two steps are completed, you should be able to run ant dist without errors.
For bundling GStreamer with the video library, use the MSVC 64-bits installer (Windows 32-bits is no longer supported in Processing 4), which can be downloaded from here. Run the install as COMPLETE. Then, change into the scripts folder and run the pack_windows_libs.bat script:
cd scripts
pack_windows_libs.bat
This should generate a windows-amd64 folder inside the library containing all the GStreamer native libraries necessary for windows development. The ant build script will include them into the video.zip package when running the dist target.
If you encounter 'BUILD FAILED. Please build the core library first...'
Go to the processing-video root folder, open build.properties, and manually set core.classpath.location to where core.jar is built.
For example:
core.classpath.location=C:/ProcessingCore/processing/core/library
compiler.classpath.location=../processing/java/mode/
As long as a recent release of GStreamer is installed on the system, and the GST environmental variables are properly set, then Processing should be able to use the system-wide GStreamer libraries, making the bundled ones unnecessary, in case you need to remove them to save space. Either the MSVC or the MinGW 64-bits versions of GStreamer are compatible with the latest version of the video library (2.2 and newer). But make sure to use at least release 1.20.3 of GStreamer.
On Linux, we can generate our GStreamer build using a system called Meson. Here there are some docs about using Meson to build GStreamer.
Meson downloads and builds all the required dependencies, and can generate GStreamer libraries ready to bundle with the video library. We have used this approach in the 2.1 release of the video library to incorporate amd64 binaries of GStreamer for Linux.
After cloning the repo with the desired version of GStreamer, you create a folder to store the build files and then follow the build process using meson and ninja:
> git clone --branch 1.20 https://gitlab.freedesktop.org/gstreamer/gstreamer.git
> mkdir build-1.20
> cd gstreamer
> meson --prefix=<full path to build-1.20> builddir
> ninja -C builddir
> meson install -C builddir
Once the build process has been completed, you can use the provided script to package the native libraries for inclusion in the video library:
> cd scripts
> pack_linux_libs.sh
The location of the build folder using with meson has to be manually set in the pack_linux_libs script using an environmental variable for that purpose.
Thanks to Gottfried Haider and the development team behind gstreamer-java, we were finally able to update the video library to use GStreamer 1.0. The 2.0-beta releases (not yet available in the Contributions Manager) are the first result of that work. There are still issues to be dealt with, see this discussion in the Pull Request that incorporated the update: https://github.com/processing/processing-video/pull/84
The 2.0 milestone lists some of the issues that should be addressed towards the stable release of the 2.0 version of the video library.
In general, the top priorities would be to ensure that camera capture is well supported by the 2.0 stable release, since that's a very important use of the library, and that direct buffer to opengl texture functionality is re-implemented in 2.0.
The 1.x branch of the video library is no longer maintained. We are currently working to make the 2.0 beta branch the new default release of the library through the Contributions Manager in the PDE. Some old notes regarding the 1.x are available here as an archived reference.