VideoProcessors v3 (Preview 1)

CHANGELOG.md

@@ -1,3 +1,46 @@
+3.0.0-beta.1 (December 4, 2024)
+===============================
+
+* Web workers are now supported for Firefox and Safari.
+
+3.0.0-preview.2 (September 16, 2024)
+====================================
+
+* The web workers can now be hosted on a different domain than that of the application, provided the `Access-Control-Allow-Origin` response header points to the domain of the application.
+  ```ts
+  import { GaussianBlurBackgroundProcessor } from '@twilio/video-processors';
+
+  /* Application is running at https://appserver.com/app */
+
+  const processor = new GaussianBlurBackgroundProcessor({
+    assetsPath: "https://assetsserver.com/path/to/assets"
+  });
+  ```
+
+3.0.0-preview.1 (August 13, 2024)
+=================================
+
+*The 3.x version of the Video Processors SDK works with `twilio-video` SDK versions `2.29.0` and above.*
+
+API Changes
+-----------
+
+* The VideoProcessors now run in web workers on Chromium-based browsers. Support for web workers on other supported browsers is upcoming. While adding a `VideoProcessor` to a `VideoTrack`, use the following `AddProcessorOptions`:
+  ```ts
+  videoTrack.addProcessor(processor, {
+    inputFrameBufferType: 'videoframe',
+    outputFrameBufferContextType: 'bitmaprenderer'
+  });
+  ```
+* The `Canvas2D` and `WebGL2` pipelines are replaced by a single hybrid pipeline. Therefore, the following APIs are no longer available:
+  * `BackgroundProcessorOptions.pipeline`
+  * `Pipeline` enum exported by `@twilio/video-processors`
+
+Performance Improvements
+------------------------
+
+*TODO(mmalavalli)*
+
 2.2.0 (July 16, 2024)
 =====================
 

LICENSE.md

@@ -1,7 +1,7 @@
 The following license applies to all parts of this software except as
 documented below.
 
-    Copyright (C) 2022 Twilio Inc.
+    Copyright (C) 2025 Twilio Inc.
     All rights reserved.
 
     Redistribution and use in source and binary forms, with or without

README.md

@@ -53,7 +53,11 @@ You can also copy `twilio-video-processors.js` from the `dist/build` folder and
 
 ### Assets
 
-In order to achieve the best performance, the VideoProcessors use WebAssembly to run TensorFlow Lite for person segmentation. You need to serve the tflite model and binaries so they can be loaded properly. These files can be downloaded from the `dist/build` folder. Check the [API docs](https://twilio.github.io/twilio-video-processors.js/interfaces/VirtualBackgroundProcessorOptions.html#assetsPath) for details and the [examples](https://github.com/twilio/twilio-video-processors.js/tree/master/examples) folder for reference.
+In order to achieve the best performance, the VideoProcessors use WebAssembly to run TensorFlow Lite for person segmentation. You need to serve the tflite model, binaries and web workers so they can be loaded properly. These files can be downloaded from the `dist/build` folder. Check the [API docs](https://twilio.github.io/twilio-video-processors.js/interfaces/VirtualBackgroundProcessorOptions.html#assetsPath) for details and the [examples](https://github.com/twilio/twilio-video-processors.js/tree/master/examples) folder for reference.
+
+#### CORS
+
+If you are serving the assets from a domain that is different from that of your application, then ensure that the `Access-Control-Allow-Origin` response header points to your application's domain.
 
 ## Usage
 

eslint.config.js

@@ -0,0 +1,20 @@
+const globals = require("globals");
+const pluginJs = require("@eslint/js");
+const tseslint = require("typescript-eslint");
+
+
+module.exports = [
+  { files: ["lib/**/*.ts"] },
+  { languageOptions: { globals: globals.browser } },
+  pluginJs.configs.recommended,
+  ...tseslint.configs.recommended,
+  {
+    rules: {
+      "@typescript-eslint/ban-ts-comment": "warn",
+      "@typescript-eslint/no-explicit-any": "warn",
+      "@typescript-eslint/no-namespace": "warn",
+      "no-unused-vars": "warn",
+      "no-undef": "warn",
+    },
+  },
+];

examples/README.md

@@ -8,9 +8,10 @@ Go to `${PROJECT_ROOT}/examples` and run `npm install`. It will install the depe
 
 Open `http://localhost:3000` in a Chrome tab. The app captures your camera upon loading and plays it in a `<video>` element. You can choose to enable, disable or update the background settings using the controls on the page. Additionally, you can specify the following url parameters:
 
+- `blurFilterRadius` - Radius of the background blur filter (default: 15)
 - `capFramerate` - Choose video capture frame rate (default: 30)
 - `capResolution=wxh` - Choose video capture resolution (default: 1280x720)
-- `debounce=true|false` - Whether to skip processing every other frame (default: false)
-- `maskBlurRadius` - Radius of the mask blur filter (default: 8 for WebGL2, 4 for Canvas2D)
-- `pipeline=Canvas2D|WebGL2` - Choose the canvas or webgl pipeline (default: WebGL2)
+- `deferInputFrameDownscale=true|false` - **(Chrome only)** Whether to calculate the person mask without waiting for the input frame to be downscaled (default: false)
+- `maskBlurRadius` - Radius of the mask blur filter (default: 8)
 - `stats=advanced|hide|show` - Show performance benchmarks (default: show)
+- `useWebWorker=true|false` - **(Chrome only)** Whether to use a web worker for background replacement (default: true)

examples/virtualbackground/index.html

@@ -44,7 +44,7 @@ <h5 class="modal-title" id="errorModalLabel">Error</h5>
         </div>
       </div>
     </div>
-    <script src="https://sdk.twilio.com/js/video/releases/2.28.1/twilio-video.js"></script>
+    <script src="twilio-video.js"></script>
     <script src="twilio-video-processors.js"></script>
     <script src="https://cdn.jsdelivr.net/npm/bootstrap@5.0.0/dist/js/bootstrap.bundle.min.js" integrity="sha384-p34f1UUtsS3wqzfto5wAAmdvj+osOnFyQFpp4Ua3gs/ZVWx6oOypYoCJhGGScy+8" crossorigin="anonymous"></script>
     <script type="module" src="https://cdn.jsdelivr.net/gh/zerodevx/zero-md@2/dist/zero-md.min.js"></script>

examples/virtualbackground/index.js

@@ -10,12 +10,13 @@ const assetsPath = '';
 const bkgImages = new Map();
 
 const defaultParams = {
+  blurFilterRadius: '15',
   capFramerate: '30',
   capResolution: '1280x720',
-  debounce: 'false',
-  pipeline: 'WebGL2',
-  maskBlurRadiusCanvas2D: '4',
-  maskBlurRadiusWebGL2: '8',
+  deferInputFrameDownscale: 'false',
+  maskBlurRadius: '8',
+  stats: 'show',
+  useWebWorker: 'true',
 };
 
 const params = {
@@ -45,35 +46,28 @@ const loadImage = async (name) => {
   return bkgImage;
 }
 
-params.maskBlurRadius = params.pipeline === 'WebGL2'
-  ? params.maskBlurRadiusWebGL2
-  : params.maskBlurRadiusCanvas2D;
-
 (async ({
   Video: { createLocalVideoTrack },
   VideoProcessors: {
     GaussianBlurBackgroundProcessor,
-    Pipeline: { Canvas2D, WebGL2 },
     VirtualBackgroundProcessor,
     isSupported,
     version,
   },
 }) => {
   const {
+    blurFilterRadius,
     capFramerate,
     capResolution,
-    pipeline,
-    debounce,
+    deferInputFrameDownscale,
     maskBlurRadius,
-    stats = 'show',
+    stats,
+    useWebWorker,
   } = params;
 
   const addProcessorOptions = {
-    inputFrameBufferType: 'video',
-    outputFrameBufferContextType: {
-      [Canvas2D]: '2d',
-      [WebGL2]: 'webgl2',
-    }[pipeline],
+    inputFrameBufferType: 'videoframe',
+    outputFrameBufferContextType: 'bitmaprenderer',
   };
 
   const capDimensions = capResolution
@@ -88,9 +82,9 @@ params.maskBlurRadius = params.pipeline === 'WebGL2'
 
   const processorOptions = {
     assetsPath,
-    pipeline,
-    debounce: JSON.parse(debounce),
+    deferInputFrameDownscale: JSON.parse(deferInputFrameDownscale),
     maskBlurRadius: Number(maskBlurRadius),
+    useWebWorker: JSON.parse(useWebWorker),
   };
 
   let videoTrack = null;
@@ -129,7 +123,10 @@ params.maskBlurRadius = params.pipeline === 'WebGL2'
       setProcessor(videoTrack, null);
     } else if (bg === 'blur') {
       if (!gaussianBlurProcessor) {
-        gaussianBlurProcessor = new GaussianBlurBackgroundProcessor(processorOptions);
+        gaussianBlurProcessor = new GaussianBlurBackgroundProcessor({
+          blurFilterRadius: Number(blurFilterRadius),
+          ...processorOptions,
+        });
         await gaussianBlurProcessor.loadModel();
       }
       setProcessor(videoTrack, gaussianBlurProcessor);
@@ -158,7 +155,6 @@ params.maskBlurRadius = params.pipeline === 'WebGL2'
       }
       const { frameRate, height, width } = videoTrack.mediaStreamTrack.getSettings();
       $stats.innerText = `Sdk version: ${version}`;
-      $stats.innerText += `\nPipeline: ${params.pipeline}`;
       $stats.innerText += `\nCapture dimensions (in): ${width} x ${height}`;
       $stats.innerText += `\nFrame rate (in): ${frameRate.toFixed(2)} fps`;
 

karma.conf.ts

@@ -111,8 +111,7 @@ export default function(config: any) {
     karmaTypescriptConfig: {
       coverageOptions: {
         exclude: [
-          /tests\/.*/,
-          /lib\/processors\/webgl2\/.*/
+          /tests\/.*/
         ]
       },
       include: [

lib/constants.ts

@@ -5,7 +5,9 @@ export const MASK_BLUR_RADIUS = 8;
 export const MODEL_NAME = 'selfie_segmentation_landscape.tflite';
 export const TFLITE_LOADER_NAME = 'tflite-1-0-0.js';
 export const TFLITE_SIMD_LOADER_NAME = 'tflite-simd-1-0-0.js';
-export const TFLITE_WORKER_NAME = 'tflite-worker.js';
+export const TWILIO_GAUSSIAN_BLUR_BACKGROUND_PROCESSOR_PIPELINE_WORKER = 'twilio-gaussian-blur-background-processor-pipeline-worker.js';
+export const TWILIO_VIRTUAL_BACKGROUND_PROCESSOR_PIPELINE_WORKER = 'twilio-virtual-background-processor-pipeline-worker.js';
+
 
 export const WASM_INFERENCE_DIMENSIONS: Dimensions = {
   width: 256,

lib/index.ts

@@ -1,6 +1,6 @@
 import { GaussianBlurBackgroundProcessor, GaussianBlurBackgroundProcessorOptions } from './processors/background/GaussianBlurBackgroundProcessor';
 import { VirtualBackgroundProcessor, VirtualBackgroundProcessorOptions } from './processors/background/VirtualBackgroundProcessor';
-import { ImageFit, Pipeline } from './types';
+import { ImageFit } from './types';
 import { isSupported } from './utils/support';
 import { version } from './utils/version';
 
@@ -10,7 +10,6 @@ if (typeof window !== 'undefined') {
     ...window.Twilio.VideoProcessors,
     GaussianBlurBackgroundProcessor,
     ImageFit,
-    Pipeline,
     isSupported,
     version,
     VirtualBackgroundProcessor,
@@ -21,7 +20,6 @@ export {
   GaussianBlurBackgroundProcessor,
   GaussianBlurBackgroundProcessorOptions,
   ImageFit,
-  Pipeline,
   isSupported,
   version,
   VirtualBackgroundProcessor,

lib/processors/Processor.ts

@@ -11,6 +11,6 @@ export abstract class Processor {
    * @param outputFrameBuffer - The output frame buffer to use to draw the processed frame.
    */
   abstract processFrame(
-    inputFrameBuffer: OffscreenCanvas | HTMLCanvasElement | HTMLVideoElement,
+    inputFrameBuffer: OffscreenCanvas | HTMLCanvasElement | HTMLVideoElement | VideoFrame,
     outputFrameBuffer: HTMLCanvasElement): Promise<void> | void;
 }