Releases: twilio/twilio-video-processors.js
3.0.0-beta.1
3.0.0-beta.1 (December 4, 2024)
- Web workers are now supported for Firefox and Safari.
3.0.0-preview.2
3.0.0-preview.2 (September 16, 2024)
- The web workers can now be hosted on a different domain than that of the application, provided the
Access-Control-Allow-Origin
response header points to the domain of the application.import { GaussianBlurBackgroundProcessor } from '@twilio/video-processors'; /* Application is running at https://appserver.com/app */ const processor = new GaussianBlurBackgroundProcessor({ assetsPath: "https://assetsserver.com/path/to/assets" });
3.0.0-preview.1
3.0.0-preview.1 (August 13, 2024)
This version works with twilio-video
SDK versions 2.29.0
and above.
API Changes
- The VideoProcessors now run in web workers on Chromium-based browsers. Support for web workers on other supported browsers is upcoming. While adding a
VideoProcessor
to aVideoTrack
, use the followingAddProcessorOptions
:videoTrack.addProcessor(processor, { inputFrameBufferType: 'videoframe', outputFrameBufferContextType: 'bitmaprenderer' });
- The
Canvas2D
andWebGL2
pipelines are replaced by a single hybrid pipeline. Therefore, the following APIs are no longer available:BackgroundProcessorOptions.pipeline
Pipeline
enum exported by@twilio/video-processors
2.2.0
2.2.0 (July 16, 2024)
Performance Improvements
- The WebGL2 pipeline now has an overall higher output frame rate, even for 720p (HD) video.
Changes
BackgroundProcessorOptions.debounce
is now set tofalse
by default.BackgroundProcessorOptions.maskBlurRadius
is now set to8
as the default for theWebGL2
pipeline, and4
for theCanvas2D
pipeline.
Bug Fixes
- Fixed trailing effect of the person mask in both Canvas2D and WebGL2 pipelines.
- Fixed a bug where changing the
maskBlurRadius
value on theVideoProcessor
was not working. - TFLite module is loaded and initialized only once, no matter how many VideoProcessor instances are created.
2.1.0
2.1.0 (December 12, 2023)
- Previously, the VideoProcessors SDK failed to compile with TypeScript 5.x. This release contains changes to support TypeScript 5.x.
- Fixed a bug where WebGL2-based VideoProcessors sometimes generated very low output fps, especially on low-powered Intel graphics cards.
2.0.0
2.0.0 (March 21, 2023)
- The VideoProcessors now work on browsers that do not support
OffscreenCanvas
. With this release, when used with twilio-video v2.27.0, the Virtual Background feature will work on browsers that supports WebGL2. See VideoTrack.addProcessor for details. - On Chrome, our tests with 640x480 VideoTracks show up to 30% reduction in CPU usage if WebGL2 is used as opposed to Canvas2D. Higher resolutions degrade the performance as compared to Canvas2D. While we work to support higher resolutions in future releases, we strongly recommend that you set the maximum resolution to 640x480 for WebGL2.
API Changes
- isSupported now returns
true
for browsers that supports canvas 2D or webgl2 rendering context. - GaussianBlurBackgroundProcessor and VirtualBackgroundProcessor's
processFrame
method now accepts different types ofinputFrameBuffer
-OffscreenCanvas
,HTMLCanvasElement
orHTMLVideoElement
. - Added the following new options
NOTES:
- Although iOS and Android browsers (Safari and Chrome) are supported, the performance of the VideoProcessors is not optimized for mobile browsers at this time. Using the VideoProcessors on a mobile browser may overpower the CPU resulting in poor quality video experiences.
- Since desktop Safari and iOS browsers do not support WebAssembly SIMD, it is recommended to use camera input dimensions of 640x480 or lower to maintain an acceptable frame rate for these browsers.
Example
See the following pages for best practice.
Other Changes
- Removing unused BodyPix related logic.
- Removing unnecessary loading of JS files after loading the model.
1.0.2
1.0.2 (November 5, 2021)
Changes
- Moving
@types/node
to devDependencies. - Fixed an issue where twilio-video-processors is throwing an exception in a server-side rendering application.
1.0.1
1.0.1 (July 12, 2021)
Bug Fixes
- Fixed an issue where the following internal classes and interfaces are being exported.
- BackgroundProcessor
- BackgroundProcessorOptions
- GrayscaleProcessor
- Processor
- Dimensions
1.0.0
1.0.0 (June 24, 2021)
1.0.0-beta.3 has been promoted to 1.0.0 GA. Twilio Video Processors will use Semantic Versioning 2.0.0 for all future changes. Additionally, this release also includes the following new features and improvements.
-
Added
isSupported
API which can be used to check whether the browser is supported or not. This API returnstrue
for chromium-based desktop browsers.import { isSupported } from '@twilio/video-processors'; if (isSupported) { // Initialize the background processors }
-
GaussianBlurBackgroundProcessor and VirtualBackgroundProcessor's
processFrame
method signature has been updated in order to improve performance. With this update, the output frame buffer should now be provided to theprocessFrame
method which will be used to draw the processed frame.Old signature:
processFrame(inputFrame: OffscreenCanvas) : Promise<OffscreenCanvas | null> | OffscreenCanvas | null;
New signature:
processFrame(inputFrameBuffer: OffscreenCanvas, outputFrameBuffer: HTMLCanvasElement) : Promise<void> | void;
-
The segmentation model has been changed from MLKit Selfie Segmentation to MediaPipe Selfie Segmentation Landscape to improve performance.
-
Added debounce logic on the image resizing step to improve performance.
1.0.0-beta.3
1.0.0-beta.3 (May 25, 2021)
Improvements
-
The VideoProcessors now use WebAssembly to run TensorFlow Lite for faster and more accurate person segmentation. You need to deploy the tflite model and binaries so the SDK can load them properly. Additionally, this improvement requires Chrome's WebAssembly SIMD support in order to achieve the best performance. WebAssembly SIMD can be turned on by visiting
chrome://flags
on versions 84 through 90. This will be enabled by default on Chrome 91+. You can also enable this on versions 84-90 for your users without turning on the flag by registering for a Chrome Origin Trial for your website. -
The segmentation model has been changed from BodyPix to MLKit Selfie Segmentation to improve segmentation accuracy.