Skip to content

Latest commit

 

History

History
314 lines (232 loc) · 15.8 KB

README_en.md

File metadata and controls

314 lines (232 loc) · 15.8 KB
Raw

libmedia

English | 中文

license

Introduction

libmedia is a tool library for processing multimedia content (such as audio, video, subtitles) on the web platform.

libmedia has typescript module and webAssembly module, and the design concept is dominated by typescript module; we implement the audio and video demux and mux layer in typescript module, so that we can process asynchronous IO. This allows the entire system to run on a non-SharedArrayBuffer environment;

The decoding and encoding modules are put into the webAssembly module. These modules can be compiled from the libavcodec module of FFmpeg, and each decoder and encoder is compiled into a separate wasm module to solve the problem of too large a compiled product. When using it, only You need to load the modules you want to use. At the same time, the codec module can use WebCodecs.

The api design of libmedia refers to the design of FFmpeg. Many data structure concepts are consistent, so you can see data structures such as AVStream, AVCodecParameters, AVFormatContext, AVPacket, AVFrame, etc. As the de facto standard in the audio and video industry, FFmpeg's design is very excellent; following the design, we can directly obtain excellent design patterns, and it also reduces the difficulty for developers to learn and understand. After all, most audio and video developers have learned about FFmpeg; of course, the main reason is that we need to make this data readable and writable in both typescript modules and webAssembly modules. The prerequisite is that its layout in memory is consistent with FFmpeg.

libmedia is designed to run on multi-threads, but can fall back to running on a single thread; so it is more friendly to multi-threaded development; developers can do multi-threaded development based on this very elegantly, after all, multi-threading is used in the audio and video field The experience is definitely much higher.

Libraries

  • avformat video and audio format libraries(flv、mp4、mpegts、matroska、oggs、mp3)
  • avcodec Audio and video codec library, mainly Wasm compiled by C/C++ (FFmpeg and other codec projects) and Web platform standard WebCodecs
  • audioresample Audio resampling (compiled by FFmpeg audio resampling module)
  • audiostretchpitch Audio speed change and pitch change processing (compiled by soundtouch)
  • videoscale video scale, format transform（compiled by FFmpeg libswscale）
  • avnetwork Web Platform network file IO related (Fetch, WebSocket, WebTransport, File)
  • avprotocol Audio and video protocols (dash, m3u8)
  • avrender Audio and video rendering (8bit, 10bit, HDR, audioWorklet, WebGL, WebGPU)
  • avpipeline Media task processing pipeline for multi-threaded parallel processing tasks

Multi-threaded

libmedia supports multi-threading, but the page needs to use SharedArrayBuffer. You can add the following two response headers to the response header of the top-level document:

  • Cross-Origin-Opener-Policy: same-origin
  • Cross-Origin-Embedder-Policy: require-corp

To enable the use of SharedArrayBuffer, if multi-threading is not supported, it will fall back to running on the main thread.

Tools

  • AVPlayer is libmedia's audio and video player implementation, supporting soft decoding, hard decoding, and MSE; it supports multiple encapsulation protocols and multiple encoding formats.online demo

  • AVTranscoder is the transcoding tool implementation of libmedia online demo

Formats

Format Input Output
flv ✅ ✅
mov ✅ ✅
mp4 ✅ ✅
mpegts ✅ ✅
matroska ✅ ✅
webm ✅ ✅
mp3 ✅ ✅
oggs ✅ ✅
ivf ✅ ✅
aac ✅ ❌
flac ✅ ❌
wav ✅ ❌
webvtt ✅ ❌

Codecs

Codecs are compiled into separate wasm modules, the decoders are in the dist/decode directory, and the encoders are in the dist/encode directory. There are three versions of the encoding and decoding wasm module: baseline, atomic, and simd. The baseline version's instruction set corresponds to the MVP version of WebAssembly, but it needs to support Mutable Globals, with the highest compatibility and the lowest performance; atomic version add the atomic operation instruction set and Bulk memory instruction set; simd version add the simd vector acceleration instruction set, has the highest performance. The current simd version is automatically optimized by the compiler, and different codecs have different effects (currently I have not seen any codec projects has optimized for the wasm simd instruction set. If you want higher acceleration effects, you may want to optimize yourself).

Compatibility support status of three versions

environment baseline atomic simd webcodecs
Chrome 74+ 75+ 91+ 94+
Firefox 61+ 79+ 89+ N/A(linux video only)
Safari 12+ 15+ 16.4+ 16.4+(video only)
Wasmtime 0.20+ 15+ 15+ N/A
Wasmer 0.7+ N/A N/A N/A
Node.js 12.0+ 16.4+ 16.4+ N/A
Deno 0.1+ 1.9+ 1.9+ N/A
wasm2c 1.0.1+ N/A N/A N/A

Supported decode codecs status

codec baseline atomic simd webcodecs(Chrome)
h264 ✅ ✅ ✅ ✅
hevc ✅ ✅ ✅ ✅ (hardware only)
vvc ✅ ✅ ✅ ❌
av1 ✅ ✅ ✅ ✅
vp8 ✅ ✅ ✅ ✅
vp9 ✅ ✅ ✅ ✅
mpeg4 ✅ ✅ ✅ ❌
aac ✅ ✅ ✅ ✅
mp3 ✅ ✅ ✅ ✅
opus ✅ ✅ ✅ ✅
flac ✅ ✅ ✅ ❌
speex ✅ ✅ ✅ ❌
vorbis ✅ ✅ ✅ ❌
ac3 ✅ ✅ ✅ ❌
eac3 ✅ ✅ ✅ ❌
dts ✅ ✅ ✅ ❌
G.711 A-law ✅ ✅ ✅ ❌
G.711 μ-law ✅ ✅ ✅ ❌

Supported encode codecs status

codec baseline atomic simd webcodecs(Chrome)
h264 ✅ ✅ ✅ ✅
hevc ❌ ✅ ✅ ❌
vvc ❌ ❌ ❌ ❌
av1 ❌ ❌ ❌ ✅
vp8 ✅ ✅ ✅ ✅
vp9 ✅ ✅ ✅ ✅
mpeg4 ✅ ✅ ✅ ❌
aac ✅ ✅ ✅ ✅
mp3 ✅ ✅ ✅ ❌
opus ✅ ✅ ✅ ✅
flac ✅ ✅ ✅ ❌
speex ✅ ✅ ✅ ❌
vorbis ✅ ✅ ✅ ❌
ac3 ✅ ✅ ✅ ❌
eac3 ✅ ✅ ✅ ❌
dts ✅ ✅ ✅ ❌
G.711 A-law ✅ ✅ ✅ ❌
G.711 μ-law ✅ ✅ ✅ ❌

API

avformat

avcodec

avpipeline

avnetwork

avplayer

avtranscoder

avutil

IO

Start

If you want to integrate this project for development, it is recommended to use this warehouse as a sub-module. The project uses the cheap library, which requires you to have some understanding of the use of cheap.

Currently, this project only supports using webpack for compilation and packaging.

Here's how to compile the AVPlayer and AVTranscoder tool

# Clone the project and all submodules
git clone git@github.com:zhaohappy/libmedia.git --recursive

# enter libmedia directory
cd libmedia

# Install dependencies
npm install

# Compile AVPlayer with development mode
npm run build-avplayer-dev

# Compile AVTranscoder with development mode
npm run build-avtranscoder-dev

# Start local http service
# Any http service will do. If it reports that edp cannot be found, you can install it globally use: npm install edp -g
edp webserver start --port=9000

# use browser access http://localhost:9000/test/avplayer.html

To debug the code in multi-threaded Worker from source, set the ENABLE_THREADS_SPLIT macro in tsconfig.json to true and recompile

{
  "cheap": {
    "defined": {
      "ENABLE_THREADS_SPLIT": true
    }
  }
}

tsconfig.json can also set other macros to tailor compilation. You can change the relevant settings according to your own needs. For details, see tsconfig.json -> cheap -> Configuration in defined

Example

examples/demux.ts is an example of demux

examples/mux.ts is an example of mux

examples/decode.ts is an example of decode

test/avplayer.html is an example of using AVPlayer and also the implementation of online demo.

test/avtranscode.html is an example of using AVTranscoder and also the implementation of online demo.

License

libmedia uses the LGPL open source license. You need to comply with the license requirements. For details, see LGPL

But some dependent libraries are under GPL license. If you use these dependent libraries, libmedia will be infected with GPL license. These dependent libraries are used by the following components:

  • dist/encoder/x264.wasm
  • dist/encoder/x264-atomic.wasm
  • dist/encoder/x264-simd.wasm
  • dist/encoder/x265-atomic.wasm
  • dist/encoder/x265-simd.wasm

Dependence Licence

  • ffmpeg: LGPL v2.1+
  • soundtouch: LGPL v2.1
  • libx264: GPL
  • libx265: GPL
  • libvorbis: BSD
  • libspeex: BSD
  • libopus: BSD
  • libvpx: BSD-3-Clause
  • libogg: BSD
  • libmp3lame: LGPL
  • kvazaar: BSD-3-Clause
  • libflac: BSD
  • fdkaac: BSD
  • dav1d: BSD 2-Clause

Copyright (C) 2024-present, Gaoxing Zhao