Whisper-CPP-Server is a high-performance speech recognition service written in C++, designed to provide developers and enterprises with a reliable and efficient speech-to-text inference engine. This project implements technology from ggml to perform inference on the open-source Whisper model. While ensuring speed and accuracy, it supports pure CPU-based inference operations, allowing for high-quality speech recognition services without the need for specialized hardware accelerators.
Real-time speech recognition and display of recognition results in the browser backend
https://github.com/litongjava/whisper-cpp-server
frontend
https://github.com/litongjava/listen-know-web
Test video
whisper-cpp-server-test.mp4
1.Pure C++ Inference Engine Whisper-CPP-Server is entirely written in C++, leveraging the efficiency of C++ for rapid processing of vast amounts of voice data, even in environments that only have CPUs for computing power.
2.High Performance Thanks to the computational efficiency of C++, Whisper-CPP-Server can offer exceptionally high processing speeds, meeting real-time or near-real-time speech recognition demands. It is especially suited for scenarios that require processing large volumes of voice data.
3.Support for Multiple Languages The service supports speech recognition in multiple languages, broadening its applicability across various linguistic contexts.
4.Docker Container Support A Docker image is provided, enabling quick deployment of the service through simple command-line operations, significantly simplifying installation and configuration processes. Deploy using the following command:
docker run -dit --name whisper-server -p 8080:8080 litongjava/whisper-cpp-server:1.0.0-large-v3
This means you can run Whisper-CPP-Server on any platform that supports Docker, including but not limited to Linux, Windows, and macOS.
4.Easy Integration for Clients Detailed client integration documentation is provided, helping developers quickly incorporate speech recognition functionality into their applications. Client Code Documentation
Whisper-CPP-Server is suitable for a variety of applications that require fast and accurate speech recognition, including but not limited to:
- Voice-driven interactive applications
- Transcription of meeting records
- Automatic subtitle generation
- Automatic translation of multi-language content
build with cmake and vcpkg
git clone https://github.com/litongjava/whisper-cpp-server.git
git submodule init
git submodule update
cmake -B cmake-build-release
cp ./ggml-metal.metal cmake-build-release
cmake --build cmake-build-release --config Release -- -j 12
macos
cmake -B cmake-build-release -DWHISPER_COREML=1run with simplest
./cmake-build-release/simplest -m models/ggml-base.en.bin test.wav
run with http-server
./cmake-build-release/whisper_http_server_base_httplib -m models/ggml-base.en.bin
run with websocket-server
./cmake-build-release/whisper_server_base_on_uwebsockets -m models/ggml-base.en.bin
copy command
mkdir bin
cp ./ggml-metal.metal bin
cp ./cmake-build-release/simplest bin
cp ./cmake-build-release/whisper_http_server_base_httplib bin
cp ./cmake-build-release/whisper_server_base_on_uwebsockets bin
cmake-build-debug/simplest -m models/ggml-base.en.bin samples/jfk.wavsimplest [options] file0.wav file1.wav ...
options:
-h, --help [default] show this help message and exit
-m FNAME, --model FNAME [models/ggml-base.en.bin] model path
-di, --diarize [false ] stereo audio diarization
Simple http service. WAV mp4 and m4a Files are passed to the inference model via http requests.
./whisper_http_server_base_httplib -h
usage: ./bin/whisper_http_server_base_httplib [options]
options:
-h, --help [default] show this help message and exit
-t N, --threads N [4 ] number of threads to use during computation
-p N, --processors N [1 ] number of processors to use during computation
-ot N, --offset-t N [0 ] time offset in milliseconds
-on N, --offset-n N [0 ] segment index offset
-d N, --duration N [0 ] duration of audio to process in milliseconds
-mc N, --max-context N [-1 ] maximum number of text context tokens to store
-ml N, --max-len N [0 ] maximum segment length in characters
-sow, --split-on-word [false ] split on word rather than on token
-bo N, --best-of N [2 ] number of best candidates to keep
-bs N, --beam-size N [-1 ] beam size for beam search
-wt N, --word-thold N [0.01 ] word timestamp probability threshold
-et N, --entropy-thold N [2.40 ] entropy threshold for decoder fail
-lpt N, --logprob-thold N [-1.00 ] log probability threshold for decoder fail
-debug, --debug-mode [false ] enable debug mode (eg. dump log_mel)
-tr, --translate [false ] translate from source language to english
-di, --diarize [false ] stereo audio diarization
-tdrz, --tinydiarize [false ] enable tinydiarize (requires a tdrz model)
-nf, --no-fallback [false ] do not use temperature fallback while decoding
-ps, --print-special [false ] print special tokens
-pc, --print-colors [false ] print colors
-pp, --print-progress [false ] print progress
-nt, --no-timestamps [false ] do not print timestamps
-l LANG, --language LANG [en ] spoken language ('auto' for auto-detect)
-dl, --detect-language [false ] exit after automatically detecting language
--prompt PROMPT [ ] initial prompt
-m FNAME, --model FNAME [models/ggml-base.en.bin] model path
-oved D, --ov-e-device DNAME [CPU ] the OpenVINO device used for encode inference
--host HOST, [127.0.0.1] Hostname/ip-adress for the service
--port PORT, [8080 ] Port number for the service
./cmake-build-debug/whisper_http_server_base_httplib -m models/ggml-base.en.bin
Test server
see request doc in doc
/inference
curl --location --request POST http://127.0.0.1:8080/inference \
--form file=@"./samples/jfk.wav" \
--form temperature="0.2" \
--form response-format="json"
--form audio_format="wav"
/load
curl 127.0.0.1:8080/load \
-H "Content-Type: multipart/form-data" \
-F model="<path-to-model-file>"
web socket server
start server
./cmake-build-debug/whisper_server_base_on_uwebsockets -m models/ggml-base.en.bin
Test server see python client
docker run -dit --name=whisper-server -p 8080:8080 -v "$(pwd)/models/ggml-base.en.bin":/models/ggml-base.en.bin litongjava/whisper-cpp-server:1.0.0 /app/whisper_http_server_base_httplib -m /models/ggml-base.en.bin
the port is 8080
curl --location --request POST 'http://127.0.0.1:8080/inference' \
--header 'Accept: */*' \
--header 'Content-Type: multipart/form-data; boundary=--------------------------671827497522367123871197' \
--form 'file=@"E:\\code\\cpp\\cpp-study\\cpp-study-clion\\audio\\jfk.wav"' \
--form 'temperature="0.2"' \
--form 'response-format="json"' \
--form 'audio_format="wav"'
docker run -dit --name whisper-server -p 8080:8080 litongjava/whisper-cpp-server:1.0.0-base-en
docker run -dit --name whisper-server -p 8080:8080 litongjava/whisper-cpp-server:1.0.0-large-v3
docker run -dit --name whisper-server -p 8080:8080 litongjava/whisper-cpp-server:1.0.0-tiny.en-q5_1