This library provides Rust bindings for OpenNMT/CTranslate2. At this time, it has only been tested and confirmed to work on macOS and Linux. Windows support is available experimentally, but it has not been thoroughly tested and may have limitations or require additional configuration.
The ct2rs crate has been tested and confirmed to work with the following models:
- BART
- BLOOM
- FALCON
- Marian-MT
- MPT
- NLLB
- GPT-2
- GPT-J
- OPT
- T5
- Whisper
Please see the respective examples for each model.
This crate also offers a streaming API that utilizes callback closures. Please refer to the example code for more information.
If you plan to use GPU acceleration, CUDA and cuDNN are available.
Please enable the cuda or cudnn feature and set the CUDA_TOOLKIT_ROOT_DIR environment variable appropriately.
Several backends are available for use: OpenBLAS, Intel MKL, Ruy, and Apple Accelerate.
- OpenBLAS: To use OpenBLAS, enable the
openblasfeature and add the path to the directory containinglibopenblas.ato theLIBRARY_PATHenvironment variable. - Intel MKL: To use Intel MKL, enable the
mklfeature and set the path to the Intel libraries in theMKLROOTenvironment variable (default is/opt/intel). - Ruy: To use Ruy, enable the
ruyfeature. - Apple Accelerate: Available only on macOS, enable the
acceleratefeature to use Apple Accelerate.
The installation of CMake is required to compile the library.
Additional notes for Windows:
it is necessary to add RUSTFLAGS=-C target-feature=+crt-static to the environment variables for compilation.
To use model files with CTranslate2, they must first be converted.
Below is an example of how to convert the nllb-200-distilled-600M model:
pip install ctranslate2 huggingface_hub torch transformers
ct2-transformers-converter --model facebook/nllb-200-distilled-600M --output_dir nllb-200-distilled-600M \
--copy_files tokenizer.json
For more details, please refer to the CTranslate2's docs.
This application is released under the MIT License. For details, see the LICENSE file.