3D virtual assistant SDK that enables you to make your 3D model into a voice-enabled chatbot. 🇯🇵日本語のREADMEはこちら
- 🍎 iOS App: OshaberiAI A Virtual Agent App made with ChatdollKit: a perfect fusion of character creation by AI prompt engineering, customizable 3D VRM models, and your favorite voices by VOICEVOX.
- 🇬🇧 Live demo English Say "Hello" to start conversation. This demo just returns what you say (echo).
- 🇯🇵 Live demo in Japanese OpenAI API Keyをご用意ください。「こんにちは」と話しかけると会話がスタートします。
- Generative AI Native: Supports multiple LLMs like ChatGPT, Anthropic Claude, Google Gemini Pro, Dify, and others, with function calling (ChatGPT/Gemini) and multimodal capabilities.
- 3D model expression: Synchronizes speech and motion, controls facial expressions and animations autonomously, supports blinking and lip-sync.
- Dialog control: Integrates Speech-to-Text and Text-to-Speech (OpenAI, Azure, Google, Watson, VOICEVOX, VOICEROID, etc.), manages dialog state (context or memory), extracts intents and routes topics, supports wakeword detection.
- Multi platforms: Compatible with Windows, Mac, Linux, iOS, Android, and other Unity-supported platforms, including VR, AR, and WebGL.
... and more! See ChatdollKit Documentation to learn details.
- ⚡ Optimized AI Dialog Processing: We've boosted response speed with parallel processing and made it easier for you to customize behavior with your own code. Enjoy faster, more flexible AI conversations!
- 🥰 Emotionally Rich Speech: Adjusts vocal tone dynamically to match the conversation, delivering more engaging and natural interactions.
- 🎤 Enhanced Microphone Control: Microphone control is now more flexible than ever! Easily start/stop devices, mute/unmute, and adjust voice recognition thresholds independently.
You can learn how to setup ChatdollKit by watching this video that runs the demo scene(including chat with ChatGPT): https://www.youtube.com/watch?v=rRtm18QSJtc
To run the demo for version 0.8, please follow the steps below after importing the dependencies:
- Open scene
Demo/Demo08
. - Select
AIAvatarVRM
object in scene. - Set OpenAI API key to following components on inspector:
- ChatGPTService
- OpenAITTSLoader
- OpenAISpeechListener
- Run on Unity Editor.
- Say "こんにちは" or word longer than 3 characters.
Download the latest version of ChatdollKit.unitypackage and import it into your Unity project after import dependencies;
Burst
from Unity Package Manager (Window > Package Manager)- UniTask(Tested on Ver.2.5.4)
- uLipSync(Tested on v3.1.0)
- For VRM model: UniVRM(v0.89.0) and VRM Extension
- JSON.NET: If your project doesn't have JSON.NET, add it from Package Manager > [+] > Add package from git URL... > com.unity.nuget.newtonsoft-json
- Azure Speech SDK: (Optional) Required for real-time speech recognition using a stream.
Add 3D model to the scene and adjust as you like. Also install required resources for the 3D model like shaders etc. In this README, I use Cygnet-chan that we can perchase at Booth. https://booth.pm/ja/items/1870320
And, import animation clips. In this README, I use Anime Girls Idle Animations Free. I believe it is worth for you to purchase the pro edition👍
Put ChatdollKit/Prefabs/ChatdollKit
or ChatdollKit/Prefabs/ChatdollKitVRM
to the scene. And, create EventSystem to use UI components.
Select Setup ModelController
in the context menu of ModelController. If NOT VRM, make sure that shapekey for blink to Blink Blend Shape Name
is set after setup. If not correct or blank, set it manually.
Select Setup Animator
in the context menu of ModelController and select the folder that contains animation clips or their parent folder. In this case put animation clips in 01_Idles
and 03_Others
onto Base Layer
for override blending, 02_Layers
onto Additive Layer
for additive blending.
Next, see the Base Layer
of newly created AnimatorController in the folder you selected. Confirm the value for transition to the state you want to set it for idle animation.
Lastly, set the value to Idle Animation Value
on the inspector of ModelController.
On the inspector of DialogController
, set Wake Word
to start conversation (e.g. hello / こんにちは🇯🇵), Cancel Word
to stop comversation (e.g. end / おしまい🇯🇵), Prompt Voice
to require voice request from user (e.g. what's up? / どうしたの?🇯🇵).
Select the speech service (OpenAI/Azure/Google/Watson) you use and set API key and some properties like Region and BaseUrl on inspector of ChatdollKit.
Attach the common components to use LLM:
- ChatdollKit/Scripts/LLM/LLMRouter
- ChatdollKit/Scripts/LLM/LLMContentSkill
Additionally, attach the specific LLM vendor component you want to use and set the API key:
- ChatdollKit/Scripts/LLM/ChatGPT/ChatGPTService
- ChatdollKit/Scripts/LLM/Claude/ClaudeService
- ChatdollKit/Scripts/LLM/Gemini/GeminiService
Select Setup VRC FaceExpression Proxy
in the context menu of VRC FaceExpression Proxy. Neutral, Joy, Angry, Sorrow and Fun face expression with all zero value and Blink face with blend shape for blink = 100.0f are automatically created.
You can edit shape keys by editing Face Clip Configuration directly or by capturing on inspector of VRCFaceExpressionProxy.
Press Play button of Unity editor. You can see the model starts with idling animation and blinking.
- Say the word you set to
Wake Word
on inspector (e.g. hello / こんにちは🇯🇵) - Your model will reply the word you set to
Prompt Voice
on inspector (e.g. what's up? / どうしたの?🇯🇵) - Say something you want to echo like "Hello world!"
- Your model will reply "Hello world"
To use Azure OpenAI Service set following info on inspector of ChatGPTService component:
- Endpoint url with configurations to
Chat Completion Url
format: https://{your-resource-name}.openai.azure.com/openai/deployments/{deployment-id}/chat/completions?api-version={api-version}
-
API Key to
Api Key
-
Set true to
Is Azure
NOTE: Model
on inspector is ignored. Engine in url is used.
See the MultiSkills
example. That is more rich application including:
- Dialog Routing:
Router
is an example of how to decide the topic user want to talk - Processing dialog:
TranslateDialog
is an example that shows how to process dialog
We are now preparing contents to create more rich virtual assistant using ChatdollKit.
Refer to the following tips for now. We are preparing demo for WebGL.
- It takes 5-10 minutes to build. (It depends on machine spec)
- Very hard to debug. Error message doesn't show the stacktrace:
To use dlopen, you need to use Emscripten’s linking support, see https://github.com/kripken/emscripten/wiki/Linking
- Built-in Async/Await doesn't work (app stops at
await
) because JavaScript doesn't support threading. Use UniTask instead. - CORS required for HTTP requests.
- Microphone is not supported. Use
ChatdollMicrophone
that is compatible with WebGL. - Compressed audio formats like MP3 are not supported. Use WAV in TTS Loaders.
- OVRLipSync is not supported. Use uLipSync and uLipSyncWebGL instead.
- If you want to show multibyte characters in message window put the font that includes multibyte characters to your project and set it to message windows.
- uLipSync (LipSync) (c)hecomi
- UniTask (async/await integration) (c)neuecc
- UniVRM (VRM) (c)VRM Consortium / (c)Masataka SUMI for MToon