v0.7.0

🤖 LLM-based Dialog Processing

✅Multiple LLMs: ChatGPT / Azure OpenAI Service, Anthropic Claude, Google Gemini Pro and others
✅Agents: Function Calling (ChatGPT / Gemini) or your prompt engineering
✅Multimodal: GPT-4V and Gemini-Pro-Vision are suppored
✅Emotions: Autonomous face expression and animation

We've developed a versatile framework that standardizes the processes of Routing, Chatting (including facial expressions and motions), and executing Tools using various Large Language Models (LLMs). This framework allows for easy customization by simply swapping out LLM-specific components.

Additionally, you can support any LLM by creating your own components that implement the ILLMService interface.

To use this new LLM-based dialog, attach LLMRouter, LLMContentSkill and ILLMService component like ChatGPTService in ChatdollKit.LLM package. Also attach function skills that extends LLMFunctionSkillBase if you build an AI agent. See DemoChatGPT that works right out-of-the-box🎁.

• Support multiple LLMs: ChatGPT, Claude and Gemini✨ #271
• Support custom request parameters and headers for LLM APIs #272
• Fix bug that iOS build failed on Xcode #273
• Fix bug that LLMFunctionSkill fails #275

NOTE: ChatdollKit.Dialog.Processor.ChatGPT* components are deprecated.

🐉 Other Changes

• Refactoring for Future tech integration and reduced dependencies #270
• Improve stability of AzureStreamVoiceRequestProvider #274
• Make microphone volume controller works with NonRecordingVRP #276
• Update demo for v0.7.0 #277

Full Changelog: v0.6.6...v0.7.0

v0.6.6

🐘 Improve stability

Previously, ChatGPT (OpenAI API) sometimes does not return a response, which prevents the character from continuing the conversation.😔
However, in this update, the system will automatically retry in the absence of a response, allowing the conversation to continue.

• Retry when no response from ChatGPT #263

💃 Autonomous animation

Set animations before starting conversation. (e.g. at Start())

chatGPTContentSkill.RegisterAnimation("waving_arm", new Model.Animation("BaseParam", 10, 3.0f));
chatGPTContentSkill.RegisterAnimation("look_away", new Model.Animation("BaseParam", 6, 3.0f, "AGIA_Layer_look_away_01", "Additive Layer"));

List them in the prompt and provide a guide on how to use them.

* You can express your emotions through the following animations:

- waving_arm
- look_away

* If you want to express emotions with gestures, insert the animation into the response message like [anim:waving_arm].

• Support autonomous animation💃 #264

😴 Multiple idling mode

You can have multiple idling mode. The default is normal. Add animations and face expressions if you want to add another mode like below:

modelController.AddIdleAnimation(new Model.Animation("BaseParam", 101, 5f), mode: "sleep");
modelController.AddIdleFace("sleep", "Blink");

Switch using ChangeIdlingModeAsync.

modelController.ChangeIdlingModeAsync("sleep");

The animation for sleep starts immediately.

• Support switching idling mode #267

🍩 Other updates

• Make it easier to set microphone device name #265
• Small changes for v0.6.6 #266
• Update demo #268

Full Changelog: v0.6.5...v0.6.6

v0.6.5

🌊 Azure OpenAI Service

To use Azure OpenAI Service set following info on inspector of ChatGPTService component:

1. Endpoint url with configurations to Chat Completion Url

format: https://{your-resource-name}.openai.azure.com/openai/deployments/{deployment-id}/chat/completions?api-version={api-version}

2. API Key to Api Key

3. Set true to Is Azure

NOTE: Model on inspector is ignored. Engine in url is used.

• Add support for Azure OpenAI Service #259

🗣️ OpenAI Speech Services

Support multiple languages and allow you to switch between them without changing configurations.

NOTE: Text-to-Speech doesn't support WebGL.

• Add support for OpenAI Speech Services #260

🚰 Real-time stream speech recognition

Use AzureStreamVoiceRequestProvider to perform real-time speech recognition.
This component depends on Azure Speech SDK.

NOTE: Microphone volume slider doesn't support controlling this component.

• Add support for stream speech recognition #261

☺️ Other small changes

• Small changes for v0.6.5 #262

Full Changelog: v0.6.4...v0.6.5

v0.6.4

🥰 Make Unagirl more kawaii

• Make Unagirl (3D model for demo) more kawaii #256

By applying eye makeup, I enhanced the attractiveness of her face.🪄

• Enhanced the double eyelids.
• Increased the volume of the lower eyelashes.

🐕 Support Retrieval Augmented Generation

• Make it possible to use request, state and user info in ChatGPTFunctionSkills #257
• Update demo and example by @uezo in #258

You can create RAG-based AI agent just by adding ChatGPT function skill. See RetrievalQASkill example that can answer based on OpenAI terms of use.

Full Changelog: v0.6.3...v0.6.4

v0.6.3

Feature updates⚡️

Minor updates but they are necessary for creating apps.

• Make it possible to set role to the next message after function calling #247
• Add support for muting microphone #248
• Add character message window #249
• Support favoring specific phrases on Google Text-to-Speech #250

Unagirl, a new demo character debuts🥰

• Unagirl debuts as a new demo character🥰 #255

See Unagirl in dress at vroid hub https://hub.vroid.com/characters/7204218679890800631/models/6961078537805052952

Other updates and fixes🐈

• Improve HTTP error debugging #251
• Use UniTask in HTTP request #252
• Fix VoiceRecorder fails in starting #253
• Fix warning message when no skill router is attached #254

Full Changelog: v0.6.2...v0.6.3

v0.6.2

🥳Create ChatGPT-based AI assistant smarter

ChatGPT is no longer just one of the various skills but is integrated as a conversation processing system of ChatdollKit.

• Add autonomous routing system by ChatGPT⚡️ #241

You can add skills much smarter: All you have to do is making the definitions of the interface of ChatGPT function calling and to implement the logic to handle the result from the function.

See also the example.
https://github.com/uezo/ChatdollKit/blob/master/Examples/ChatGPT/FunctionSkills/WeatherSkill.cs

🌎 Better experience on WebGL

• Support ChatGPT stream mode in WebGL #244

👍 Other updates

• Add support for ChatGPT function calling #239
• Support starting conversation without prompt #243
• Fix bug that ChatGPT func is not registered #245
• Fix performing response bugs #246

Full Changelog: v0.6.1...v0.6.2

v0.6.1

⚡️ Improve performance, stability and developer experience

• Make GetDialogToken public #225
• Improve message window #226
• Add example of using ChatGPT in streaming mode #227
• Start idling when no animations on start #230 #231
• Avoid errors when setting unregistered face expressions #232 #233
• Add missing inheritance to SkillRouterBase #234
• Make compatible easily with the latest VOICEVOX #235
• Improve ChatGPT response speed🚅 #236
• Update demo for v0.6.1 #237

🐞 Fix bug

• Fix bug that iOS build fails #218
• Fix bug that voice cache doesn't work #224

Full Changelog: v0.6...v0.6.1

v0.6

❤️ Emotion Simulation powered by ChatGPT

Add ChatGPTEmotionSkill as an out-of-the-box example that express the emotion as face expressions while conversation.
Also add sample motions while thinking (waiting response from ChatGPT) and support multiple idle animations (change pose periodically).
You can build rich 3D AI virtual assistant lightning fast⚡️

• Add example of ChatGPT with emotion engine by @uezo in #209
• Support multiple idle animations by @uezo in #214

💡 Improve prompt engineering interface

Support one-shot prompt: system message content and a pair of user and assistant message content as an example turn. You can configure it on inspector, and even runtime for they are the public fields.

• Support prompt engineering for ChatGPT by @uezo in #207
• Easy to customize chatgptskill by @uezo in #208

🚅 Support runtime VRM loading

Support configuration at runtime. This enables you to build apps that uses user own VRM.

• Support runtime VRM loading by @uezo in #205

⚡️ Improve performance and stability

Improve FPS about 30% on my machine and stability of controlling animation gets better. They are basically internal refactoring but that includes some destructive changes.

• Use animator parameters to set animation instead of state name.

// ~ v0.5.3
response.AddAnimation("AnimationName");

// v0.6
response.AddAnimation("BaseParam", 6);

• PreGap and PostGap no longer available for the argument of Animation and FaceExpression. Use duration to adjust timing.

• Improve animation performance⚡️ by @uezo in #210
• Make FaceExpression feature runs in Update() by @uezo in #211
• Simplify the interface to control model by @uezo in #212

Other changes

• Add Azure OpenAI's ChatGPT Skill by @nakazax in #202
• Use uLipSync as the default LipSync library by @uezo in #203
• Change Azure Text-to-Speech default voice by @uezo in #204
• Make it possible to observe the status of dialog by @uezo in #206
• Fix build error in uLipSyncHelper by @uezo in #213
• Messagebox in hierarchy by @uezo in #215
• Make FaceClipConfiguration create various expressions by @uezo in #216

New Contributors

• @nakazax made their first contribution in #202

Full Changelog: v0.5.3...v0.6

v0.5.3

ChatGPT support✨

Skill for contextual conversation with ChatGPT and it's demo is included!
You can develop and start talking 3D model based virtual agent lightning fast⚡️

• Add ChatGPT ready example and demo #198

Add UI components😊

Add slider for microphone volume and FPS indicator.

• Add UI Components (Mic volume and FPS) #199

Others

Fix some bugs for WebGL platform.

• Fix some build errors #197

v0.5.2

Separate ChatdollKit components from avatar object

Separate ChatdollKit components from avatar object to manage components easier; setting up, changing 3D model or something like that use cases. At this version, the developer can complete configuration by just adding ChatdollKit prefab to the hierarchy and configuring some components attached to this prefab. #190

Support uLipSync configuration

To set up uLipSync select uLipSync on the inspector of ModelController and then start Setup ModelController. #191

Demo

• Add a skill using GPT3 based chatting API by OpenAI. #193
• Out-of-the-Box demo for Oculus VR. Enjoy chat with the ChatdollKit demo girl in VR. #195