TEN AI Agent Demo

This app is powered by the technology of Realtime Communication, Realtime Transcription, a Large Language Model (LLM), and Text to Speech extensions. The TEN Framework makes the workflow super easy! The Unity Demo resembles the web demo and acts as the mobile frontend to the AI Agent. You may ask the Agent any general question.

The Plugin is exported as a reusable package that can be imported on any Agora RTC projects. Download the package and import into your project.

Prerequisites:

• Agora Developer account
• Agora Video SDK for Unity (v4.2.6 or up)
• TEN Frameworks Agent
• Unity 2021 or up

Implicit requirements by TEN Framework:

• Text to Speech Support (API Key from Azure Speech Service)

• LLM Support (e.g. API key from OpenAI)

Setups

TEN Agent Server

First you should have gotten the TEN Agent working in your environment. You will just need the Server (ten_agent_dev) running for this application.

Note, the last verified commit is ac7fd7a7a76d09d018513989d32b37ba7685e652.

To run this Demo

0. Clone this project.

1. Download and import the Agora Video SDK for Unity.

2. Add both TENEntryScreen scene and TENDemoScene to the Build Settings. The TENEntryScreen scene should be on top.

3. Fill out the information for Config input asset.

4. Play the demo on Editor or build it to device platform of your choice.

Alternatively, Android apk

Go to the Release section, download the apk file and experience the AI Agent.

To use the prefab for your own project

0. Download the TENClientFramework.unitypackage file from the Releases section; import it to your project.

1. Download and import the Agora Video SDK for Unity.

2. Drag the prefabs into your project and use them connect to your controller code.

   • ChatController
   • SphereVisual
   • TENManager

3. Follow the Modification Steps below.

4. Fill out the information for Config input asset.

5. Play the demo on Editor or build it to device platform of your choice.

Modification Steps to An Existing Project

1. Pass Scriptable object to AppConfig:

using Agora.TEN.Client;

[SerializeField]
TENConfigInput TENConfig; // input from Editor

// Call this function in your Init step
void  SetConfig()
{
	AppConfig.Shared.SetValue(TENConfig);
	// obtain channel name before this call
	AppConfig.Shared.Channel = UtilFunctions.GenRandomString("agora_", 5);
}

Note you should provide a channel name in your app logic to use among the participanting users. In this 1-to-1 chat project, we provide an util function GenRandomString() to ganerate the channel name.

Don't forget to drop the TENConfigInput Asset into Inspector field for TENConfig. See demo picture in the previous section.

2. Hook the prefabs up in your main logic:

[SerializeField]
internal  IChatTextDisplay  TextDisplay;  // ChatController
[SerializeField]
internal  TENSessionManager  TENSession;  // TENManager
[SerializeField]
internal  SphereVisualizer  Visualizer;   // SphereVisual

3. Use TENSession.GetToken() to get token before joining channel

async void GetTokenAndJoin()
{
    AppConfig.Shared.RtcToken = await TENSession.GetToken();
    JoinChannel();  // your join channel call that uses the rtc token
}

4. In your initialization step, setup sound visualization, which will automatically configure the Agora RTC engine for audio data capture.

	Visualizer?.Init(RtcEngine);

5. Start the TEN session on OnJoinChannelSuccess()

// _app is the instance of your controller class
_app.TENSession.StartSession(connection.localUid);

6. Disable/Enable the sound visualizer, a component of a gameobject that represents the AI Agent:

• Disable it during initialization, e.g. SetupUI() or Start()

	Visualizer?.gameObject.SetActive(false);

• Enable it when the AI Agent user joins the channel:

public override void OnUserJoined(RtcConnection connection, uint uid, int elapsed)
{
    if (uid == AppConfig.Shared.AgentUid) {
        _app.Visualizer?.gameObject.SetActive(true);
	}
}

7. Register handler OnStreamMessage:

• SDK ver 4.4.0

public  override  void  OnStreamMessage(RtcConnection  connection, uint  remoteUid, int  streamId, byte[] data, ulong  length, ulong  sentTs)
{
	string  str = System.Text.Encoding.UTF8.GetString(data, 0, (int)length);
	_app.TextDisplay.ProcessTextData(remoteUid, str);
	_app.TextDisplay.DisplayChatMessages(_app.LogText.gameObject);
}

• SDK ver 4.2.6

public override void OnStreamMessage(RtcConnection connection, uint remoteUid, int streamId, byte[] data, uint length, System.UInt64 sentTs)
{
    string str = System.Text.Encoding.UTF8.GetString(data, 0, (int)length);
    _app.TextDisplay.ProcessTextData(remoteUid, str);
    _app.TextDisplay.DisplayChatMessages(_app.LogText.gameObject);
}

8. OnDestroy() or logic to stop.

TENSession.StopSession();

References

For reference, it is worthwhile to check out the following resources:

• TEN Framework docs
• TEN IOS Client
• Agora SDK API references.

License

MIT License