Welcome to Ubiq-Genie

Ubiq-Genie is a framework that enables you to build server-assisted collaborative mixed reality applications with Unity using the Ubiq framework. This is particularly useful for building multi-user applications that require server-side processing such as generative models, conversational agents, and real-time transcription. For more information, please refer to the Ubiq-Genie paper.

Note

Before starting with Ubiq-Genie, we recommend that you familiarize yourself with the Ubiq framework. For more information, see Ubiq's documentation and website. Ubiq-Genie currently uses Ubiq v1.0.0-pre7.

Initial Setup

These instructions will get you a copy of the project up and running to run the samples and to start building your own applications. Ubiq-Genie supports Windows, macOS, and Linux. Ubiq-Genie has a server-client architecture, which means you may need to run the server on a separate machine from the Unity client.

Server (Node.js)

0. Install Node.js (v20 or later) and Python (v3.10 or later).

1. Clone this repository somewhere on your machine (either local or remote).

2. Open a terminal in the Node folder and run npm install to install the dependencies.

3. Install the Python dependencies by navigating to the Node/services folder and running pip install -r requirements.txt. If you are using a virtual environment, activate it before running the command. Please ensure that you have the correct PyTorch and CUDA versions installed (see the PyTorch website for more information).

Client (Unity)

1. Install Unity 2022.3.32f1 or later.

2. Clone this repository somewhere on your local machine.

3. Add the Unity folder to Unity Hub and open the project in Unity.

4. Navigate to Package Manager, click the Ubiq package (com.ucl.ubiq), navigate to the "Samples" tab, and import the "Demo (XRI)" sample. This will add the Unity XR Interaction Toolkit package to the project, as well as some scripts used by the Ubiq-Genie sample applications.

Note

Read the README file in the corresponding folder in the Node/apps folder for further setup instructions. For a list of available samples, see the Samples section below.

Documentation

For more information on how to use Ubiq-Genie, please refer to the README files in the Node folder.

Samples

The Node/apps folder contains a number of samples that demonstrate how to use Ubiq-Genie, which each utilize one or more services defined in the Node/services folder. For more information on how to use these samples, please refer to the README files in the corresponding folders. Currently, the following collaborative sample applications are available:

• Texture Generation: generates a texture based on voice-based input and an optional ray to select target objects
• Multi-user Conversational Agent: a conversational agent that can be interacted with by multiple users
• Transcription: transcribes and audio of each user in the room in separate files

For a demo video of the samples, please refer to the Ubiq-Genie demo video.

Support

For any questions, please use the Discussions tab on GitHub or send a message in the ubiq-genie channel in the Ubiq Discord server. For bug reports, please use the Issues tab on GitHub.