Text to Image Demo

This repo contains demo for generating image based on user prompt. It runs model locally for the sake of PoC and is not suitable for production. Require NVIDIA GPU to run.

TODO:

• Implement quantized Stable Diffusion 3.5.
• Implement quantized Flux.1 Schnell.
• Implement factory pattern to handle models.
• Add model switching feature.
• Implement NiceGUI ui.storage to store session states.
• Better code modularization.
• Add brush and inpainting features.
• Add models outputs comparison feature.
• Use javascript for brush to bring the feature to client side.
• To explore Auto1111sdk. (Dec 27, 2024: Only support Stable Diffusion as of the date, so won't implement soon.)

How to install

1. Install Conda
2. Create environment: conda create -n sd python=3.11
3. Activate environment: conda activate sd
4. Install Pytorch
5. Install the required libraries: pip install -r requirements.txt

Huggingface Access Token & Model Registration

1. Follow this to create access token.
2. Visit the following links to register:
   • Flux.1 Schnell
   • Stable Diffusion 3.5 Large Turbo
   • Stable Diffusion 3.5 Medium Turbo TensorArt
   • Stable Diffusion 2 Inpainting
   • Flux.1 Fill dev
3. In terminal, after activating sd environment, type huggingface-cli login and key in the created access token to login.

How to run

1. Run the following command: python src/main.py
2. Open the Local URL (default: http://localhost:8080/) in a browser.

How to use

1. Select "Flux.1" in the top dropdown list and click "INITIALIZE T2I MODEL" button to load the text to image model.
2. Describe what you want to draw in the text box.
3. Click "DRAW" button and wait for the image to be generated.
4. If modification is desired, select "Stable Diffusion 2 Inpaint" in the bottom dropdown list and click "INITIALIZE FILL MODEL" button to load inpainting model.
5. Brush the image to mask the area you want to modify.
6. Describe what to add to the area in the text box.
7. Click "FILL" to apply the modification.

Example

Quantized Stable Diffusion 3.5 Large Turbo (StabilityAI)	Stable Diffusion 3.5 Medium Turbo (TensorArt)	Quantized Flux.1 Schnell (Black Forest Labs)

Original Image generated by Quantized Stable Diffusion 3.5 Large Turbo	Brush the image with a mask on the canvas directly	Fill a spaceship to the region with Stable Diffusion 2 Inpainting