Watchlist: Interesting new Models / Papers / Tech

[Acly]

This is a list of AI models, papers and ComfyUI nodes which might be candidates for integration into the plugin.

Guide to posting feature suggestions and requests:

• You found out about a project or new model? You want to post a link because you think it will make a good addition?
  • You're welcome to post links here in this discussion.
• You want to share an evaluation or image comparisons?
  • Feel free to open a separate discussion topic about your results .
• You want to contribute by implementing a new feature (in code)?
  • It's a good idea to create an "enhancement" issue to let everyone know.

Important

Please do not open separate issues / discussions just to post a link!

This helps avoid clutter. Keep in mind there are many shiny new things coming out all the time, and realistically only a fraction of them will be integrated.

How to make it happen?

In order of most likely success:

1. Implement it! (If you're unsure, discuss it first)
2. Evaluate and compare techniques (eg in ComfyUI), share images and workflows. It's a significant portion of the work, so it helps!
3. Comment/upvote in this discussion. One comment per feature allows people to thumbs-up to show support.

Base Models

Project	Architecture	Params	License	Comment
Kolors	U-Net, GLM	3.5B	Non-commercial	Inpaint, IP-Adapter
PixArt-Σ	DiT, T5	0.6B	Open RAIL++-M
Hunyuan-DiT	DiT, T5	1.5B	Tencent-Hunyuan (limited)
AuraFlow	DiT/Flow, T5	6.8B	Apache 2.0	Beta 0.1 version
UltraPixel	Cascade, CLIP	1B/3.5B + 0.7B/1.5B	Apache 2.0
Flux [dev]	DiT/Flow, T5	12B	Non-commercial	✅ v1.22.0 (experimental)
Flux [schnell]	DiT/Flow, T5	12B	Apache 2.0	✅ v1.22.0 (experimental)

Control Layer

Project	Description	Comment
MistoLine / AnyLine	Line art detector and ControlNet	✅ v1.19.0
Xinsir CN models	Scribble, OpenPose, Canny for SDXL	✅ v1.19.0
IC-Light	Lighting/shading control	requires Layer-Diffusion
PuLID	Face adapter
Recolor	SDXL Control Lora
PhotoMaker	Face adapter (SDXL)	#388 No insightface

Performance

Project	Description	Comment
Trajectory Consistency Distillation (TCD)	Disstilled LoRA models + sampler for low-step (SD1.5, SDXL)	❌ LoRA, ❓ Sampler
Phased Consistency Model (PCM)	Distilled LoRA models (SD1.5, SDXL), allows CFG	Models, already works
Distribution Matching Distillation (DMD2)	Distilled 4-step LoRA models (SD1.5, SDXL)	❌ Prefer Hyper-SD
Automatic CFG	Rescaled CFG, can disable CFG to boost performance
Hyper-SD	Distilled 1/4/8-step LoRA models, allows CFG	✅ v1.19.0
HiDiffusion	High resolutions w/o 2-pass at improved speed	#651 ComfyUI Node
Align-Your-Steps (AYS)	Custom scheduler for low-step	✅ v1.20.0
T-GATE	Attention caching
Improved ToMe (ToDo)	Token merging

Other

Project	Description	Comment
Layer-Diffusion	Alpha channel for image generation	#453
BrushNet	Inpaint extension model (SD1.5, SDXL)	ComfyUI node
ToonCrafter	Animation keyframe interpolation	ComfyUI node
SUPIR	Upscale model, diffusion	ComfyUI node
APISR	Upscale model, transformer	ComfyUI node

[syl1130]

nice work,please

[moonwhaler]

Vote: Control Layer - Xinsir CN models

[moonwhaler]

Vote: Other - Layer-Diffusion

[Rogal80]

Vote: MistoLine

[Rogal80]

Vote: Improved ToMe

[Towering-Toska]

Vote: Layer-Diffusion. Layer-Diffusion sounds exactly like what I would now like the most after using Krita-AI for a night. I wasn't aware of this type of plugin, I wonder if we can already manually install and use that with the Krita plugin.

[DaVinchik]

We need supir!

[xingewh]

InstantID。InstantID seems to perform better than FaceID.
(https://github.com/InstantID/InstantID)

[dkqjhoifh]

yes

[ETCore7]

Vote: Other - Tooncrafter. The video shows interesting examples of multiple frames generated between two key frames. Natural and consistent.

[Drakosha405]

what do you think about this. comparable to individual models or not, if the results are good then it can save a lot of space on the ssd.
https://huggingface.co/xinsir/controlnet-union-sdxl-1.0

[Acly]

Yes it sounds great. According to instruction you specify an ID depending on which mode you want, which isn't possible yet in ComfyUI as far as I know. It works without doing it, but maybe results can be better if you provide it.

[Corruptinator]

Not sure if this would be an interesting candidate to implement into the plugin but any chance the AuraFlow Ai could be implemented?
https://huggingface.co/fal/AuraFlow?ref=blog.fal.ai

[moonwhaler]

@Acly: This seems pretty interesting: background removal using InSPyReNet (https://github.com/plemeri/InSPyReNet).
There's also a Comfy Node for that: https://github.com/john-mnz/ComfyUI-Inspyrenet-Rembg?tab=readme-ov-file

I am already using the "transparent background" implementation (https://github.com/plemeri/transparent-background), but having this in Krita directly could be pretty interesting.

Although, I admit this might be a bit different from the diffusion implementation you have done with Krita Diffusion.

[Drakosha405]

https://huggingface.co/blog/quanto-diffusers

good development and hopefully optimized versions of Aura Flow and ControlNet versions for it will appear soon. and then it is quite possible to consider adding it to the plugin

[waynemadsen]

https://huggingface.co/black-forest-labs/FLUX.1-dev

I'm sure flux support is on your radar and I look forward to building in krita with it!

[Acly]

Added a table for base models: Kolors, PixArt, Hunyuan-DiT, AuraFlow, Flux

They come up a lot, but none of them have any kind of ecosystem or commitment from open-source community so far. Kolors is the only one with inpaint and IP-Adapter support (both coming from the model authors directly).

[Marksusu]

https://huggingface.co/2vXpSwA7/iroiro-lora/blob/main/test_controlnet2/CN-posetest_v1.safetensors
https://x.com/nana_tsukisuwa/status/1832544135766769960

这个可以用手绘简笔骨骼图生成人物动作

[ibow1983]

[Flux [dev]!! ControlNet for SDXL！！We need them!!!

[syl1130]

https://huggingface.co/Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro ,
https://huggingface.co/XLabs-AI/flux-controlnet-collections
and https://huggingface.co/XLabs-AI/flux-ip-adapter

[waynemadsen]

I'd prefer to see a GGUF/unet implementation native in Krita before ControlNet on flux. But that's just my 8GB of VRAM crying in a corner...

[eglepere]

I'd really like to see SUPIR. The current upscale implementation feels lacking

[Leroy-X]

Vote: Base Models - Kolors

[waynemadsen]

flux controlnet upscaler:
https://huggingface.co/jasperai/Flux.1-dev-Controlnet-Upscaler
same user has a good depth and normals map controlnet for flux as well.

[RanjithGadde447]

vote SUPIR UPSCALE

[waynemadsen]

FLUX ipadapter would be a great addition. It doesn't work as strongly as the style ipadapter for SDXL, but anything is better than the way FLUX resists 'style by' prompts.

https://huggingface.co/XLabs-AI/flux-ip-adapter

I suggest reading this paper: https://sandner.art/prompting-art-and-design-styles-in-flux-in-forge-and-comfyui/