Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Image edit using DALL·E #462

Open
wants to merge 6 commits into
base: main
Choose a base branch
from
Open

Image edit using DALL·E #462

wants to merge 6 commits into from

Conversation

gilcu3
Copy link
Contributor

@gilcu3 gilcu3 commented Nov 16, 2023

The results using DALL·E 2 are not impressive, but at least it will be ready when openai releases edit support for DALL·E 3.

The feature works by sending an image to the bot with caption "/edit this is my prompt". Then the bots replies with the image converted to PNG. You must download this image, modify it (by using the telegram image editor or any other), and attach it in a reply message to the previous message by the bot, with caption "/edit".

Certainly not the most user-friendly, but given the telegram bot restrictions I could not figure a better way.

@gilcu3
Copy link
Contributor Author

gilcu3 commented Nov 18, 2023

This is how the interactions currently look like, both images as photos or files are accepted.
User:
i1
Bot replies to the previous user message:
i2
User replies to the previous bot message:
i3
i4
And the end result:
result

I think it could be done in just one interaction now, but initially I thought doing three to allow editing using telegram itself.

@iamjackg
Copy link

I think it makes a lot of sense to do it in multiple steps to allow for editing in Telegram itself. I love your approach of an image diff, so people can use any color to draw the mask: it's very intuitive. However it could cause an issue if the image happens to have that same color under the brush stroke. Maybe the "Reply to this message" response should recommend using an uncommon color like magenta?

@gilcu3
Copy link
Contributor Author

gilcu3 commented Nov 24, 2023

@iamjackg Thanks for the feedback. Instead of explaining more in the reply message, we could put it in the README :) Anyway, what I am doing currently is to do the best effort (cannot be perfect because of compression) to find the differences, and then passing the minimal rectangle that covers those differences as a mask, so it should work in most cases even if the underlying image has the same color of the brush (not all of course).

@gilcu3
Copy link
Contributor Author

gilcu3 commented Apr 4, 2024

After some news this week it seems image editing will (is already) support Dalle-3, therefore probably it is a good time to get this merged. @n3d1117 could you take a look here?

@gilcu3 gilcu3 changed the title Image edit using DALL·E 2 (WIP) Image edit using DALL·E Apr 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants