How to input base64 image

[mcgrimm]

Self Checks

• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
• Please do not modify this template :) and fill in all the required fields.

1. Is this request related to a challenge you're experiencing? Tell me about your story.

Hello , i have an ERP system which want to send image to DIFY , but the image should in format of base64, currently i can't find any documentation about the base64 image input, i would like to run API to pass the base64 file to workflow.
can anyone have any suggestion or hints? i can't find any documentation in the API introduction so far. thank you very much.
Dify is the great tool , i hope i can use it in the industrial areas, the the integration ability should be very important.
thanks.

2. Additional context or comments

No response

3. Can you help us with this feature?

• I am interested in contributing to this feature.

[dosubot[bot]]

Hey @mcgrimm! 👋 I'm here to help you with bug fixes, answer questions, and even guide you on becoming a contributor. If you need anything, just give me a shout!

I found some relevant information that might help you with handling base64 images in Dify. The Dify API documentation provides methods for decoding base64 encoded images, which include checking if the encoded string is plain base64 without a MIME type prefix, and extracting MIME type and image data from a base64 encoded string with a MIME type prefix. Here's a snippet of the relevant code:

@staticmethod
def _decode_image(base64_image: str) -> tuple[str, bytes]:
    if DallE3Tool._is_plain_base64(base64_image):
        return "image/png", base64.b64decode(base64_image)
    else:
        return DallE3Tool._extract_mime_and_data(base64_image)

@staticmethod
def _is_plain_base64(encoded_str: str) -> bool:
    return not encoded_str.startswith("data:image")

@staticmethod
def _extract_mime_and_data(encoded_str: str) -> tuple[str, bytes]:
    mime_type = encoded_str.split(";")[0].split(":")[1]
    image_data_base64 = encoded_str.split(",")[1]
    decoded_data = base64.b64decode(image_data_base64)
    return mime_type, decoded_data

These methods can help you convert base64 images to a format suitable for API upload ^[1].

Additionally, there are similar discussions and issues that might provide more context or solutions:

1. Beginner Question: How to send Base64 encoded JPG from Make to Dify and process it with LLM? - This is an unsolved discussion.
2. I can't upload the image on dify workflow via api - Another unsolved discussion.

These might offer additional insights or approaches to your problem.

To continue talking to Dosu, mention @dosu.

[mcgrimm]

@dosu thanks for quick respond . i still struggle how to pass the base 64 string into LLM , i did try firework.ai , their api playground tell me how to setup the json with base64 string, and run very well. so i think dify should have such capability. , just fyi below is the api of firework.ai.

URI uri = URI.create("https://api.fireworks.ai/inference/v1/chat/completions");
HttpClient client = HttpClient.newHttpClient();
HttpRequest request = HttpRequest.newBuilder()
.uri(uri)
.header("Accept", "application/json")
.header("Content-Type", "application/json")
.header("Authorization", "Bearer <API_KEY>")
.POST(HttpRequest.BodyPublishers.ofString("""{
"model": "accounts/fireworks/models/llama-v3p2-90b-vision-instruct",
"max_tokens": 16384,
"top_p": 1,
"top_k": 40,
"presence_penalty": 0,
"frequency_penalty": 0,
"temperature": 0.6,
"messages": [
{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {
"url": "data:image/jpeg;base64,iVBORw0KGgoAAAANSUhEUgAABnUAAAkhCAYAAACaNmvNAAAACXBIWXMAAB7CAAAewgFu0HU+AAAgAElEQVR4nOzdeZxc1X3m/885d6mlV0mtfRcSoAUESGITSGK1CWPHduwQO16zTDKTyTaJk3GcXxxnsWcSZ5v4l2RiJ3a8gOPEjg0Gg0HsAgFCKwIJAdrV2qWWeqnl3nPmj6pqVbdaoiVaSO153q9X0VW37nK6u0pC56nv9xjvvUdEREREREREREREREQuaPZ8D0BERERERERERERERETenEIdERERERERERERERGRYUChjoiIiIiIiIiIiIiIyDCgUEdERERERERERERERGQYUKgjIiIiIiIiIiIiIiIyDCjUERERERERERERERERGQYU6oiIiIiIiIiIiIiIiAwDCnVERERERERERERERESGAYU6IiIiIiIiIiIiIiIiw0DovT/rg091rDEGY8xZn1dERERERERERERERET6Cge740ABTm1b/+estQp1REREREREREREREREhpDar4mIiIiIiIiIiIiIiAwDg67UgdO3WxMREREREREREREREZFz54zarznngBMhTv+vNQp5REREREREREREREREhtZZhzoDBTsKc0RERERERERERERERM6Nt9R+TSGOiIiIiIiIiIiIiIjI28MOdsdapU7tVlOr0FHAIyIiIiIiIiIiIiIicu6cVfs1ay3e+5Mqd0REREREREREREREROTcGHSlDoBzrjfMqQU6qtIRERERERERERERERE5984o1BEREREREREREREREZHzY8D2a/2rcWqt12pVOUEQ9N76H1NjrfIiERERERERERERERGRoRLWt07rv0ZOfajjvccYg7UWay1BEAwY3NT2ExERERERERERERERkaGjchoREREREREREREREZFhQKGOiIiIiIiIiIiIiIjIMKBQR0REREREREREREREZBhQqCMiIiIiIiIiIiIiIjIMKNQREREREREREREREREZBhTqiIiIiIiIiIiIiIiIDAMKdURERERERERERERERIYBhToiIiIiIiIiIiIiIiLDQHi2B3rv8d4P5VhERERERERERERERETkFAYd6hhjCMMTuzvneu9ba3v3McYM4fBEREREREREREREREQEzjDUeSvPi4iIiIiIiIiIiIiIyNnTmjoiIiIiIiIiIiIiIiLDgEIdERERERERERERERGRYUChjoiIiIi