IMAP emails

[ardudi]

I use Wemos D1 mini as an IoT device. The device should receive once in a time an email with few parameters which are contained in the message body. Message subject line also composed of specific text.
I have modified Read_Single_Email_Loop example to reduce memory usage and consumption of the device.
My email Inbox contains only 2 unread emails. Surprisingly, the example I use doesn't return and indication for unread emails (return unseen=0). Also I define a search criteria to search sublect line content but always return 0. I am attching here my modified code. You may see that I defined ENABLE_ERROR_STRING, imap_data.search.criteria = F("SUBJECT ardudi@gmail.com") and added ESP_MAIL_PRINTF("Search Criteria Found: %d\n", sFolder.searchCount());
As you may see, I also captured the serial consule output during runtime. Would appreciate any comment on this.
Modified_Read_Single_Email_Loop2.zip
‏‏Modified_Read_Single_Email_Loop2 - consule.txt

[mobizt]

You asked the same question as on Arduino forum that I already answered.

There are misunderstanding in your usage and library concept when I look at your providing code.

If you read all comments in examples and test it enough to see how search and fetch examples work, you can adapt both of them to your code easily.

Library works with IMAP command to make request and parsing response which a task can be a combination of multiple IMAP commands which need to be done sequentially.

It is recommended that you should read the RFC standard related to IMAP/SMTP commands for using this library properly.

Instead of asking here, you should check my latest reply with code that works based on your requirement in Arduino forum.

Library provides a method for accessing the message in user mailbox via MailClient.readMail.

Library provides the 3 classes for transportation and data that passed to readMail funcion, ie.e. IMAP_Session (transport medium), Session_Config (session configuration) and IMAP_Data (user data that control the IMAP operation).

There are two IMAP commands that mainly used in this library which have different purposes i.e. SEARCH and FETCH.

The SEARCH command will be used for searching the messages with some criteria which IMAP server returns only UID (if UID prodided in search criteria) or messages sequence number (if no UID keyword provided) in the response.

Library will perform some additional operations by parsing that search result for UID or message number and collect them in atrray and fetching the message by its UID or number one by one (using FETCH command).

Only message header will be fetched for faster operation and minimum memory usage as brief information of messages header only stored in the result i.e. message items data (list).

You will see the debugging message about the searching operation when you test with search example.

To determine which types of operation is being perform, call headerOnly method from IMAP_Session, i.e., imap.headerOnly() which will return true when search operation is being perform.

To read entire message content (header and body) of the message items returns from search operation, you need to store the UID or sequence number of messages in an array and iterate fetching the message by UID or number in that array one by one.

The FETCH command will be used when you know which message to read or fetch.

Library will perform fetch when one of these options are set.
imap_data.fetch.sequence_set.string, imap_data.fetch.uid, and imap_data.fetch.number

Now look at your code.

When you use this line for searching, imap_data.search.criteria = F("SUBJECT ardudi@gmail.com");

It will perform searching only when the function MailClient.readMail was executed.

I see you are trying to print irrelevant information of search count in the function printSelectedMailboxInfo like this.

ESP_MAIL_PRINTF("Search Criteria Found: %d\n", sFolder.searchCount());

Using sFolder.searchCount() here is not make sense because when imap.selectFolder was executed followed by printSelectedMailboxInfo, the information from selected (opened) mailbox will be printed, no search was done here.

In the loop, you are trying to fetch message by providing the message number to imap_data.fetch.number.

The message number you provide is the count-down/count-up value initiated from total messages in your mailbox (using imap.selectedFolder().msgCount()).

This will fetch whole messages in your opened mailbox one by one, then the debug info will be something like this in case only two messages stored in your mailbox.

#### Fetch message 1, Number: 2
..
..

#### Fetch message 2, Number: 1
..
..

#### Fetch message 3, Number: 1
..
..

#### Fetch message 4, Number: 2
..
..

#### Fetch message 5, Number: 1
..
..

#### Fetch message 6, Number: 1
..
..

#### Fetch message 7, Number: 2
..
..

If your mailbox contains 100 messages, the debug message will show Number: 100 to Number: 1 during count down and Number: 1 to Number: 100 during count up.

Based on your needs, do something like this is not make sense as search does not perform and imap.selectedFolder().msgCount() returns the total message in your mailbox instead of number of messages from search result.

You should start with search, then store the UID or number of messages from search result. (imap.data().msgItems which is the array (vector) that can be iterated through the indexes to get UID with imap.data().msgItems[x].UID or message number with imap.data().msgItems[x].msgNo)

Once you keep the UID or message number of your messages result into array already, now you can iterate fetching that message to get the whole message data later.

Here is the code that I modified from my latest post in Arduino forum based on your needs.

In this code we work with message UID instead of message number.

#include <Arduino.h>
#include <ESP8266WiFi.h>

#include <ESP_Mail_Client.h>

#define WIFI_SSID "<ssid>"
#define WIFI_PASSWORD "<password>"

#define IMAP_HOST "<host>"

#define IMAP_PORT esp_mail_imap_port_993

#define AUTHOR_EMAIL "<email>"
#define AUTHOR_PASSWORD "<password>"

void imapCallback(IMAP_Status status);

void printMessageData();

IMAPSession imap;

// Max messages in the search result
int max_result = 10;

// Array to store the UID of messages in search result
int msg_uid[10];

void setup()
{

    Serial.begin(115200);

    Serial.println();

    WiFi.begin(WIFI_SSID, WIFI_PASSWORD);

    Serial.print("Connecting to Wi-Fi");

    while (WiFi.status() != WL_CONNECTED)
    {
        Serial.print(".");
        delay(300);
    }
    Serial.println();
    Serial.print("Connected with IP: ");
    Serial.println(WiFi.localIP());
    Serial.println();

    MailClient.networkReconnect(true);

    imap.debug(1);

    imap.callback(imapCallback);

    Session_Config config;

    config.server.host_name = IMAP_HOST;
    config.server.port = IMAP_PORT;
    config.login.email = AUTHOR_EMAIL;
    config.login.password = AUTHOR_PASSWORD;

    IMAP_Data imap_data;
    
    // Clear all these fetch options to perform search
    imap_data.fetch.uid.clear();
    imap_data.fetch.number.clear();
    imap_data.fetch.sequence_set.string.clear();

    imap_data.search.unseen_msg = true;

    // Don't download all to filesystem
    imap_data.download.header = false;
    imap_data.download.text = false;
    imap_data.download.html = false;
    imap_data.download.attachment = false;
    imap_data.download.inlineImg = false;

    // Store html/text message body in IMAPSession object
    imap_data.enable.html = true;
    imap_data.enable.text = true;

    imap_data.enable.recent_sort = true;

    // Max messages in the search result
    imap_data.limit.search = max_result;

    imap_data.limit.msg_size = 2048;

    imap_data.limit.attachment_size = 1024 * 1024 * 5;

    if (!imap.connect(&config, &imap_data))
        return;

    if (!imap.selectFolder(F("INBOX")))
        return;

    // We search the messages first to get its UID and stored in msg_uid.
    imap_data.search.criteria = F("SEARCH SUBJECT ardudi@gmail.com");
    MailClient.readMail(&imap, false /* keep session open for fetching message in opened mailbox later */);

    // We already get the search result message, fetch it

    // Fetch the messages using UID stored in msg_uid one by one
    for (int i = 0; i < max_result; i++)
    {
        imap_data.search.criteria.clear();

        // Now Fech message by UID stored in msg_uid
        imap_data.fetch.uid = msg_uid[i];
        MailClient.readMail(&imap, false /* keep session open for fetching message in opened mailbox later */);
    }

    imap.closeSession();
    imap.empty();
}

void loop()
{
}

void imapCallback(IMAP_Status status)
{
    Serial.println(status.info());

    if (status.success())
    {
        // If this is the search result (imap contains only header info),
        // store the message UID that we can fetch for its body later.
        if (imap.headerOnly())
        {
            max_result  = imap.data().msgItems.size();
            for (size_t i = 0; i < imap.data().msgItems.size(); i++)
            {
                msg_uid[i] = imap.data().msgItems[i].UID;
            }
        }
        else
        {
            // This is the fetch result, print the whole message (header + body)
            printMessageData();
        }

        imap.empty();
    }
}

void printMessageData()
{

    IMAP_MSG_Item msg = imap.data().msgItems[0]; // msgItems contains only one message from fetch

    Serial.println("****************************");

    ESP_MAIL_PRINTF("Number: %d\n", msg.msgNo);
    ESP_MAIL_PRINTF("UID: %d\n", msg.UID);
    ESP_MAIL_PRINTF("Messsage-ID: %s\n", msg.ID);
    ESP_MAIL_PRINTF("Flags: %s\n", msg.flags);

    ESP_MAIL_PRINTF("Attachment: %s\n", msg.hasAttachment ? "yes" : "no");

    if (strlen(msg.acceptLang))
        ESP_MAIL_PRINTF("Accept Language: %s\n", msg.acceptLang);
    if (strlen(msg.contentLang))
        ESP_MAIL_PRINTF("Content Language: %s\n", msg.contentLang);
    if (strlen(msg.from))
        ESP_MAIL_PRINTF("From: %s\n", msg.from);
    if (strlen(msg.sender))
        ESP_MAIL_PRINTF("Sender: %s\n", msg.sender);
    if (strlen(msg.to))
        ESP_MAIL_PRINTF("To: %s\n", msg.to);
    if (strlen(msg.cc))
        ESP_MAIL_PRINTF("CC: %s\n", msg.cc);
    if (strlen(msg.date))
    {
        ESP_MAIL_PRINTF("Date: %s\n", msg.date);
        ESP_MAIL_PRINTF("Timestamp: %d\n", (int)MailClient.Time.getTimestamp(msg.date));
    }
    if (strlen(msg.subject))
        ESP_MAIL_PRINTF("Subject: %s\n", msg.subject);
    if (strlen(msg.reply_to))
        ESP_MAIL_PRINTF("Reply-To: %s\n", msg.reply_to);
    if (strlen(msg.return_path))
        ESP_MAIL_PRINTF("Return-Path: %s\n", msg.return_path);
    if (strlen(msg.in_reply_to))
        ESP_MAIL_PRINTF("In-Reply-To: %s\n", msg.in_reply_to);
    if (strlen(msg.references))
        ESP_MAIL_PRINTF("References: %s\n", msg.references);
    if (strlen(msg.comments))
        ESP_MAIL_PRINTF("Comments: %s\n", msg.comments);
    if (strlen(msg.keywords))
        ESP_MAIL_PRINTF("Keywords: %s\n", msg.keywords);

    if (strlen(msg.text.content))
        ESP_MAIL_PRINTF("Text Message: %s\n", msg.text.content);
    if (strlen(msg.text.charSet))
        ESP_MAIL_PRINTF("Text Message Charset: %s\n", msg.text.charSet);
    if (strlen(msg.text.transfer_encoding))
        ESP_MAIL_PRINTF("Text Message Transfer Encoding: %s\n", msg.text.transfer_encoding);
    if (strlen(msg.html.content))
        ESP_MAIL_PRINTF("HTML Message: %s\n", msg.html.content);
    if (strlen(msg.html.charSet))
        ESP_MAIL_PRINTF("HTML Message Charset: %s\n", msg.html.charSet);
    if (strlen(msg.html.transfer_encoding))
        ESP_MAIL_PRINTF("HTML Message Transfer Encoding: %s\n\n", msg.html.transfer_encoding);

    Serial.println();
}

Finally, you can't use #define ENABLE_ERROR_STRING in your sketch because this macro is for global configuration in file ESP_Mail_FS.h

If you define it here in your code, it will be seen only in your sketch. Library can't see this definition.

This macro is already defined globally here.

Library will look for definitions in library provided ESP_Mail_FS.h or Custom_ESP_Mail_FS.h you created.

Edit:
Fix typo and remove unnecessary code.

[mobizt]

I'm not quite sure that what information you used to justify that you can't get unread message from search or fetch examples.

If you mean Unseen Message Index: 0 from this information, it's not correct.

Info of the selected folder
Total Messages: 2
UID Validity: 1
Predicted next UID: 138
Search Criteria Found: 0
Unseen Message Index: 0
Highest Modification Sequence: 20759
Flags: \Answered, \Flagged, \Draft, \Deleted, \Seen, $NotPhishing, $Phishing

The above information is the properties of selected (opened) mailbox and is not related to searching or fetching operation.

The Unseen Message Index from method sFolder.unseenIndex() means the first message number of messages that unread or unseen (no \Seen flag was added or set).

The default value of unseen index is 0 which means no unread messages in selected mailbox, or all messages have been read.

The valid index of unseen message is 1 to total messages stored in selected mailbox.

[ardudi]

Thank you so much for your explanation above, As you may understand, I am not so strong in C++ and actually didn't understand in deep the IMAP procedure. Your above code and your explanation were made perfectly. Now the code performs exactly what I want.
Just one question. This statement : imap_data.search.criteria = F("SEARCH SUBJECT ardudi@gmail.com"); works very well. can the syntax of "SEARCH SUBJECT ardudi@gmail.com" be used as a string varaible of better spit into "SEARCH SUBJECT" + string varaibale?

[mobizt]

Yes, you can use any type of string or string concatenation.
F macro is just for demonstrating that it supports flash string too.

[ardudi]

Great. I'll give a try. Thanks again. I do ecellant support!

[ardudi]

Sorry Suwatchai. What I wanted to say is that you are doing great support. I have a new question rgerading the statement: strlen(msg.text.content). Could it be that two control characters (CR,LF) are added to the end of msg.text.content ? I see that strlen(msg.text.content) return the size of the txt message freater by two. I was looking at an external consule monitor using hex dispaly and got that result.

[mobizt]

No, it depends on your message body. Library does not change it unless decode if it encoded.

[ardudi]

I'll be checking this with different email client and report back.

[mobizt]

I found this issue in v3.1.1 afterward.

This also cause another IMAP issue of missing text message in message with both text and html contents and without attachments.

I will fix it and update soon.

[ardudi]

Very well. I'll wait. I also saw another issue. When I used msg.deleteMessages for a specific email in gmail using msg.UID or message number, although I got indication that the message was removed, nothing happend in my Gmail inbox. I am working on this...

[mobizt]

There is no issue about delete messages function, only the \Deleted flag will be added to message and message will only move to trash if you don't expunge it.

The second parameter of method imap.deleteMessages can set to true to delete messages permanently.

The Boolean status returns from the deleteMessages function depends on the server response.

[ardudi]

Please see my partial code vs. its consule monitor. I made a kinf of 'correction' to the CR+LF extra characters that were added to the message text:
My code:
if (strlen(msg.from))
ESP_MAIL_PRINTF("From: %s\n", msg.from);
if (strlen(msg.sender))
ESP_MAIL_PRINTF("Sender: %s\n", msg.sender);
if (strlen(msg.to))
ESP_MAIL_PRINTF("To: %s\n", msg.to);
if (strlen(msg.cc))
ESP_MAIL_PRINTF("CC: %s\n", msg.cc);
if (strlen(msg.date))
{
ESP_MAIL_PRINTF("Date: %s\n", msg.date);
}
if (strlen(msg.subject))
ESP_MAIL_PRINTF("Subject: %s\n", msg.subject);
// if (strlen(msg.references))
// ESP_MAIL_PRINTF("References: %s\n", msg.references);
// if (strlen(msg.comments))
// ESP_MAIL_PRINTF("Comments: %s\n", msg.comments);
// if (strlen(msg.keywords))
// ESP_MAIL_PRINTF("Keywords: %s\n", msg.keywords);
if (strlen(msg.text.content))
{
int messageLen = strlen(msg.text.content)-2;// there are additional CR and LF (0x0d, 0x0a)
char messageEnd = msg.text.content[messageLen-1];
Serial.print(F("--->>> Message length: "));
Serial.print(messageLen);
Serial.print(F("; last char: "));
Serial.println(messageEnd);
Serial.println(strlen(msg.text.content));

    ESP_MAIL_PRINTF("Text Message: %s\n", msg.text.content);
    if ((messageLen % 3 != 0) || (messageEnd != ';'))
    {
      Serial.print(F("--->>> Message Text (UID = "));
      Serial.print(msg.UID);
      Serial.println(F(") is wrong, remove message..."));
      if (imap.deleteMessages(msg.msgNo,true))
      {
        Serial.print(F("Message "));
        Serial.print(msg.msgNo);
        Serial.println(F(" deleted"));
      }
    }
}

Consule output:

Reading messages...

Finished reading Email

---

Number: 1
UID: 136
From: Dudi Argaman ardudi@hotmail.com
To: "abc@gmail.com" abc@gmail.com
Date: Sun, 18 Jun 2023 15:05:35 +0000
Subject: ardudi@gmail.com
--->>> Message length: 8; last char: 5
10
Text Message: E1;L8;D5

--->>> Message Text (UID = 136) is wrong, remove message...

Open the mailbox folder...

Deleting message(s)...

Message 1 deleted

Connecting to IMAP server...

IMAP server connected

Checking the capability...

Logging in...

Send client identification...

Open the mailbox folder...

Fetch message 1, UID: 136

Get Flags...

Reading messages...

Finished reading Email

---

Number: 1
UID: 136
From: Dudi Argaman ardudi@hotmail.com
To: "abc@gmail.com" abc@gmail.com
Date: Sun, 18 Jun 2023 15:05:35 +0000
Subject: ardudi@gmail.com
--->>> Message length: 8; last char: 5
10
Text Message: E1;L8;D5

--->>> Message Text (UID = 136) is wrong, remove message...

Open the mailbox folder...

Deleting message(s)...

Message 1 deleted

Fetch message 2, UID: 142

Get Flags...

Finished reading Email

[ardudi]

I forgot to mention that the specific email shown above was not removed from the email server. It still exist in Inbox and is deleted by my code again and again with no real result!

[mobizt]

This is not correct usage, imap.deleteMessages(msg.msgNo,true).

You should follow these delete messages examples to use library correctly.

The first argument is the pointer to MessageList or string of sequence set.

By passing msg.msgNo which is integer variable to the method, the compiler can compile but it treats that argument as address or pointer to somewhere in memory that does not exist.

FYI

• The deleteMessages is silent method (response was ignored) which server accepts the request and returns success response if the request is valid. That's why there's nothing happened for nonexistent messages.
• Some servers delete the message permanently without expunge command needed.

[ardudi]

Sorry for my late respond. Your explnation was great. I got the point. I am busy now with the project integration and put the email support on hold for a while. I'll return to it later to make the sw integration as well...

[mobizt]

The line break issue fixed in v3.1.12.

[ardudi]

Thanks.🙏 I'll give a try.