Lazy value serialization

[mirecta]

Hi for example i want send jpeg image from ESP32 camera (through for example MQTT) using Json, so i must encode jpeg data as base64. And its vaste of memory store jpeg in memory again even bigger because encoded as base64. So i patched ArduinoJson library to do it incrementaly
i created Abstract class DynamicData

class DynamicData {
 public:
  typedef std::function<void(const char* s, size_t n)> WriteJsonFunc;
  typedef std::function<void(const uint8_t* s, size_t n)> WriteRawFunc;

  virtual void writeJsonTo(WriteJsonFunc writeFunc) = 0;
  virtual void writeRawTo(WriteRawFunc writeFunc) {}
  virtual size_t sizeJson() {
    return 0;
  }
  virtual size_t sizeRaw() {
    return 0;
  }
};

writeJsonTo write data for Json serializer using pointer to write func
writeRawTo write data for MessagePack serializer using pointer to write func

and then i for example create Base64 encoder inherited from Dynamic data and i use it

#include <ArduinoJson.h>
#include <stdio.h>

class Base64 : public DynamicData {
 public:
  Base64(const char* data, size_t size) : data(data), dataSize(size) {}
  Base64() : data(0), dataSize(0) {}

  void encodeTriplet(const char (&input)[3], char (&output)[4]) {
    output[0] = base64_chars[(input[0] & 0xfc) >> 2];
    output[1] =
        base64_chars[((input[0] & 0x03) << 4) + ((input[1] & 0xf0) >> 4)];
    output[2] =
        base64_chars[((input[1] & 0x0f) << 2) + ((input[2] & 0xc0) >> 6)];
    output[3] = base64_chars[input[2] & 0x3f];
  }

  virtual size_t sizeRaw() {
    return dataSize;
  }

  virtual size_t sizeJson() {
    return 4 * (1 + ((sizeRaw() - 1) / 3));
  }

  virtual void writeJsonTo(DynamicData::WriteJsonFunc writer) {
    uint j = 0;
    char input[3];
    char output[4];

    for (uint i = 0; i < sizeRaw(); ++i) {
      input[j++] = data[i];
      if (j == 3) {
        encodeTriplet(input, output);
        writer(output, 4);
        j = 0;
      }
    }
    if (j != 0) {
      for (uint i = j; i < 3; ++i) {
        input[i] = '\0';
      }
      encodeTriplet(input, output);

      for (uint i = j + 1; i < 4; ++i) {
        output[i] = '=';
      }
      writer(output, 4);
    }
  }

 private:
  const char* data;
  size_t dataSize;
  static const char* base64_chars;
};
const char* Base64::base64_chars =
    "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/";


int main() {
  char output[1280];
  const char binaryData[] =
      "test big data in memory which cand be stored encoded as base64 again "
      "because we must save memory!!";
  size_t size =
      strlen(binaryData);  // only for example (i must know size of binary data)

  DynamicJsonDocument doc(1024);
  doc["sensor"] = "gps";
  doc["binary"] = Base64(binaryData, size);

  serializeJson(doc, output);

  printf("%s\n ", output);
  return 0;
}

and after compile and run i got

{"sensor":"gps","binary":"dGVzdCBiaWcgZGF0YSBpbiBtZW1vcnkgd2hpY2ggY2FuZCBiZSBzdG9yZWQgZW5jb2RlZCBhcyBiYXNlNjQgYWdhaW4gYmVjYXVzZSB3ZSBtdXN0IHNhdmUgbWVtb3J5ISE="}

so it works perfectelly.

Do you want patch ? It is usefull for others ?
it is in my fork ...

[bblanchon]

Hi @mirecta,

I expect to add a similar feature in the future but not right now.
Now is not the right time because it would require C++11 and virtual functions, two things that are forbidden by ArduinoJson 6 design constraints.

This feature should not be called "Incremental serializer" (#206, #210, #1690) and but "Lazy value serialization" (#1713).

BTW, using JSON to store an image is extremely inefficient.
You should consider alternative solutions.

Best regards,
Benoit

[mirecta]

I am not store image but i want send it through , i must send it into home assitant , before i was use message pack, but home assistant not support it