Proposal: Proto file generation for Protocol Buffers SerDes

[MohamedSabthar]

Summary

Ballerina SerDes module allows serializing/deserializing Ballerina anydata type to send them efficiently over the wire. The Proto3Schema class of serdes module uses the protocol buffer as the underlying technology to perform serialization and deserialization of ballerina anydata types. It does so by dynamically generating a proto3 message definition for the given ballerina type. This proposal proposes an API to generate a proto3 file from the dynamically generated proto3 message definition.

Goals

• Provide an API to generate proto3 files from the dynamically generated message definition
• Enable a language-neutral communication mechanism between ballerina and other languages

Motivation

A proto3 file generated using the proposed API can be used by network clients/ software written with other languages to generate native language bindings. These generated native language bindings can be used to send/receive messages to/from the ballerina side. Therefore, the proposed API provides a way to enable a language-neutral communication mechanism between ballerina and other languages. For example, if software written in java wants to send messages which need to be converted to a ballerina anydata subtype then the following steps can be done

1. Define a type in ballerina side
2. Generate the proto3 file using the proposed API
3. Use the generated porto3 file and generate java code using the protobuf compiler
4. Use the generated java code and prepare the proto messages
5. Serialize the message on the Java side and send the serialized bytes to the ballerina side
6. On the Ballerina side, deserialize the received bytes using the Proto3Schema class and the type defined in step 1. Deserialization results in a value having the same type as step 1.

Description

The enhancement is to introduce an API to generate proto3 files so that the generated proto3 file can be used by clients/software written with other languages to receive/send messages from/to ballerina clients.

API

Approach 1

public class Proto3Schema {
    *Schema;

    // init()
    // serialize()
    // deserialize()

    public isolated function generateProtoFile(string filePath) returns Error? = @java:Method {
        'class: "io.ballerina.stdlib.serdes"
    } external;
}

example:

import ballerina/serdes;

type Module record {
    string name;
    string|int id;
    int? stars;
    Contributor[] contributors;
};

type Contributor record {
    string username;
    byte[] img;
};

function main() returns error? {
    serdes:Proto3Schema serdes = check new (Module);
    check serdes.generateProtoFile("Module.proto");
}

The generateProtoFile function takes the file path and writes the file in the specified path or returns an Error on failure. On success the above example should produce a proto3 file with the content provided below.

syntax = "proto3";

message Module {
  message stars___UnionBuilder {
     sint64 int___unionField  = 1;
     bool nullField  = 2;
  }

  message Contributor {
     bytes img  = 1;
     string username  = 2;
  }

  message id___UnionBuilder {
     sint64 int___unionField  = 1;
     string string___unionField  = 2;
  }

   repeated Contributor contributors  = 1;
   id___UnionBuilder id  = 2;
   string name  = 3;
   stars___UnionBuilder stars  = 4;
}

For more information about the ballerina type to protobuf message mapping refer serdes spec.

Approach 2

public class Proto3Schema {
    *Schema;

    public isolated function init(typedesc<anydata> ballerinaDataType, string? filePath = ()) returns Error? {
        check generateSchema(self, ballerinaDataType);
        if filePath is string {
            check generateProtoFile(filePath);
        }
    }

    // serialize()
    // deserialize()

    private isolated function generateProtoFile(string filePath) returns Error? = @java:Method {
        'class: "io.ballerina.stdlib.serdes"
    } external;
}

example:

import ballerina/serdes;

type Module record {
    string name;
    string|int id;
    int? stars;
    Contributor[] contributors;
};

type Contributor record {
    string username;
    byte[] img;
};

function main() returns error? {
    serdes:Proto3Schema serdes = check new (Module, "Module.proto");
}

The init function takes an optional filePath argument and writes the file in the specified path or returns an Error on failure.

Approach Comparison

Approach 1
pros

• The function name clearly explains the intention and only does one thing which is generating a proto file
• Does not break the current API provided by Proto3Schema class

cons

• Introduces a new function to the Proto3Schema class

Approach 2
Someone can feel that generating a proto file is part of the initialization of the schema.
In that case, maybe passing an optional argument to the init function is a better way to generate the proto file.

pros

• Does not break the current API provided by Proto3Schema class

cons

• The init function does more than one thing. It generates a schema and generates the proto file
• The optional argument filePath may confuse the user

[MohamedSabthar]

We are closing this issue because the serdes module has been deprecated.