How to generate connector for World Bank API using Ballerina OpenAPI tool

[aneeshafedo]

The response payload structure of World Bank API is an heterogeneous array which includes two elements. First element is a record. second element is an array of another record type.

Ex:

[
    {
        "page": 1,
        "pages": 6,
        "per_page": 50 
        ...
    },
    [
        {
            "indicator": {
                "id": "SP.POP.TOTL",
                "value": "Population, total"
            },
            "country": {
                "id": "ZH",
                "value": "Africa Eastern and Southern"
            },
            "countryiso3code": "AFE",
            ...
        },
        {
            ...
        },
        ....
    ]
]

To map the response payload to a Ballerina type there are two approaches.

1. Array with oneOf schema as items

Suggested OpenAPI schema

CountryPopulationResponse:
      type: array
      items:
        oneOf:
          - type: array
            items:
              $ref: "#/components/schemas/CountryPopulation"
          - $ref: "#/components/schemas/PaginationData"
    PaginationData:
      type: object
      properties:
        page:
          type: integer
        pages:
          type: integer
        total:
          type: integer
        ...
    CountryPopulation:
      type: object
      description: "Represent country population."
      required:
        - "indicator"
        - "country"
        - "date"
        - "value"
      properties:
        indicator:
          "$ref": "#/components/schemas/Indicator"
          description: "World bank indicator"
        country:
          "$ref": "#/components/schemas/Country"
          description: "Country"
          ...

Suggested types generation

public type CountryPopulationResponse (CountryPopulation[]|PaginationData)[]; 

public type PaginationData record {
    int page;
    int pages;
    int per_page;
};

public type CountryPopulation record {
    # Data indicator
    Indicator indicator;
    # Represent a Country
    Country country;
    ...
};

Generating an array type like CountryPopulationResponse is not yet supported by the Ballerina OpenAPI tool.

Suggested remote function generation

    remote isolated function getPopulation(string? date = (), string format = "json", int page = 1, int perPage = 50) returns CountryPopulationResponse|error {
        string path = string `/country/all/indicator/SP.POP.TOTL`;
        map<anydata> queryParam = {"date": date, "format": format, "page": page, "per_page": perPage};
        path = path + check getPathForQueryParam(queryParam);
        CountryPopulationResponse response = check self.clientEp->get(path, targetType = CountryPopulationResponse);
        return response;
    }

However this approach returns an error

Error : Incompatible type for parameter 'targetType' with inferred typedesc value: expected 'typedesc<(ballerina/http:1.1.0-beta.2:Response|ballerina/http:1.1.0-beta.2:PayloadType)>', found 'typedesc<(aneesha/worldbank:0.1.0:CountryPopulation[]|aneesha/worldbank:0.1.0:MetaData)[]>

Sample code to access data

    CountryPopulationResponse result = check baseClient->getPopulation();
    CountryPopulation[] populationArr = <CountryPopulation[]> result[1];
    io:println(populationArr[0].country);

2. Tuple of two elements

Following the standard given in OpenAPI specification 3.1, schema of a tuple can be defined as below.

CountryPopulationResponse:
      type: array
      prefixItems:
        - $ref: "#/components/schemas/PaginationData"
        - type: array
          items:
            $ref: "#/components/schemas/CountryPopulation"
      minItems: 2
      maxItems: 2

Suggested types generation

public type CountryPopulationResponse [PaginationData, CountryPopulation[]]; 

Generating a tuple from OpenAPI schema is not yet supported by Ballerina OpenAPI tool (Not supporting OpenAPI Spec 3.1)

Suggested remote function generation

    remote isolated function getPopulation(string? date = (), string format = "json", int page = 1, int perPage = 50) returns CountryPopulationResponse|error {
        string path = string `/country/all/indicator/SP.POP.TOTL`;
        map<anydata> queryParam = {"date": date, "format": format, "page": page, "per_page": perPage};
        path = path + check getPathForQueryParam(queryParam);
        CountryPopulationResponse response = check self.clientEp->get(path, targetType = CountryPopulationResponse);
        return response;
    }

However this approach returns an error

ERROR [client.bal:(30:90,30:115)] incompatible types: expected 'typedesc<(ballerina/http:2.0.0:Response|ballerina/http:2.0.0:PayloadType)>', found 'typedesc<[aneesha/worldbank:0.1.0:PaginationData,aneesha/worldbank:0.1.0:CountryPopulation[]]>'

Sample code to access data

    CountryPopulationResponse [pagination, arr] = check baseClient->getPopulation();
    io:println(arr[0].country);

Conclusion

Approach 1

Advantages	Disadvantages
Easy to implement in the Ballerina OpenAPI tool	Type binding not supported by Ballerina

Approach 2

Advantages	Disadvantages
True interpretation of the data model	Type binding not supported by Ballerina
	Ballerina OpenAPI tool does not support OpenAPI spec 3.1

@indikasampath2000 @abeykoon @lnash94 Appreciate your inputs

[aneeshafedo]

Issue related to Ballerina is discussed here

[lnash94]

As discussed offline, It is nice to have a second approach because its OAS concept is directly mapping correctly to ballerina concepts as well.

But as we figured out, OAS uses oneOf type for the array to represent the mixed-array-type (refer) since the suggested first method is correct mapping for the array which has oneOf type. Because current OpenAPI tool implementation takes one type of the array instead of taking mixed types array. It needs to fix on the OpenAPI tool also as you mentioned.

Considering advantage and disadvantages of both approaches in the OpenAPI tool perspective, if we can provide support first to the first approach would be really helpful to other connectors with array type with oneOf data binding and creating a task for supporting the second approach once the OpenAPI parser is available with the new OpenAPI 3.1.validations. WDYT :) ?

[indikasampath2000]

As we identified during the offline discussion, approach one would be a bit complex to understand by the ordinary programmer. The main problem is to cast to the exact data type he wanted before accessing the fields.

In that aspect, the second approach seems more appealing, and we should support it when the parser is updated to swagger 3.1. We should stick to the current implementation until that. The only change to address is that the previously optional fields to required by looking at the API response.