Will this project support int128_t and uint128_t ?

[gzliudan]

When will this project support int128_t and uint128_t ?

[nlohmann]

Thanks for the question!

Via template arguments, it is possible to set the types of the integers to 128 bit types:

using json128 = nlohmann::basic_json<std::map, std::vector, std::string, bool, __int128_t, __uint128_t>;

However, some parts of the library make silent assumptions on the integer size. For instance, the lexer is calling std::strtoull to convert from a string to an integer. When called on a very large number like 18446744073709551619, std::strtoull sets errno to ERANGE.

This indicates an overflow according to the documentation:

The strtoul() function returns either the result of the conversion or, if there was a leading minus sign, the negation of the result of the conversion represented as an unsigned value, unless the original (nonnegated) value would overflow; in the latter case, strtoul() returns ULONG_MAX and sets errno to ERANGE. Precisely the same holds for strtoull() (with ULLONG_MAX instead of ULONG_MAX).

I am not sure whether this is the only spot with such an assumption. If it is, then 128-bit integer support would "just" mean providing some function that can parse a 128-bit integer from a string.

[nlohmann]

Update:

I tried a naive implementation from https://stackoverflow.com/a/45619917 to replace std::strtoull:

__uint128_t atouint128_t(const char *s)
{
    const char *p = s;
    __uint128_t val = 0;

    while (*p >= '0' && *p <= '9') {
        val = (10 * val) + (*p - '0');
        p++;
    }
    return val;
}

With that, I could successfully roundtrip 18446744073709551619 to JSON and back to a __uint128_t.

[nlohmann]

I started to implement 128-bit integer support: https://github.com/nlohmann/json/tree/issue2228

[gzliudan]

Wonderful !!! Hope to use this feature in next release. By the way, how about __float128 support?

[nlohmann]

I think this is a bit more tricky, because we would code for

1. string to __float128
2. __float128 to string

[nlohmann]

Update: This seems to be possible with libquadmath, but it is only available for GCC. I am not aware of portable versions that work with Clang or MSVC.

[gzliudan]

clang has a data type named long double which is 16 bytes float number, and function snprintf supports this data type use length modifier L:
https://en.cppreference.com/w/cpp/io/c/fprintf

GCC and MSVC also support long doube, but seems is 12 bytes and 80 bit.

[gzliudan]

Will you create a commit to support int128_t and uint128_t? If yes, how about time?

[nlohmann]

I currently have no timeline for the next release.

Note I work on this project in my free time. Feel free to sponsor my work.

[gzliudan]

Your professional and in time support is worthy of praise . Do you have alipay or wechat account?

[nlohmann]

I'm afraid not, but Paypal and GitHub Sponsors.

[gzliudan]

I mailed 12EUR to your paypal account just now through a friend, since I have no paypal account and no credit card.

[nlohmann]

Thanks a lot!!! 🙏

[mjcrawford22]

Is there still work for the 128 bit integer? Really we need a uint64, but a int128 would cover it since there is some work already.

[gregmarr]

@mjcrawford22 unsigned values are already uint64_t by default.

[mjcrawford22]

I figured out where our anomaly lies. The JSON Schema Validator we use, and a very good one at that, uses your number_integer_t type for their integer validation, which the type signed 64 bit. The JSON specification does not specific number of digits and with/without leading sign.

[pboettch]

It's currently not in the switch, but if I put it there upon your suggestion it would never be hit.

This switch is used at schema-parsing-time; at that moment we only know that the schema requires an integer, due to lack of support for unsigned in json-schema, we can only use number_integer here.

[gregmarr]

It's currently not in the switch, but if I put it there upon your suggestion it would never be hit.

It is in the switch, it falls through to the number_integer case. I just added the make_shared...unsigned line.

This switch is used at schema-parsing-time; at that moment we only know that the schema requires an integer, due to lack of support for unsigned in json-schema, we can only use number_integer here.

That's unfortunate.

So the issue happens at https://github.com/pboettch/json-schema-validator/blob/08d8a52a8a1353b6bc953b63f7c6763e74e6047b/src/json-validator.cpp#L876-L878 when the number_unsigned_t value is extracted as number_integer_t before the comparisons?

If so, brainstorming here, could you rename numeric to floating and add another version as integer that takes both number_integer_t and number_unsigned_t as template parameters, such as TS and TU? You'd likely need both TS and TU versions of maximum_ and minimum_. Then validate could do a runtime check using instance.is_number_unsigned() and select either a TS or TU version of the function. It would mean a fair amount of code duplication, but it would get you the bigger range.

[pboettch]

It's currently not in the switch, but if I put it there upon your suggestion it would never be hit.

It is in the switch, it falls through to the number_integer case. I just added the make_shared...unsigned line.

Sorry, can't read anymore.

So the issue happens at https://github.com/pboettch/json-schema-validator/blob/08d8a52a8a1353b6bc953b63f7c6763e74e6047b/src/json-validator.cpp#L876-L878 when the number_unsigned_t value is extracted as number_integer_t before the comparisons?

Yes.

If so, brainstorming here, could you rename numeric to floating and add another version as integer that takes both number_integer_t and number_unsigned_t as template parameters, such as TS and TU? You'd likely need both TS and TU versions of maximum_ and minimum_. Then validate could do a runtime check using instance.is_number_unsigned() and select either a TS or TU version of the function. It would mean a fair amount of code duplication, but it would get you the bigger range.

Good idea, OP had the same idea, the issue is not only the type, but rather the conditional-stuff related to a json-schema-type: mainly minimum/maximum range-checks: Even if the instance has been parsed as unsigned by json, I have to compare it to a minimum or a maximum which might be negative (field in json-schema). So I have to get a signed int.

We could have the idea to say, if min and max are positive, let's use unsigned. But then we also have to care about the case for the instance is actually negative and min and max are positive. Then there is the multipleOf-check.

I'm hesitant to write this code just to win 1-bit precision on a 64-bit integer for when what OP actually wants is 128 or more bits to store his data.

[gregmarr]

Good idea, OP had the same idea, the issue is not only the type, but rather the conditional-stuff related to a json-schema-type: mainly minimum/maximum range-checks: Even if the instance has been parsed as unsigned by json, I have to compare it to a minimum or a maximum which might be negative (field in json-schema). So I have to get a signed int.

Ah, yes, you would need to store flags for the minimum and maximum so that you know whether they were signed or unsigned in the schema document, and do the comparisons appropriately.

We could have the idea to say, if min and max are positive, let's use unsigned. But then we also have to care about the case for the instance is actually negative and min and max are positive. Then there is the multipleOf-check.

The multipleOf is already float, so it already has a different range, so you can probably leave that one as-is, at least for now.

I'm hesitant to write this code just to win 1-bit precision on a 64-bit integer for when what OP actually wants is 128 or more bits to store his data.

Is there still work for the 128 bit integer? Really we need a uint64, but a int128 would cover it since there is some work already.

Looks like @mjcrawford22 thought that 128 bit would be the easier way to solve it, but 64 bit unsigned would work too. @mjcrawford22 can you confirm that you need larger than the max value of a 64 bit signed, but less than the max value of 64 bit unsigned? (If that's still too much work, then that's fine, just trying to help figure out if it's even feasible.)

[mjcrawford22]

Yes, We need the 64 bit unsigned as a part of validation for our specific application. The 64 bit signed is not adequate for this application.

As a plug for 128 bit support, most if not all processors, including GPUs, and compilers, now support 128 bit integers, but I took a quick grep 64, and I understand it is all over the place and clearly not trivial.