low.js and node-matter?

[javierbarellano4v]

Would node-matter work with low.js?

[Apollon77]

In fact my answer is a clear "it could" ;-)

node-matter is 95% "pure JS" and uses some, already abstracted away, Node.js relevant APIs. These are: Networking/UDP, Date/Timer, Crypto, (and soon) FS ... If low.js is Node.js conform in these places then it is a yes.

When I look at https://www.neonious-iot.com/lowjs/documentation/nodejs-api.html then Crypto might get interesting.

The good news is that because of the existing abstractions, even if Crypto is not directly compatible, it can be manually implemented and plugged in (soon, because brypto needs to get abstracted - on todo list) for the special case withut any issue.

At the bottom line:

• seems to work, but you need to implement the missing crypto methods there or find a "pure JS" alternative

[javierbarellano4v]

@Apollon77 Appreciate the detailed explanation! It's "future use" appears promising.

[jdesai61]

Very timely post. I am actually tring to make it work with lowjs - one issue I am running into is BigInt. I am trying to replace it with JSBI.BigInt and see what the next roadblock is (besides crypto). Basic issue is that lowjs supports ES 5. Where as node-matter uses ES 6. Lowjs (or babel) can "transpile" from ES 6 to ES 5, but certain features are not supported (BigInt is the first one I ran into).

[Apollon77]

what happens if you change the typescript generator target to es5? never tried this.

Bit the low.js webpage states "Node.JS API, JavaScript ES 6" ??

Onn the other side ... last change of that project is 2 years ago ... is that really a meaningful goal one?

[jdesai61]

what happens if you change the typescript generator target to es5? never tried this.

I tried and it leaves BigInt usage as is. BTW I did have to add "downlevelIteration": true to to tsconfig.json for ES5.

[jdesai61]

I tried using JSBI.BigInt in node-matter code, but it is a bit more than I can chew because @project-chip/matter.js project is using bigint as well and BigInt there needs to be replaced there as well.

[JimBuzbee]

Note that Bellard's MIT licensed, embeddable QuickJS supports BigInt. It hasn't had an update for a couple of years either, but the developer's mailing list appears to be active this year. Says to have almost complete ES2020 support

https://bellard.org/quickjs/

[jdesai61]

Thanks - I want to run this on ESP32 class platform and lowjs seems to be the logcal choice now. As QuickJS currently provides MAC/Linux binaries and would have to be compiled/verified on these small systems.

[JimBuzbee]

Looks like at least a couple of people have ported Quckjs to an ESP32 platform. I don't know anything about them, but here's one of the two found on github:

https://github.com/binzume/esp32quickjs

[jdesai61]

@JimBuzbee Thanks - I will try it out.

[Apollon77]

Please ceate an issue in matter.js and we take care, but that might need some time

[mfucci]

low.js support: as stated above the two main bottlenecks will be for BigInt and Crypto support.

BigInt considerations

Matter protocol uses unsigned and signed 64-bit integer (up to 2^64 – 1).
This doesn't fit into a js Number, since they are represented under by a double-precision floating-point format, which includes the exponent, so they can only represent accurately unsigned integers up to 2^53 – 1.
So I decided to represent those values with a BigInt since this is a native ES6 object that will work to represent them.

However:

1. as you can see, just a few bits are missing to represent a 64-bit integer so two Numbers will work
2. I have only seen unsigned 64-bit integers used in Matter specs, not signed 64-bit integers and they were always used for IDs, no mathematical operations were performed on them => they means that they can be represented by a string instead or array of bytes

So maybe there is no need for BigInt support after all in Matter in practice...
@jdesai61: do you want to dig through the specs to see actual use cases for signed / unsigned 64-bit integer to hold values rather than IDs?
Using string instead of BigInt for IDs is a simple change in matter.js...

Crypto considerations

node-matter uses either built-in node.js crypto or crypto based on elliptic library for missing features
elliptic library is not using BigInt internally, it uses bn.js (maybe to consider as a default implementation for BigInt?)
For adding web support, it might be good to add a pure js implementation of all the crypto and to override those with native implementation for specific targets (node.js / browsers / low.js).
I would recommend this approach:

1. rewrite a pure js implementation of all the crypto using a few open source libraries: it will run against the same crypto unit tests already written for the node.js crypto
2. override the crypto primitives with the native implementation in low.js (I think only AES if I remember well)

[jdesai61]

I haven't yet checked into the specs, but I did a quick dirty test to modify matter.js and node-matter to use JSBI instead of BigInt. There doesn't seem to be any math, but comparison operator between BigInt and number had to be addressed and JSBI refused to auto convert to Number, so there are more changes than I had hoped. But the changes seem to work and I can run modified code with nodejs and pair with Home Assistant.

I am not having much luck running node-matter code with lowjs or quickjs (sans crypto) because they don't support CommonJS and I am not very familiar with different ways of packaging modules (I have tried webpack - but still struggling).

Files modified in matter.js :

        package.json
        packages/matter.js/src/cluster/GeneralCommissioningCluster.ts
        packages/matter.js/src/common/FabricId.ts
        packages/matter.js/src/common/NodeId.ts
        packages/matter.js/src/tlv/TlvCodec.ts
        packages/matter.js/src/tlv/TlvNumber.ts
        packages/matter.js/src/util/DataReader.ts
        packages/matter.js/src/util/DataWriter.ts
        
[matter.js-bigint-jsbi-changes.txt](https://github.com/mfucci/node-matter/files/10994829/matter.js-bigint-jsbi-changes.txt)
[node-matter-bigint-jsbi-changes.txt](https://github.com/mfucci/node-matter/files/10994830/node-matter-bigint-jsbi-changes.txt)
packages/matter.js/src/util/Number.ts

Files modified in node-matter.js:

        package.json
        src/Device.ts
        src/matter/MatterController.ts
        src/matter/certificate/CertificateManager.ts
        src/matter/cluster/GeneralCommissioningCluster.ts
        src/matter/cluster/server/GeneralCommissioningServer.ts
        src/matter/common/FabricId.ts
        src/matter/common/NodeId.ts
        src/matter/session/SessionManager.ts

See attached for exact diffs (it's quick and dirty hack - so bear with me)
node-matter-bigint-jsbi-changes.txt
matter.js-bigint-jsbi-changes.txt

[mfucci]

BigInt: I would suggest to replace all references to BigInt with a Integer64 interface and have a BigNumber implementation for it (like it has been done with Network or Time). After this is merge, it will be easy to add other implementations.

CommonJS: matter.js supports CommonJs and ES modules. The module system is used to resolve js imports and this is completely unrelated to webpack which is a packing system for web applications.

[jdesai61]

I was trying to use webpack to package all code in one file - and hence import from file(s) is taken care of by webpack (using babel or such).

[Apollon77]

I will put the Integer64 interface on my list

[jdesai61]

I understand that I don't need to make any BigInt changes for running with quickjs, but I can't figure out how to modify tsconfig.json to remove CommonJS because it is provided by nodejs but not available in quickjs

[Apollon77]

Maybe the chances are better once we are done with the node-matter->matter.js merges because then the "pure JS" parts will be all matter.js with different build targets already and in fact you just need to adjust the niodejs specific oarts ... but we are not there yet

[JimBuzbee]

This stuff is all new to me, but it looks like someone wrote a barebones(?) version of CommonJS for quicks?

https://github.com/IvanGaravito/quickjs-require

[jdesai61]

@JimBuzbee thanks - tried that but that is a JavaScript and I can't figure out how to convert/adapt it to TypeScript.

[jdesai61]

It seems like I can get around to not having require() by using webpack - but using webpack I ran into following issues:

crypto
net
os
dgram
child_process

I understand that lowjs will provide some of these APIs, with varying levels of support - however, it only supports ES5 (despite document page mentioning ES6)

QuickJS does support ES2020, ES6 among others. However, it seems to support only "os" and "std".
Will continue to dig at both options. There are other ports of QuickJS with extensive API support, but they don't run on ESP32 platforms.