CLI changes to follow best SW engineering practices and be backward compatible

[dmfiore]

Hi,

I don't know if this topic has been raised. I was wondering what the current OT policy is on backward compatibility.
I did a couple of searches on 'backward compatibility' and 'CLI changes' but didn't find anything.

As a SQA engineer, I think it is important that changes to OT do not break what exists.
For CLI syntax changes, this can cause a lot of disruption for many engineers developing test automation over the evolution of this protocol.

I am new to this forum, but not the protocol. @jwhui is there a best practices policy in this respect for OT and backward compatibility?
Thank you,
-D

[jwhui]

@dmfiore , thanks for raising this question.

OpenThread does not currently have any policy for the CLI. Historically, the CLI was primarily used as a simple mechanism for quick testing during software and development. Given that the CLI operates on resource-constrained devices, we wanted to minimize resource requirements for the CLI. At the same time, given that Thread is a relatively new technology, we did not want to constrain ourselves to design decisions that were made in the past as we obtain new learnings and insights moving forward.

That said, we could be more diligent in maintaining a CHANGES list for the CLI. Similar to what we do with the OpenThread API. Would that help?

[jwhui]

A point I was trying to push is yes, a flag became invalid. Can that be handled gracefully vs hard fault ? For something that once was valid. While it is CLI, it is also attached to the API. I do not know if that translates to an invalid value over the air. But it could possibly happen under these conditions of validation.

I believe the CLI returns an error to indicate that the b flag is no longer valid and does not generate a hard fault. In this particular case, I believe returning an error is the proper behavior since users of the b flag should be aware that this functionality no longer exists in the Thread standard.

Ideally, any test infrastructure leveraging the CLI should check for and report CLI errors. If that is the case, I believe this issue could have been addressed with relatively little effort. This is how we implement all of the OpenThread continuous integration test scripts.

[dmfiore]

Agreed on the test infrastructure, our teams continually move in that direction upon changes of either env, cli, or other, yes.

There's one thing here running around my head...
What happens when a version 1.1 in the field talks to a version 3.1 in the field for which multiple API changes have been made over time?
How is interoperability discussed and handled for these cases ? Do we test any version interoperability with OT?
I appreciate the discussions, and we can take offline, though I think this is a good thread to discuss intent and some philosophy?
-D

[jwhui]

Thread Protocol interoperability is very different from API and CLI backward compatibility. With the Thread Protocol, we need to make sure that devices can continue to communicate over the air. The Thread Group's Certification program tests that a given device is backward compatible with all prior Thread protocol versions. The OpenThread project also includes continuous integration tests for backwards compatibility.

[simonlingoogle]

Agree with @jwhui and @abtink on the general principles for dealing with CLI compatibilities.

I had several issues regarding CLI compatibility, too.
These are the general strategies I have been using to address CLI compatibility issues:

• Check the API version and choose different commands accordingly.
• Check if a command exists before running it. (e.g. masterkey)

Overall, I think we are already trying to make the CLI commands and output stable. And it's not difficult to handle when it's not.

[dmfiore]

Thanks for all the feedback! Good info to absorb.
-D