Best way to handle packets of TM of varying rates

[gmarchetx]

In our application, we need to resort to packets of fixed structure to facilitate interfacing with YAMCS, while the TM channels are emitted at wildly different rates and the datarate needs to be minimised.

One way to do this is to group TM by expected rates and send them to YAMCS using a fixed format, one YAMCS port per group. TlmPacketizer seems to be a good candidate for this, however, either

• we need some kind of router at the packet level to split the packet stream into different paths based on the packet IDs; we could make it if it does not exist but either way I think all packets are emitted at the same time, which would defeat the purpose of grouping by rate.
  or
• we use one TlmPacketizer per group of TM; however, it seems like TlmPacketizer is configured by the autocoder (via its setPacketList() method) from {Deployment}Packets.xml in the Top folder of the application, which seems to imply from its structure and name that only one TlmPacketizer is expected.

I would very much like to leverage {Deployment}Packets.xml because it would facilitate generating the parsing pattern for YAMCS using fprime-gds's XML dictionaries loader (into packet template objects).

What would be the best way to go about this?

[timcanham]

So to clarify, you would want something like:

1. Packet 1 @ 1Hz
2. Packet 2 @ 10Hz
3. Packet 3 @ 0.5Hz?

[timcanham]

What is TM?

[timcanham]

TlmPacketizer works by checking each cycle to see if any of the channels in the packet have been updated, and if so, it sends the packet with the time stamp set to the moment it was sent. That means:

1. Channels that have never been written will have a value of zero
2. Channels that were written one or twice in the past will appear to have a time stamp of the time of the generation of the packet, even though they may not have been written in a while.

This is a tradeoff between TlmChan, which will have accurate timestamps for each channel but the generation of the packets containing the channels is not as deterministic as TlmPacketizer.

[gmarchetx]

Yes, I confirm this is my current understanding of the TlmPacketizer (but reminders are always appreciated, especially for the Discussion in itself). So far we have the choice between:

• using TlmChan but writing a piece of ground code that essentially does what TlmPacketizer does on the ground side (with the difference that we would expose TLM_NAME and TLM_NAME_TIME instead of TLM_NAME alone and ignore YAMCS's packet reception time) to send it to YAMCS. Feels like duplicated efforts, but above all the built-in time series will not make sense since they are based on packet reception time.
• using TlmPacketizer with the issues highlighted in the question

The tradeoff is still open, but currently I am interested in TlmPacketizer because that is where the most uncertainty lies. Depending on what the anticipated effort turns out to be, it may be more interesting to go with the first option.

P.S: Apologies, I use TM for what Fprime calls telemetry channels.

[zimri-leisher]

Don't mean to butt in, but if I understand your problem right, you might be able to solve this problem by using a single TlmPacketizer.

1. Make a rate group for each telemetry rate (1 Hz, 10 Hz, 0.5 Hz, etc)
2. Have the rate group call a Sched port on each component which has telemetry at the given rate, and have that Sched port only call tlmWrite_XYZ for each channel at that rate. For example, see BufferManager.SchedIn.
3. Have TlmPacketizer.Run called at a frequency which is the lowest common denominator of the telemetry rates (in this case, 10 Hz)
4. Have TlmPacketizer::PACKET_UPDATE_MODE set to PACKET_UPDATE_ON_CHANGE

This should allow you to have several different rates of telemetry coming from the same TlmPacketizer.

[timcanham]

To add to what @zimri-leisher says, you can compose the packets so that channels that update at a particular rate are all in one packet so it gets sent at the update rate. Another thing to note about packets is that you can define multiple sets of packet XML files and feed the output C++ to individual TlmPacketizer instances. You could tie them to different rate groups. Each XML file is supposed to completely specify a place for all defined channels, but ones you aren't using for a given TlmPacketizer instance can be placed in the <ignore> section.

[gmarchetx]

Thanks for your answers.

I looked more deeply and it does seem like one TlmPacketizer could do the trick, with packets grouped by expected refresh rate. Correct me if I am wrong, but I do not think there is a F' or GDS component that is able to split packets based on their packet ID, so we would need to make it. Then, defining the packets breakdown would need to be maintained for every addition or modification to the telemetry set, because automating this process would not be worth the high effort. Finally, I have the intuition we will have to pay close attention to the clock arrangement to be able to send telemetry without a delay of 1 sample.

With this in mind, we went back to the datarate budget and it seems we will be able to use TlmChan and make the other end instead. The datarate penalty is large (from 2x to as high as 14x depending on the average telemetry channel serialised length, 6x in our case) but we should be able to accommodate it. This assumes we change type sizes for channel IDs to 16 bits, and packet descriptors to 8 bits*. If the TlmChan is clocked to the fastest telemetry channel, and if we forward the packets as soon as they are received using memory values for channels not present in the packet, we should be able to have reasonable time series in YAMCS with just staircase interpolations. Each channel will be presented as a value and a generation timestamp in the YAMCS database, for more accurate post processing. Let me know if this does not sound like a good plan, as we intend to share the YAMCS interface when it is done.

I will mark @zimri-leisher's post as answer as I am not really answering the question. Thanks @timcanham for the clarifications.

*: I wonder why the latter was 32bits, it seems to be a small enum. Similarly, I wondered why there were only 1000 slots for TM in the TlmPacketizer when it says it should be higher than the number of telemetry channels, of which the ID type is by default 32 bits.

[timcanham]

Note that we are planning to tie things like channel ID/descriptors to FPP variables so they can be shared with the F Prime GDS, but that hasn't happened yet. If you change the sizes, you won't be able to use the F Prime GDS until those updates are made. Yes, the packet descriptor is 32 bits because that's what we picked early for room to grow, but it could be smaller.

[gmarchetx]

Thanks for the info. Changing those sizes would be made after initial development when the YAMCS interface becomes our primary interface so in our case this should not be a problem - after all we were considering TlmPacketizer which, if I'm not mistaken, is not compatible with GDS as it is.