Fields randomly reordered when updating Elasticsearch templates

[DaveCTurner]

As per elastic/elasticsearch#74501 (comment) Elasticsearch considers the order of fields in a mapping defined in an index template to be important. This means that if a Beat puts a template that differs from the existing template only by reordering the JSON then Elasticsearch will treat it as a genuine template update and not a no-op. However we believe that Beats don't fix the order of fields when updating the template, and different instances may put templates with different field orders. This means Beats may cause many unnecessary cluster state updates when creating templates.

#24332 likely means that this isn't a big deal in most cases any more, but it's still a problem if Beats are configured to overwrite templates at startup.

(In fact Elasticsearch treats the order of properties in every JSON object within the mapping to be important. It's not just about the fields)

Please could we fix the order of JSON object properties when creating templates? If you don't care about the order then sorting them alphabetically would be fine.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[botelastic]

Hi!
We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!

[DaveCTurner]

👍 still relevant AFAIK

[cmacknz]

This means Beats may cause many unnecessary cluster state updates when creating templates.

#24332 likely means that this isn't a big deal in most cases any more, but it's still a problem if Beats are configured to overwrite templates at startup.

Can you help us understand the impact in more detail so we know how to prioritize this? Is this a nice to have performance optimization? Or is this actively causing incidents?

[DaveCTurner]

We haven't had any escalations about this for a while (a year or so I reckon) so this seems more of a nice-to-have at this stage.

[botelastic]

Hi!
We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!