Command manager timeseries index model

[AlexRuiz7]

Description

In order to schedule the management of the commands by the Command Manager plugin, we need to add a @timestamp field to the index data model. This field is part of the ECS.

This field will be initialized by the plugin. In other words, the value of the field depicts the exact moment on which the command is indexed. We'll also include a field to store the result of delivery_timestamp = timestamp + timeout.

Functional requirements

• The commands index template has mappings for the @timestamp field.
• The commands index template has mappings for the delivery_timestamp field.
• The command manager initializes the commands with a timestamp generated during runtime.
• The command manager job scheduler uses the timestamp to sort the commands by time.

Implementation restrictions

• Regenerate the index template using our tools under wazuh-indexer/ecs.
• The dates follow the date_time_no_millis format. See Full date formats.

Plan

• Add @timestamp to the index data model (wazuh-indexer/ecs + generator)
• Update the index data model (wazuh-indexer/plugins/setup)
• Generate and add the timestamp to the command object before indexing.
• Sort commands by its delivery timeout wazuh-indexer-plugins#155

[QU3B1M]

We decided to add the @timestamp, and delivery_timestamp fields at the document "top level" (same level than the agent, or command groups), the pertinent modifications were made on both repositories, and now are under testing.

Wazuh-indexer updates

The ECS definitions of the command index were modified, creating a base.yml file which stores the delivery_timestamp field definition, and applying the corresponding modifications on the subset.yml file.
The event_generator was updated and the index template was successfully generated

Command index pattern

{
  "index_patterns": [
    ".commands*"
  ],
  "mappings": {
    "date_detection": false,
    "dynamic": "strict",
    "properties": {
      "@timestamp": {
        "type": "date"
      },
      "agent": {
        "properties": {
          "groups": {
            "ignore_above": 1024,
            "type": "keyword"
          }
        }
      },
      "command": {
        "properties": {
          "action": {
            "properties": {
              "args": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "name": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "version": {
                "ignore_above": 1024,
                "type": "keyword"
              }
            }
          },
          "order_id": {
            "ignore_above": 1024,
            "type": "keyword"
          },
          "request_id": {
            "ignore_above": 1024,
            "type": "keyword"
          },
          "result": {
            "properties": {
              "code": {
                "type": "short"
              },
              "data": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "message": {
                "ignore_above": 1024,
                "type": "keyword"
              }
            }
          },
          "source": {
            "ignore_above": 1024,
            "type": "keyword"
          },
          "status": {
            "ignore_above": 1024,
            "type": "keyword"
          },
          "target": {
            "properties": {
              "id": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "type": {
                "ignore_above": 1024,
                "type": "keyword"
              }
            }
          },
          "timeout": {
            "type": "short"
          },
          "user": {
            "ignore_above": 1024,
            "type": "keyword"
          }
        }
      },
      "delivery_timestamp": {
        "type": "date"
      }
    }
  },
  "order": 1,
  "settings": {
    "index": {
      "hidden": true,
      "number_of_replicas": "0",
      "number_of_shards": "1",
      "query.default_field": [
        "command.source",
        "command.target.type",
        "command.status",
        "command.action.name"
      ],
      "refresh_interval": "5s"
    }
  }
}

Wazuh-indexer-plugins updates

Added timestamp and deliveryTimestamp attributes to the command-manager Document model, the attributes are mapped to the @timestamp and delivery_timestamp fields corresponding to each case.

Updated command index template on both plugins' resources.

---

Command created with a POST request to the command-manager API

{
    "took": 32,
    "timed_out": false,
    "_shards": {
        "total": 1,
        "successful": 1,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": {
            "value": 1,
            "relation": "eq"
        },
        "max_score": 1.0,
        "hits": [
            {
                "_index": ".commands",
                "_id": "fZuAdJMBy-t6tcAPACYT",
                "_score": 1.0,
                "_source": {
                    "agent": {
                        "groups": [
                            "groups000"
                        ]
                    },
                    "command": {
                        "source": "Engine",
                        "user": "TestUser",
                        "target": {
                            "type": "agent",
                            "id": "TestTarget"
                        },
                        "action": {
                            "name": "restart",
                            "args": [
                                "/test/path/fake/args"
                            ],
                            "version": "v4"
                        },
                        "timeout": 30,
                        "status": "PENDING",
                        "order_id": "fJuAdJMBy-t6tcAPACYT",
                        "request_id": "e5uAdJMBy-t6tcAPACYS"
                    },
                    "@timestamp": 1732826365971,
                    "delivery_timestamp": 1732826395971
                }
            }
        ]
    }
}

[QU3B1M]

Updated timestamp and deliveryTimestamp to be of type ZonedDateTime and use the OpenSearch DateUtils library, the timestamp format applied is date_time_no_millis (YYYY-MM-DDTHH:mm:ssZ)