ci: elasticsearch: Upload script index map examples

Add several examples for `upload_test_results_es.py` script usage
with ElasticSearch index map files for the following use cases:

 * Twister test results.

 * Twister test results with recordings.

 * Memory Footprint data (`twister-footprint.json`).

Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>

scripts/ci/es_upload/README.md

@@ -0,0 +1,156 @@
+# ElasticSearch index map examples
+
+This directory contains [ElasticSearch data store](https://github.com/elastic/elasticsearch)
+index map examples for use by [upload_test_results_es.py](../upload_test_results_es.py)
+script for [Twister](https://docs.zephyrproject.org/latest/develop/test/twister.html)
+JSON reports upload.
+
+An index map file allows to create destination index files on ElasticSearch server
+with explicit scheme having all required Twister JSON report objects associated
+to proper data types, eventually to store the expected document structure.
+Besides, it allows to track changes in Twister JSON report scheme
+and the corresponding data scheme in the same source code repository.
+An alternative way is to have index files created at the ElasticSearch server
+by other tools, or rely on default data mapping with potential type misalignments.
+
+The command examples below are simplified with parameters essential for data transformations.
+For other command line options and more details run the upload script with `--help`,
+or read its source code with inline comments there.
+
+Use these examples as a reference for your setup. Always pay attention to possible changes
+in the current Twister report format and in the upload script itself.
+Tune resulting data scheme and size depending on your particular needs.
+
+It is recommended to try the upload script with `--dry-run` option first
+to check the resulting data without its actual upload.
+
+You should set environment variables `ELASTICSEARCH_SERVER` and `ELASTICSEARCH_KEY`
+to connect with your server running the upload script.
+
+
+## Index creation with a map file
+
+Execute the upload script once for the first time to create your index at the ElasticSearch
+server with the appropriate map file, for example:
+```bash
+python3 ./scripts/ci/upload_test_results_es.py --create-index \
+  --index YOUR_INDEX_NAME \
+  --map-file YOUR_INDEX_MAP.json
+```
+
+
+## Upload transformations
+
+The upload script has several command line options to change `twister.json` data:
+exclude excess data, extract substrings by regular expressions, change data structure
+making it suitable for Kibana default visual components
+(see `--help` for more details):
+
+* `--exclude` removes excess testsuite properties not needed to store them
+  at Elasticsearch index.
+
+* `--transform` applies regexp group parsing rules to string properties extracting
+  derived object properties.
+
+* `--flatten` changes testsuite data structure in regard of one of its list components:
+  either `testcases` or `recording`, so each item there becomes an independent data record
+  inheriting all other testsuite properties, whereas the children object's properties
+  are renamed with the parent object's name as a prefix:
+  `testcases_` or `recording_` respectively.
+  Only one testsuite property object can be flattened this way per index upload.
+  All other testsuite objects will be treated according to the index structure.
+
+* Other command line options: `--flatten-dict-name`, `--flatten-list-names`,
+`--flatten-separator`, `--transpose-separator`, `--escape-separator`.
+
+
+## Index map examples and use cases
+
+
+### Twister test results
+
+Create the index:
+```bash
+python3 ./scripts/ci/upload_test_results_es.py --create-index \
+    --index zephyr-test-example \
+    --map-file zephyr_twister_index.json
+```
+
+Upload Twister test results 'as-is', without additional transformations:
+```bash
+python3 ./scripts/ci/upload_test_results_es.py \
+    --index zephyr-test-example \
+    ./twister-out/**/twister.json
+```
+
+
+### Twister tests with recording
+
+Store test results with `recording` data entries, for example from
+[Kernel Timer Behavior](https://github.com/zephyrproject-rtos/zephyr/tree/main/tests/kernel/timer/timer_behavior)
+test suite.
+
+Create the index:
+```bash
+python3 ./scripts/ci/upload_test_results_es.py --create-index \
+    --index zephyr-test-recording-example-1 \
+    --map-file zephyr_twister_flat_recording_metrics_index.json
+```
+
+Upload data with 'flattened' test suites creating documents for each `recording` data entry.
+```bash
+python3 ./scripts/ci/upload_test_results_es.py \
+    --index zephyr-test-recording-example-1 \
+    --exclude path run_id \
+    --flatten recording \
+    ./twister-out/**/twister.json
+```
+
+
+### Twister test with recording and extracting more data
+
+Store test results with `recording` data entries which are also scanned by regular expressions
+to extract embedded values, for example
+[Kernel Latency Benchmarks](https://github.com/zephyrproject-rtos/zephyr/tree/main/tests/benchmarks/latency_measure)
+test suite.
+
+Create the index:
+```bash
+python3 ./scripts/ci/upload_test_results_es.py --create-index \
+    --index zephyr-test-recording-example-2 \
+    --map-file zephyr_twister_flat_recording_index.json
+```
+
+Upload data with 'flattened' test suites creating documents for each `record` data entry
+with 3 additional data properties extracted by regular expressions.
+```bash
+python3 ./scripts/ci/upload_test_results_es.py \
+    --index zephyr-test-recording-example-2 \
+    --exclude path run_id \
+    --flatten recording \
+    --transform "{ 'recording_metric': '(?P<recording_metric_object>[^\.]+)\.(?P<recording_metric_action>[^\.]+)\.(?P<recording_metric_details>[^ -]+)' }" \
+    ./twister-out/**/twister.json
+```
+
+
+### Memory Footprint results
+
+To store Memory Footprint data reported in `twister-footprint.json` (see Twister `--footprint-report`).
+
+Create the index:
+```bash
+python3 ./scripts/ci/upload_test_results_es.py --create-index \
+    --index zephyr-memory-footprint-example \
+    --map-file zephyr_twister_flat_footprint_index.json
+```
+
+Upload data with 'flattened' footprint entries creating documents for each of them.
+```bash
+python3 ./scripts/ci/upload_test_results_es.py  \
+    --index zephyr-memory-footprint-example \
+    --exclude path run_id \
+    --flatten footprint \
+    --flatten-list-names "{'children':'name'}" \
+    --transform "{ 'footprint_name': '^(?P<footprint_area>([^\/]+\/){0,2})(?P<footprint_path>([^\/]*\/)*)(?P<footprint_symbol>[^\/]*)$' }" \
+    ../footprint_data/**/twister_footprint.json
+```

scripts/ci/es_upload/zephyr_twister_flat_footprint_index.json

@@ -0,0 +1,158 @@
+{
+    "properties": {
+        "environment": {
+            "properties": {
+                "commit_date": {
+                    "type": "date"
+                },
+                "os": {
+                    "type": "keyword",
+                    "ignore_above": 256
+                },
+                "run_branch": {
+                    "type": "keyword",
+                    "ignore_above": 256
+                },
+                "run_workflow": {
+                    "type": "keyword",
+                    "ignore_above": 256
+                },
+                "run_id": {
+                    "type": "keyword",
+                    "ignore_above": 256
+                },
+                "run_attempt": {
+                    "type": "integer"
+                },
+                "run_date": {
+                    "type": "date"
+                },
+                "toolchain": {
+                    "type": "keyword",
+                    "ignore_above": 256
+                },
+                "zephyr_version": {
+                    "type": "keyword",
+                    "ignore_above": 256
+                },
+                "options": {
+                    "type": "object",
+                    "enabled": true,
+                    "dynamic": true,
+                    "subobjects": true
+                }
+            }
+        },
+        "name": {
+            "type": "keyword",
+            "ignore_above": 256
+        },
+        "component": {
+            "type": "keyword",
+            "ignore_above": 256
+        },
+        "sub_component": {
+            "type": "keyword",
+            "ignore_above": 256
+        },
+        "arch": {
+            "type": "keyword",
+            "ignore_above": 256
+        },
+        "platform": {
+            "type": "keyword",
+            "ignore_above": 256
+        },
+        "used_ram": {
+            "type": "unsigned_long"
+        },
+        "used_rom": {
+            "type": "unsigned_long"
+        },
+        "available_ram": {
+            "type": "unsigned_long"
+        },
+        "available_rom": {
+            "type": "unsigned_long"
+        },
+        "retries": {
+            "type": "integer"
+        },
+        "status": {
+            "type": "keyword",
+            "ignore_above": 256
+        },
+        "build_time": {
+            "type": "float"
+        },
+        "reason": {
+            "type": "text",
+            "fields": {
+                "keyword": {
+                    "type": "keyword",
+                    "ignore_above": 256
+                }
+           }
+        },
+        "log": {
+            "type": "text"
+        },
+        "footprint_name": {
+            "type": "text",
+            "fields": {
+                "keyword": {
+                    "type": "keyword",
+                    "ignore_above": 256
+                }
+            }
+        },
+        "footprint_name_depth": {
+            "type": "integer"
+        },
+        "footprint_identifier": {
+            "type": "text",
+            "fields": {
+                "keyword": {
+                    "type": "keyword",
+                    "ignore_above": 256
+                }
+            }
+        },
+        "footprint_area": {
+            "type": "text",
+            "fields": {
+                "keyword": {
+                    "type": "keyword",
+                    "ignore_above": 256
+                }
+            }
+        },
+        "footprint_path": {
+            "type": "text",
+            "fields": {
+                "keyword": {
+                    "type": "keyword",
+                    "ignore_above": 256
+                }
+            }
+        },
+        "footprint_symbol": {
+            "type": "text",
+            "fields": {
+                "keyword": {
+                    "type": "keyword",
+                    "ignore_above": 256
+                }
+            }
+        },
+        "footprint_address": {
+            "type": "unsigned_long"
+        },
+        "footprint_size": {
+            "type": "long"
+        },
+        "footprint_total_size": {
+            "type": "unsigned_long"
+        }
+    }
+}