Restructure a log message into custom dimensions

[omers]

Dear community,

I have the following log message:

{
  "data_source_name": "TEST",
  "data_source_type": "shipper",
  "filename": "parser.py",
  "host": "DEVMACHINE",
  "hostname": "DEVMACHINE",
  "is_dry_run": false,
  "is_enabled": true,
  "lineno": 101,
  "log_id": "b91f4a11-e9ec-4f97-96fc-96e611d94b07",
  "module": "parser",
  "name": "parse_service",
  "node_id": "main",
  "pid": 15488,
  "processName": "MainProcess",
  "service_name": "MASTER",
  "shipper": "vector",
  "location": "us-east-2",
  "completed_at": "2021-09-30T14:00:00Z",
  "timestamp": "2021-09-30T14:00:00Z"
}

I want to reserve some of the json keys, and put all other keys in a "customDimenssions" key, for example:

{
  "processName": "MainProcess",
  "service_name": "MASTER",
  "shipper": "vector",
  "location": "us-east-2",
  "completed_at": "2021-09-30T14:00:00Z",
  "timestamp": "2021-09-30T14:00:00Z",
  "customDimenssions": {
    "data_source_name": "TEST",
    "data_source_type": "shipper",
    "filename": "parser.py",
    "host": "DEVMACHINE",
    "hostname": "DEVMACHINE",
    "is_dry_run": false,
    "is_enabled": true,
    "lineno": 101,
    "log_id": "b91f4a11-e9ec-4f97-96fc-96e663d94b07",
    "module": "parser",
    "name": "parse_service",
    "node_id": "main",
    "pid": 15488
  }
}

Vector shall accept any log message, it will reserve the "known" keys, and place all the rest in the "customDimenssions" key.

Many thanks

[jszwedko]

Maybe something like this?

https://playground.vrl.dev/?state=eyJwcm9ncmFtIjoidmFsdWUgPSAuXG5cbnZhbHVlID0ge1xuICBcInByb2Nlc3NOYW1lXCI6IGRlbCgucHJvY2Vzc05hbWUpLFxuICBcInNlcnZpY2VfbmFtZVwiOiBkZWwoLnNlcnZpY2VfbmFtZSksXG4gIFwic2hpcHBlclwiOiBkZWwoLnNoaXBwZXIpLFxuICBcImxvY2F0aW9uXCI6IGRlbCgubG9jYXRpb24pLFxuICBcImNvbXBsZXRlZF9hdFwiOiBkZWwoLmNvbXBsZXRlZF9hdCksXG4gIFwidGltZXN0YW1wXCI6IGRlbCgudGltZXN0YW1wKSxcbn1cblxudmFsdWUuY3VzdG9tRGltZW5zc2lvbnMgPSAuXG5cbi4gPSB2YWx1ZSIsImV2ZW50Ijp7ImRhdGFfc291cmNlX25hbWUiOiJURVNUIiwiZGF0YV9zb3VyY2VfdHlwZSI6InNoaXBwZXIiLCJmaWxlbmFtZSI6InBhcnNlci5weSIsImhvc3QiOiJERVZNQUNISU5FIiwiaG9zdG5hbWUiOiJERVZNQUNISU5FIiwiaXNfZHJ5X3J1biI6ZmFsc2UsImlzX2VuYWJsZWQiOnRydWUsImxpbmVubyI6MTAxLCJsb2dfaWQiOiJiOTFmNGExMS1lOWVjLTRmOTctOTZmYy05NmU2MTFkOTRiMDciLCJtb2R1bGUiOiJwYXJzZXIiLCJuYW1lIjoicGFyc2Vfc2VydmljZSIsIm5vZGVfaWQiOiJtYWluIiwicGlkIjoxNTQ4OCwicHJvY2Vzc05hbWUiOiJNYWluUHJvY2VzcyIsInNlcnZpY2VfbmFtZSI6Ik1BU1RFUiIsInNoaXBwZXIiOiJ2ZWN0b3IiLCJsb2NhdGlvbiI6InVzLWVhc3QtMiIsImNvbXBsZXRlZF9hdCI6IjIwMjEtMDktMzBUMTQ6MDA6MDBaIiwidGltZXN0YW1wIjoiMjAyMS0wOS0zMFQxNDowMDowMFoifSwiaXNfanNvbmwiOmZhbHNlLCJlcnJvciI6bnVsbH0%3D

[omers]

Thank you @jszwedko! This is an elegant solution that I hadn't considered - much simpler than my attempts using for_each. This approach would be a valuable addition to the official documentation.