Help Needed: Adding account-id and instance-id to Vector's S3 Sink key_prefix

[block-erichter]

Hello everyone,

I'm currently working with Vector and trying to configure the S3 sink to include both the account-id and instance-id in the key_prefix. I've experimented with several approaches, including using remap, lua, and environment variables, but haven't found a solution that is both obvious and clean.

Here's what I've tried so far:

Remap: Attempted to use the remap transform to modify the data before it reaches the sink, but I couldn't find a straightforward way to inject these values directly into the key_prefix.

Lua: Considered using a Lua script to manipulate the data, but integrating this with the S3 sink configuration proved challenging.

Environment Variables: Explored setting environment variables for these IDs and referencing them in the configuration, but this approach didn't seem to integrate well with the key_prefix.

Has anyone successfully managed to include dynamic values like account-id and instance-id in the key_prefix for an S3 sink in Vector? If so, could you please share your approach or any tips you might have?

Any help or guidance would be greatly appreciated!

Thank you!

[jszwedko]

Hi @block-erichter ,

Environment Variables: Explored setting environment variables for these IDs and referencing them in the configuration, but this approach didn't seem to integrate well with the key_prefix.

I'm curious what challenges you ran into with this? It should be possible to interpolate environment variables in the key_prefix.

Is your aim to use the account-id / instance-id of the instance running Vector? If so, you could use the aws_ec2_metadata transform to annotate events with metadata from the instance and then refer to that in the key_prefix. However, the environment variable approach is likely to be simpler so I'm curious what stopped you from using it.

[block-erichter]

I attempted to use the aws_ec2_metdata but this fails vector --validate. Here is an example configuration that one would expect to work, using aws_ec2_metadata. However, testing shows this does not work as expected. I will say, that region does work in the key_prefix.

└─$ cat /etc/vector/vector.yaml
sources:
  journald:
    type: journald
  secure_log:
    type: file
    include:
      - "/var/log/secure"
    read_from: beginning
  dnf_log:
    type: file
    include:
      - "/var/log/dnf.log"
    read_from: beginning
  cloud-init_log:
    type: file
    include:
      - "/var/log/cloud-init.log"
    read_from: beginning
  sudo_log:
    type: file
    include:
      - "/var/log/sudo.log"
    read_from: beginning

transforms:
  add_metadata:
    type: aws_ec2_metadata
    inputs:
      - journald
      - secure_log
      - dnf_log
      - cloud-init_log
      - sudo_log
    fields:
      - account-id
      - ami-id
      - availability-zone
      - instance-id
      - instance-type
      - local-hostname
      - local-ipv4
      - public-hostname
      - public-ipv4
      - role-name
      - region
      - subnet-id
      - tags
      - vpc-id
sinks:
  s3:
    type: aws_s3
    inputs:
      - add_metadata
    bucket: "REDACTED"
    region: "us-west-2"
    compression: gzip
    encoding:
      codec: json
    key_prefix: "date=%Y-%m-%d/region={{ region }}/account_id={{ account-id }}/"
    auth:
      assume_role: "arn:aws:iam::REDACTED"

└─$ sudo vector validate /etc/vector/vector.yaml
√ Loaded ["/etc/vector/vector.yaml"]

Component errors
----------------
x Sink "s3": Invalid field path in template "account-id" (see https://vector.dev/docs/reference/configuration/template-syntax/)

[jszwedko]

Ah, you'll need to quote account-id because it has a special character: {{ "account-id" }} like:

 key_prefix: >
   date=%Y-%m-%d/region={{ region }}/account_id={{ "account-id" }}/

[block-erichter]

Awesome! This works now! Thanks so much for taking a look! 🍻