fluentd output format json flat-json

[paukstis]

Hi, in the very first versions (0.1.xx) the fluentd worker record format was packed as json, now in recent versions it's flat-json. I guess this is not an issue, but when fluentd output is Opensearch or Elasticsearch, then with flat-json the number of fields quickly reaches limits (1000 by default). Anyone in similar setup ?

[dmachard]

Hi, could you share more details? I’m a bit surprised, as the KV JSON model typically doesn’t contain as many as 1000 fields.

Could it be related to https://www.elastic.co/guide/en/elasticsearch/reference/current/subobjects.html ? Do you see any other specific error messages in Elasticsearch?

I don't see any errors with my quick lab described here https://github.com/dmachard/go-dnscollector/blob/main/docs/_integration/elasticsearch/README.md

[paukstis]

Thank you for the pointer to subobjects in Elastisearch, that setting makes sense. I'm actually on Opensearch and a similar setting I guess would be the flat_object type field.
The problematic field in flat-json is dns.resource-records, where with default settings and mappings, every field such as
dns.resource-records.an.0.class
dns.resource-records.an.1.class
becomes a new field, and then it quickly explodes fields count, no errors in the logs.

When I set mapping as flat_object, then Opensearch throws error:
2024-11-12 11:29:17 +0100 [warn]: #0 dump an error event: error_class=Fluent::Plugin::OpenSearchErrorHandler::OpenSearchError error="400 - Rejected by OpenSearch [error type]: mapper_parsing_exception [reason]: 'Could not dynamically add mapping for field [dns.resource-records.an.0.class]. Existing mapping for [dns.resource-records.an] must be of type object but found [flat_object].'" location=nil tag="dnstap.message" time=2024-11-12 11:28:48.153776418 +0100 record={"network.family"=>"IPv4", "network.ip-defragmented"=>false, "dns.flags.tc"=>false, "dnstap.identity"=>"dns", "dns.resource-records.an.0.class"=>"IN", "dns.opcode"=>0,

[dmachard]

Thank you for the feedback!

I can look into adding a new setting to configure the separator in flat-json, which would make it possible to replace . with _.

Could you open an issue to track this future change?

[paukstis]

I don't think separator change would solve the issue in my case, Opensearch would still see these as multiple separate fields.

As a workaround, at the moment, I switched the output to tcp, with mode: json, and I parse the input with json parser with fluentd. It adds a bit of work, but the result is ok, with limited fields count, like in the very first versions of go-dnscollector.