Docs reference a setting incorrectly

[nerophon]

Version: 7.8.1

The following docs have an incorrect reference:
https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html
https://www.elastic.co/guide/en/beats/filebeat/7.8/configuring-ingest-node.html

They show:

output:
  pipeline:

Whereas they should show:

output:
  parameters.pipeline:

If the incorrect reference is used, it results in unexpected behaviour.

[elasticmachine]

Pinging @elastic/integrations-services (Team:Services)

[elasticmachine]

Pinging @elastic/obs-docs (Team:Docs)

[bmorelli25]

Related to #20342 and https://discuss.elastic.co/t/filebeat-output-elasticsearch-pipeline-not-working/240167.

[dedemorton]

This looks like a product bug, not a documentation bug. The documented configuration works correctly on Metricbeat. Since the output settings for all the beats are essentially the same, I would expect them to use the same config syntax.

[kvch]

@dedemorton is correct, it is a product problem if the configuration snippets in the reference doc pages do not work in this case.

@nerophon What is the exact configuration which leads to issues? None of the configurations you have shared is supposed to work with Beats.

The supported configurations are the following:

output.elasticsearch:
  hosts: ["your-host:9200"]
  pipeline: your-pipeline

output.elasticsearch:
  hosts: ["your-host:9200"]
  pipelines:
    - pipeline: your-pipeline-1
      when.contains:
        message: something
    - pipeline: your-pipeline-2
      when.contains:
        message: anything

[dedemorton]

@kvch I think Nicholas was shortening the example to show the parts that didn't work. I checked in 7.9, and the configuration that you show works on Metricbeat, but does not seem to work on Filebeat unless you use parameters.pipeline instead of pipeline. I see old forum posts and GitHub issues going back to 2016 about a similar problem, so I wonder if this is some kind of regression (#1962).

Should I create a troubleshooting topic that describes the workaround since this seems to be a recurrent problem? I hesitate to recommend a non-standard config, though.

Either way, I will close this issue because it sounds like this is not a documentation problem.

[kvch]

@nerophon I cannot reproduce the issue locally. Could you please share more details e.g. exact configuration in a separate issue? What is the unexpected issue you are seeing?

[dedemorton]

Closing this issue because it does not seem to be a doc bug, and the dev team is unable to reproduce it.

[gbanasiak]

I'd like to reopen this as pipeline configuration in Elasticsearch output continues to be problematic for Filebeat users. Based on #20342, #21548 and #22062 it seems that since 7.8 up until now (7.15) the following gets reported:

• pipeline doesn't work, but parameters.pipeline does
• pipelines (conditional pipelines) do not work

#22062 has full configuration example, and says conditional pipelines work for one type of input, but do not work for another (!). Perhaps this explains why the pipeline problem is elusive in Filebeat - it might require specific combination of inputs or modules to surface.

Can we remove these options from Filebeat documentation of Elasticsearch output until we fix these long open issues or clarify when the options are expected to work and when not?

[rschirin]

Hi all,
I would give you my little experience with ELK 8.2.2. I am using Filebeat (input: gcp pub/sub, output: elasticsearch), I am still facing the problem if I specify the pipeline in the output.elasticsearch plugin.
Everything works fine if I specify the pipeline in the type.input

[dedemorton]

The writing team needs more input from PMs and/or product development team regarding how to proceed with documenting this problem.

• Is this a product bug or docs issue?
• If product bug, will it be fixed?
• How should we fix the problem in the docs?
  • Change the config docs to show parameters.pipeline instead of pipeline? For Filebeat only? Any other Beats?
  • Remove the pipelines (conditional pipelines) section (from Filebeat? Other docs?)
  • Keep the pipelines documentation but clarify the current limitations?
  • Instead of changing the config docs or adding limitations, document this as a known problem and workaround in the release notes?
  • Something else?

My apologies if I'm missing some details here but I don't get to work on Beats docs very often these days.

[dedemorton]

@pierrehilbert This issue is very old. Can you assign this to someone to follow up on? Or close it if it's no longer valid. I am going to remove my name from the assignee field because I will not be working on these changes. Thanks!

[botelastic]

Hi!
We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!