[CNCF] Proposal to Restructure Fluentd and Fluent Bit Under the Fluent Organization

[edsiper]

Restructure Fluentd and Fluent Bit Under the Fluent Organization

The following is an open discussion about the upcoming proposal to restructure the Fluentd and Fluent Bit projects under the Fluent organization for CNCF.

I have asked CNCF TOC for guidance and the restructure proposal idea has been provided by them.

After feedback and adjustments, we will run an open voting process. This is the first step. Feel free to jump in and share your thoughts.

Introduction

The Fluentd project has been pivotal in addressing the complexities of Telemetry Data management across infrastructure by enabling reliable collection, processing, and delivery of Logs, while integrating seamlessly with other ecosystems. Since its inception, Fluentd has significantly contributed to this mission. Established in 2011 and donated to the Cloud Native Computing Foundation (CNCF) in 2016, Fluentd achieved CNCF graduation in 2019. This milestone marked its widespread adoption by leading cloud providers such as AWS, Google, and Microsoft, solidifying its role in enterprise logging.

Evolution of Technology and User Demands

As technology and user expectations have evolved, so has the demand for high-performance solutions capable of handling diverse types of Telemetry data. Fluent Bit, created in 2015, emerged as a response to these needs. Designed to be lightweight and efficient, Fluent Bit offers enhanced performance and the capability to manage not only logs but also metrics and traces. This adaptability has led to its increased adoption, making it a preferred choice for many users, and has positioned it at the forefront of innovation within our organization.

Proposal for Restructuring

Given the strategic and technological shift towards Fluent Bit, we propose a restructuring of the Fluent organization for CNCF. The recommendation is to unify both Fluentd and Fluent Bit projects under the broader Fluent umbrella. In doing so, both projects will continue to address different use cases and capabilities, while allowing for future expansion with other projects that can fill the observability and telemetry gaps as identified in the upcoming CNCF Radar report.

This restructuring would ensure that the Fluent organization remains self-governing, with both Fluentd and Fluent Bit retaining their unique roles. It would also allow the governing body of Fluent to manage how to promote each project and its contributions in the broader ecosystem. The benefits of this restructuring will be felt at the Fluent level, rather than being specific to either Fluentd or Fluent Bit.

Note that this only affects the structure at a CNCF level, the Fluent organization already exists in Github so there are no breaking changes.

Next Steps Proposed

• Restructure Fluentd and Fluent Bit as part of the broader "Fluent" organization.
• Both projects will exist independently under the Fluent organization, governed by the higher-level Fluent governance structure.
• Update the GOVERNANCE files for both Fluentd and Fluent Bit to reflect this new structure.
• Open the voting process once the proposed governance structure is reviewed and PRs are clear.

F.A.Q

Are there any License changes involved?

NO. Both Fluentd and Fluent Bit are open-source projects hosted by the Cloud Native Computing Foundation under the terms of the Apache v2 License. We are vendor-neutral projects.

Will this have any side effects on developers?

No, everything continues as usual.

Are there any name changes for the projects ?

No, Fluentd and Fluent Bit will keep it's names. This is just a proposal to change the structure at CNCF level.

[edsiper]

cc: @ashie @daipom @kenhys @repeatedly @frsyuki @cosmo0920 @fujimotos @tagomoris @PettitWesley

[kenhys]

Basically I think it is practical approach to resolve current situation, but I'm not sure because the following step is a bit ambiguous.

Restructure Fluentd and Fluent Bit as part of the broader "Fluent" organization.

In the previous discussion #4471, switching the primary and sub-project was suggested.
In this time, this approach is changed toward co-existence oriented.

Did you coordinate with CNCF already? and have you learned any new facts about "Restructure" procedures?
If so, I'd like to know a little more about it.

FYI: I have shared this feedback with the CNCF TOC and they have provided good guidance. I will create another discussion with a different approach to continue there.

Oh, I've overlooked above post. anyway as I don't know well about actual "Restructure" procedures so it is happy if you explain further more.

[edsiper]

While I don't manage all the internal details on how CNCF handles this internally, today in the Landscape only Fluentd as a single project repository is exposed:

By restructuring into a Fluent organization (similar to how the projects are organized on GitHub), the Landscape would then point to the Fluent organization itself, which includes both Fluentd and Fluent Bit as distinct projects. Here’s what would change with this approach:

• Unified Governance: Establishing a clear governance structure for the Fluent organization as a whole.
• Expanded Visibility: Broadening the marketing scope, making Fluent Bit more visible alongside Fluentd.
• Extended Scope: While Fluentd is focused primarily on logs, the “Fluent” organization could encompass a wider range of observability tools, addressing needs beyond just logging.

The first step is to get a public agreement on this, vote, and then work with CNCF, specifically with TAG Contributor Strategy to help us with governance details.

[daipom]

Thanks for consulting with CNCF and considering a good course of action!

I agree with the approach of having Fluent Bit coexist with Fluentd.
I agree with the content of the "Next Steps Proposed."

then work with CNCF

Regarding this point, I would appreciate it if we could discuss the detail steps later.

Thanks!

[erikbledsoe]

What would be the impact, if any, on project stage? Would Fluent Bit be recognized as graduated separate from Fluentd, or would it need to apply for graduation?

Are there any other CNCF projects that have a similar organizational structure?

[edsiper]

There have been no changes in graduation status. I think this is a unique case but I don't manage all the details of all projects.

[tagomoris]

I know there are more and more fluent bit use cases in the real world, and yes, it's been a part of the ecosystem for a long time.
I agree with this proposal.

[repeatedly]

For me, this proposal is good for Fluentd and Fluent Bit ecosystem.
Fluent Bit is now important project in cloud native and container use-cases, so it should be visible at CNCF level.
Keep growing together 💪

[cosmo0920]

I agree with this proposal. Because cloud native and containerized environment are more and more important. So, both of Fluentd and Fluent Bit should be visible in CNCF projects page.

[ashie]

Thanks for considering revised proposal!
I also agree to move this forward, I understand that exposing Fluent Bit as CNCF project is important for the community.
To tell the truth, the previous proposal is hard to response for me because it was difficult to understand for the audience (especially for Fluentd users). The new proposal seems to be resolving the issues.

[edsiper]

adding other maintainers for visibility: @sonots @okkez @toru-takahashi @fujimotos