Project direction / contributions

[macsux]

I've recently been doing a lot of work building out building blocks for .NET Kubernetes integration, with the goal of providing the right components to create Kubernetes aware apps, creation of controllers, operators, and .net specific k8s automation. A lot of this work has been contributed to the official KubernetesClient and is currently hanging as open PRs (about 10k LOC total).

A lot of the code that is in there is actually abstract enough to be made easily compatible with both projects, which I may want to do. I'm finding that the structure of this project is definitely easier for the developer given it's hand-coded nature, but also introduces more overhead from a maintenance perspective to stay up to date with Kubernetes.

I have a few questions on general project philosophy and direction

1. Has there been an effort to make this part of the official Kubernetes org to improve community participation in this project?
2. Both projects seem to focus primarily on being "clients" rather than providing an SDK to build new features on top of K8S (controllers, informers, operators, etc). What is the scope and vision of this project?
3. Seems like the project hasn't been getting a lot of updates over the last year. What is the maintainer's plans for continuing to support and improve this?
4. Why does the code require an explicit List type per object type? It seems implementation is the same in all of them, and KubeResourceListV1 can be used directly if it wasn't abstract (Kind/APIVersion can be derived from the generic argument's metadata).
5. Most of the ResourceClients look like they can be generated as well since most APIs follow the standard URL scheme. There's an open PR from me to add something like this to
   KubernetesClient Generic methods kubernetes-client/csharp#412
   Looking at them, there can be a generic version created for most objects and manually extended where necessary. This would obviously lead to less maintenance, while still allowing nice organization.
6. Why is model generator written in python?

[tintoy]

Hi - thanks for the reminder, I’ve been meaning to give this project some attention (my current job doesn’t deal with K8s at all so it’s kinda fallen off the radar a bit).

I’ve only just woken up, so let me have a coffee and then I’ll read this fully and do my best to answer these questions 🙂

[macsux]

Looking through the code it actually seems like there is a DynamicClient, but it's weird that it's not using a generics interface and instead relies on kind/api version being parameters.

IMO those things are fundamentally type information and if you already have a class structure that represents it, this is just metadata for that type. Those things should ideally be hidden from the user as they are more of serialization metadata then actual data. Also, it's highly desirable to make it seamless to the user to convert between different versions of resources (as you may wanna support multiple at the same time without having to explicitly fill out two nearly identical model structures). This was implemented in kubernetes-client/csharp#420 to allow direct casting between all versions of the same Kind, which opens up a path to dynamically select an appropriate version based on what the server is expecting. Seems like this was partially tackled in DynamicClient, but IMO should apply to everything.

[tintoy]

I’ve tried using generics for the clients, but the problem is that the APIs for various resource types are subtly inconsistent. Some of them will return the resource state when calling a given operation, whereas others just return a v1/Status.

And third-party APIs don’t have to follow the rules at all unfortunately.

Still, I’m open to anyone’s efforts to find a consistent or sufficiently-flexible way to handle these scenarios that doesn’t compromise the client’s ergonomics :)

[tintoy]

BTW, I'm most definitely open to collaboration / contributions! Basically, as long as the ergonomics (or more generally the developer experience) are preserved or enhanced I'm up for pretty much anything :)

Going to go and get a coffee shortly, then will come and re-read this (it's Saturday and it's been a busy week so it may take me a little while to get back up to speed).

[tintoy]

OK, question by question:

1. Some bits of this client (notably the WebSockets implementation) have been contributed to the official client. Having spoken to the folks who maintain the official client, I get the impression they want to leverage the same client/model generation mechanism for all their language bindings. I don't like the ergonomics of their generated client or models, so I built and then kept using this one.
2. KubeClient does have a couple of other integration points (e.g. ASP.NET Core configuration from ConfigMaps and Secrets) which are in common use, and I have an old half-finished K8s controller framework (that uses Rx or Akka.NET, can't remember which one) lying around somewhere but that's not production-grade (I found it an interesting challenge but didn't have a use case for it at the time, and I generally try to avoid releasing any functionality without a clear use case). It would be nice to have something like controller and operator frameworks that layer on top of the client (the client's general philosophy is one of layered extensibility).
3. My current job doesn't involve K8s at all, so the only time I make changes myself is if there is a bug or some new model or feature requested by one of the library's users. A couple of people have made contributions as well. It's not dead, just not under active enhancement. But if you have ideas and would like to contribute I'd certainly be open to that.
4. Philosophically speaking, I suppose this one falls under purism vs pragmatism; generics are great (hence the KubeResourceListV1<TResource> base class for resource list models), but I've also found it useful to have a concrete model class corresponding to each combination of kind and apiVersion (makes dynamic client scenarios a lot easier to implement since resolving model types doesn't have to care if it's a list or not, and handles "special case" lists such as APIGroupListV1 or APIResourceListV1). You are correct that the kind could be inferred from the item type (except for special cases but they probably don't count anyway), but at this stage that would be a breaking change for consumers and I'd need to see some concrete benefits before I'd be convinced that making this change is a good idea. I do try to keep an open mind though :)
5. You'd think so, but my experience is that they only sort of follow the standard scheme and have too many quirks to make this worthwhile. Having said that, perhaps as you say generating some of the standard operations for each resource client (where those operations do correctly follow the pattern in terms of behaviour and API shape) might be helpful? I'd be fine with this if you (or anyone else) would like to have a go at it.
6. Because I needed something quick-and-dirty, and I'd already written a parser for K8s YAML in Python before (and didn't want to waste time yak-shaving). I'm not fussed if this wasn't in Python, it's just that what I have works and I figured that the time it would take to rewrite it could be better spent elsewhere on the project :)

[jonstelly]

First a slightly offtopic question for @macsux - Do you have any links to implementing kubernetes operators in .NET? I don't need to do that right now but I've looked around a few times and haven't really found much info.

FWIW, I chose this project over the "official" one for a few reasons:

1. The official dotnet client repository's summary is "Work In Progress" which is a minor thing but it was something I noticed.
2. If you look on nuget it seems like they've been averaging about 10-15 releases per month for the last 3 months, but there are several cases where they pushed 3-5 nuget packages in one day.
3. They have no release notes.
4. 1+2+3 makes it very tough to feel like they're pushing stable releases and makes me wonder how I'd even find a stable release in the sea of updates given that there's no changelog / release notes.
5. I'm all for code generation but code generating a 176,000 line source file makes it tough to dig into their code if needed. It looks like they use AutoRest. I've never used that before so I'm not sure if it's possible to break out the generated code into multiple source files but it would be nice to see source files that made it easier to find code when needed. Their generated files are so big you can't even view them with syntax highlighting on github (minor point, but still at least an annoyance).
6. This tintoy client code was very easy to jump into, understand, and submit PRs for and Adam has been very responsive in reviewing, making suggestions and approving PRs. I wouldn't have guessed he was no longer using Kubernetes given his responsiveness.

I'll also say I've had bad experiences in the past with projects that try to provide clients for many different programming languages and dotnet isn't viewed as a critical one. Where the APIs look crooked and things don't get fixed "because we code generate that." The kubernetes client seems to have good dotnet developers on it and the code isn't bad, so this is more about explaining my bias than pointing out a fault in that project.

All that said, I agree that consolidating efforts is good and you make some good points in general. If there's anything I can do to help, I'm happy to join whatever efforts we feel make sense.

[macsux]

I'm currently working towards building the necessary building blocks to allows this to happen. While it started as a desire to build controllers with .NET, it morphed a little into a broader initiative. We're building a framework that will offer LINQ abstraction over "live query", allowing you not only to get the current state of resources but to synchronize and respond to changes to the result set of the query over time. This is equivalent to informers in for Kubernetes. When it's released it will include provider for Kubernetes. This will be made as part of Steeltoe.
You can read how it's gonna work here:
https://stakhov.pro/informers-intro/
(though I've made a turn to base it on IAsyncEnumerable vs IObservable since the article was written)

This will make writing controller loops super simple and let us expand into higher-level functional operators with ease. You should be able to try it out over the coming weeks. I'm mainly building many of these blocks for myself as I have many ideas for controllers/operators in .NET space.

In regards to the official client, I've been actively contributing to it, but I do agree there are issues. I personally disagree with the approach of both projects:

• the official client has the benefit of staying up to date with the official release because they have a low maintenance burden. They also have a strongly typed method to work with all objects. The downside is all the ones you already mentioned. I do agree that the lack of proper release management is not great. This makes large PRs difficult to push in, stabilize, and validate. This is one of the main reasons much of this stuff will land under steeltoe umbrella.
• kubeclient has a much better API structure, but a much larger maintenance burden because it codes all clients by hand. Some resources are not exposed as clients, even though they exist as part of core k8s. IMO it can be significantly improved to generate clients for things that follow standard patterns and only hand-code those that deviate (custom routes, subresources, etc).

Both projects tend to lack from lack of contributors. If there were 20+ ACTIVE contributors to the official client then a better case could be made to sacrifice some automation for better API. Alas, that is not the case - both libraries suffer from low number of maintainers. Having said that being part of Kubernetes SIG definitely puts a significant momentum behind the official client, even though from where I'm standing it's codebase is inferior from that of KubeClient. There are a bunch of PRs I submitted against the official client to add support for installing CRDs, generic client, and some other things.

I would LOVE to work on towards getting to a point where if KubeClient models AND clients can be autogenerated to follow the same pattern already in this library instead of ugly mess that AutoReset generates. If this is achieved, there's a case to be made that it aligns with the official project's philosophy, and should be integrated into it. While I love this codebase here, an architect in me would be very nervous to take on dependency with a single part-time contributor.

I think going forward my contributions are gonna be split between official client and Steeltoe. Steeltoe has a team of full-time engineers from VMWare and a rigid release structure.