Maturity and community support for kube-rs

[unmarshall]

Hello,

I am one of the maintainers of Gardener and one of the teams that i work in is considering using Rust to develop a new component. The only equivalent of client-go/controller-runtime in Rust at the moment is kube-rs and therefore to be able to use it productively i have some fundamental questions.

• As per https://kube.rs/maintainers/ there are 4 maintainers but as per the KubeCon session there are 2 active maintainers and rest are ad-hoc/part-time contributors. Currently this project is a CNCF Sanbox project. Can you please provide more clarity on:

  • Roadmap of this project and plans to graduate this out of sandbox?
  • Community support and its rate of adoption for production grade solutions?

• Do you already have a document where you compare kube-rs with client-go or controller-runtime? These are currently the most heavily used frameworks to create k8s controllers in golang and have a massive community support. Essentially i am looking at what is not there in kube-rs yet. For example, I can already see issue#1108 which is currently open, hinting that there is no equivalent to envtest yet.

• For HA k8s controllers, one of the most basic requirements is an ability to do leader election. Unfortunately as per Support Leaderelection like client-go #485 it was decided to not include it as part of core framework. I understand that the philosophy of the Rust language is to keep the core minimal and use external crates. However one cannot put a language and a framework in the same bucket. To find the best crates we usually go to https://blessed.rs/crates but there i see no crate being listed for leader election in k8s context. This makes it super hard for adoption as now the onus is on the consumer to choose a crate and then may later on find issues with it and then move to another one or create our own. This only raises the bar for Rust adoption in k8s. Example is RUSTSEC-2024-0384: instant is unmaintained #1635 where you start hunting for crates time and again.

It will be great if we can get some answers as that will help us take a decision on whether we can at present use kube-rs to develop highly scalable components. Looking forward to adopting Rust and your support.

Best Regards,
Madhav

[clux]

Hey, there. Some quick informal answers here.

Roadmap; we do technically have a roadmap but it's mostly ironing out /issues rather than a whole bunch of new features at the moment. A couple of outstanding things that we're considering getting done before 1.0 is stream sharing and potentially k8s-pb integration (protobuf transport), but beyond that, we're generally feature complete as a client (ws, subresources, exec, client-gold requirements), and the runtime is the stronger side (as that's generally been the needs for all the maintainers).

Comparison; There's no direct comparison doc (afaik) to controller-runtime / client-go, but we do follow their precedents when they make sense (often they do not in rust, but we get a lot of go migrators' input). Sometimes we are faster at getting things available to users; websocket support (what we first implemented in 2021 for exec/attach, only reached kubectl in 1.31), streaming list (which we've had for >a year whereas it just reached client-go). This is generally the exception though, and the go side tends to act faster (as it's where it's originating after all). Thankfully, upstream core api changes are rare, so kube can mostly focus on tuning, handling issues, and importing new structs on kubernetes release.

Community support is essential to edge cases or outstanding issues. Lots of implementors from linkerd / mirrord / aws / stackable / datadog / various GUI implementors like aptakube have helped give back in areas where the few active maintainers have experience.

Leader election is one of those things we would like to bless an implementation, but nothing feels ready yet. There are hints at crates in https://kube.rs/controllers/availability/#leader-election though. Envtest has been brought up a couple of times, but the community is generally moved to kwok or k3d for integration or mocking as outlined in https://kube.rs/controllers/testing/ instead. YMMV.

I encourage you to look at https://kube.rs/ ask actual adopters if you are on the fence. It's rust, so not everything is going to be smooth sailing as the language is (arguably) harder and the ecosystem is newer; but when it compiles it generally works.

As for CNCF and sandbox.. It's not actually a huge benefit to graduate, and it's not something we prioritise atm. This is a volunteer library effort (unlike most other CNCF projects). We don't sell anything, but we all use Rust and want the experience to be smooth.

[unmarshall]

Thanks for a detailed response. I will have a look at the roadmap and will also have a look at the open issues to get an understanding of where this project is w.r.t maturity. Since most people developing/enhancing k8s or on-k8s usually use golang, therefore it is tricky to convince an organisation/people to use Rust given that the ecosystem of libraries and frameworks around golang for k8s are community driven and highly mature. Even though i personally find Rust as a better language for writing highly scalable controllers its framework support is still a work-in-progres but very promising indeed.