Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: First release #1

Merged
merged 43 commits into from
Jul 8, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
c39ca07
Draft implementation
alex-karpenko Jun 30, 2024
43c4c50
Add tests
alex-karpenko Jun 30, 2024
3660c8c
Improve error handling
alex-karpenko Jun 30, 2024
f70b2fb
Add LeaseManager tests
alex-karpenko Jul 1, 2024
d9938f3
Fix locking approach
alex-karpenko Jul 2, 2024
0f97f6c
Add resourceVersion updates
alex-karpenko Jul 2, 2024
a343c84
Revert resourceVersion updates
alex-karpenko Jul 2, 2024
0cc44f6
Improve LeaseManager tests
alex-karpenko Jul 2, 2024
66d49eb
Improve complex LeaseManager tests
alex-karpenko Jul 3, 2024
1e536b8
Change seconds-long durations to explicit u64
alex-karpenko Jul 3, 2024
da499ff
Improve state locking
alex-karpenko Jul 3, 2024
1f73e72
Implement Lease auto-creation
alex-karpenko Jul 3, 2024
d47633b
Fix clippy warning
alex-karpenko Jul 3, 2024
5a4b821
MSRV to 1.61
alex-karpenko Jul 3, 2024
7e2337c
Fix ci workflow
alex-karpenko Jul 3, 2024
05cfaf6
Fix ci workflow
alex-karpenko Jul 3, 2024
84ef835
MSRV to 1.70
alex-karpenko Jul 3, 2024
46f457e
MSRV to 1.75
alex-karpenko Jul 3, 2024
b56cf11
Debug many_managers_1st_expires_then_someone_locks test
alex-karpenko Jul 3, 2024
982b613
Improve locking
alex-karpenko Jul 4, 2024
9d23c3d
Improve locking
alex-karpenko Jul 4, 2024
7729feb
Rename some resources
alex-karpenko Jul 4, 2024
8c86369
Rename crate
alex-karpenko Jul 4, 2024
0ad37d1
Improve ci
alex-karpenko Jul 4, 2024
2310185
Implement conflict backoff timer
alex-karpenko Jul 5, 2024
93d0be2
Improve tests of conflict backoff timer
alex-karpenko Jul 5, 2024
a514717
Refactor state
alex-karpenko Jul 5, 2024
9ac0a49
Implement watch method
alex-karpenko Jul 5, 2024
9312799
Implement watch method
alex-karpenko Jul 5, 2024
8aa23b9
Remove LockConflict error from public API
alex-karpenko Jul 6, 2024
f03e074
Implement watch method tests
alex-karpenko Jul 6, 2024
fc8d631
Fix dependencies
alex-karpenko Jul 6, 2024
83a6006
Make watch method test harder
alex-karpenko Jul 6, 2024
d500e82
Add LeaseManagerBuilder
alex-karpenko Jul 6, 2024
cd29c3c
Improve tracing messages and tests
alex-karpenko Jul 6, 2024
80163b6
Fix tests
alex-karpenko Jul 6, 2024
6f8ab76
Fix grammar
alex-karpenko Jul 6, 2024
e92ec68
Update documentation
alex-karpenko Jul 6, 2024
024d7b0
Update documentation
alex-karpenko Jul 7, 2024
1cc3d57
Merge branch 'feat/initial-release' of github.com:alex-karpenko/kube-…
alex-karpenko Jul 7, 2024
41eb06f
Exclude k8s-openapi feature from dependencies
alex-karpenko Jul 7, 2024
f80cc10
Update documentation
alex-karpenko Jul 7, 2024
34d2c73
Update documentation
alex-karpenko Jul 8, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .cargo/config.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
[env]
K8S_OPENAPI_ENABLED_VERSION = "1.26"
18 changes: 18 additions & 0 deletions .github/workflows/audit.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
name: Security audit
on:
workflow_dispatch:
schedule:
- cron: "17 5 * * 5"
push:
branches:
- main
paths:
- "**/Cargo.toml"
pull_request:

jobs:
security-audit:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: EmbarkStudios/cargo-deny-action@v1
58 changes: 58 additions & 0 deletions .github/workflows/ci.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
name: Continuous integration
on:
workflow_dispatch:
pull_request:
schedule:
- cron: "30 16 * * 5"
env:
CARGO_TERM_COLOR: always
CI: true

jobs:
ci:
runs-on: ubuntu-latest
strategy:
matrix:
rust:
- stable
- beta
- 1.75.0
k8s:
- v1.26
- latest

steps:
- uses: actions/checkout@v4
- uses: dtolnay/rust-toolchain@stable
name: Setup toolchain
with:
toolchain: ${{ matrix.rust }}
components: rustfmt,clippy

- uses: Swatinem/rust-cache@v2

- name: Fmt
run: cargo fmt --all -- --check

- name: Clippy
run: cargo clippy --all-targets -- -D warnings

- name: Build
run: cargo build

- name: Light tests
# env:
# RUST_LOG: kube_lease=debug
run: cargo test --lib --all

- uses: nolar/setup-k3d-k3s@v1
with:
version: ${{matrix.k8s}}
k3d-name: kube
github-token: ${{ secrets.GITHUB_TOKEN }}
k3d-args: "--no-lb --no-rollback --k3s-arg --disable=traefik,servicelb,metrics-server@server:*"

- name: Heavy tests
# env:
# RUST_LOG: kube_lease_manager=debug
run: cargo test --lib --all -- --ignored
58 changes: 58 additions & 0 deletions .github/workflows/coverage.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
name: Code coverage report
on:
workflow_dispatch:
pull_request:
push:
branches:
- main
- rel-v*

env:
CARGO_TERM_COLOR: always

jobs:
codecov:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Setup for testing
run: |
mkdir grcov
curl -sL https://github.com/mozilla/grcov/releases/latest/download/grcov-x86_64-unknown-linux-gnu.tar.bz2 | (cd grcov ; tar jxf -)
rustup toolchain add nightly --component llvm-tools-preview
rustup override set nightly

- uses: Swatinem/rust-cache@v2
- uses: nolar/setup-k3d-k3s@v1
with:
version: latest
k3d-name: kube
github-token: ${{ secrets.GITHUB_TOKEN }}
k3d-args: "--no-lb --no-rollback --k3s-arg --disable=traefik,servicelb,metrics-server@server:*"

- name: Test
env:
CI: true
RUSTFLAGS: -Cinstrument-coverage
LLVM_PROFILE_FILE: kube-lease-manager-%p-%m.profraw
RUST_LOG: kube_lease_manager=debug
run: cargo test --lib --all -- --include-ignored

- name: Generate coverage
run: |
grcov/grcov $(find . -name "kube-lease-manager-*.profraw" -print) \
--branch \
--ignore-not-existing \
--binary-path ./target/debug/ \
-s src \
-t lcov \
--ignore "/*" \
--excl-line '^\s*\.await\??;?$' --excl-br-line '^\s*\.await\??;?$' \
-o lcov.info

- name: Upload coverage reports to Codecov
uses: codecov/codecov-action@v4
with:
token: ${{ secrets.CODECOV_TOKEN }}
fail_ci_if_error: true
file: ./lcov.info
64 changes: 64 additions & 0 deletions .github/workflows/publish.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
name: Release
permissions:
contents: write
on:
workflow_dispatch:
push:
tags: ["v[0-9]+.[0-9]+.[0-9]+"]
env:
CARGO_TERM_COLOR: always

jobs:
generate-changelog:
name: Generate changelog
runs-on: ubuntu-latest
outputs:
release_body: ${{ steps.git-cliff.outputs.content }}
steps:
- name: Checkout
uses: actions/checkout@v4
with:
fetch-depth: 0

- name: Generate release changelog
uses: orhun/git-cliff-action@v3
id: git-cliff
with:
config: cliff.toml
args: -v --latest --strip header --github-token ${{ secrets.GITHUB_TOKEN }}
env:
OUTPUT: CHANGELOG.md
GITHUB_REPO: ${{ github.repository }}

release:
name: Release to Github
needs: generate-changelog
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4

- name: Release
uses: softprops/action-gh-release@v2
with:
body: "${{ needs.generate-changelog.outputs.release_body }}"
name: "Release ${{ github.ref_name }}"

publish:
name: Publish to Crates.io
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4

- name: Setup toolchain
uses: dtolnay/rust-toolchain@stable
with:
profile: minimal
toolchain: stable
override: true

- name: Publish to Crates.io
run: cargo publish --token ${CRATES_TOKEN}
env:
CRATES_TOKEN: ${{ secrets.CRATES_TOKEN }}
8 changes: 8 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
# will have compiled files and executables
debug/
target/
.DS_Store

# Remove Cargo.lock from gitignore if creating an executable, leave it for libraries
# More information here https://doc.rust-lang.org/cargo/guide/cargo-toml-vs-cargo-lock.html
Expand All @@ -12,3 +13,10 @@ Cargo.lock

# MSVC Windows builds of rustc generate these, which store debugging information
*.pdb

# Idea
.idea/

# Local development artifacts
TODO.md
.vscode/
47 changes: 47 additions & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
[package]
name = "kube-lease-manager"
authors = ["Oleksii Karpenko <alexkarpenko@yahoo.com>"]
categories = ["api-bindings", "asynchronous"]
description = "Ergonomic and durable leader election using Kubernetes Lease API."
edition = "2021"
rust-version = "1.75"
homepage = "https://github.com/alex-karpenko/kube-lease-manager"
keywords = ["kubernetes", "async", "lease", "leader", "election"]
license = "MIT"
readme = "README.md"
repository = "https://github.com/alex-karpenko/kube-lease-manager"
version = "0.1.0"
exclude = [
".github/**",
".vscode/**",
"TODO.md",
"Cargo.lock",
"target/**",
".gitignore",
".cargo/**",
]

[dependencies]
k8s-openapi = { version = "0.22.0" }
kube = { version = "0.92.1", default-features = false, features = ["client"] }
rand = { version = "0.8.5" }
thiserror = "1.0.61"
tokio = { version = "1.38.0", default-features = false, features = [
"sync",
"time",
"macros",
] }
tracing = { version = "0.1.40", default-features = false, features = ["std"] }

[dev-dependencies]
futures = { version = "0.3.30", default-features = false, features = [
"async-await",
] }
kube = { version = "0.92.1", default-features = false, features = [
"rustls-tls",
] }
tokio = { version = "1.38.0", default-features = false, features = [
"rt-multi-thread",
] }
tracing-subscriber = { version = "0.3.18" }
uuid = { version = "1.9.1", features = ["v4"] }
121 changes: 119 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,2 +1,119 @@
# kube-lease
Wrapper for Kubernetes Lease API, siplifies leader election.
# kube-lease-manager

Ergonomic and durable leader election using Kubernetes Lease API.

<p>
<a href="https://github.com/alex-karpenko/kube-lease-manager/actions/workflows/ci.yaml" rel="nofollow"><img src="https://img.shields.io/github/actions/workflow/status/alex-karpenko/kube-lease-manager/ci.yaml?label=ci" alt="CI status"></a>
<a href="https://github.com/alex-karpenko/kube-lease-manager/actions/workflows/audit.yaml" rel="nofollow"><img src="https://img.shields.io/github/actions/workflow/status/alex-karpenko/kube-lease-manager/audit.yaml?label=audit" alt="Audit status"></a>
<a href="https://github.com/alex-karpenko/kube-lease-manager/actions/workflows/publish.yaml" rel="nofollow"><img src="https://img.shields.io/github/actions/workflow/status/alex-karpenko/kube-lease-manager/publish.yaml?label=publish" alt="Crates.io publishing status"></a>
<a href="https://docs.rs/kube-lease-manager" rel="nofollow"><img src="https://img.shields.io/docsrs/kube-lease-manager" alt="docs.rs status"></a>
<a href="https://crates.io/crates/kube-lease-manager" rel="nofollow"><img src="https://img.shields.io/crates/v/kube-lease-manager" alt="Version at Crates.io"></a>
<a href="https://github.com/alex-karpenko/kube-lease-manager/blob/HEAD/LICENSE" rel="nofollow"><img src="https://img.shields.io/crates/l/kube-lease-manager" alt="License"></a>
</p>

`kube-lease-manager` is a high-level helper to facilitate leader election using
[Lease Kubernetes resource](https://kubernetes.io/docs/reference/kubernetes-api/cluster-resources/lease-v1/).
It ensures that only a single instance of the lease managers holds the lock at any moment of time.

Some of the typical use cases:
* automatic coordination of leader election between several instances (Pods) of Kubernetes controllers;
* ensure only a single instance of concurrent jobs is running right now;
* exclusive acquiring of shared resource.

## Features

* `LeaseManager` is a central part of the crate.
This is a convenient wrapper around a Kubernetes `Lease` resource to manage all aspects of leader election process.
* Provides two different high-level approaches to lock and release lease:
fully automated or partially manual lock control.
* Uses [Server-Side-Apply](https://kubernetes.io/docs/reference/using-api/server-side-apply/)
approach to update lease state that facilitates conflict detection and resolution
and makes impossible concurrent locking.
* Tolerate configurable time skew between nodes of the Kubernetes cluster.
* Behavioral parameters of the lease manager are easily and flexibly configurable.
* Uses well-known and highly appreciated [kube](https://crates.io/crates/kube)
and [Tokio](https://crates.io/crates/tokio)
crates to access Kubernetes API and coordinate asynchronous tasks execution.
* You don't need to use low-level Kubernetes API.
* Uses Tokio [`tracing`](https://crates.io/crates/tracing) carte to provide event logs.

Please visit [crate's documentation](https://docs.rs/kube-lease-manager/) to get details and more examples.

---

As mentioned above, `kube-lease-manager` provides two possible ways to manage lease lock:
1. _Fully automated_: you create `LeaseManager` instance and run its `watch()` method.
It returns [Tokio watch channel](https://docs.rs/tokio/1.38.0/tokio/sync/watch/index.html) to watch on state changes
Besides that it runs an unattended background task
which permanently tries to lock lease if it's free and publish changed state to the channel.
The task finishes if the channel is closed.
2. _Partially manual_: you create `LeaseManager`
instance and use its `changed()`
and `release()` methods to control lock.
`changed()` tries to lock lease as soon as it becomes free and returns actual lock state when it's changed.
Your responsibilities are:
- to keep `changed()` running (it's a `Future`) to ensure lock is refreshing while it's in use;
- to call `release()` when you don't need the lock and want to make it free for others.

First way ensures that lease is locked (has a holder) at any moment of time.
Second makes possible to acquire and release lock when you need it.

## Example

The simplest example using first locking approach:
```rust
use kube::Client;
use kube_lease_manager::LeaseManagerBuilder;
use std::time::Duration;

#[tokio::main]
async fn main() {
// Use default Kube client
let client = Client::try_default().await.unwrap();
// Create the simplest LeaseManager with reasonable defaults using convenient builder.
// It uses Lease resource called `test-watch-lease`.
let manager = LeaseManagerBuilder::new(client, "test-auto-lease")
.build()
.await
.unwrap();

let (mut channel, task) = manager.watch().await;
// Watch on the channel for lock state changes
tokio::select! {
_ = channel.changed() => {
let lock_state = *channel.borrow_and_update();

if lock_state {
// Do something useful as a leader
println!("Got a luck!");
}
}
_ = tokio::time::sleep(Duration::from_secs(10)) => {
println!("Unable get lock during 10s");
}
}

// Explicitly close the control channel
drop(channel);
// Wait for the finish of the manager and get it back
let _manager = tokio::join!(task).0.unwrap().unwrap();
}
```

Please visit [crate's documentation](https://docs.rs/kube-lease-manager/) to get more examples and usage details.

## TODO

- [ ] Provide some real and useful examples.

## Credits

The author was inspired on this piece of work by two other crates that provide similar functionality:
[kubert](https://crates.io/crates/kubert) and [kube-leader-election](https://crates.io/crates/kube-leader-election).
Both of them are great, thanks to the authors.
But both have something missing for one of my projects.
So it was a reason to create this one.

## License

This project is licensed under the [MIT license](LICENSE).
Loading