Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added instructions to update kubeconfig after manual certs renewal or cluster upgrade #7890

Merged
merged 20 commits into from
Jun 7, 2024
Merged
Show file tree
Hide file tree
Changes from 11 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
73 changes: 71 additions & 2 deletions docs/content/en/docs/clustermgmt/security/manually-renew-certs.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,7 @@ ${IMAGE_ID} tmp-cert-renew \
sudo etcdctl --cacert=/etc/etcd/pki/ca.crt --cert=/etc/etcd/pki/etcdctl-etcd-client.crt --key=/etc/etcd/pki/etcdctl-etcd-client.key member list
{{< /tab >}}
{{< tab header="Bottlerocket" lang="bash" >}}
ETCD_CONTAINER_ID=$(ctr -n k8s.io c ls | grep -w "etcd-io" | cut -d " " -f1)
ETCD_CONTAINER_ID=$(ctr -n k8s.io c ls | grep -w "etcd-io" | cut -d " " -f1 | tail -1)
ctr -n k8s.io t exec -t --exec-id etcd ${ETCD_CONTAINER_ID} etcdctl \
--cacert=/var/lib/etcd/pki/ca.crt \
--cert=/var/lib/etcd/pki/server.crt \
Expand Down Expand Up @@ -153,7 +153,17 @@ ${IMAGE_ID} tmp-cert-renew \
{{< /tab >}}
{{< /tabpane >}}

3. If you have external etcd nodes, manually replace the `apiserver-etcd-client.crt` and `apiserver-etcd-client.key` file in `/etc/kubernetes/pki` (or `/var/lib/kubeadm/pki` in Bottlerocket) folder with the files you saved from any etcd node.
3. If you have external etcd nodes, manually replace the `server-etcd-client.crt` and `apiserver-etcd-client.key` file in `/etc/kubernetes/pki` (or `/var/lib/kubeadm/pki` in Bottlerocket) folder with the files you saved from any etcd node.
saiteja313 marked this conversation as resolved.
Show resolved Hide resolved

- **For Bottlerocket**:

```
cp apiserver-etcd-client.key /tmp/
cp server-etcd-client.crt /tmp/
sudo sheltie
cp /run/host-containerd/io.containerd.runtime.v2.task/default/admin/rootfs/tmp/apiserver-etcd-client.key /var/lib/kubeadm/pki/
cp /run/host-containerd/io.containerd.runtime.v2.task/default/admin/rootfs/tmp/server-etcd-client.crt /var/lib/kubeadm/pki/
```

4. Restart static control plane pods.

Expand Down Expand Up @@ -219,3 +229,62 @@ etcd:
- https://xxx.xxx.xxx.xxx:2379
- https://xxx.xxx.xxx.xxx:2379
```

### What do I do if my local kubeconfig has expired?

Your local kubeconfig used to interact with the cluster contains a certificate that expires after 1 year. When you rotate cluster certificates, a new kubeconfig with a new certificate is created as a Secret in the cluster. If you do not retrieve the new kubeconfig and your local kubeconfig certificate expires, you will receive the following error:
saiteja313 marked this conversation as resolved.
Show resolved Hide resolved

```
Error: Couldn't get current Server API group list: the server has asked for the client to provide credentials error: you must be logged in to the server.
This error typically occurs when the cluster certificates have been renewed or extended during the upgrade process. To resolve this issue, you need to update your local kubeconfig file with the new cluster credentials.
```

You can extract your new kubeconfig using the following steps.

1. SSH to one of the Control Plane nodes and run the following command to validate connection with API Server, export kubeconfig from `${CLUSTER_NAME}-kubeconfig` secret object (`eksa-system` namespace) using kubectl and copy kubeconfig file to `/tmp` directory.
saiteja313 marked this conversation as resolved.
Show resolved Hide resolved
saiteja313 marked this conversation as resolved.
Show resolved Hide resolved

```
ssh -i <YOUR_PRIVATE_KEY> <USER_NAME>@<YOUR_CONTROLPLANE_IP> # USER_NAME should be ec2-user for bottlerocket, ubuntu for Ubuntu ControlPlane machine Operating System

```

{{< tabpane >}}
{{< tab header="Ubuntu or RHEL" lang="bash" >}}

export CLUSTER_NAME="<YOUR_CLUSTER_NAME_HERE>"

cat /var/lib/kubeadm/admin.conf
export KUBECONFIG="/var/lib/kubeadm/admin.conf"
saiteja313 marked this conversation as resolved.
Show resolved Hide resolved

kubectl get secret ${CLUSTER_NAME}-kubeconfig -n eksa-system -o yaml > new-admin.kubeconfig
saiteja313 marked this conversation as resolved.
Show resolved Hide resolved

cat new-admin.kubeconfig > /tmp/new-admin-decoded.kubeconfig

{{< /tab >}}

{{< tab header="Bottlerocket" lang="bash" >}}

# You would need to be in the admin container when you ssh to the Bottlerocket machine
# open a root shell
sudo sheltie

cat /var/lib/kubeadm/admin.conf
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you mean to cut the cat /var/lib/kubeadm/admin.conf line, since you run it again a few lines down to redirect the content into another file. Likewise, the export KUBECONFIG... line just below here doesn't seem to be consumed anywhere, unless I'm missing something.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right. This KUBECONFIG export is not needed. removing it.

export KUBECONFIG="/var/lib/kubeadm/admin.conf"

cat /var/lib/kubeadm/admin.conf > /run/host-containerd/io.containerd.runtime.v2.task/default/admin/rootfs/tmp/new-admin.kubeconfig

{{< /tab >}}
{{< /tabpane >}}

2. From your admin machine, download the kubeconfig file from the ControlPlane node and use it to access your Kubernetes Cluster.

```
ssh <ADMIN_MACHINE_IP>

export CONTROLPLANE_IP=""
saiteja313 marked this conversation as resolved.
Show resolved Hide resolved
scp -i <keypair>@${CONTROLPLANE_IP}:/tmp/new-admin.kubeconfig .
saiteja313 marked this conversation as resolved.
Show resolved Hide resolved

ls -ltr
saiteja313 marked this conversation as resolved.
Show resolved Hide resolved
export KUBECONFIG="new-admin.kubeconfig"
saiteja313 marked this conversation as resolved.
Show resolved Hide resolved

kubectl get pods
Original file line number Diff line number Diff line change
Expand Up @@ -184,7 +184,7 @@ aws eks tag-resource \

## Delete Subscriptions

>**_NOTE_** Only inactive subscriptions can be deleted. Deleting inactive subscriptions removes them from the AWS Management Console view and API responses.
>**_NOTE_** Only inactive subscriptions can be deleted. Deleting inactive subscriptions removes them from the AWS Management Console view and API responses. To delete any Active Subscriptions, Please create a Support Case with AWS Support team.
saiteja313 marked this conversation as resolved.
Show resolved Hide resolved

### AWS Management Console

Expand Down