How to rename a cluster without losing data

[Kisuke-CZE]

Hi,

I would like to change cluster name (I have only one node in a ssh-proxy role). But I cannot find any working procedure how to do that.

I tried to stop teleport service. Edited /etc/teleport.yaml and changed names on all places, including:

auth_service:
  cluster_name: jumphost.example.net

But in web interface and tctl status cluster name remains teleport-proxy.example.net.

Is there any way to change that without loosing data? Cause if I remove /var/lib/teleport I will lose all data (users, their passwords and roles).

[webvictim]

Unfortunately there is no way to easily change the name of the cluster without losing data. This is because the cluster name is encoded into the identity of the certificate authorities within Teleport and changing it requires them to be regenerated from scratch.

Here's something you can try that might work.

1. Back up your existing users and roles to a YAML file like this:

sudo tctl get users --with-secrets > /tmp/backup.yaml
echo "---" | sudo tee -A /tmp/backup.yaml
sudo tctl get roles --with-secrets >> /tmp/backup.yaml

2. Move /var/lib/teleport out of the way to clear your cluster name/database:

sudo mv /var/lib/teleport /var/lib/teleport-old

3. Start Teleport up again as a fresh cluster, using the backup file as a bootstrap:

sudo teleport start --bootstrap /tmp/backup.yaml

This should create you a new cluster with the name as set in /etc/teleport.yaml.

At this point, you should be able to log in with your old username/password. Check the cluster name and it should be updated. All your old roles should also still exist.

This may not work, so just be warned.

[webvictim]

If your chart values have persistence.enabled: true (which is the default) then there is a PersistentVolumeClaim in the Kubernetes cluster which holds your data: https://github.com/gravitational/teleport/blob/v11.1.2/examples/chart/teleport-cluster/templates/deployment.yaml#L277-L282

You should still be able to run the tctl commands inside the Teleport pod (using kubectl exec) to get the backup file as per the instructions above.

[jullianow]

That's what I thought. But I don't use a local persistence. Just the empty one.
However, even using local persistence, I also use Firestore and Cloud Storage (my deployment is on GKE).
Could there be something in one of these two persistence that I need to clean up?

[webvictim]

If you're using Firestore then the users and roles will be persisted there. The process looks like this for Firestore:

• run the same tctl commands to pull the data out
• shut down your Teleport cluster
• then either:
  • delete (or rename) the Firestore collections for backend and audit log
  • or change the names used in Teleport's config
• restart Teleport using the --bootstrap command above

[jullianow]

Thanks

[tukusejssirs]

I have checked the output of tctl get all --with-secrets. I think that we can backup all kind configs except for cert_authority (I get users, roles and github kinds).

I wish there would be an option to:

• option A:
  • get a list of kinds currently configured (e.g. tctl get kinds; the output could be a comma-separated list):
    • also include an option to exclude a specific kind (e.g. -K, --exclude-kinds=KIND, where KIND is a comma-separated list of kinds);
  • get multiple kinds without the need to get them in separate commands and them merge them (e.g. tctl get users,roles,github);
• option B:
  • get all kinds with an option to exclude a kind (e.g. tctl get all -K cert_authority).

---

Until the support for the above is added to tctl, my backup instructions are as follows:

file='/tmp/backup.yml'
readarray -t kinds <<< "$(sudo tctl get all | grep -Po '^kind: \K.*$' | sort | uniq | grep -v 'cert_authority')"

for i in "${!kinds[@]}"; do
  # Make sure `file` is empty before backing the data up
  [ i = 0 ] && echo -n > "$file"
  
  # Back the current kind up
  sudo tctl get "${kinds[i]}" --with-secrets >> "$file"
  
  # Append the separator if the currently backed up kind is not the last one 
  [ "${kinds[i]}" != "${!kinds[-1]}" ] && echo '---' >> "$file"
done

Regarding restoration in Docker, I created a temporary Docker container in which I overrode the entrypoint, recreated Teleport data, stopped and removed the machine and started the main container. Update whatever is different in your case (container name, backup filename and path, …).

# Stop Teleport cluster if it is running
docker stop teleport

# Remove (or rename) the Teleport data if it exists
rm -rf /root/teleport/data

# Start Teleport up again as a fresh cluster, using the backup file as a bootstrap
# Note: The backup file should be located in '/root/teleport/config/backup.yml', otherwise you need to modify the `--entrypoint` value below.
# TODO: This container will run in the foreground. Kill it after it starts up (you can verify if users, logins and GitHub connector are added using web ui or `tctl`).
docker run -it \
  --rm \
  --restart no \
  --network host \
  -v /root/teleport/config:/etc/teleport:ro \
  -v /root/teleport/data:/var/lib/teleport:rw \
  --entrypoint '/usr/bin/dumb-init' \
  public.ecr.aws/gravitational/teleport:12 \
  /usr/local/bin/teleport start -c /etc/teleport/teleport.yaml --bootstrap /etc/teleport/backup.yml

# Start `teleport` service
docker start teleport

[mind-drive]

How about the sessions and recording?
How can I back it up?

[webvictim]

Assuming you're using local storage, you should be able to keep a backup of the /var/lib/teleport/log directory which contains both audit logs and session recordings, and put this back in place once you've set up the new cluster name.