nodEvac

---

A Ganeti cluster node evacuation tool.

The main idea of this tool is to give someone (more specifically a technician)
a Web UI to evacuate a node from a Ganeti cluser and poweroff the machine
without requiring SSH access to the hosts.

Written in Flask and uses Celery for async tasks (with Redis as a broker).

Assumptions

This project makes the following assumptions:

• You have one or more Ganeti clusters
• You have a Rapi user for each of the clusters and the RAPI port is accesible
• You have access to the IPMI interface of each machine in order to issue poweroff/on commands
• The hosts are pingable from the server the project runs in order to detect host status
• Alls hosts share the same IPMI credentials
• The IPMI hostname is assigned on each node with an ipmi: Ganeti tag
• Any non mirrored VM on the cluster (file,plain templates) will be shutdown during evacuation (not implemented yet)

Limitations

Current limitations (mainly Ganeti's) that I would like to find a workaround for:

• You can't failover/demote a master node (this is by design)
  • This means we can't shutdown a master node
• You can't properly readd a node to the cluster - we fallback to modify --offline no which is not clean

Instalation

• Clone the project

git clone https://github.com/alexkiousis/nodEvac.git

• Create a virtualenv

virtualenv .venv
source .venv/bin/activate

• Install requirements

pip install -r requirements.txt

• Install redis

apt-get install redis-server

• Install ipmitool

apt-get install ipmitool

• Create a config file named cluster_config.py in the root of the repository with the Ganeti credentials
  • See ganeti_utils.py for more clear instructions
• Run celery

celery -A tasks.celery_app_worker --log-level=info -E --concurrency=10

• Run nodEvac

./nodEvac.py

Dependencies

Full dependency state is in the requirements.txt file.

These are the explanations for the root dependencies:

• Flask: the backend software the project runs on
• Celery: a worker queue for running asyncronus jobs
  • all ganeti jobs and ipmitool commands run through celery
• redis: library to communicate with a redis instance
  • used as a broker for celery and for job state caching for the application
• pycurl and simplejson: are dependencies for the Ganeti rapi client library in contrib