A Conode is a Collective Authority Node and is a server in the cothority. Conodes are linked together to form a cothority. They can run decentralized protocols, and to offer services to clients.
The conode in this repository includes all protocols and services and can be run either for local tests or on a public server. The currently running conodes are available under http://status.dedis.ch.
To operate a Conode, one needs to correctly set up a host and run the Conode program. The following chapters describe the requirements and needed environment to correctly run the conode program, as well as general instruction on how to operate it.
Once you have a conode up and running, you can inform us at dedis@epfl.ch and we will include your conode in the DEDIS-cothority.
📖 Table of Contents
- Introduction
- Running a conode
- Maintaining a conode
- For the lazy ones: A survival guide to install your server with Ubuntu 18.04
- 24/7 availability
- 512MB of RAM and 1GB of disk-space
- a public IP-address and two open ports
Two distinct communication schemes need to be configured:
- conode-conode communication, and
- Client to conode communication.
The conode-conode communication happens for consensus-based transactions in the cothority, for example when the Conodes need to reach a consensus to store a transaction on the skip-chain. The client-conode communication happens when something from the outside of cothority performs a query. It happens when someone wants to store a value on the skip-chain. In this case, the client will contact the cothority via the client-conode communication scheme.
As we will discover later, conode-conode communication is automatically secured
via TLS when you use the configuration from conode setup
unchanged. However,
conode-client communication happens on the next port up from the conode-conode
port and it defaults to WebSockets inside of HTTP. Therefore, it is highly
recommended to arrange for this port to be wrapped in TLS as well.
The current recommended way to add HTTPS to the WebSocket port is to use a web server like Apache or Nginx in reverse proxy mode to forward connections from port 443 to the WebSocket port, which is the conode's port plus 1.
Here is an example config for Apache using a Let's Encrypt certificate:
<IfModule mod_ssl.c>
<VirtualHost *:443>
ServerName excellent.example.com
# If conode is running on port 7000, non-TLS WebSocket is on 7001,
# so the reverse proxy points there.
ProxyPass / ws://localhost:7001/
SSLCertificateFile /etc/letsencrypt/live/excellent.example.com/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/excellent.example.com/privkey.pem
Include /etc/letsencrypt/options-ssl-apache.conf
</VirtualHost>
</IfModule>
And here is a version with Nginx:
location / {
server_name example.com;
# ...
ssl_certificate /etc/nginx/ssl/example.com.certificate.pem;
ssl_certificate_key /etc/nginx/ssl/example.com.key.pem;
# ...
location /conode/ {
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_pass "http://localhost:7771/";
}
}
As we will see later during the configuration phase, we will have to advertise
this configuration in the public.toml
configuration file with the URL
field.
In this case, the configuration would be URL ="https://excellent.example.com"
for Apache and URL = "example.com/conode"
for Nginx. More on that later.
During the setup phase, the conode program creates its public/private key and prompts the user with some questions:
- PORT: the indicated port and port+1 to be used for communication.
- IP-address: if it cannot detect your IP-address, it will ask for it. This usually means that something is wrong. Perhaps you didn't allow your firewall to accept incoming connections.
- Description: any description you want to share with the world.
- Folder: press
<enter>
for the default folder.
Once the interactive setup is done, the program has created two configuration files:
- The public configuration file (public.toml), which holds the public key, network information, and a description. This file is the one that should be sent to other cothority operators to request access to the cothority.
- The private configuration file (private.toml), which holds the server config, including the private key and network configuration, like the server's public address on the network. The server will listen to this port, as well as to this port + 1 (for conode-conode and conode-client connections, respectively).
Warning: Never (!!!) share the file private.toml
with anybody, as it
contains the private key of your conode.
There are two options to run the configuration setup: using the command line or using docker.
First, we need to get the conode program. The recommended way for getting the conode program is to download it from the official releases on GitHub (replace with the latest version):
$ wget https://github.com/dedis/cothority/releases/download/v3.1.3/conode-v3.1.3.tar.gz
$ tar -xvf conode-v3.1.3.tar.gz
Otherwise, you can build it from the sources if you have the latest version of go:
$ git clone https://github.com/dedis/cothority.git
$ cd conode
$ go install ./conode
Then, you can launch the interactive setup configuration by running
$ conode setup
Once done, the configuration files (the public.toml and private.toml) are saved in the default location or in the one specified during the interactive setup.
The default locations are the following depending on the operating system:
- Linux:
$HOME/.config/conode
- MacOS:
$HOME/Library/Application Support/conode
- Windows:
%AppData%\Conode
If you have docker installed and running on your host, you can fire the interactive setup with the following command:
$ mkdir ~/conode_data
$ docker run -it --rm -p 7770-7771:7770-7771 --name conode -v ~/conode_data:/conode_data dedis/conode:latest ./conode setup
This will prompt the interactive setup and set the 2 configuration files in the
~/conode_data
directory.
There are a few things you can do now that you have your configuration files.
This first one is to update the URL
field in the public.toml file in case you
set up a WebSocket over TLS connection for the conode-client communication
during the Environment setup.
Another option concerns the conode-conode communication that runs on TLS. If you
would like the conode to run TLS on the WebSocket interface, you can tell it
where to find the private key and a certificate for that key in the
private.toml
file with the following fields:
WebSocketTLSCertificate = "/etc/fullchain.pem"
WebSocketTLSCertificateKey = "/etc/privkey.pem"
In this case, it is up to you to get get a certificate from a certificate
authority and to update fullchain.pem
when needed to renew the certificate.
Using the Let's Encrypt CA and the certbot
client program, you can get free
certificates for domains which you control. certbot
writes the files it
creates into /etc/letsencrypt
.
If the user you use to run the conode has the rights to read from the directory where Let's Encrypt writes the private key and the current certificate, you can arrange for conode to share the TLS certificate used by the server as a whole:
WebSocketTLSCertificate = "/etc/letsencrypt/live/conode.example.com/fullchain.pem"
WebSocketTLSCertificateKey = "/etc/letsencrypt/live/conode.example.com/privkey.pem"
Let's Encrypt certificates expire every 90 days, so you will need
to restart your conode when the fullchain.pem
file is refreshed.
Once the setup is done with one of the two options, you can finally run your conode depending on the previously chosen option.
Simply launch the following:
$ conode server
It is recommended to use more verbose logging and this can be done with the -d
option. It is also possible to specify a different location than the default one
for the private.toml file with the -c
option. If you used Docker to set up your
conode, then the private.toml file is not in the default location. You can then
use:
$ conode -d 2 -c ~/conode_data/private.toml server
You can use the -h
for a description of the available commands.
If you want to run the server in the background, you can use the screen
program:
$ screen -S conode -d -m conode server
To enter the screen, type screen -r conode
. You can detach from it with
<ctrl-a> d
.
Note: The logs are not saved by default. If you want to keep a trace of the
logs, it is recommended to use tee
:
$ conode server | tee logfile.txt
Type the following to start the conode program with docker:
$ docker run --restart always -d -p 7770-7771:7770-7771 --name conode -v ~/conode_data:/conode_data dedis/conode:latest
Because it will run detached, you can use docker logs -f conode
to see the
logs. It will be restarted on the next boot as well. To stop it, you can use
docker stop
. Then, you must remove it with docker rm <id>
(the id is found
with docker ps -a
).
Pro tip: you can use the following options for docker in order to gracefully
handle the logs and prevent a disk saturation:
--log-opt max-size=10m --log-opt max-file=4 --log-opt compress=true
If you have systemd, you can simply copy the conode.service
file and add it to
your systemd-startup. Of course, you should do this as a non-root user:
$ wget https://raw.githubusercontent.com/dedis/cothority/conode/conode.service
$ systemctl --user enable conode.service
$ systemctl --user start conode
Unfortunately, systemd doesn't allow a user to run a service at system startup, and all user services get stopped once the user logs out!
For development purposes, the run_nodes.sh
script can be used to launch
multiple conodes. For example, the following command:
$ ./run_nodes -d tmp -v 2 -n 5
will run 5 conodes and save their files in the tmp directory with verbosity of 2. The file containing all the public configurations will be in "tmp/public.toml".
There are two important files to backup:
- The
private.toml
file containing the conode's configuration along with its private key, and - the
<id>.db
file containing the database, where<id>
is in fact the sha256 of the public keys.
On linux, the private.toml
file is located in
$HOME/.config/conode/private.toml
, and the database file is in
$HOME/.local/share/conode/<id>.db
.
When using docker, everything is stored in $HOME/conode_data
.
The DB file is a BoltDB file, and more information about considerations while backing them up is in Database backup.
If you have a backup of the private.toml file and a recent backup of the .db file, you can put them onto a new server, and start the conode. The IP address in the private.toml file must match the IP address on the server.
To facilitate IP address switches, it is recommended that the public IP address for the leader of critical skipchains should be a virtual address. For example, if you have two servers:
- 10.0.0.2 conode-live, also with secondary address 10.0.0.1
- 10.0.0.3 conode-standby
You can keep both servers running, and use scp to move the DB file from
conode-live to conode-standby. Both servers should have the same private.toml
file, which includes the line Address = "tcp://10.0.0.1:7770"
If conode-live is down and unrecoverable, you can add 10.0.0.1 as a secondary address to conode-standby and start the conode on it. From this moment on, you must be sure that conode-live does not boot, or if it does, that it does not have the secondary address on it anymore. You could do so by not adding the secondary address to boot-time configs and only move it manually.
The address 10.0.0.1 will be in the Roster of any skipchains, and nodes which are following that skipchain will still be able to contact the leader, even if it is now running on a different underlying server.
Note: The address part of a server identity has name resolution applied to it. Thus it would be possible to set the roster of a skipchain to include a server identity like "tcp://conode-main.example.com:6979" and then change the definition of conode-main.example.com in DNS to change the IP address of the main.
You can check if the configuration file is correct with:
conode -d 3 check ~/.config/conode/public.toml
You can start multiple nodes on the same server by using one user per node and set up the nodes as described above. Be sure to change the port-numbers and remember that two ports are used.
For most of the apps, you need at least 3 running nodes. Once you have them up
and running, you will need a roster.toml
that includes all the
public.toml
-files from your conodes:
cat ../*/conode_data/public.toml > roster.toml
The only existing cothority for the moment is available at http://status.dedis.ch. You can send us an email at dedis@epfl.ch to be added to this list.
To create your docker-image and use it, you can create it like this:
$ go get github.com/dedis/cothority
$ cd $(go env GOPATH)/src/github.com/dedis/cothority/conode
$ make docker
If you use make docker_run
the first time, a directory called conode_data
will be created and you will be asked for a port - use 7770 or adapt the
Makefile - and a description of your node. Your public and private key for the
conode will be stored in conode_data
. If you run make docker_run
again, the
stored configuration will be used.
To stop the docker, simply run make docker_stop
or kill the docker-container.
All the configuration is stored in conode_data
For the latest and greatest version of the conode, you can replace
conode:latest
with conode:dev
and you should get a stable, but changing
conode. This means, that to use all the functionalities you need to update the
apps and follow the latest conode:dev
container regularly.
For creating a new docker image, there are two commands:
make docker_dev
- creates a docker image with the currently checked out versions on your machine.make docker BUILD_TAG=v3.0.0-pre1
- creates a docker image from source at tag BUILD_TAG.
In this section, we provide "as-is" instructions to set up a conode server from scratch on Ubuntu 18.04 witch nginx, letsencrypt, and docker. We assume that you start with a fresh install and are logged as root.
Update and set up SSH
# Update
$ sudo apt update
$ sudo apt upgrade
# Set up a new user, staying as root is frightening
$ adduser deployer
$ usermod -aG sudo deployer
$ su deployer
$ sudo apt-get install vim
$ sudo vim /etc/ssh/sshd_config
# We update the ssh config to improve security. Here are the things we update:
> Port 44
> PermitRootLogin no
> PasswordAuthentication no
> UsePAM no
# SSH is only allowed with pub/priv key. We must then allow us to use our key:
$ mkdir ~/.ssh # if this is a fresh account, the folder might not exist yet
$ vim ~/.ssh/authorized_keys
> *add pub key*
# Allow connecting with the new SSH port
$ sudo ufw allow 44
$ sudo service ssh restart
Install docker
Those directly come from the official guide.
$ sudo apt-get install \
apt-transport-https \
ca-certificates \
curl \
gnupg-agent \
software-properties-common
$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
$ sudo apt-key fingerprint 0EBFCD88
> 9DC8 5822 9FC7 DD38 854A E2D8 8D81 803C 0EBF CD88
$ sudo add-apt-repository \
"deb [arch=amd64] https://download.docker.com/linux/ubuntu \
$(lsb_release -cs) \
stable"
$ sudo apt-get update
$ sudo apt-get install docker-ce docker-ce-cli containerd.io
$ sudo docker run hello-world
Install Nginx
$ sudo apt install nginx
$ sudo ufw allow "Nginx Full"
$ sudo ufw status
$ systemctl status nginx
Setup Nginx virtual host
# First create the location where the website will be stored:
$ sudo mkdir -p /var/www/example.com/public_html
$ sudo chown -R $USER:$USER /var/www/example.com/public_html/
$ vim /var/www/example.com/public_html/index.html
> *insert what you like*
# Create the virtual host from the default one
$ sudo cp /etc/nginx/site-available/default /etc/nginx/site-available/example.com
$ sudo vim /etc/nginx/site-available/example.com
> *update for the domain and location we created (do not need to setup ssl yet)*
# Disable the default host
$ sudo rm /etc/nginx/sites-enabled/default
# Activate our new virtual host
$ sudo ln -s /etc/nginx/sites-available/example.com /etc/nginx/sites-enabled/
$ sudo service nginx reload
Here is a complete example of an Nginx config (without using letsencrypt):
server {
listen 80 default_server;
listen [::]:80 default_server;
server_name example.com www.example.com;
return 301 "https://example.com${request_uri}";
}
server {
listen 443 default_server ssl;
listen [::]:443 default_server ssl;
ssl on;
ssl_certificate /etc/nginx/ssl/example.com.certificate.pem;
ssl_certificate_key /etc/nginx/ssl/example.com.key.pem;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_prefer_server_ciphers on;
ssl_ciphers 'EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH';
root /var/www/example.com/public_html/;
index index.html index.htm index.nginx-debian.html;
server_name example.com www.example.com;
location / {
try_files $uri $uri/ =404;
}
location /conode/ {
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_pass "http://localhost:7771/";
}
}
Install letsencrypt
$ sudo add-apt-repository ppa:certbot/certbot
$ sudo apt install python-certbot-nginx
$ sudo certbot --nginx -d example.com
$ sudo certbot renew --dry-run
Setup Nginx for the conode
$ sudo vim /etc/nginx/site-available/swisscloud.cothority.net
> *add the following in the main server block:*
location /conode/ {
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_pass "http://localhost:7771/";
}
Use docker without root
$ sudo groupadd docker
$ sudo gpasswd -a $USER docker
$ sudo service docker restart
Run the conode
$ sudo ufw allow 7770
$ mkdir conode_data
$ docker run -it --rm -p 7770-7771:7770-7771 --name conode -v ~/conode_data:/conode_data dedis/conode:latest ./conode setup
$ vim ~/conode_data/public.toml
> *update the URL field*
$ docker run --restart always -d -p 7770-7771:7770-7771 --name conode -v ~/conode_data:/conode_data --log-opt max-size=10m --log-opt max-file=4 --log-opt compress=true dedis/conode:latest
Run watchtower
Watchtower automatically updates the container if it finds a new version.
$ docker run -d \
--name watchtower \
-v /var/run/docker.sock:/var/run/docker.sock \
containrrr/watchtower