Skip to content

Commit

Permalink
Merge pull request #33 from gfieni/feat-kadeploy-api
Browse files Browse the repository at this point in the history
Major rework for new api version use
  • Loading branch information
gfieni authored Jun 21, 2019
2 parents e3318b7 + 005203a commit 1e91684
Show file tree
Hide file tree
Showing 9 changed files with 680 additions and 363 deletions.
98 changes: 55 additions & 43 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,29 +1,30 @@
[![CircleCI](https://circleci.com/gh/Spirals-Team/docker-machine-driver-g5k.svg?style=svg)](https://circleci.com/gh/Spirals-Team/docker-machine-driver-g5k)
[![CircleCI](https://circleci.com/gh/Spirals-Team/docker-machine-driver-g5k.svg?style=shield)](https://circleci.com/gh/Spirals-Team/docker-machine-driver-g5k)
[![GitHub release](https://img.shields.io/github/release/Spirals-Team/docker-machine-driver-g5k.svg)](https://github.com/Spirals-Team/docker-machine-driver-g5k/releases)
[![License](https://img.shields.io/github/license/Spirals-Team/docker-machine-driver-g5k.svg)](https://opensource.org/licenses/Apache-2.0)

# docker-machine-driver-g5k
A Docker Machine driver for the Grid5000 testbed infrastructure. It can be used to provision a Docker machine on a node of the Grid5000 infrastructure.
[Docker Machine](https://docs.docker.com/machine/install-machine) driver for the [Grid'5000 testbed infrastructure](https://www.grid5000.fr/).

## Requirements
* [Docker](https://www.docker.com/products/overview#/install_the_platform)
* [Docker Machine](https://docs.docker.com/machine/install-machine)
* [Go tools (Only for installation from sources)](https://golang.org/doc/install)
* [Go tools (only for installation from sources)](https://golang.org/doc/install)

You need a Grid5000 account to use this driver. See [this page](https://www.grid5000.fr/mediawiki/index.php/Grid5000:Get_an_account) to create an account.
You need a Grid'5000 account to use this driver. See [this page](https://www.grid5000.fr/mediawiki/index.php/Grid5000:Get_an_account) to create an account.

## VPN
**You need to be connected to the Grid5000 VPN to create and access your Docker node.**
**You need to be connected to the Grid'5000 VPN to create and access your Docker node.**
**Do not forget to configure your DNS or use OpenVPN DNS auto-configuration.**
**Please follow the instructions from the [Grid5000 Wiki](https://www.grid5000.fr/mediawiki/index.php/VPN).**
**Please follow the instructions from the [Grid'5000 Wiki](https://www.grid5000.fr/mediawiki/index.php/VPN).**

## Installation from GitHub releases
Binary releases for Linux, MacOS and Windows using x86/x86_64 CPU architectures are available in the [releases page](https://github.com/Spirals-Team/docker-machine-driver-g5k/releases).
Binary releases for Linux, MacOS and Windows are available in the [releases page](https://github.com/Spirals-Team/docker-machine-driver-g5k/releases).
On Linux and MacOS, you can use the following commands to install or upgrade the driver:
```bash
# download the binary for your OS and CPU architecture :
# download the binary for your OS and CPU architecture:
sudo curl -L -o /usr/local/bin/docker-machine-driver-g5k "<link to release>"

# grant execution rigths to the driver for everyone :
# grant execution rigths to the driver for everyone:
sudo chmod +x /usr/local/bin/docker-machine-driver-g5k
```

Expand All @@ -45,36 +46,36 @@ export PATH=$PATH:$GOPATH/bin
### Driver-specific command line flags

#### Flags description
* **`--g5k-username` : Your Grid5000 account username (required)**
* **`--g5k-password` : Your Grid5000 account password (required)**
* **`--g5k-username` : Your Grid'5000 account username (required)**
* **`--g5k-password` : Your Grid'5000 account password (required)**
* **`--g5k-site` : Site where the reservation of the node will be made (required)**
* `--g5k-walltime` : Duration of the resource reservation (in "HH:MM:SS" format)
* `--g5k-walltime` : Duration of the resource reservation (in `HH:MM:SS` format)
* `--g5k-image` : Name of the system image to deploy on the node
* `--g5k-resource-properties` : [Resource selection with OAR properties](#resource-properties)
* `--g5k-make-resource-reservation` : [Make a resource reservation for the given start date](#resource-reservation)
* `--g5k-use-resource-reservation` : [Use a resource reservation (need to be an existing job ID)](#resource-reservation)
* `--g5k-skip-vpn-checks` : Skip the VPN client connection and DNS configuration checks
* `--g5k-select-node-from-reservation` : [Hostname of the node to use from the reservation](#resource-reservation)
* `--g5k-reuse-ref-environment` : [Reuse the Grid'5000 reference environment instead of re-deploying the node](#grid5000-reference-environment-reuse)
* `--g5k-job-queue` : [Specify the job queue (besteffort queue is NOT supported)](#job-queues)
* `--g5k-job-queue` : [Specify the job queue (the `besteffort` queue is NOT supported)](#job-queues)
* `--g5k-external-ssh-public-keys` : SSH public key(s) allowed to connect to the node (in authorized_keys format)
* `--g5k-keep-resource-at-deletion` : [Keep the allocated resource when removing the machine](#resource-reservation)

#### Flags usage
| Option | Environment | Default value |
|-----------------------------------|---------------------------------|-----------------------|
| `--g5k-username` | `G5K_USERNAME` | |
| `--g5k-password` | `G5K_PASSWORD` | |
| `--g5k-site` | `G5K_SITE` | |
| `--g5k-walltime` | `G5K_WALLTIME` | "1:00:00" |
| `--g5k-image` | `G5K_IMAGE` | "debian9-x64-std" |
| `--g5k-resource-properties` | `G5K_RESOURCE_PROPERTIES` | |
| `--g5k-make-resource-reservation` | `G5K_MAKE_RESOURCE_RESERVATION` | |
| `--g5k-use-resource-reservation` | `G5K_USE_RESOURCE_RESERVATION` | |
| `--g5k-skip-vpn-checks` | `G5K_SKIP_VPN_CHECKS` | False |
| `--g5k-reuse-ref-environment` | `G5K_REUSE_REF_ENVIRONMENT` | False |
| `--g5k-job-queue` | `G5K_JOB_QUEUE` | "default" |
| `--g5k-external-ssh-public-keys` | `G5K_EXTERNAL_SSH_PUBLIC_KEYS` | |
| `--g5k-keep-resource-at-deletion` | `G5K_KEEP_RESOURCE_AT_DELETION` | False |
| Flag name | Environment variable | Default value |
|--------------------------------------|------------------------------------|-----------------------|
| `--g5k-username` | `G5K_USERNAME` | |
| `--g5k-password` | `G5K_PASSWORD` | |
| `--g5k-site` | `G5K_SITE` | |
| `--g5k-walltime` | `G5K_WALLTIME` | "1:00:00" |
| `--g5k-image` | `G5K_IMAGE` | "debian9-x64-std" |
| `--g5k-resource-properties` | `G5K_RESOURCE_PROPERTIES` | |
| `--g5k-make-resource-reservation` | `G5K_MAKE_RESOURCE_RESERVATION` | |
| `--g5k-use-resource-reservation` | `G5K_USE_RESOURCE_RESERVATION` | |
| `--g5k-select-node-from-reservation` | `G5K_SELECT_NODE_FROM_RESERVATION` | |
| `--g5k-reuse-ref-environment` | `G5K_REUSE_REF_ENVIRONMENT` | False |
| `--g5k-job-queue` | `G5K_JOB_QUEUE` | "default" |
| `--g5k-external-ssh-public-keys` | `G5K_EXTERNAL_SSH_PUBLIC_KEYS` | |
| `--g5k-keep-resource-at-deletion` | `G5K_KEEP_RESOURCE_AT_DELETION` | False |

#### Resource properties
You can use [OAR properties](http://oar.imag.fr/docs/2.5/user/usecases.html#using-properties) to only select a node that matches your hardware requirements.
Expand All @@ -89,12 +90,12 @@ More information about usage of OAR properties are available on the [Grid'5000 W
#### Resource reservation
You can either do a job submission to reserve resources as soon as possible (this is the default mode) or do an advance reservation for a specific date/time.

To do a resource reservation, you need to use the `--g5k-make-resource-reservation` flag and provide a starting date/time in either the 'YYYY-MM-DD HH:MM:SS' date format or an UNIX timestamp.
To do a resource reservation, you need to use the `--g5k-make-resource-reservation` flag and provide a starting date/time in either the `YYYY-MM-DD HH:MM:SS` date format or an UNIX timestamp.
Don't forget to save the job ID of your reservation in order to be able to create a machine when the resources will be available.

To use a resource reservation, set the `--g5k-use-resource-reservation` flag with the job ID of an existing reservation.
This will create a machine, deploy an OS image and provision Docker on the node.
Please note that the job must be in `running` state in order for the machine to be created, otherwise the driver will wait until the job start.
In case the reservation have multiple nodes, you need to select one using the `--g5k-select-node-from-reservation` flag, otherwise the first node will be taken.
This will create a machine, deploy an OS image and provision Docker on the node. Please note that the job must be in `running` state in order for the machine to be created, otherwise the driver will wait until the job start.

By default the resource is automatically deallocated when you remove a machine using the `rm` command.
However, you can use the `g5k-keep-resource-at-deletion` flag when creating the machine to keep the resource allocated even when the machine is removed.
Expand All @@ -120,18 +121,17 @@ Error with pre-create check: "Error when submitting new job: The server returned
See [this page](https://www.grid5000.fr/mediawiki/index.php/Grid5000:UsagePolicy#Rules_for_the_production_queue) for more information about the production queue.

### Usage examples
An example of node provisioning reusing the Grid'5000 standard environment:
An example reusing the Grid'5000 standard environment:
```bash
docker-machine create -d g5k \
--g5k-username "user" \
--g5k-password "********" \
--g5k-site "lille" \
--engine-storage-driver "overlay2" \
--g5k-reuse-ref-environment \
test-node
```

An example of node provisioning deploying the `debian9-x64-min` environment on the node:
An example deploying the `debian9-x64-min` environment image on the node:
```bash
docker-machine create -d g5k \
--g5k-username "user" \
Expand All @@ -141,15 +141,15 @@ docker-machine create -d g5k \
test-node
```

An example of node provisioning using environment variables:
An example using environment variables to configure the driver:
```bash
export G5K_USERNAME="user"
export G5K_PASSWORD="********"
export G5K_SITE="lille"
docker-machine create -d g5k test-node
```

An example with resource properties (node in cluster `chimint` with more thant 8GB of RAM and at least 4 CPU cores):
An example using resource properties (node in `chimint` cluster having more than 8GB of RAM and at least 4 CPU cores):
```bash
docker-machine create -d g5k \
--g5k-username "user" \
Expand All @@ -159,7 +159,7 @@ docker-machine create -d g5k \
test-node
```

An example of doing a resource reservation of 1 node for `8 hours` starting the `2019-01-01` at `20:00:00`:
An example doing a resource reservation of 1 node for `8 hours` starting the `2019-01-01` at `20:00:00`:
```bash
docker-machine create -d g5k \
--g5k-username "user" \
Expand All @@ -170,22 +170,34 @@ docker-machine create -d g5k \
test-node
```

An example of using a resource reservation:
An example using the first node of a resource reservation having the `1234567` job ID:
```bash
docker-machine create -d g5k \
--g5k-username "user" \
--g5k-password "********" \
--g5k-site "lille" \
--g5k-use-resource-reservation 1234567 \
--g5k-use-resource-reservation "1234567" \
test-node
```
```

An example adding multiple external SSH keys (your keys can be of any supported format, and may be longer than the following example):
An example using the `chifflet-2.lille.grid5000.fr` node of a resource reservation having the `1234567` job ID:
```bash
docker-machine create -d g5k \
--g5k-username "user" \
--g5k-password "********" \
--g5k-site "lille" \
--g5k-use-resource-reservation "1234567" \
--g5k-select-node-from-reservation "chifflet-2.lille.grid5000.fr" \
test-node
```

An example adding two external SSH keys (the keys can be of any supported type and size):
```bash
docker-machine create -d g5k \
--g5k-username "user" \
--g5k-password "********" \
--g5k-site "lille" \
--g5k-external-ssh-public-keys "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFLs3JzUYn7LbHE+SzJNoMvYbasnhjlen0k6dFs801DT test-ed25519" \
--g5k-external-ssh-public-keys "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgQC5qQt/nzGW19uCb9CDVEvP93LZ2mu3rd7drPP1nLf1pzLwlL2U2ksfwDCjMWU0P7KA6tB4scI+4dhxj07t0Z8g4TsMGYhbG0kjf7tWN73DombB4h/zobo2GvVoMg0NBLTP4peXLYAEofTYc0g7OWtJicAzLwcMzHsitDjjBwCKHQ== test-rsa"
--g5k-external-ssh-public-keys "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgQC5qQt/nzGW19uCb9CDVEvP93LZ2mu3rd7drPP1nLf1pzLwlL2U2ksfwDCjMWU0P7KA6tB4scI+4dhxj07t0Z8g4TsMGYhbG0kjf7tWN73DombB4h/zobo2GvVoMg0NBLTP4peXLYAEofTYc0g7OWtJicAzLwcMzHsitDjjBwCKHQ== test-rsa" \
test-node
```
28 changes: 24 additions & 4 deletions api/api.go
Original file line number Diff line number Diff line change
@@ -1,12 +1,15 @@
package api

import (
"net/url"
gopath "path"

"github.com/go-resty/resty"
)

const (
// G5kAPIFrontend is the link to the Grid'5000 API frontend
G5kAPIFrontend = "https://api.grid5000.fr/stable"
g5kAPIhostname string = "api.grid5000.fr"
g5kAPIversion string = "4.0"
)

// Client is a client to the Grid'5000 REST API
Expand All @@ -25,9 +28,26 @@ func NewClient(username, password, site string) *Client {
}
}

// Request returns a configured resty request
func (c *Client) Request() *resty.Request {
// getRequest returns a configured resty request
func (c *Client) getRequest() *resty.Request {
return resty.R().
SetHeader("Accept", "application/json").
SetBasicAuth(c.Username, c.Password)
}

// getBaseURL returns the Grid'5000 API base url
func (c *Client) getBaseURL() *url.URL {
return &url.URL{
Scheme: "https",
Host: g5kAPIhostname,
Path: gopath.Join(g5kAPIversion, "sites", c.Site),
}
}

// getEndpoint construct and returns the API endpoint for the given api name and path
func (c *Client) getEndpoint(api string, path string, params url.Values) string {
url := c.getBaseURL()
url.Path = gopath.Join(url.Path, api, path)
url.RawQuery = params.Encode()
return url.String()
}
85 changes: 0 additions & 85 deletions api/deployment.go

This file was deleted.

Loading

0 comments on commit 1e91684

Please sign in to comment.