Merge pull request #824 from gkvijay/v2plugin

configure controlURL and ctrl_ip for v2plugin
contiv · Apr 14, 2017 · d225f91 · d225f91
2 parents aef4c74 + b092b82
commit d225f91
Show file tree

Hide file tree

Showing 4 changed files with 100 additions and 20 deletions.
diff --git a/install/v2plugin/README.md b/install/v2plugin/README.md
@@ -0,0 +1,77 @@
+# Docker 1.13/17.03 v2plugin support
+
+Docker 1.13/17.03 supports legacy plugins (binaries/containers) and docker managed plugins (v2plugin) using docker plugin commands. Contiv binaries (netplugin and netmaster) and Contiv container (contiv/netplugin) support legacy plugin mode. In addition, Contiv can be run as v2plugin (contiv/v2plugin).
+
+## Classic Swarm and Swarm-mode
+In Classic Swarm or Legacy Docker Swarm, swarm binary/container running on each host forms a cluster using an external key-value store. Network plugins can run in the legacy mode. Docker versions upto 1.12 supported only this mode for remote drivers. In this mode docker engine is started with a cluster-store option. Docker 1.13.1 also supports this mode and the plugin can be run legacy plugin or v2plugin. Contiv binaries/containers will be supported in this legacy mode. 
+
+Docker introduced swarm-mode in 1.12 where the docker engines form a cluster without using an external key-value store. Swarm-mode in Docker 1.12 only supported docker overlay network driver. In this mode the swarm is initialized using docker swarm commands. From Docker 1.13.1, remote network drivers implemented as v2plugins are also supported in swarm-mode. Contiv v2plugin support for docker swarm-mode is still in progress.
+
+## v2plugin 
+Docker managed [plugins](https://docs.docker.com/engine/extend/) are run as runc containers and are managed using docker plugin commands. Docker engine running in [swarm-mode](https://docs.docker.com/engine/swarm/) requires the remote drivers to implement v2plugin architecture.
+
+## Contiv plugin install
+Contiv plugin config options should be specified if it is different from default:
+docker plugin install contiv/v2plugin:0.1 ARG1=VALUE1 ARG2=VALUE2 ...
+```
+ARG           : DESCRIPTION                                   : DEFAULT VALUE
+--------------:-----------------------------------------------:----------------------
+iflist        : VLAN uplink interface used by OVS             : ""
+cluster_store : Etcd or Consul cluster store url              : etcd://localhost:2379
+ctrl_ip       : Local IP address to be used by netplugin      : none
+                for control communication
+vtep_ip       : Local VTEP IP address to be used by netplugin : none
+plugin_role   : In 'master' role, plugin runs netmaster       : master
+                and netplugin
+listen_url    : Netmaster url to listen http requests on      : ":9999"
+control_url   : Netmaster url for control messages            : ":9999"
+dbg_flag      : To enable debug mode, set to '-debug'         : ""
+```
+### docker store
+Docker certified contiv plugin is avaliable on [Docker Store](https://store.docker.com/plugins/803eecee-0780-401a-a454-e9523ccf86b3?tab=description).
+```
+docker plugin install store/contiv/v2plugin:1.0.0-beta.3 iflist=eth1,eth2
+```
+### docker hub
+Contiv plugin released from contiv repo is also pushed to docker hub. iflist has the list of data interfaces used for vlan networks in contiv.
+```
+docker plugin install contiv/v2plugin:1.0.0-beta.3 iflist=eth1,eth2
+```
+### vagrant dev/demo setup
+To create a plugin from [contiv repo](https://github.com/contiv/netplugin), enable v2plugin and run docker in swarm-mode, use the Makefile target demo-v2plugin
+```
+make demo-v2plugin
+```
+
+## Contiv plugin-modes
+Contiv plugin runs both netplugin and netmaster by default. Contiv v2plugin can be run with only netplugin by setting the plugin_role to slave.
+```
+docker plugin install contiv/v2plugin:1.0.0-beta.3 iflist=eth1,eth2 plugin_role=slave
+```
+
+## Contiv plugin workflow
+  1. Etcd cluster should be brought up on the hosts on localhost:2379. If a different port (or Consul) is used, cluster-store option needs to be specified in the plugin install command.
+
+  2. Install contiv v2plugin
+  ```
+  docker plugin install contiv/v2plugin:1.0.0-beta.3 iflist=<data ifs used for vlan networks>
+  ( allow/grant the install permissions when prompted )
+
+  docker plugin ls
+  # to see if the plugin is installed properly and enabled
+  ```
+  3. Debug logs
+  ```
+  # bootup logs are in /var/run/contiv/log/plugin_bootup.log
+  # netplugin and netmaster logs are also in /var/run/contiv/log
+  ```
+  4. Continue with the regular workflow to create networks and run containers
+  ```
+  # create networks using netctl commands
+  netctl network create contiv-net -s 100.1.1.1/24 -g 100.1.1.100
+
+  # run containers
+  docker run -itd --net=contiv-net --name=c1 alpine /bin/sh
+  docker run –it –rm –net=contiv-net –name=c2 alpine ping –c2 c1
+  ```
+
diff --git a/install/v2plugin/config.template b/install/v2plugin/config.template
@@ -13,7 +13,7 @@
           "Settable": [
              "value"
           ],
-          "Value": "-debug"
+          "Value": ""
        },
        {
           "Description": "VLAN uplink interface used by OVS",
@@ -63,6 +63,14 @@
           ],
           "Value": ":9999"
        },
+       {
+          "Description": "Netmaster url for control messages",
+          "Name": "control_url",
+          "Settable": [
+             "value"
+          ],
+          "Value": ":9999"
+       },
        {
           "Description": "Network Driver name for requests to dockerd. Should be same as name:tag of the plugin",
           "Name": "plugin_name",

diff --git a/install/v2plugin/startcontiv.sh b/install/v2plugin/startcontiv.sh
@@ -23,6 +23,9 @@ fi
 if [ $listen_url != ":9999" ]; then
     listen_url_cfg="-listen-url=$listen_url"
 fi
+if [ $control_url != ":9999" ]; then
+    control_url_cfg="-control-url=$control_url"
+fi
 
 echo "Loading OVS" >> $BOOTUP_LOGFILE
 (modprobe openvswitch) || (echo "Load ovs FAILED!!! " >> $BOOTUP_LOGFILE && while true; do sleep 1; done)
@@ -36,29 +39,29 @@ echo "  Creating OVS DB" >> $BOOTUP_LOGFILE
 (ovsdb-tool create  /etc/openvswitch/conf.db /usr/share/openvswitch/vswitch.ovsschema) || (while true; do sleep 1; done)
 
 echo "  Starting OVSBD server " >> $BOOTUP_LOGFILE
-ovsdb-server --remote=punix:/var/run/openvswitch/db.sock --remote=db:Open_vSwitch,Open_vSwitch,manager_options --private-key=db:Open_vSwitch,SSL,private_key --certificate=db:Open_vSwitch,SSL,certificate --bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert --log-file=/var/run/contiv/log/ovs-db.log -vsyslog:dbg -vfile:dbg --pidfile --detach /etc/openvswitch/conf.db >> $BOOTUP_LOGFILE
+ovsdb-server --remote=punix:/var/run/openvswitch/db.sock --remote=db:Open_vSwitch,Open_vSwitch,manager_options --private-key=db:Open_vSwitch,SSL,private_key --certificate=db:Open_vSwitch,SSL,certificate --bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert --log-file=/var/log/openvswitch/ovs-db.log -vsyslog:dbg -vfile:dbg --pidfile --detach /etc/openvswitch/conf.db >> $BOOTUP_LOGFILE
 echo "  Starting ovs-vswitchd " >> $BOOTUP_LOGFILE
-ovs-vswitchd -v --pidfile --detach --log-file=/var/run/contiv/log/ovs-vswitchd.log -vconsole:err -vsyslog:info -vfile:info &
+ovs-vswitchd -v --pidfile --detach --log-file=/var/log/openvswitch/ovs-vswitchd.log -vconsole:err -vsyslog:info -vfile:info &
 ovs-vsctl set-manager tcp:127.0.0.1:6640 
 ovs-vsctl set-manager ptcp:6640
 
-echo "Started OVS" >> $BOOTUP_LOGFILE
+echo "Started OVS, logs in /var/log/openvswitch/" >> $BOOTUP_LOGFILE
 
 echo "Starting Netplugin " >> $BOOTUP_LOGFILE
 while true ; do
     echo "/netplugin $dbg_flag -plugin-mode docker -vlan-if $iflist -cluster-store $cluster_store $ctrl_ip_cfg $vtep_ip_cfg" >> $BOOTUP_LOGFILE
     /netplugin $dbg_flag -plugin-mode docker -vlan-if $iflist -cluster-store $cluster_store $ctrl_ip_cfg $vtep_ip_cfg &> /var/run/contiv/log/netplugin.log
-    echo "CRITICAL : Net Plugin has exited, Respawn in 5s" >> $BOOTUP_LOGFILE
+    echo "CRITICAL : Netplugin has exited, Respawn in 5s" >> $BOOTUP_LOGFILE
     sleep 5
     echo "Restarting Netplugin " >> $BOOTUP_LOGFILE
 done &
 
 if [ $plugin_role == "master" ]; then
     echo "Starting Netmaster " >> $BOOTUP_LOGFILE
     while  true ; do
-        echo "/netmaster $dbg_flag -plugin-name=$plugin_name -cluster-store=$cluster_store $listen_url_cfg " >> $BOOTUP_LOGFILE
-        /netmaster $dbg_flag -plugin-name=$plugin_name -cluster-store=$cluster_store $listen_url_cfg &> /var/run/contiv/log/netmaster.log
-        echo "CRITICAL : Net Master has exited, Respawn in 5s" >> $BOOTUP_LOGFILE
+        echo "/netmaster $dbg_flag -plugin-name=$plugin_name -cluster-store=$cluster_store $listen_url_cfg $control_url_cfg " >> $BOOTUP_LOGFILE
+        /netmaster $dbg_flag -plugin-name=$plugin_name -cluster-store=$cluster_store $listen_url_cfg $control_url_cfg &> /var/run/contiv/log/netmaster.log
+        echo "CRITICAL : Netmaster has exited, Respawn in 5s" >> $BOOTUP_LOGFILE
         echo "Restarting Netmaster " >> $BOOTUP_LOGFILE
         sleep 5
     done &

diff --git a/scripts/python/api/tnode.py b/scripts/python/api/tnode.py
@@ -67,26 +67,18 @@ def runCmd(self, cmd, timeout=None):
             print "Ignoring EOF errors executing command"
             return [], [], 0
 
-    # Install v2plugin on vagrant node
-    def installV2Plugin(self, args=""):
-        ssh_object = self.sshConnect(self.username, self.password)
-        command = "docker plugin install " + os.environ.get("CONTIV_V2PLUGIN_NAME", "") + " --grant-all-permissions iflist=eth2,eth3 " + " plugin-name=" + os.environ["CONTIV_V2PLUGIN_NAME"] + args + " > /tmp/netplugin.log 2>&1"
-        self.npThread = threading.Thread(target=ssh_exec_thread, args=(ssh_object, command))
-        # npThread.setDaemon(True)
-        self.npThread.start()
-
     # Create v2plugin on vagrant node
     def createV2Plugin(self, args=""):
         ssh_object = self.sshConnect(self.username, self.password)
-        command = "docker plugin create " + os.environ.get("CONTIV_V2PLUGIN_NAME", "") + " /opt/gopath/src/github.com/contiv/netplugin/install/v2plugin/ " + args + ">> /tmp/netplugin.log 2>&1"
+        command = "docker plugin create " + os.environ.get("CONTIV_V2PLUGIN_NAME", "contiv/v2plugin:0.0") + " /opt/gopath/src/github.com/contiv/netplugin/install/v2plugin/ " + args + ">> /tmp/netplugin.log 2>&1"
         self.runCmd(command)
 
     # Enable v2plugin on vagrant node
     def enableV2Plugin(self, args=""):
         ssh_object = self.sshConnect(self.username, self.password)
-        command = "docker plugin set " + os.environ.get("CONTIV_V2PLUGIN_NAME", "") + " iflist=eth2,eth3 plugin_name=" + os.environ["CONTIV_V2PLUGIN_NAME"] + args + " >> /tmp/netplugin.log 2>&1"
+        command = "docker plugin set " + os.environ.get("CONTIV_V2PLUGIN_NAME", "contiv/v2plugin:0.0") + " ctrl_ip="+ self.addr + " control_url=" + self.addr + ":9999 iflist=eth2,eth3 plugin_name=" + os.environ.get("CONTIV_V2PLUGIN_NAME", "contiv/v2plugin:0.0") + args + " >> /tmp/netplugin.log 2>&1"
         self.runCmd(command)
-        command = "docker plugin enable " + os.environ.get("CONTIV_V2PLUGIN_NAME", "") +  args + " >> /tmp/netplugin.log 2>&1"
+        command = "docker plugin enable " + os.environ.get("CONTIV_V2PLUGIN_NAME", "contiv/v2plugin:0.0") +  args + " >> /tmp/netplugin.log 2>&1"
         self.npThread = threading.Thread(target=ssh_exec_thread, args=(ssh_object, command))
         # npThread.setDaemon(True)
         self.npThread.start()
@@ -123,7 +115,7 @@ def runCmdThread(self, command):
 
     # Stop v2plugin by force rm 
     def stopV2Plugin(self, args=""):
-        command = "docker plugin rm -f " + os.environ.get("CONTIV_V2PLUGIN_NAME", "") + "> /tmp/netplugin.log 2>&1"
+        command = "docker plugin rm -f " + os.environ.get("CONTIV_V2PLUGIN_NAME", "contiv/v2plugin:0.0") + "> /tmp/netplugin.log 2>&1"
         self.runCmd(command)
 
     # Stop netplugin by killing it