Skip to content
This repository has been archived by the owner on Mar 28, 2018. It is now read-only.

Unable to run Clear Conntainers on Ubuntu 16.04 Server #1060

Open
vinay-y opened this issue Aug 18, 2017 · 16 comments
Open

Unable to run Clear Conntainers on Ubuntu 16.04 Server #1060

vinay-y opened this issue Aug 18, 2017 · 16 comments
Labels

Comments

@vinay-y
Copy link

vinay-y commented Aug 18, 2017

Unable to run Clear Containers on Ubuntu 16.04 Server using docker as it gets timed out waiting for container.

As compatibility for Docker 17.06 is not yet supported as given here, I have setup Docker 17.03.2.

I started a docker daemon using this.
sudo dockerd --add-runtime cor=/usr/bin/cc-oci-runtime.sh --default-runtime=cor --storage-driver=vfs

Output of sudo docker info is as follows.

Containers: 4
 Running: 0
 Paused: 0
 Stopped: 4
Images: 1
Server Version: 17.03.2-ce
Storage Driver: vfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: cor runc
Default Runtime: cor
Init Binary: docker-init
containerd version: 4ab9917febca54791c5f071a9d1f404867857fcc
runc version: 54296cf40ad8143b62dbcaa1d90e520a2136ddfe
init version: 949e6fa
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.12.7
Operating System: Ubuntu 16.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 23.53 GiB
Name: SDN-NFV-Server
ID: 3VQW:Q5C5:5OGL:WVHZ:NSQY:6M3H:LJ6Z:Z5L4:YER6:62IT:3YYY:J6CP
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

Now, when I do sudo docker run -ti --runtime=runc hello-world, everything works fine.
But, when I do sudo docker run -ti hello-world, the output after two minutes is the following.

docker: Error response from daemon: containerd: container did not start before the specified timeout.

The error on the docker daemon is the following.

ERRO[0186] containerd: start container                   error=containerd: container did not start before the specified timeout id=d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b
ERRO[0186] Create container failed with error: containerd: container did not start before the specified timeout 
ERRO[0186] Handler for POST /v1.27/containers/d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b/start returned error: containerd: container did not start before the specified timeout

Output of journalctl -u cc-proxy.service is this.

-- Logs begin at Fri 2017-08-18 21:54:28 IST, end at Fri 2017-08-18 22:09:34 IST. --
Aug 18 22:00:57 SDN-NFV-Server systemd[1]: Stopped Clear Containers Proxy.
Aug 18 22:00:57 SDN-NFV-Server systemd[1]: Started Clear Containers Proxy.
Aug 18 22:00:57 SDN-NFV-Server cc-proxy[1831]: I0818 22:00:57.467281    1831 proxy.go:381] proxy started
Aug 18 22:04:00 SDN-NFV-Server cc-proxy[1831]: I0818 22:04:00.988991    1831 proxy.go:71] [client #1] client connected
Aug 18 22:04:01 SDN-NFV-Server cc-proxy[1831]: I0818 22:04:01.298320    1831 proxy.go:81] [client #1] hello(containerId=d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b,ctlSerial=/var/run/cc-oci-runtime/d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b/ga-ctl.sock,ioSerial=/var/run/cc-oci-runtime/d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b/ga-tty.sock,console=/var/run/cc-oci-runtime/d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b/console.sock)
Aug 18 22:06:00 SDN-NFV-Server cc-proxy[1831]: I0818 22:06:00.877399    1831 proxy.go:71] [client #2] client connected
Aug 18 22:06:00 SDN-NFV-Server cc-proxy[1831]: I0818 22:06:00.877635    1831 proxy.go:81] [client #2] attach(containerId=d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b)
Aug 18 22:06:00 SDN-NFV-Server cc-proxy[1831]: I0818 22:06:00.878182    1831 proxy.go:81] [client #2] hyper(cmd=destroypod, data={})

The content of /run/cc-oci-runtime/cc-oci-runtime.log is the following.

2017-08-18T16:34:00.968559Z:2041:cc-oci-runtime:debug:cc-oci-runtime 2.2.4  called as: /usr/bin/cc-oci-runtime create --bundle /var/run/docker/libcontainerd/d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b --console /dev/pts/2 --pid-file /run/docker/libcontainerd/containerd/d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b/init/pid d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b
2017-08-18T16:34:00.968842Z:2041:cc-oci-runtime:debug:path '/var/run/docker/libcontainerd/d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b' resolved to '/run/docker/libcontainerd/d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b'
2017-08-18T16:34:00.968908Z:2041:cc-oci-runtime:debug:using config_file /run/docker/libcontainerd/d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b/config.json
2017-08-18T16:34:00.973991Z:2041:cc-oci-runtime:debug:path '/var/lib/docker/vfs/dir/e8becfdc6df83ab9f5cb63ff1282b24c6481e56a17f4953390c9f9bfe9bbe685' resolved to '/var/lib/docker/vfs/dir/e8becfdc6df83ab9f5cb63ff1282b24c6481e56a17f4953390c9f9bfe9bbe685'
2017-08-18T16:34:00.974170Z:2041:cc-oci-runtime:debug:Reading VM configuration from /usr/share/defaults/cc-oci-runtime/vm.json
2017-08-18T16:34:00.982747Z:2041:cc-oci-runtime:debug:path '/usr/bin/qemu-lite-system-x86_64' resolved to '/usr/bin/qemu-lite-system-x86_64'
2017-08-18T16:34:00.985866Z:2041:cc-oci-runtime:debug:path '/usr/share/clear-containers/clear-containers.img' resolved to '/usr/share/clear-containers/clear-16670-containers.img'
2017-08-18T16:34:00.985972Z:2041:cc-oci-runtime:debug:path '/usr/share/clear-containers/vmlinux.container' resolved to '/usr/share/clear-containers/vmlinux-4.9.35-62.container'
2017-08-18T16:34:00.986180Z:2041:cc-oci-runtime:debug:OCI spec versions: config=1.0.0-rc2-dev, runtime=1.0.0-rc1
2017-08-18T16:34:00.986285Z:2041:cc-oci-runtime:debug:creating directory /var/run/cc-oci-runtime/d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b
2017-08-18T16:34:00.986397Z:2041:cc-oci-runtime:debug:Created rootfs bind mount object for container d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b
2017-08-18T16:34:00.986453Z:2041:cc-oci-runtime:debug:Added rootfs bind mount for container d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b
2017-08-18T16:34:00.986552Z:2041:cc-oci-runtime:debug:mounting /var/lib/docker/vfs/dir/e8becfdc6df83ab9f5cb63ff1282b24c6481e56a17f4953390c9f9bfe9bbe685 of type bind onto /var/run/cc-oci-runtime/d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b/workload//d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b/rootfs with options '' and flags 0x1000
2017-08-18T16:34:00.986649Z:2041:cc-oci-runtime:debug:setting up namespaces
2017-08-18T16:34:00.986787Z:2041:cc-oci-runtime:debug:created mount namespace
2017-08-18T16:34:00.987627Z:2041:cc-oci-runtime:debug:created network namespace
2017-08-18T16:34:00.987690Z:2041:cc-oci-runtime:debug:ignoring uts namespace request
2017-08-18T16:34:00.987738Z:2041:cc-oci-runtime:debug:ignoring pid namespace request
2017-08-18T16:34:00.987785Z:2041:cc-oci-runtime:debug:ignoring ipc namespace request
2017-08-18T16:34:00.987832Z:2041:cc-oci-runtime:debug:finished namespace setup
2017-08-18T16:34:00.987890Z:2041:cc-oci-runtime:debug:Mounting mount /var/run/cc-oci-runtime/d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b/workload/31356b6379ce2018-resolv.conf for mnt_dir /etc/resolv.conf
2017-08-18T16:34:00.987963Z:2041:cc-oci-runtime:debug:mounting /var/lib/docker/containers/d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b/resolv.conf of type bind onto /var/run/cc-oci-runtime/d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b/workload/31356b6379ce2018-resolv.conf with options '' and flags 0x45000
2017-08-18T16:34:00.988053Z:2041:cc-oci-runtime:debug:Mounting mount /var/run/cc-oci-runtime/d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b/workload/8c6bf0ced6e6fcb7-hostname for mnt_dir /etc/hostname
2017-08-18T16:34:00.988126Z:2041:cc-oci-runtime:debug:mounting /var/lib/docker/containers/d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b/hostname of type bind onto /var/run/cc-oci-runtime/d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b/workload/8c6bf0ced6e6fcb7-hostname with options '' and flags 0x45000
2017-08-18T16:34:00.988223Z:2041:cc-oci-runtime:debug:Mounting mount /var/run/cc-oci-runtime/d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b/workload/b27dca258d03970f-hosts for mnt_dir /etc/hosts
2017-08-18T16:34:00.988297Z:2041:cc-oci-runtime:debug:mounting /var/lib/docker/containers/d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b/hosts of type bind onto /var/run/cc-oci-runtime/d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b/workload/b27dca258d03970f-hosts with options '' and flags 0x45000
2017-08-18T16:34:00.988393Z:2041:cc-oci-runtime:debug:connecting to proxy cc-proxy
2017-08-18T16:34:00.988728Z:2041:cc-oci-runtime:debug:connected to proxy socket /var/run/cc-oci-runtime/proxy.sock
2017-08-18T16:34:00.989047Z:2041:cc-oci-runtime:debug:hypervisor child pid is 2042
2017-08-18T16:34:00.989186Z:2042:cc-oci-runtime:debug:disconnecting from proxy
2017-08-18T16:34:00.989373Z:2042:cc-oci-runtime:debug:reading hypervisor command-line length from pipe
2017-08-18T16:34:00.989357Z:2041:cc-oci-runtime:debug:shim process running with pid 2043
2017-08-18T16:34:00.989500Z:2043:cc-oci-runtime:debug:shim child waiting for proxy socket fd on fd 6
2017-08-18T16:34:00.989735Z:2041:cc-oci-runtime:debug:created state file /var/run/cc-oci-runtime/d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b/state.json
2017-08-18T16:34:01.071447Z:2041:cc-oci-runtime:debug:building hypervisor command-line
2017-08-18T16:34:01.071708Z:2041:cc-oci-runtime:debug:Discovering container interfaces
2017-08-18T16:34:01.071794Z:2041:cc-oci-runtime:debug:Interface := [lo]
2017-08-18T16:34:01.071863Z:2041:cc-oci-runtime:debug:Interface := [eth0]
2017-08-18T16:34:01.071929Z:2041:cc-oci-runtime:debug:Interface := [lo]
2017-08-18T16:34:01.072001Z:2041:cc-oci-runtime:debug:Interface := [eth0]
2017-08-18T16:34:01.072095Z:2041:cc-oci-runtime:debug:IP := [172.17.0.2]
2017-08-18T16:34:01.072171Z:2041:cc-oci-runtime:debug:IP := [255.255.0.0]
2017-08-18T16:34:01.072245Z:2041:cc-oci-runtime:debug:MTU for interface eth0: 1500

2017-08-18T16:34:01.072317Z:2041:cc-oci-runtime:debug:Interface := [lo]
2017-08-18T16:34:01.072384Z:2041:cc-oci-runtime:debug:Interface := [eth0]
2017-08-18T16:34:01.072453Z:2041:cc-oci-runtime:debug:IP := [fe80::42:acff:fe11:2]
2017-08-18T16:34:01.072520Z:2041:cc-oci-runtime:debug:netlink_get_default_gw
2017-08-18T16:34:01.072614Z:2041:cc-oci-runtime:debug:table=254
2017-08-18T16:34:01.072687Z:2041:cc-oci-runtime:debug:IP := [172.17.0.1]
2017-08-18T16:34:01.072754Z:2041:cc-oci-runtime:debug:gateway : 172.17.0.1
2017-08-18T16:34:01.072870Z:2041:cc-oci-runtime:debug:ifname=eth0
2017-08-18T16:34:01.072945Z:2041:cc-oci-runtime:debug:table=254
2017-08-18T16:34:01.073015Z:2041:cc-oci-runtime:debug:IP := [172.17.0.0]
2017-08-18T16:34:01.073082Z:2041:cc-oci-runtime:debug:destination : 172.17.0.0
2017-08-18T16:34:01.073160Z:2041:cc-oci-runtime:debug:ifname=eth0
2017-08-18T16:34:01.073231Z:2041:cc-oci-runtime:debug:table=255
2017-08-18T16:34:01.073297Z:2041:cc-oci-runtime:debug:table=255
2017-08-18T16:34:01.073363Z:2041:cc-oci-runtime:debug:table=255
2017-08-18T16:34:01.073428Z:2041:cc-oci-runtime:debug:table=255
2017-08-18T16:34:01.073494Z:2041:cc-oci-runtime:debug:table=255
2017-08-18T16:34:01.073559Z:2041:cc-oci-runtime:debug:table=255
2017-08-18T16:34:01.073624Z:2041:cc-oci-runtime:debug:table=255
2017-08-18T16:34:01.073693Z:2041:cc-oci-runtime:debug:[1] networks discovered
2017-08-18T16:34:01.073760Z:2041:cc-oci-runtime:debug:Running command : iptables-save
2017-08-18T16:34:01.075544Z:2041:cc-oci-runtime:debug:Running commands for dropping iptables
2017-08-18T16:34:01.075645Z:2041:cc-oci-runtime:debug:arg: '/bin/sh'
2017-08-18T16:34:01.075720Z:2041:cc-oci-runtime:debug:arg: '-c'
2017-08-18T16:34:01.075790Z:2041:cc-oci-runtime:debug:arg: 'iptables -P INPUT ACCEPT && iptables -P FORWARD ACCEPT && iptables -P OUTPUT ACCEPT && iptables -t nat -F && iptables -t mangle -F && iptables -F && iptables -X'
2017-08-18T16:34:01.085339Z:2041:cc-oci-runtime:debug:MTU set for interface ceth0 as 1500

2017-08-18T16:34:01.085441Z:2041:cc-oci-runtime:debug:netlink_link_add_bridge beth0
2017-08-18T16:34:01.085510Z:2041:cc-oci-runtime:debug:Turning off multicast snooping for bridge beth0
2017-08-18T16:34:01.085823Z:2041:cc-oci-runtime:debug:netlink_link_set_addr eth0
2017-08-18T16:34:01.085894Z:2041:cc-oci-runtime:debug:macaddr 02:00:ca:fe:00:00
2017-08-18T16:34:01.086026Z:2041:cc-oci-runtime:debug:netlink_link_set_master 2 3
2017-08-18T16:34:01.086212Z:2041:cc-oci-runtime:debug:netlink_link_set_master 8 3
2017-08-18T16:34:01.086402Z:2041:cc-oci-runtime:debug:netlink_link_enable[1] ceth0
2017-08-18T16:34:01.086509Z:2041:cc-oci-runtime:debug:netlink_link_enable[1] eth0
2017-08-18T16:34:01.086584Z:2041:cc-oci-runtime:debug:netlink_link_enable[1] beth0
2017-08-18T16:34:01.086746Z:2041:cc-oci-runtime:debug:network configuration complete
2017-08-18T16:34:01.086860Z:2041:cc-oci-runtime:debug:using /usr/share/defaults/cc-oci-runtime/hypervisor.args
2017-08-18T16:34:01.096717Z:2041:cc-oci-runtime:debug:guest agent ctl socket: /var/run/cc-oci-runtime/d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b/ga-ctl.sock
2017-08-18T16:34:01.096784Z:2041:cc-oci-runtime:debug:guest agent tty socket: /var/run/cc-oci-runtime/d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b/ga-tty.sock
2017-08-18T16:34:01.096923Z:2041:cc-oci-runtime:debug:checking child setup (blocking)
2017-08-18T16:34:01.097041Z:2042:cc-oci-runtime:debug:reading hypervisor command-line from pipe
2017-08-18T16:34:01.097234Z:2042:cc-oci-runtime:debug:running command:
2017-08-18T16:34:01.097310Z:2042:cc-oci-runtime:debug:arg: '/usr/bin/qemu-lite-system-x86_64'
2017-08-18T16:34:01.097378Z:2042:cc-oci-runtime:debug:arg: '-name'
2017-08-18T16:34:01.097443Z:2042:cc-oci-runtime:debug:arg: 'e65a82562cfc'
2017-08-18T16:34:01.097507Z:2042:cc-oci-runtime:debug:arg: '-machine'
2017-08-18T16:34:01.097572Z:2042:cc-oci-runtime:debug:arg: 'pc-lite,accel=kvm,kernel_irqchip,nvdimm'
2017-08-18T16:34:01.097638Z:2042:cc-oci-runtime:debug:arg: '-device'
2017-08-18T16:34:01.097709Z:2042:cc-oci-runtime:debug:arg: 'nvdimm,memdev=mem0,id=nv0'
2017-08-18T16:34:01.097777Z:2042:cc-oci-runtime:debug:arg: '-object'
2017-08-18T16:34:01.097848Z:2042:cc-oci-runtime:debug:arg: 'memory-backend-file,id=mem0,mem-path=/usr/share/clear-containers/clear-16670-containers.img,size=235929600'
2017-08-18T16:34:01.097917Z:2042:cc-oci-runtime:debug:arg: '-m'
2017-08-18T16:34:01.097981Z:2042:cc-oci-runtime:debug:arg: '2G,slots=2,maxmem=3G'
2017-08-18T16:34:01.098045Z:2042:cc-oci-runtime:debug:arg: '-kernel'
2017-08-18T16:34:01.098110Z:2042:cc-oci-runtime:debug:arg: '/usr/share/clear-containers/vmlinux-4.9.35-62.container'
2017-08-18T16:34:01.098176Z:2042:cc-oci-runtime:debug:arg: '-append'
2017-08-18T16:34:01.098241Z:2042:cc-oci-runtime:debug:arg: 'root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k panic=1 console=hvc0 console=hvc1 initcall_debug init=/usr/lib/systemd/systemd systemd.unit=cc-agent.target iommu=off quiet systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket systemd.show_status=false cryptomgr.notests net.ifnames=0 ip=::::::d3b785dd3ec6::off::'
2017-08-18T16:34:01.098339Z:2042:cc-oci-runtime:debug:arg: '-smp'
2017-08-18T16:34:01.098407Z:2042:cc-oci-runtime:debug:arg: '2,sockets=1,cores=2,threads=1'
2017-08-18T16:34:01.098474Z:2042:cc-oci-runtime:debug:arg: '-cpu'
2017-08-18T16:34:01.098538Z:2042:cc-oci-runtime:debug:arg: 'host'
2017-08-18T16:34:01.098603Z:2042:cc-oci-runtime:debug:arg: '-rtc'
2017-08-18T16:34:01.098666Z:2042:cc-oci-runtime:debug:arg: 'base=utc,driftfix=slew'
2017-08-18T16:34:01.098730Z:2042:cc-oci-runtime:debug:arg: '-no-user-config'
2017-08-18T16:34:01.098803Z:2042:cc-oci-runtime:debug:arg: '-nodefaults'
2017-08-18T16:34:01.098868Z:2042:cc-oci-runtime:debug:arg: '-global'
2017-08-18T16:34:01.098933Z:2042:cc-oci-runtime:debug:arg: 'kvm-pit.lost_tick_policy=discard'
2017-08-18T16:34:01.098997Z:2042:cc-oci-runtime:debug:arg: '-device'
2017-08-18T16:34:01.099085Z:2042:cc-oci-runtime:debug:arg: 'virtio-serial-pci,id=virtio-serial0'
2017-08-18T16:34:01.099153Z:2042:cc-oci-runtime:debug:arg: '-device'
2017-08-18T16:34:01.099217Z:2042:cc-oci-runtime:debug:arg: 'virtconsole,chardev=charconsole0,id=console0'
2017-08-18T16:34:01.099283Z:2042:cc-oci-runtime:debug:arg: '-chardev'
2017-08-18T16:34:01.099338Z:2042:cc-oci-runtime:debug:arg: 'socket,path=/var/run/cc-oci-runtime/d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b/console.sock,server,nowait,id=charconsole0,signal=off'
2017-08-18T16:34:01.099391Z:2042:cc-oci-runtime:debug:arg: '-chardev'
2017-08-18T16:34:01.099448Z:2042:cc-oci-runtime:debug:arg: 'socket,id=procsock,path=/var/run/cc-oci-runtime/d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b/process.sock,server,nowait'
2017-08-18T16:34:01.099517Z:2042:cc-oci-runtime:debug:arg: '-chardev'
2017-08-18T16:34:01.099582Z:2042:cc-oci-runtime:debug:arg: 'socket,id=charch0,path=/var/run/cc-oci-runtime/d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b/ga-ctl.sock,server,nowait'
2017-08-18T16:34:01.099650Z:2042:cc-oci-runtime:debug:arg: '-device'
2017-08-18T16:34:01.099714Z:2042:cc-oci-runtime:debug:arg: 'virtserialport,bus=virtio-serial0.0,nr=1,chardev=charch0,id=channel0,name=sh.hyper.channel.0'
2017-08-18T16:34:01.099781Z:2042:cc-oci-runtime:debug:arg: '-chardev'
2017-08-18T16:34:01.099846Z:2042:cc-oci-runtime:debug:arg: 'socket,id=charch1,path=/var/run/cc-oci-runtime/d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b/ga-tty.sock,server,nowait'
2017-08-18T16:34:01.099914Z:2042:cc-oci-runtime:debug:arg: '-device'
2017-08-18T16:34:01.099977Z:2042:cc-oci-runtime:debug:arg: 'virtserialport,bus=virtio-serial0.0,nr=2,chardev=charch1,id=channel1,name=sh.hyper.channel.1'
2017-08-18T16:34:01.100044Z:2042:cc-oci-runtime:debug:arg: '-uuid'
2017-08-18T16:34:01.100108Z:2042:cc-oci-runtime:debug:arg: '0c55e194-8518-473c-a34a-e65a82562cfc'
2017-08-18T16:34:01.100172Z:2042:cc-oci-runtime:debug:arg: '-qmp'
2017-08-18T16:34:01.100237Z:2042:cc-oci-runtime:debug:arg: 'unix:/var/run/cc-oci-runtime/d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b/hypervisor.sock,server,nowait'
2017-08-18T16:34:01.100305Z:2042:cc-oci-runtime:debug:arg: '-nographic'
2017-08-18T16:34:01.100368Z:2042:cc-oci-runtime:debug:arg: '-vga'
2017-08-18T16:34:01.100432Z:2042:cc-oci-runtime:debug:arg: 'none'
2017-08-18T16:34:01.100495Z:2042:cc-oci-runtime:debug:arg: '-netdev'
2017-08-18T16:34:01.100558Z:2042:cc-oci-runtime:debug:arg: 'tap,ifname=ceth0,script=no,downscript=no,id=ceth0,vhost=on'
2017-08-18T16:34:01.100624Z:2042:cc-oci-runtime:debug:arg: '-device'
2017-08-18T16:34:01.100688Z:2042:cc-oci-runtime:debug:arg: 'driver=virtio-net-pci,netdev=ceth0,mac=02:42:ac:11:00:02'
2017-08-18T16:34:01.100753Z:2042:cc-oci-runtime:debug:arg: '-device'
2017-08-18T16:34:01.100818Z:2042:cc-oci-runtime:debug:arg: 'virtio-9p-pci,fsdev=workload9p,mount_tag=rootfs'
2017-08-18T16:34:01.100885Z:2042:cc-oci-runtime:debug:arg: '-fsdev'
2017-08-18T16:34:01.100950Z:2042:cc-oci-runtime:debug:arg: 'local,id=workload9p,path=/var/run/cc-oci-runtime/d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b/workload,security_model=none'
2017-08-18T16:34:01.113438Z:2041:cc-oci-runtime:debug:child setup successful
2017-08-18T16:34:01.286281Z:2041:cc-oci-runtime:debug:CTL created event: 3
2017-08-18T16:34:01.286408Z:2041:cc-oci-runtime:debug:CTL created event: 1
2017-08-18T16:34:01.286509Z:2041:cc-oci-runtime:debug:communicating with proxy
2017-08-18T16:34:01.286580Z:2041:cc-oci-runtime:debug:sending message (length 447) to proxy socket
2017-08-18T16:34:01.286634Z:2041:cc-oci-runtime:debug:writing message data to proxy socket: {"id":"hello","data":{"containerId":"d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b","ctlSerial":"/var/run/cc-oci-runtime/d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b/ga-ctl.sock","ioSerial":"/var/run/cc-oci-runtime/d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b/ga-tty.sock","console":"/var/run/cc-oci-runtime/d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b/console.sock"}}
2017-08-18T16:36:00.869590Z:2084:cc-oci-runtime:debug:cc-oci-runtime 2.2.4  called as: /usr/bin/cc-oci-runtime delete d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b
2017-08-18T16:36:00.871458Z:2043:cc-oci-runtime:critical:failed to read proxy socket fd
2017-08-18T16:36:00.876648Z:2084:cc-oci-runtime:debug:connecting to proxy cc-proxy
2017-08-18T16:36:00.877115Z:2084:cc-oci-runtime:debug:connected to proxy socket /var/run/cc-oci-runtime/proxy.sock
2017-08-18T16:36:00.877245Z:2084:cc-oci-runtime:debug:communicating with proxy
2017-08-18T16:36:00.877297Z:2084:cc-oci-runtime:debug:sending message (length 105) to proxy socket
2017-08-18T16:36:00.877331Z:2084:cc-oci-runtime:debug:writing message data to proxy socket: {"id":"attach","data":{"containerId":"d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b"}}
2017-08-18T16:36:00.877841Z:2084:cc-oci-runtime:debug:proxy msg length: 37
2017-08-18T16:36:00.877893Z:2084:cc-oci-runtime:debug:message read from proxy socket: {"success":true,"data":{"version":1}}
2017-08-18T16:36:00.877993Z:2084:cc-oci-runtime:debug:msg received: {"success":true,"data":{"version":1}}
2017-08-18T16:36:00.878044Z:2084:cc-oci-runtime:debug:communicating with proxy
2017-08-18T16:36:00.878074Z:2084:cc-oci-runtime:debug:sending message (length 58) to proxy socket
2017-08-18T16:36:00.878099Z:2084:cc-oci-runtime:debug:writing message data to proxy socket: {"id":"hyper","data":{"hyperName":"destroypod","data":{}}}

I think the issue is with critical:failed to read proxy socket fd.
Tried running on Ubuntu 16.04 Server and Desktop VMs on same machine. Same error in both.

Thanks for the help.

@grahamwhaley
Copy link

@chavafg @jcvenegas Hi - any thoughts here? I suspect we have seen the non-existent proxy socket before maybe.
@vinay-y for our info, how did you install CC. Presumably following:
https://github.com/01org/cc-oci-runtime/blob/master/documentation/Installing-Clear-Containers-on-Ubuntu.md
or
https://github.com/01org/cc-oci-runtime/blob/master/documentation/Quickstart-Clear-Containers-in-a-Ubuntu-VM.md

@grahamwhaley
Copy link

oh, and thanks for the pretty comprehensive report @vinay-y :-)

@jcvenegas
Copy link
Contributor

Hi @vinay-y , agree pretty comprehensive report, for what I can see the timeout was caused because the runtime was waiting for the VM started correctly, but seems that never happend and docker called to delete commmand.

017-08-18T16:34:01.286634Z:2041:cc-oci-runtime:debug:writing message data to proxy socket: {"id":"hello","data":2017-08-18T16:36:00.869590Z:2084:cc-oci-runtime:debug:cc-oci-runtime 2.2.4  called as: /usr/bin/cc-oci-runtime delete d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b
2017-08-18T16:36:00.871458Z:2043:cc-oci-runtime:critical:failed to read proxy socket fd
  1. Could you enable cc-proxy debug logs to get more information about it:
sudo sed -i 's,ExecStart=/usr/libexec/cc-proxy,ExecStart=/usr/libexec/cc-proxy -v 3 ,g' /usr/lib/systemd/system/cc-proxy.service
sudo systemctl daemon-reload
sudo systemctl restart cc-proxy

Then run a container again and collect again the cc-proxy logs.
2. You mentinued you are using VMs, what is the hypervisor you are using?

@vinay-y
Copy link
Author

vinay-y commented Aug 21, 2017

Hi.
@grahamwhaley I used the first one.
https://github.com/01org/cc-oci-runtime/blob/master/documentation/Installing-Clear-Containers-on-Ubuntu.md

@jcvenegas A clarification: All the logs I have uploaded are for a physical machine. In addition to testing on this machine I tried on qemu-kvm VMs on the same machine. My goal is to run containers on VMs. But current priority is to get it running on physical machine.

After enabling cc-proxy debug logs, output of journalctl -u cc-proxy.service is

-- Logs begin at Sat 2017-08-19 00:59:48 IST, end at Mon 2017-08-21 21:47:31 IST. --
Aug 21 21:42:29 SDN-NFV-Server systemd[1]: Stopped Clear Containers Proxy.
Aug 21 21:42:29 SDN-NFV-Server systemd[1]: Started Clear Containers Proxy.
Aug 21 21:42:29 SDN-NFV-Server cc-proxy[5000]: I0821 21:42:29.392438    5000 proxy.go:381] proxy started
Aug 21 21:44:08 SDN-NFV-Server cc-proxy[5000]: I0821 21:44:08.358831    5000 proxy.go:71] [client #1] client connected
Aug 21 21:44:08 SDN-NFV-Server cc-proxy[5000]: I0821 21:44:08.686025    5000 proxy.go:81] [client #1] hello(containerId=d5d517005372ce688e3dc82fd1bb2b0a741abe6c91cb8f5d6032fbb6193bc63a,ctlSerial=/var/run/cc-oci-runtime/d5d517005372ce688e3dc82fd1bb2b0a741abe6c91cb8f5d6032fbb6193bc63a/ga-ctl.sock,ioSerial=/var/run/cc-oci-runtime/d5d517005372ce688e3dc82fd1bb2b0a741abe6c91cb8f5d6032fbb6193bc63a/ga-tty.sock,console=/var/run/cc-oci-runtime/d5d517005372ce688e3dc82fd1bb2b0a741abe6c91cb8f5d6032fbb6193bc63a/console.sock)

The logs for same command I provided in previous comment are probably wrong as cc-proxy.service was not active then. Only cc-proxy.socket was active.
But, this time because of restart cc-proxy command, cc-proxy.service started and these are correct logs.

Thanks again.

@jcvenegas
Copy link
Contributor

@vinay-y thank you, seems that we did not get enough information from proxy logs :(.

You could you try running the following script in order to detect if your system could a Clear Container:

curl -O https://download.clearlinux.org/current/clear-linux-check-config.sh
bash ./clear-linux-check-config.sh  container

Could you verify if is there any qemu instance running ?

pgrep qemu-lite

Also could you provide cpu and memory information from your host ?

@vinay-y
Copy link
Author

vinay-y commented Aug 21, 2017

@jcvenegas Thanks for this. Maybe this was the problem.
This was the output for the check script.

Checking if host is capable of running Clear Linux* OS for Intel® Architecture in a container

SUCCESS: Intel CPU
SUCCESS: 64-bit CPU (lm)
SUCCESS: Streaming SIMD Extensions v4.1 (sse4_1)
SUCCESS: Virtualisation support (vmx)
SUCCESS: Kernel module kvm
SUCCESS: Kernel module kvm_intel
SUCCESS: Nested KVM support
FAIL: Unrestricted guest KVM support

Can you please tell me how to enable unrestricted guest kvm support?
My CPU model name : Intel(R) Xeon(R) CPU E5507 @ 2.27GHz
What more information do you need? I am pretty sure qemu-lite was running.

@mcastelino
Copy link
Contributor

@vinay-y Can you try a quick test on that system to confirm that it may not be able to support Clear Containers. Can you try running the following command as root or under sudo

qemu-lite-system-x86_64 -machine pc-lite,accel=kvm,kernel_irqchip,nvdimm \
-cpu host -m 256,maxmem=1G,slots=2 -smp 2 \
-no-user-config -nodefaults -rtc base=utc,driftfix=slew \
-global kvm-pit.lost_tick_policy=discard \
-kernel /usr/share/clear-containers/vmlinux.container \
-append "reboot=k panic=1 rw tsc=reliable no_timer_check noreplace-smp root=/dev/pmem0p1 init=/usr/lib/systemd/systemd initcall_debug rootfstype=ext4 rootflags=dax,data=ordered dhcp rcupdate.rcu_expedited=1 clocksource=kvm-clock console=hvc0 single iommu=false debug" \
-device virtio-serial-pci,id=virtio-serial0 \
-chardev stdio,id=charconsole0 \
-device virtconsole,chardev=charconsole0,id=console0 \
-nographic \
-object memory-backend-file,id=mem0,share,mem-path=/usr/share/clear-containers/clear-containers.img,size=235929600 \
-device nvdimm,memdev=mem0,id=nv0 -no-reboot

If your platform can actually run the VM, you will end up on the command prompt.

You are in rescue mode. After logging in, type "journalctl -xb" to view
system logs, "systemctl reboot" to reboot, "systemctl default" or ^D to boot
into default mode.
Press Enter for maintenance
(or press Control-D to continue):

If this hangs, then please send us the kernel boot log.

@jovizhangwei
Copy link

Hi guys, I encountered the same issue in my local ubuntu machine.

The error is same: "cc-oci-runtime:critical:failed to read proxy socket fd", no other error reported/

And I tested my machine by above command "bash ./clear-linux-check-config.sh container" as @jcvenegas mentioned, the result is SUCCESS for all items.

I also checked the qemu command as @mcastelino posted, it shows nothing, is there have some issues about my machine?

Please give us some tips, thanks.

@vinay-y
Copy link
Author

vinay-y commented Aug 22, 2017

Hi @mcastelino,
I ran the command you gave and ended on the command prompt without any output.
I don't understand the second block is - "rescue mode" block. What is it for?

synerg@SDN-NFV-Server:~$ sudo qemu-lite-system-x86_64 -machine pc-lite,accel=kvm,kernel_irqchip,nvdimm \
> -cpu host -m 256,maxmem=1G,slots=2 -smp 2 \
> -no-user-config -nodefaults -rtc base=utc,driftfix=slew \
> -global kvm-pit.lost_tick_policy=discard \
> -kernel /usr/share/clear-containers/vmlinux.container \
> -append "reboot=k panic=1 rw tsc=reliable no_timer_check noreplace-smp root=/dev/pmem0p1 init=/usr/lib/systemd/systemd initcall_debug rootfstype=ext4 rootflags=dax,data=ordered dhcp rcupdate.rcu_expedited=1 clocksource=kvm-clock console=hvc0 single iommu=false debug" \
> -device virtio-serial-pci,id=virtio-serial0 \
> -chardev stdio,id=charconsole0 \
> -device virtconsole,chardev=charconsole0,id=console0 \
> -nographic \
> -object memory-backend-file,id=mem0,share,mem-path=/usr/share/clear-containers/clear-containers.img,size=235929600 \
> -device nvdimm,memdev=mem0,id=nv0 -no-reboot
[sudo] password for synerg: 
synerg@SDN-NFV-Server:~$

Thanks.

@vinay-y
Copy link
Author

vinay-y commented Aug 22, 2017

Hi @jovizhangwei ,
You recently closed an issue here about this.
There was a comment that your friend resolved your problem. What did you do? Were you able to run it finally?
Thanks.

@jovizhangwei
Copy link

Hi @vinay-y , I guess that issue is not same as this one, the error is also not same, we didn't found clear root cause of that issue, but probably caused by multi-version installed and misused.

@mcastelino
Copy link
Contributor

@vinay-y you should not end up on your system's command prompt. Rather you should have ended up inside the container environment. If you ended up on the system command prompt, it means that the VM did not launch. However I am surprised that you did not see any other output.

I think your system (Nehalem) cannot support clear containers as it lacks Unrestricted Guest mode support.

@egernst egernst added the CC 2.1 label Aug 22, 2017
@jovizhangwei
Copy link

@mcastelino , is there have any simpler way to check the machine lacks Unrestricted Guest mode? I have some issue with @vinay-y , the qemu command didn't print anything.

@jiangpengcheng
Copy link

I ran into the same error on centos7.3, I follow the https://github.com/01org/cc-oci-runtime/blob/master/documentation/Installing-Clear-Containers-on-Centos-7.md to install clear container

the error logs are almost the same. and the result of sh clear-linux-check-config.sh container is

Checking if host is capable of running Clear Linux* OS for Intel® Architecture in a container

SUCCESS: Intel CPU
SUCCESS: 64-bit CPU (lm)
SUCCESS: Streaming SIMD Extensions v4.1 (sse4_1)
SUCCESS: Virtualisation support (vmx)
SUCCESS: Kernel module kvm
SUCCESS: Kernel module kvm_intel
SUCCESS: Nested KVM support
SUCCESS: Unrestricted guest KVM support
SUCCESS: Kernel module vhost
SUCCESS: Kernel module vhost_net

but the result of sh clear-linux-check-config.sh host is

Checking if host is capable of running Clear Linux* OS for Intel® Architecture

SUCCESS: Intel CPU
SUCCESS: 64-bit CPU (lm)
SUCCESS: Streaming SIMD Extensions v4.1 (sse4_1)
FAIL: EFI firmware

when I ran command:

qemu-system-x86_64 -machine pc-lite,accel=kvm,kernel_irqchip,nvdimm \
-cpu host -m 256,maxmem=1G,slots=2 -smp 2 \
-no-user-config -nodefaults -rtc base=utc,driftfix=slew \
-global kvm-pit.lost_tick_policy=discard \
-kernel /usr/share/clear-containers/vmlinux.container \
-append "reboot=k panic=1 rw tsc=reliable no_timer_check noreplace-smp root=/dev/pmem0p1 init=/usr/lib/systemd/systemd initcall_debug rootfstype=ext4 rootflags=dax,data=ordered dhcp rcupdate.rcu_expedited=1 clocksource=kvm-clock console=hvc0 single iommu=false debug" \
-device virtio-serial-pci,id=virtio-serial0 \
-chardev stdio,id=charconsole0 \
-device virtconsole,chardev=charconsole0,id=console0 \
-nographic \
-object memory-backend-file,id=mem0,share,mem-path=/usr/share/clear-containers/clear-containers.img,size=235929600 \
-device nvdimm,memdev=mem0,id=nv0 -no-reboot

I got no results just like vinay-y.

looking forward to your help

@jiangpengcheng
Copy link

jiangpengcheng commented Sep 19, 2017

@devimc not yet, I will try it now and give feedback


I tried the CC 3.0, and it works as expected.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

No branches or pull requests

8 participants