Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Ansible provisioning never returns #169

Open
jacklund opened this issue Feb 1, 2022 · 2 comments
Open

Ansible provisioning never returns #169

jacklund opened this issue Feb 1, 2022 · 2 comments

Comments

@jacklund
Copy link

jacklund commented Feb 1, 2022

Overview of the Issue

I'm creating a ubuntu 21.10 raspberry pi image using ansible 2.9.6. My ansible playbook is very simple (so far), but it hangs when it tries to run echo ~root && sleep 0, which, AFAICT, ansible runs to test the connection. When I try to connect via ssh to the packer ssh communicator and run a command, the command output returns but the connection never does, which is what I think ansible is waiting on.

Reproduction Steps

% sudo PACKER_DEBUG=1 packer build packer/ubuntu_server_21.10_arm64.json.pkr.hcl

Packer version

v1.7.8

Simplified Packer Buildfile

source "arm" "raspberry_pi_k8s" {
  file_checksum_type    = "sha256"
  file_checksum_url     = "http://cdimage.ubuntu.com/releases/21.10/release/SHA256SUMS"
  file_target_extension = "xz"
  file_unarchive_cmd    = ["xz", "--decompress", "$ARCHIVE_PATH"]
  file_urls             = ["http://cdimage.ubuntu.com/releases/21.10/release/ubuntu-21.10-preinstalled-server-arm64+raspi.img.xz"]
  image_build_method    = "reuse"
  image_chroot_env      = ["PATH=/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin"]
  image_partitions {
    filesystem   = "fat"
    mountpoint   = "/boot/firmware"
    name         = "boot"
    size         = "256M"
    start_sector = "2048"
    type         = "c"
  }
  image_partitions {
    filesystem   = "ext4"
    mountpoint   = "/"
    name         = "root"
    size         = "2.8G"
    start_sector = "526336"
    type         = "83"
  }
  image_path                   = "ubuntu-21.10.img"
  image_size                   = "3.1G"
  image_type                   = "dos"
  qemu_binary_destination_path = "/usr/bin/qemu-aarch64-static"
  qemu_binary_source_path      = "/usr/bin/qemu-aarch64-static"
}

build {
  sources = ["source.arm.raspberry_pi_k8s"]

  provisioner "ansible" {
    playbook_file = "ansible/playbook.yml"
    extra_arguments = [ "-vvvv" ]
  }
}

Ansible Playbook

---
- name: 'Provision image'
  hosts: default
  become: true
  gather_facts: no
  vars_files:
    - vars.yml

  tasks:
    - name: Debug
      debug:
        msg: "Testing 1 2 3"

    - name: Create user group
      group:
        name: "{{ username }}"
        state: present

Operating system and Environment details

Ubuntu 20.04 on x86_64

Log Fragments and crash.log files

Packer log

Output of manually running ssh command

% sudo ssh -v 127.0.0.1 -o Port=34669 -o 'IdentityFile="/tmp/ansible-key544648103"' 'pwd' 
OpenSSH_8.2p1 Ubuntu-4ubuntu0.4, OpenSSL 1.1.1f  31 Mar 2020
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug1: Connecting to 127.0.0.1 [127.0.0.1] port 34669.
debug1: Connection established.
debug1: identity file /tmp/ansible-key544648103 type -1
debug1: identity file /tmp/ansible-key544648103-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.4
debug1: Remote protocol version 2.0, remote software version Go
debug1: no match: Go
debug1: Authenticating to 127.0.0.1:34669 as 'root'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ssh-rsa
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ssh-rsa SHA256:9ZU07UhwYA8vXs1g6QILOxwnzOYM4peD+AX1ZoB1tks
debug1: Host '[127.0.0.1]:34669' is known and matches the RSA host key.
debug1: Found key in /root/.ssh/known_hosts:16
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /tmp/ansible-key544648103  explicit
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /tmp/ansible-key544648103
debug1: Authentication succeeded (publickey).
Authenticated to 127.0.0.1 ([127.0.0.1]:34669).
debug1: channel 0: new [client-session]
debug1: Entering interactive session.
debug1: pledge: network
debug1: Sending environment.
debug1: Sending env LANG = en_US.UTF-8
debug1: Sending command: pwd
/
<Command hangs here>
^Cdebug1: channel 0: free: client-session, nchannels 1
Killed by signal 2.
@goldyfruit
Copy link

goldyfruit commented Mar 16, 2022 •
edited
Loading

Same here, here is the Packer output:

==> arm.recovery: Retrieving rootfs_archive
==> arm.recovery: Trying https://downloads.raspberrypi.org/raspios_lite_arm64/images/raspios_lite_arm64-2022-01-28/2022-01-28-raspios-bullseye-arm64-lite.zip?archive=false
==> arm.recovery: Trying https://downloads.raspberrypi.org/raspios_lite_arm64/images/raspios_lite_arm64-2022-01-28/2022-01-28-raspios-bullseye-arm64-lite.zip?archive=false&checksum=file%3Ahttps%3A%2F%2Fdownloads.raspberrypi.org%2Fraspios_lite_arm64%2Fimages%2Fraspios_lite_arm64-2022-01-28%2F2022-01-28-raspios-bullseye-arm64-lite.zip.sha256
==> arm.recovery: https://downloads.raspberrypi.org/raspios_lite_arm64/images/raspios_lite_arm64-2022-01-28/2022-01-28-raspios-bullseye-arm64-lite.zip?archive=false&checksum=file%3Ahttps%3A%2F%2Fdownloads.raspberrypi.org%2Fraspios_lite_arm64%2Fimages%2Fraspios_lite_arm64-2022-01-28%2F2022-01-28-raspios-bullseye-arm64-lite.zip.sha256 => /root/.cache/packer/258574cfe2c5e2a35c3bbc9affc3e113d9d98e63.zip
    arm.recovery: unpacking /root/.cache/packer/258574cfe2c5e2a35c3bbc9affc3e113d9d98e63.zip to recovery.img
    arm.recovery: resizing the image file recovery.img to 6G
    arm.recovery: expanding partition no. 2 on the image recovery.img
    arm.recovery: mapping image recovery.img to free loopback device
    arm.recovery: image recovery.img mapped to /dev/loop2
    arm.recovery: running resize2fs on /dev/loop2p2
    arm.recovery: mounting /dev/loop2p2 to /tmp/3188529091
    arm.recovery: mounting /dev/loop2p1 to /tmp/3188529091/boot
    arm.recovery: running extra setup
    arm.recovery: mounting /dev with: [mount --bind /dev /tmp/3188529091/dev]
    arm.recovery: mounting /devpts with: [mount -t devpts /devpts /tmp/3188529091/dev/pts]
    arm.recovery: mounting proc with: [mount -t proc proc /tmp/3188529091/proc]
    arm.recovery: mounting binfmt_misc with: [mount -t binfmt_misc binfmt_misc /tmp/3188529091/proc/sys/fs/binfmt_misc]
    arm.recovery: mounting sysfs with: [mount -t sysfs sysfs /tmp/3188529091/sys]
    arm.recovery: running the provision hook
==> arm.recovery: Provisioning with shell script: /tmp/packer-shell576595168
    arm.recovery: [all]
    arm.recovery: dtoverlay=vc4-kms-v3d
    arm.recovery: max_framebuffers=1
    arm.recovery: arm_64bit=1
    arm.recovery: over_voltage=8
    arm.recovery: arm_freq=2100
    arm.recovery: gpu_freq=650
    arm.recovery: enable_uart=1
    arm.recovery: start_x=1
    arm.recovery: gpu_mem=256
    arm.recovery: initial_turbo=60
    arm.recovery: gpio=14=op,dl
    arm.recovery: dtoverlay=disable-wifi
    arm.recovery: dtoverlay=gpio-fan,gpiopin=21,temp=65000
    arm.recovery: force_turbo=1
    arm.recovery: initial_turbo=60
    arm.recovery: avoid_warnings=2
    arm.recovery: dtparam=i2c_arm=on
    arm.recovery: dtparam=spi=on
    arm.recovery: dtoverlay=gpio-ir,gpio_pin=26
    arm.recovery: dtoverlay=gpio-ir-tx,gpio_pin=19
    arm.recovery: avoid_warnings=2
    arm.recovery: disable_splash=1
    arm.recovery:
==> arm.recovery: Provisioning with shell script: /tmp/packer-shell440055694
    arm.recovery: console=serial0,115200 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait quiet init=/usr/lib/raspi-config/init_resize.sh
==> arm.recovery: Provisioning with shell script: /tmp/packer-shell292882634
    arm.recovery: en_US.UTF-8 UTF-8
==> arm.recovery: Provisioning with shell script: /tmp/packer-shell857059794
    arm.recovery: deb [arch=arm64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/debian bullseye test
    arm.recovery:
==> arm.recovery: Provisioning with Ansible...
    arm.recovery: Setting up proxy adapter for Ansible....
==> arm.recovery: Executing Ansible: ansible-playbook -e packer_build_name="recovery" -e packer_builder_type=arm -vvvv -e ansible_ssh_private_key_file=/tmp/ansible-key042511905 -i /tmp/packer-provisioner-ansible556372748 /build/ansible/recovery.yml
    arm.recovery: ansible-playbook 2.9.6
    arm.recovery:   config file = /etc/ansible/ansible.cfg
    arm.recovery:   configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
    arm.recovery:   ansible python module location = /usr/lib/python3/dist-packages/ansible
    arm.recovery:   executable location = /usr/bin/ansible-playbook
    arm.recovery:   python version = 3.8.10 (default, Nov 26 2021, 20:14:08) [GCC 9.3.0]
    arm.recovery: Using /etc/ansible/ansible.cfg as config file
    arm.recovery: setting up inventory plugins
    arm.recovery: host_list declined parsing /tmp/packer-provisioner-ansible556372748 as it did not pass its verify_file() method
    arm.recovery: script declined parsing /tmp/packer-provisioner-ansible556372748 as it did not pass its verify_file() method
    arm.recovery: auto declined parsing /tmp/packer-provisioner-ansible556372748 as it did not pass its verify_file() method
    arm.recovery: Parsed /tmp/packer-provisioner-ansible556372748 inventory source with ini plugin
    arm.recovery: Loading callback plugin default of type stdout, v2.0 from /usr/lib/python3/dist-packages/ansible/plugins/callback/default.py
    arm.recovery:
    arm.recovery: PLAYBOOK: recovery.yml *********************************************************
    arm.recovery: Positional arguments: /build/ansible/recovery.yml
    arm.recovery: verbosity: 4
    arm.recovery: connection: smart
    arm.recovery: timeout: 10
    arm.recovery: become_method: sudo
    arm.recovery: tags: ('all',)
    arm.recovery: inventory: ('/tmp/packer-provisioner-ansible556372748',)
    arm.recovery: extra_vars: ('packer_build_name="recovery"', 'packer_builder_type=arm', 'ansible_ssh_private_key_file=/tmp/ansible-key042511905')
    arm.recovery: forks: 5
    arm.recovery: 1 plays in /build/ansible/recovery.yml
    arm.recovery:
    arm.recovery: PLAY [default] *****************************************************************
    arm.recovery: META: ran handlers
    arm.recovery:
    arm.recovery: TASK [Add repositories] ********************************************************
    arm.recovery: task path: /build/ansible/recovery.yml:6
    arm.recovery: <127.0.0.1> ESTABLISH SSH CONNECTION FOR USER: root
    arm.recovery: <127.0.0.1> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o Port=45941 -o 'IdentityFile="/tmp/ansible-key042511905"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="root"' -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/d901886dc9 127.0.0.1 '/bin/sh -c '"'"'echo ~root && sleep 0'"'"''

And same conclusion as @jacklund, when running the SSH command from the container, the command returns the output but I have to hit a key to exit.

root@46c5f7adc081:/tmp# ssh -v 127.0.0.1 -o Port=45941 -o 'IdentityFile="/tmp/ansible-key042511905"' 'ip a'
OpenSSH_8.2p1 Ubuntu-4ubuntu0.4, OpenSSL 1.1.1f  31 Mar 2020
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug1: Connecting to 127.0.0.1 [127.0.0.1] port 45941.
debug1: Connection established.
debug1: identity file /tmp/ansible-key042511905 type -1
debug1: identity file /tmp/ansible-key042511905-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.4
debug1: Remote protocol version 2.0, remote software version Go
debug1: no match: Go
debug1: Authenticating to 127.0.0.1:45941 as 'root'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ssh-rsa
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ssh-rsa SHA256:mSYV+OZHs6abq3PkdEQdsZbqyfDCvFAzPtGnTOdvRSA
debug1: Host '[127.0.0.1]:45941' is known and matches the RSA host key.
debug1: Found key in /root/.ssh/known_hosts:1
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /tmp/ansible-key042511905  explicit
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /tmp/ansible-key042511905
debug1: Authentication succeeded (publickey).
Authenticated to 127.0.0.1 ([127.0.0.1]:45941).
debug1: channel 0: new [client-session]
debug1: Entering interactive session.
debug1: pledge: network
debug1: Sending environment.
debug1: Sending command: ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
32: eth0@if33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.17.0.2/16 brd 172.17.255.255 scope global eth0
       valid_lft forever preferred_lft forever

debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: channel 0: free: client-session, nchannels 1
Transferred: sent 2424, received 1892 bytes, in 3.5 seconds
Bytes per second: sent 696.0, received 543.3
debug1: Exit status 0

@goldyfruit
Copy link

Just got this:

==> arm.recovery: failed to handshake

@brucellino brucellino mentioned this issue Jul 21, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jacklund @goldyfruit