Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ansible Networking Workshop error #2160

Open
JamesG-Projects opened this issue Jul 2, 2024 · 4 comments
Open

Ansible Networking Workshop error #2160

JamesG-Projects opened this issue Jul 2, 2024 · 4 comments
Assignees

Comments

@JamesG-Projects
Copy link

JamesG-Projects commented Jul 2, 2024

Problem Summary

Workshop Name: Ansible Automation Platform 2 Networking Automation Workshop

Workshop URL: http://m4j6m.example.opentlc.com/

Description:

The workshop provides 3 links -

  1. VS Code access
  2. Automation Controller
  3. SSH access

In some workshop user instances, when trying to access the VS Code URL, you get the message "This site cannot provide a secure connection. student3.m4j6m.example.opentlc.com sent an invalid response. ERR_SSL_PROTOCOL_ERROR".
To circumvent this error you can change "https" to "http", and the login page will load. However, if you then input the password, it says the password is incorrect and you cannot access the VS Code browser link.

In the event that a workshop user can successfully access the https link, this issue does not happen.
Reproducing the issue is unreliable. Some instances work and some do not.

This is not an issue for the Automation Controller or the SSH access, only the VS Code link.

Issue Type

Bug

Extra vars file

N/A

Ansible Playbook Output

N/A

Ansible Version

[student@ansible-1 ~]$ ansible --version
ansible [core 2.15.5]
config file = /etc/ansible/ansible.cfg
configured module search path = ['/home/student/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /usr/local/lib/python3.9/site-packages/ansible
ansible collection location = /home/student/.ansible/collections:/usr/share/ansible/collections
executable location = /usr/local/bin/ansible
python version = 3.9.16 (main, Sep 12 2023, 00:00:00) [GCC 11.3.1 20221121 (Red Hat 11.3.1-4)] (/usr/bin/python3)
jinja version = 3.1.2
libyaml = True

Ansible Configuration

[student@ansible-1 ~]$ ansible-config dump --only-changed
ACTION_WARNINGS(/etc/ansible/ansible.cfg) = False
COLLECTIONS_ON_ANSIBLE_VERSION_MISMATCH(/etc/ansible/ansible.cfg) = ignore
CONFIG_FILE() = /etc/ansible/ansible.cfg
DEFAULT_HOST_LIST(/etc/ansible/ansible.cfg) = ['/home/student/lab_inventory/hosts']
DEFAULT_STDOUT_CALLBACK(/etc/ansible/ansible.cfg) = yaml
DEFAULT_TIMEOUT(/etc/ansible/ansible.cfg) = 60
DEPRECATION_WARNINGS(/etc/ansible/ansible.cfg) = False
DEVEL_WARNING(/etc/ansible/ansible.cfg) = False
HOST_KEY_CHECKING(/etc/ansible/ansible.cfg) = False
INTERPRETER_PYTHON(/etc/ansible/ansible.cfg) = auto_silent
PERSISTENT_COMMAND_TIMEOUT(/etc/ansible/ansible.cfg) = 200
PERSISTENT_CONNECT_TIMEOUT(/etc/ansible/ansible.cfg) = 200
RETRY_FILES_ENABLED(/etc/ansible/ansible.cfg) = False
SYSTEM_WARNINGS(/etc/ansible/ansible.cfg) = False

Ansible Execution Node

CLI Ansible (Ansible Core)

Operating System

Windows & MacOS

@IPvSean IPvSean self-assigned this Jul 3, 2024
@IPvSean
Copy link
Contributor

IPvSean commented Jul 3, 2024

I am trying to see what is going on, recreating now in mye nv

@IPvSean
Copy link
Contributor

IPvSean commented Jul 8, 2024

OK I have been hammering on this most of Friday and today (Monday July 8th). I don't have a fix yet.... but these types of unreliable errors that are hard to reproduce are a giant pain.... it is really hard to figure out what is going on. In parallel I am ordering a bunch of workbenches from demo.redhat.com to see if I can get one to mess up.

Unrelated... but also causing pains are the Cisco IOS-XE support for SSH keys... I am testing a fix right now to move us to ED25519 vs ssh-rsa. My first few tests are favorable and will get rid of a lot of errors people have with ssh legacy SSH keys.

Anyways.... some findings for this specific issue with code-server and the workshop:

  • The UUID for the failed workshop is m4j6m
  • I can see a failure on July 2nd, but then a success rerun July 4th

image

The task that fails on July 2nd->

TASK [infra.controller_configuration.groups : Managing Controller Groups | Wait for finish the Controller Groups management] ***
changed: [m4j6m-student1-ansible-1] => (item=Create/Update Controller Group cisco | Wait for finish the Controller Group creation)
changed: [m4j6m-student2-ansible-1] => (item=Create/Update Controller Group cisco | Wait for finish the Controller Group creation)
changed: [m4j6m-student3-ansible-1] => (item=Create/Update Controller Group cisco | Wait for finish the Controller Group creation)
changed: [m4j6m-student4-ansible-1] => (item=Create/Update Controller Group cisco | Wait for finish the Controller Group creation)
changed: [m4j6m-student6-ansible-1] => (item=Create/Update Controller Group cisco | Wait for finish the Controller Group creation)
changed: [m4j6m-student5-ansible-1] => (item=Create/Update Controller Group cisco | Wait for finish the Controller Group creation)
changed: [m4j6m-student7-ansible-1] => (item=Create/Update Controller Group cisco | Wait for finish the Controller Group creation)
changed: [m4j6m-student8-ansible-1] => (item=Create/Update Controller Group cisco | Wait for finish the Controller Group creation)
changed: [m4j6m-student11-ansible-1] => (item=Create/Update Controller Group cisco | Wait for finish the Controller Group creation)
changed: [m4j6m-student12-ansible-1] => (item=Create/Update Controller Group cisco | Wait for finish the Controller Group creation)
changed: [m4j6m-student10-ansible-1] => (item=Create/Update Controller Group cisco | Wait for finish the Controller Group creation)
changed: [m4j6m-student9-ansible-1] => (item=Create/Update Controller Group cisco | Wait for finish the Controller Group creation)
changed: [m4j6m-student13-ansible-1] => (item=Create/Update Controller Group cisco | Wait for finish the Controller Group creation)
changed: [m4j6m-student14-ansible-1] => (item=Create/Update Controller Group cisco | Wait for finish the Controller Group creation)
changed: [m4j6m-student15-ansible-1] => (item=Create/Update Controller Group cisco | Wait for finish the Controller Group creation)
changed: [m4j6m-student16-ansible-1] => (item=Create/Update Controller Group cisco | Wait for finish the Controller Group creation)
changed: [m4j6m-student17-ansible-1] => (item=Create/Update Controller Group cisco | Wait for finish the Controller Group creation)
changed: [m4j6m-student20-ansible-1] => (item=Create/Update Controller Group cisco | Wait for finish the Controller Group creation)
changed: [m4j6m-student18-ansible-1] => (item=Create/Update Controller Group cisco | Wait for finish the Controller Group creation)
changed: [m4j6m-student21-ansible-1] => (item=Create/Update Controller Group cisco | Wait for finish the Controller Group creation)
changed: [m4j6m-student19-ansible-1] => (item=Create/Update Controller Group cisco | Wait for finish the Controller Group creation)
changed: [m4j6m-student1-ansible-1] => (item=Create/Update Controller Group arista | Wait for finish the Controller Group creation)
changed: [m4j6m-student3-ansible-1] => (item=Create/Update Controller Group arista | Wait for finish the Controller Group creation)
changed: [m4j6m-student2-ansible-1] => (item=Create/Update Controller Group arista | Wait for finish the Controller Group creation)
changed: [m4j6m-student7-ansible-1] => (item=Create/Update Controller Group arista | Wait for finish the Controller Group creation)
changed: [m4j6m-student11-ansible-1] => (item=Create/Update Controller Group arista | Wait for finish the Controller Group creation)
changed: [m4j6m-student4-ansible-1] => (item=Create/Update Controller Group arista | Wait for finish the Controller Group creation)
changed: [m4j6m-student5-ansible-1] => (item=Create/Update Controller Group arista | Wait for finish the Controller Group creation)
changed: [m4j6m-student6-ansible-1] => (item=Create/Update Controller Group arista | Wait for finish the Controller Group creation)
changed: [m4j6m-student8-ansible-1] => (item=Create/Update Controller Group arista | Wait for finish the Controller Group creation)
changed: [m4j6m-student12-ansible-1] => (item=Create/Update Controller Group arista | Wait for finish the Controller Group creation)
changed: [m4j6m-student10-ansible-1] => (item=Create/Update Controller Group arista | Wait for finish the Controller Group creation)
changed: [m4j6m-student13-ansible-1] => (item=Create/Update Controller Group arista | Wait for finish the Controller Group creation)
changed: [m4j6m-student15-ansible-1] => (item=Create/Update Controller Group arista | Wait for finish the Controller Group creation)
changed: [m4j6m-student14-ansible-1] => (item=Create/Update Controller Group arista | Wait for finish the Controller Group creation)
changed: [m4j6m-student19-ansible-1] => (item=Create/Update Controller Group arista | Wait for finish the Controller Group creation)
changed: [m4j6m-student16-ansible-1] => (item=Create/Update Controller Group arista | Wait for finish the Controller Group creation)
changed: [m4j6m-student9-ansible-1] => (item=Create/Update Controller Group arista | Wait for finish the Controller Group creation)
changed: [m4j6m-student17-ansible-1] => (item=Create/Update Controller Group arista | Wait for finish the Controller Group creation)
changed: [m4j6m-student21-ansible-1] => (item=Create/Update Controller Group arista | Wait for finish the Controller Group creation)
changed: [m4j6m-student18-ansible-1] => (item=Create/Update Controller Group arista | Wait for finish the Controller Group creation)
changed: [m4j6m-student20-ansible-1] => (item=Create/Update Controller Group arista | Wait for finish the Controller Group creation)
changed: [m4j6m-student1-ansible-1] => (item=Create/Update Controller Group juniper | Wait for finish the Controller Group creation)
changed: [m4j6m-student3-ansible-1] => (item=Create/Update Controller Group juniper | Wait for finish the Controller Group creation)
changed: [m4j6m-student2-ansible-1] => (item=Create/Update Controller Group juniper | Wait for finish the Controller Group creation)
changed: [m4j6m-student11-ansible-1] => (item=Create/Update Controller Group juniper | Wait for finish the Controller Group creation)
changed: [m4j6m-student7-ansible-1] => (item=Create/Update Controller Group juniper | Wait for finish the Controller Group creation)
changed: [m4j6m-student4-ansible-1] => (item=Create/Update Controller Group juniper | Wait for finish the Controller Group creation)
changed: [m4j6m-student5-ansible-1] => (item=Create/Update Controller Group juniper | Wait for finish the Controller Group creation)
changed: [m4j6m-student10-ansible-1] => (item=Create/Update Controller Group juniper | Wait for finish the Controller Group creation)
changed: [m4j6m-student8-ansible-1] => (item=Create/Update Controller Group juniper | Wait for finish the Controller Group creation)
changed: [m4j6m-student12-ansible-1] => (item=Create/Update Controller Group juniper | Wait for finish the Controller Group creation)
changed: [m4j6m-student6-ansible-1] => (item=Create/Update Controller Group juniper | Wait for finish the Controller Group creation)
changed: [m4j6m-student13-ansible-1] => (item=Create/Update Controller Group juniper | Wait for finish the Controller Group creation)
changed: [m4j6m-student19-ansible-1] => (item=Create/Update Controller Group juniper | Wait for finish the Controller Group creation)
changed: [m4j6m-student15-ansible-1] => (item=Create/Update Controller Group juniper | Wait for finish the Controller Group creation)
changed: [m4j6m-student17-ansible-1] => (item=Create/Update Controller Group juniper | Wait for finish the Controller Group creation)
changed: [m4j6m-student14-ansible-1] => (item=Create/Update Controller Group juniper | Wait for finish the Controller Group creation)
changed: [m4j6m-student20-ansible-1] => (item=Create/Update Controller Group juniper | Wait for finish the Controller Group creation)
changed: [m4j6m-student16-ansible-1] => (item=Create/Update Controller Group juniper | Wait for finish the Controller Group creation)
changed: [m4j6m-student21-ansible-1] => (item=Create/Update Controller Group juniper | Wait for finish the Controller Group creation)
changed: [m4j6m-student18-ansible-1] => (item=Create/Update Controller Group juniper | Wait for finish the Controller Group creation)
changed: [m4j6m-student9-ansible-1] => (item=Create/Update Controller Group juniper | Wait for finish the Controller Group creation)
changed: [m4j6m-student1-ansible-1] => (item=Create/Update Controller Group routers | Wait for finish the Controller Group creation)
changed: [m4j6m-student3-ansible-1] => (item=Create/Update Controller Group routers | Wait for finish the Controller Group creation)
changed: [m4j6m-student2-ansible-1] => (item=Create/Update Controller Group routers | Wait for finish the Controller Group creation)
changed: [m4j6m-student11-ansible-1] => (item=Create/Update Controller Group routers | Wait for finish the Controller Group creation)
changed: [m4j6m-student7-ansible-1] => (item=Create/Update Controller Group routers | Wait for finish the Controller Group creation)
changed: [m4j6m-student5-ansible-1] => (item=Create/Update Controller Group routers | Wait for finish the Controller Group creation)
changed: [m4j6m-student4-ansible-1] => (item=Create/Update Controller Group routers | Wait for finish the Controller Group creation)
changed: [m4j6m-student10-ansible-1] => (item=Create/Update Controller Group routers | Wait for finish the Controller Group creation)
changed: [m4j6m-student8-ansible-1] => (item=Create/Update Controller Group routers | Wait for finish the Controller Group creation)
changed: [m4j6m-student12-ansible-1] => (item=Create/Update Controller Group routers | Wait for finish the Controller Group creation)
changed: [m4j6m-student19-ansible-1] => (item=Create/Update Controller Group routers | Wait for finish the Controller Group creation)
changed: [m4j6m-student13-ansible-1] => (item=Create/Update Controller Group routers | Wait for finish the Controller Group creation)
changed: [m4j6m-student15-ansible-1] => (item=Create/Update Controller Group routers | Wait for finish the Controller Group creation)
FAILED - RETRYING: [m4j6m-student6-ansible-1]: Managing Controller Groups | Wait for finish the Controller Groups management (30 retries left).
changed: [m4j6m-student17-ansible-1] => (item=Create/Update Controller Group routers | Wait for finish the Controller Group creation)
changed: [m4j6m-student20-ansible-1] => (item=Create/Update Controller Group routers | Wait for finish the Controller Group creation)
changed: [m4j6m-student21-ansible-1] => (item=Create/Update Controller Group routers | Wait for finish the Controller Group creation)
changed: [m4j6m-student14-ansible-1] => (item=Create/Update Controller Group routers | Wait for finish the Controller Group creation)
changed: [m4j6m-student16-ansible-1] => (item=Create/Update Controller Group routers | Wait for finish the Controller Group creation)
changed: [m4j6m-student18-ansible-1] => (item=Create/Update Controller Group routers | Wait for finish the Controller Group creation)
changed: [m4j6m-student9-ansible-1] => (item=Create/Update Controller Group routers | Wait for finish the Controller Group creation)
failed: [m4j6m-student1-ansible-1] (item=Create/Update Controller Group network | Wait for finish the Controller Group creation) => {"__group_job_async_results_item": {"__controller_groups_item": {"children": ["routers"], "inventory": "Workshop Inventory", "name": "network", "variables": {"restore_credential": "Workshop Credential", "restore_inventory": "Workshop Inventory", "restore_project": "Workshop Project"}}, "ansible_job_id": "j156787352862.21826", "ansible_loop_var": "__controller_groups_item", "changed": false, "failed": 0, "finished": 0, "results_file": "/root/.ansible_async/j156787352862.21826", "started": 1}, "ansible_job_id": "j156787352862.21826", "ansible_loop_var": "__group_job_async_results_item", "attempts": 1, "changed": false, "finished": 1, "msg": "Could not find groups with name routers", "results_file": "/root/.ansible_async/j156787352862.21826", "started": 1, "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
failed: [m4j6m-student3-ansible-1] (item=Create/Update Controller Group network | Wait for finish the Controller Group creation) => {"__group_job_async_results_item": {"__controller_groups_item": {"children": ["routers"], "inventory": "Workshop Inventory", "name": "network", "variables": {"restore_credential": "Workshop Credential", "restore_inventory": "Workshop Inventory", "restore_project": "Workshop Project"}}, "ansible_job_id": "j859423898414.21486", "ansible_loop_var": "__controller_groups_item", "changed": false, "failed": 0, "finished": 0, "results_file": "/root/.ansible_async/j859423898414.21486", "started": 1}, "ansible_job_id": "j859423898414.21486", "ansible_loop_var": "__group_job_async_results_item", "attempts": 1, "changed": false, "finished": 1, "msg": "Could not find groups with name routers", "results_file": "/root/.ansible_async/j859423898414.21486", "started": 1, "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
failed: [m4j6m-student2-ansible-1] (item=Create/Update Controller Group network | Wait for finish the Controller Group creation) => {"__group_job_async_results_item": {"__controller_groups_item": {"children": ["routers"], "inventory": "Workshop Inventory", "name": "network", "variables": {"restore_credential": "Workshop Credential", "restore_inventory": "Workshop Inventory", "restore_project": "Workshop Project"}}, "ansible_job_id": "j989485026492.21631", "ansible_loop_var": "__controller_groups_item", "changed": false, "failed": 0, "finished": 0, "results_file": "/root/.ansible_async/j989485026492.21631", "started": 1}, "ansible_job_id": "j989485026492.21631", "ansible_loop_var": "__group_job_async_results_item", "attempts": 1, "changed": false, "finished": 1, "msg": "Could not find groups with name routers", "results_file": "/root/.ansible_async/j989485026492.21631", "started": 1, "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
failed: [m4j6m-student7-ansible-1] (item=Create/Update Controller Group network | Wait for finish the Controller Group creation) => {"__group_job_async_results_item": {"__controller_groups_item": {"children": ["routers"], "inventory": "Workshop Inventory", "name": "network", "variables": {"restore_credential": "Workshop Credential", "restore_inventory": "Workshop Inventory", "restore_project": "Workshop Project"}}, "ansible_job_id": "j466050216929.21465", "ansible_loop_var": "__controller_groups_item", "changed": false, "failed": 0, "finished": 0, "results_file": "/root/.ansible_async/j466050216929.21465", "started": 1}, "ansible_job_id": "j466050216929.21465", "ansible_loop_var": "__group_job_async_results_item", "attempts": 1, "changed": false, "finished": 1, "msg": "Could not find groups with name routers", "results_file": "/root/.ansible_async/j466050216929.21465", "started": 1, "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
failed: [m4j6m-student11-ansible-1] (item=Create/Update Controller Group network | Wait for finish the Controller Group creation) => {"__group_job_async_results_item": {"__controller_groups_item": {"children": ["routers"], "inventory": "Workshop Inventory", "name": "network", "variables": {"restore_credential": "Workshop Credential", "restore_inventory": "Workshop Inventory", "restore_project": "Workshop Project"}}, "ansible_job_id": "j739768768972.21481", "ansible_loop_var": "__controller_groups_item", "changed": false, "failed": 0, "finished": 0, "results_file": "/root/.ansible_async/j739768768972.21481", "started": 1}, "ansible_job_id": "j739768768972.21481", "ansible_loop_var": "__group_job_async_results_item", "attempts": 1, "changed": false, "finished": 1, "msg": "Could not find groups with name routers", "results_file": "/root/.ansible_async/j739768768972.21481", "started": 1, "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
changed: [m4j6m-student5-ansible-1] => (item=Create/Update Controller Group network | Wait for finish the Controller Group creation)
changed: [m4j6m-student4-ansible-1] => (item=Create/Update Controller Group network | Wait for finish the Controller Group creation)
changed: [m4j6m-student10-ansible-1] => (item=Create/Update Controller Group network | Wait for finish the Controller Group creation)
changed: [m4j6m-student8-ansible-1] => (item=Create/Update Controller Group network | Wait for finish the Controller Group creation)
changed: [m4j6m-student12-ansible-1] => (item=Create/Update Controller Group network | Wait for finish the Controller Group creation)
failed: [m4j6m-student19-ansible-1] (item=Create/Update Controller Group network | Wait for finish the Controller Group creation) => {"__group_job_async_results_item": {"__controller_groups_item": {"children": ["routers"], "inventory": "Workshop Inventory", "name": "network", "variables": {"restore_credential": "Workshop Credential", "restore_inventory": "Workshop Inventory", "restore_project": "Workshop Project"}}, "ansible_job_id": "j365342199221.21364", "ansible_loop_var": "__controller_groups_item", "changed": false, "failed": 0, "finished": 0, "results_file": "/root/.ansible_async/j365342199221.21364", "started": 1}, "ansible_job_id": "j365342199221.21364", "ansible_loop_var": "__group_job_async_results_item", "attempts": 1, "changed": false, "finished": 1, "msg": "Could not find groups with name routers", "results_file": "/root/.ansible_async/j365342199221.21364", "started": 1, "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
changed: [m4j6m-student13-ansible-1] => (item=Create/Update Controller Group network | Wait for finish the Controller Group creation)
changed: [m4j6m-student15-ansible-1] => (item=Create/Update Controller Group network | Wait for finish the Controller Group creation)
changed: [m4j6m-student17-ansible-1] => (item=Create/Update Controller Group network | Wait for finish the Controller Group creation)
changed: [m4j6m-student20-ansible-1] => (item=Create/Update Controller Group network | Wait for finish the Controller Group creation)
changed: [m4j6m-student21-ansible-1] => (item=Create/Update Controller Group network | Wait for finish the Controller Group creation)
changed: [m4j6m-student16-ansible-1] => (item=Create/Update Controller Group network | Wait for finish the Controller Group creation)
changed: [m4j6m-student14-ansible-1] => (item=Create/Update Controller Group network | Wait for finish the Controller Group creation)
failed: [m4j6m-student18-ansible-1] (item=Create/Update Controller Group network | Wait for finish the Controller Group creation) => {"__group_job_async_results_item": {"__controller_groups_item": {"children": ["routers"], "inventory": "Workshop Inventory", "name": "network", "variables": {"restore_credential": "Workshop Credential", "restore_inventory": "Workshop Inventory", "restore_project": "Workshop Project"}}, "ansible_job_id": "j171681188278.21375", "ansible_loop_var": "__controller_groups_item", "changed": false, "failed": 0, "finished": 0, "results_file": "/root/.ansible_async/j171681188278.21375", "started": 1}, "ansible_job_id": "j171681188278.21375", "ansible_loop_var": "__group_job_async_results_item", "attempts": 1, "changed": false, "finished": 1, "msg": "Could not find groups with name routers", "results_file": "/root/.ansible_async/j171681188278.21375", "started": 1, "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
changed: [m4j6m-student9-ansible-1] => (item=Create/Update Controller Group network | Wait for finish the Controller Group creation)
changed: [m4j6m-student6-ansible-1] => (item=Create/Update Controller Group routers | Wait for finish the Controller Group creation)
changed: [m4j6m-student6-ansible-1] => (item=Create/Update Controller Group network | Wait for finish the Controller Group creation)

It is weird that it is failing for 7 (seven) hosts but the others are fine, because they should all be identical. It also.... "sorta' matches that "some hosts" are broken..... There seems to be some sort of "order of operations" with using controller configuration collection to pre-provision groups.

However... I am not sure why this would affect visual studio code (or code-server in this case). These roles are not connected in anyway...

when I look at the code-server code everything ran correctly....

My thoughts:

  • I am not 100% positive that the populate_controller role failure affects code-server at all... I need to replicate both failures. It is just interesting how random hosts were affected... and its also hard to reproduce.
  • It is possible that some memory or storage issue affected the hosts... so when hammered with API requests the machine just failed... which would then fail code-server... again I can't prove this yet...
  • I originally thought this was a lets encrypt issue which would explain the intermittent failures but nothing in logs is pointing at that so far.

I wish I was 100% certain what is going on... this workshop type is in the process of being migrated to agnosticd and the demo.redhat.com native software so I have not been maintaining it as frequently as I used to... so this is also probably slowing down mean time to resolution. Keep reporting any issue you are seeing and I will continue to hammer on this.

@IPvSean
Copy link
Contributor

IPvSean commented Jul 8, 2024

OK it is related b/c its a handler.... the handler to restart code-server is never run

I was able to recreate

RUNNING HANDLER [../roles/code_server_native : Restart code-server] ************
changed: [kwtpc-student3-ansible-1]
changed: [kwtpc-student2-ansible-1]
changed: [kwtpc-student4-ansible-1]
[WARNING]: Could not match supplied host pattern, ignoring: automation_hub

In this example, student1 and studen5 are missing from the code-server restart, b/c they failed on the identical task

failed: [kwtpc-student1-ansible-1] (item=Create/Update Controller Group network | Wait for finish the Controller Group creation) => {"__group_job_async_results_item": {"__controller_groups_item": {"children": ["routers"], "inventory": "Workshop Inventory", "name": "network", "variables": {"restore_credential": "Workshop Credential", "restore_inventory": "Workshop Inventory", "restore_project": "Workshop Project"}}, "ansible_job_id": "j685691272827.21906", "ansible_loop_var": "__controller_groups_item", "changed": false, "failed": 0, "finished": 0, "results_file": "/root/.ansible_async/j685691272827.21906", "started": 1}, "ansible_job_id": "j685691272827.21906", "ansible_loop_var": "__group_job_async_results_item", "attempts": 1, "changed": false, "finished": 1, "msg": "Could not find groups with name routers", "results_file": "/root/.ansible_async/j685691272827.21906", "started": 1, "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}

@IPvSean
Copy link
Contributor

IPvSean commented Jul 8, 2024

So the reason this fails is because when I use the controller_configuration content collection

controller_groups:
  - name: cisco
    inventory: "Workshop Inventory"
    variables:
      ansible_network_os: ios
      ansible_connection: network_cli
  - name: arista
    inventory: "Workshop Inventory"
    variables:
      ansible_network_os: eos
      ansible_connection: network_cli
      ansible_become: true
      ansible_become_method: enable
  - name: juniper
    inventory: "Workshop Inventory"
    variables:
      ansible_network_os: junos
      ansible_connection: netconf
  - name: routers
    inventory: "Workshop Inventory"
    children:
      - cisco
      - arista
      - juniper
  - name: network
    inventory: "Workshop Inventory"
    children:
      - routers
    variables:
      restore_inventory: "Workshop Inventory"
      restore_credential: "Workshop Credential"
      restore_project: "Workshop Project"

It is a dictionary...

dictionaries are commonly not ordered.... so the problem

failed: [kwtpc-student1-ansible-1] (item=Create/Update Controller Group network | Wait for finish the Controller Group creation) => {"__group_job_async_results_item": {"__controller_groups_item": {"children": ["routers"], "inventory": "Workshop Inventory", "name": "network", "variables": {"restore_credential": "Workshop Credential", "restore_inventory": "Workshop Inventory", "restore_project": "Workshop Project"}}, "ansible_job_id": "j685691272827.21906", "ansible_loop_var": "__controller_groups_item", "changed": false, "failed": 0, "finished": 0, "results_file": "/root/.ansible_async/j685691272827.21906", "started": 1}, "ansible_job_id": "j685691272827.21906", "ansible_loop_var": "__group_job_async_results_item", "attempts": 1, "changed": false, "finished": 1, "msg": "Could not find groups with name routers", "results_file": "/root/.ansible_async/j685691272827.21906", "started": 1, "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}

happens SOMETIMES because it will try to configure the group network before the group routers has been created (which has like a 1 in 5 chance... of happening) or 20% it seems.... looking at rough math above.... 20% of 20 student is 4 and we had 7 hosts messed up, so this roughly now makes sense to me.

I have gotten rid of those two groups from using controller configuration and forced them into manual adds at the end in the right order. Will send fixes tomorrow.

IPvSean added a commit to IPvSean/workshops that referenced this issue Jul 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants