Failed to connect to validate entity - Are entities not added automatically?

[nagyt234]

Description of problem

• What did you do?

I've installed basic sensu-backend with sensu-puppet with an agent on the same machine (moni2....) and with an agent on another machine (drawmgt....). Both machines are VMs.

• What happened?

I see in the dashboard, that the events of the entity, where the backend and the agent are on the same VM, are working correctlly. However the agent on the separate machine fails already at running puppet with the error message "Failed to connect to validate entity". From the debug log I can see, that the connection to the backend is working correctly, but querying the entity fails with RESPONSE: 404 {"message":"not found","code":2}.

• What did you expect to happen?

The backend should register new agents automatically.

• How can someone reproduce the problem?

Use the following manifest files, where moni2... is the backend and drawmgt... is the agent::

node "moni2.mydomain.org" {
  class { 'sensu':
    password => 'xXxXxX',
    use_ssl => false,
    agent_entity_config_password => 'xXxXxX',
  }
  include sensu::backend
  sensu_check { 'check-cpu':
    ensure        => 'present',
    command       => 'check-cpu.rb -w 75 -c 90',
    interval      => 60,
    subscriptions => ['linux'],
    runtime_assets => ['sensu-plugins-cpu-checks','sensu-ruby-runtime'],
  }
  sensu_bonsai_asset { 'sensu-plugins/sensu-plugins-cpu-checks':
    rename => 'sensu-plugins-cpu-checks',
    ensure => 'present',
  }
  sensu_bonsai_asset { 'sensu/sensu-ruby-runtime':
    rename => 'sensu-ruby-runtime',
    ensure => 'present',
  }
  class { 'sensu::agent':
    subscriptions => ['linux'],
  }
}

node "drawmgt.zg-3.mydomain.org" {
  class { 'sensu':
    api_host                     => 'moni2.mydomain.org',
    agent_entity_config_password => 'xXxXxX',
    password => 'xXxXxX',
  }
  class { 'sensu::agent':
    backends      => ['moni2.mydomain.org:8081'],
    subscriptions => ['linux'],
  }
}

Command used and debugging output

• What does your puppet config look like (including any hiera config)

See the manifests files above

• Is this a masterless or master based puppet setup?

masterless

Platform and version information

• Your OS: Ubuntu 18.04.5 LTS, running on a Vagarnt virtual machine.
• Your Ruby version: ruby is not installed
• Your version of Puppet: 6.25.1 open source
• Your version of Sensu: 6.6.3-5588
• Your version of this module: 5.8.0

Anything else to add that you think will be helpful?

The debug output of the puppet agent:

Debug: Executing: '/bin/systemctl is-active -- sensu-agent'
Debug: Executing: '/bin/systemctl is-enabled -- sensu-agent'
Debug: method=get url=https://moni2.mydomain.org:8080/api/core/v2/namespaces/default/entities/drawmgt.zg-3.mydomain.org path=entities/drawmgt.zg-3.mydomain.org
Debug: Sensu API: Using basic auth of admin:xXxXxX
Debug: RESPONSE: 401
{"Code":5,"Message":"bad credentials"}
Debug: method=get url=https://moni2.mydomain.org:8080/auth path=/auth
Debug: Sensu API: Using basic auth of admin:xXxXxX
Debug: RESPONSE: 200
{"access_token":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2NDMxODYyNjUsImp0aSI6ImM5NTY3M2ZiMTJjYmY2NzBlYjE4NzUwYTUxOTQ3NTcwIiwiaXNzIjoiaHR0cDovL21vbmkyLnNvZnR4cy5jaDo4MDgwIiwic3ViIjoiYWRtaW4iLCJncm91cHMiOlsiY2x1c3Rlci1hZG1pbnMiLCJzeXN0ZW06dXNlcnMiXSwicHJvdmlkZXIiOnsicHJvdmlkZXJfaWQiOiJiYXNpYyIsInByb3ZpZGVyX3R5cGUiOiIiLCJ1c2VyX2lkIjoiYWRtaW4ifSwiYXBpX2tleSI6ZmFsc2V9.-LlED39V0fSLHufsHCq0PvdTFa8k5GMZ7VuaXBQIp-M","expires_at":1643186265,"refresh_token":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJqdGkiOiIwNTIzYTA5MjE1OGMyYTljZDFmMzBkYjc4ZGYxYWNhOSIsInN1YiI6ImFkbWluIiwiZ3JvdXBzIjpudWxsLCJwcm92aWRlciI6eyJwcm92aWRlcl9pZCI6IiIsInByb3ZpZGVyX3R5cGUiOiIiLCJ1c2VyX2lkIjoiIn0sImFwaV9rZXkiOmZhbHNlfQ.CoaF7Cw2BXXDxaBZ-MjXcjnBHPFDJGUaHwmovBroP-s"}

Debug: method=get url=https://moni2.mydomain.org:8080/api/core/v2/namespaces/default/entities/drawmgt.zg-3.mydomain.org path=entities/drawmgt.zg-3.mydomain.org
Debug: Sensu API: Using token eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2NDMxODYyNjUsImp0aSI6ImM5NTY3M2ZiMTJjYmY2NzBlYjE4NzUwYTUxOTQ3NTcwIiwiaXNzIjoiaHR0cDovL21vbmkyLnNvZnR4cy5jaDo4MDgwIiwic3ViIjoiYWRtaW4iLCJncm91cHMiOlsiY2x1c3Rlci1hZG1pbnMiLCJzeXN0ZW06dXNlcnMiXSwicHJvdmlkZXIiOnsicHJvdmlkZXJfaWQiOiJiYXNpYyIsInByb3ZpZGVyX3R5cGUiOiIiLCJ1c2VyX2lkIjoiYWRtaW4ifSwiYXBpX2tleSI6ZmFsc2V9.-LlED39V0fSLHufsHCq0PvdTFa8k5GMZ7VuaXBQIp-M
Debug: RESPONSE: 404
{"message":"not found","code":2}
Notice: Failed to connect to validate entity drawmgt.zg-3.mydomain.org; sleeping 2 seconds before retry

5 TIMES the last messages

Notice: Failed to connect validate entity drawmgt.zg-3.mydomain.org within timeout window of 10 seconds; giving up.
Debug: /Stage[main]/Sensu::Agent/Sensu_agent_entity_validator[drawmgt.zg-3.mydomain.org]: Nothing to manage: no ensure and the resource doesn't exist
Debug: Prefetching sensu_api resources for sensu_agent_entity_config
Debug: Fetching namespaces via Sensu API
Debug: method=get url=https://moni2.mydomain.org:8080/api/core/v2/namespaces path=namespaces
Debug: Sensu API: Using token eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2NDMxODYyNjMsImp0aSI6IjEyZGQxNmE5ODJmNGNkNGFiOTJjMzFmYTkyMDM2NDQzIiwiaXNzIjoiaHR0cDovL21vbmkyLnNvZnR4cy5jaDo4MDgwIiwic3ViIjoicHVwcGV0LWFnZW50X2VudGl0eV9jb25maWciLCJncm91cHMiOlsicHVwcGV0OmFnZW50X2VudGl0eV9jb25maWciLCJzeXN0ZW06dXNlcnMiXSwicHJvdmlkZXIiOnsicHJvdmlkZXJfaWQiOiJiYXNpYyIsInByb3ZpZGVyX3R5cGUiOiIiLCJ1c2VyX2lkIjoicHVwcGV0LWFnZW50X2VudGl0eV9jb25maWcifSwiYXBpX2tleSI6ZmFsc2V9.jPQ6E47AHFjTN31AkxKmi9lhQo8zO0qwGtLdogAECQc
Debug: RESPONSE: 200
[{"name":"default"}]
Debug: BODY: [
  {
    "name": "default"
  }
]
Debug: method=get url=https://moni2.mydomain.org:8080/api/core/v2/namespaces/default/entities path=entities
Debug: Sensu API: Using token eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2NDMxODYyNjMsImp0aSI6IjEyZGQxNmE5ODJmNGNkNGFiOTJjMzFmYTkyMDM2NDQzIiwiaXNzIjoiaHR0cDovL21vbmkyLnNvZnR4cy5jaDo4MDgwIiwic3ViIjoicHVwcGV0LWFnZW50X2VudGl0eV9jb25maWciLCJncm91cHMiOlsicHVwcGV0OmFnZW50X2VudGl0eV9jb25maWciLCJzeXN0ZW06dXNlcnMiXSwicHJvdmlkZXIiOnsicHJvdmlkZXJfaWQiOiJiYXNpYyIsInByb3ZpZGVyX3R5cGUiOiIiLCJ1c2VyX2lkIjoicHVwcGV0LWFnZW50X2VudGl0eV9jb25maWcifSwiYXBpX2tleSI6ZmFsc2V9.jPQ6E47AHFjTN31AkxKmi9lhQo8zO0qwGtLdogAECQc
Debug: RESPONSE: 200
[{"entity_class":"agent","system":{"hostname":"moni2","os":"linux","platform":"ubuntu","platform_family":"debian","platform_version":"18.04","network":{"interfaces":[{"name":"lo","addresses":["127.0.0.1/8","::1/128"]},{"name":"enp0s3","mac":"02:11:f7:37:ca:bc","addresses":["10.0.2.15/24","fe80::11:f7ff:fe37:cabc/64"]},{"name":"enp0s8","mac":"08:00:27:7e:c5:e1","addresses":["172.16.3.101/12","fe80::a00:27ff:fe7e:c5e1/64"]}]},"arch":"amd64","libc_type":"glibc","vm_system":"vbox","vm_role":"guest","cloud_provider":"","processes":null},"subscriptions":["entity:moni2.mydomain.org","linux"],"last_seen":1643185965,"deregister":false,"deregistration":{},"user":"agent","redact":["password","passwd","pass","api_key","api_token","access_key","secret_key","private_key","secret"],"metadata":{"name":"moni2.mydomain.org","namespace":"default"},"sensu_agent_version":"6.6.3"}]
Debug: BODY: [
  {
    "entity_class": "agent",
    "system": {
      "hostname": "moni2",
      "os": "linux",
      "platform": "ubuntu",
      "platform_family": "debian",
      "platform_version": "18.04",
      "network": {
        "interfaces": [
          {
            "name": "lo",
            "addresses": [
              "127.0.0.1/8",
              "::1/128"
            ]
          },
          {
            "name": "enp0s3",
            "mac": "02:11:f7:37:ca:bc",
            "addresses": [
              "10.0.2.15/24",
              "fe80::11:f7ff:fe37:cabc/64"
            ]
          },
          {
            "name": "enp0s8",
            "mac": "08:00:27:7e:c5:e1",
            "addresses": [
              "172.16.3.101/12",
              "fe80::a00:27ff:fe7e:c5e1/64"
            ]
          }
        ]
      },
      "arch": "amd64",
      "libc_type": "glibc",
      "vm_system": "vbox",
      "vm_role": "guest",
      "cloud_provider": "",
      "processes": null
    },
    "subscriptions": [
      "entity:moni2.mydomain.org",
      "linux"
    ],
    "last_seen": 1643185965,
    "deregister": false,
    "deregistration": {
    },
    "user": "agent",
    "redact": [
      "password",
      "passwd",
      "pass",
      "api_key",
      "api_token",
      "access_key",
      "secret_key",
      "private_key",
      "secret"
    ],
    "metadata": {
      "name": "moni2.mydomain.org",
      "namespace": "default"
    },
    "sensu_agent_version": "6.6.3"
  }
]
Debug: Fetching entity drawmgt.zg-3.mydomain.org via Sensu API
Debug: method=get url=https://moni2.mydomain.org:8080/api/core/v2/namespaces/default/entities/drawmgt.zg-3.mydomain.org path=entities/drawmgt.zg-3.mydomain.org
Debug: Sensu API: Using token eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2NDMxODYyNjMsImp0aSI6IjEyZGQxNmE5ODJmNGNkNGFiOTJjMzFmYTkyMDM2NDQzIiwiaXNzIjoiaHR0cDovL21vbmkyLnNvZnR4cy5jaDo4MDgwIiwic3ViIjoicHVwcGV0LWFnZW50X2VudGl0eV9jb25maWciLCJncm91cHMiOlsicHVwcGV0OmFnZW50X2VudGl0eV9jb25maWciLCJzeXN0ZW06dXNlcnMiXSwicHJvdmlkZXIiOnsicHJvdmlkZXJfaWQiOiJiYXNpYyIsInByb3ZpZGVyX3R5cGUiOiIiLCJ1c2VyX2lkIjoicHVwcGV0LWFnZW50X2VudGl0eV9jb25maWcifSwiYXBpX2tleSI6ZmFsc2V9.jPQ6E47AHFjTN31AkxKmi9lhQo8zO0qwGtLdogAECQc
Debug: RESPONSE: 404
{"message":"not found","code":2}
Error: Unable to query entity data for entity drawmgt.zg-3.mydomain.org: Resource not found at URL https://moni2.mydomain.org:8080/api/core/v2/namespaces/default/entities/drawmgt.zg-3.mydomain.org: Net::HTTPNotFound
Error: /Stage[main]/Sensu::Agent/Sensu::Agent::Subscription[linux]/Sensu_agent_entity_config[sensu::agent::subscription linux]/ensure: change from 'absent' to 'present' failed: Unable to query entity data for entity drawmgt.zg-3.mydomain.org: Resource not found at URL https://moni2.mydomain.org:8080/api/core/v2/namespaces/default/entities/drawmgt.zg-3.mydomain.org: Net::HTTPNotFound
Debug: Sensu::Agent::Subscription[linux]: Resource is being skipped, unscheduling all events
Debug: Class[Sensu::Agent]: Resource is being skipped, unscheduling all events

[treydock]

Is sensu:: agent_entity_config_password identical on both nodes?

Another thing to check is on the backend server run sensuctl entity list and see if the entity is there for the drawmgt.zg-3.mydomain.org host. It might be the Sensu Agent registered using a different name if maybe the host's FQDN doesn't actually show as drawmgt.zg-3.mydomain.org. If entity names don't match can force it with sensu::agent::entity_name.

[nagyt234]

Yes, the passwords are identical. I guess, otherwise I could not get the access token at the beginning of the log above with HTTP response 200.

sensuctl entity list doesn't list the entity at all (I've known that), and that is the problem. Who and how should register the entity with this puppet module?

[treydock]

The sensu-agent daemon, when it starts, will register the entity with the sensu-backend and then after that Puppet is able to check the entity exists and perform other operations after the agent has registered the entity. I would check the logs for sensu-agent to see if anything indicates errors with registering the agent. If the entity list is empty then it also sounds like there was an issue with the backend's sensu-agent registering its entity too.

[nagyt234]

What I see, that the sensu agent is running:

# ps -ef | grep sensu
sensu     3788     1  0 15:47 ?        00:00:00 /usr/sbin/sensu-agent start -c /etc/sensu/agent.yml

The sensu agent log /var/log/sensu/sensu-client.log is empty.

I've started the agent in a terminal and I see the following messages more time:

{"component":"agent","error":"x509: certificate signed by unknown authority","level":"error","msg":"reconnection attempt failed","time":"2022-01-27T16:17:59Z"}
{"component":"agent","level":"info","msg":"connecting to backend URL \"wss://moni2.mydomain.org:8081\"","time":"2022-01-27T16:18:00Z"}

The SSL certificate of moni2.mydomain.org is Let's Encrypt certificate, so I do n ot understand this problem. I can connect to https://moni2.mydomain.org:8081/ with wget, I get obviously 401 Unauthorized. OK, I've started now the agent with the option --insecure-skip-tls-verify, then I got the following messages:

{"component":"agent","header":"Accept: application/octet-stream","level":"debug","msg":"setting header","time":"2022-01-27T16:39:18Z"}
{"component":"agent","error":"handshake failed with status 400: Bad Request\nwebsocket: the client is not using the websocket protocol: 'upgrade' token not found in 'Connection' header\n","level":"error","msg":"reconnection attempt failed","time":"2022-01-27T16:39:18Z"}

So I think, the problem is, that the API ports are forwarded with apache rproxy to the VM and apache gives the SSL connection.
I'm going to holiday now, so I'll investigate the problem further from the 8th of February.