Teleport 6.0 web not working behind ALB with ACM ssl cert.

[aduston-rapid]

I had previously deployed a cluster in AWS following the ha autoscale cluster terraform example. This worked well, but I'm in the process of building out some new infrastructure where I want to use teleport for ssh access, and I want upgrade to the latest version as part of that process.

I was able to get version 5.2.1 running, but when I tried updating the cluster to 6.0.1 I couldn't access the web interface. The target groups on the load balancer seemed to be getting a good response from the /readyz endpoint because they were marked as healthy, but the proxy was returning a 502 error when trying to load teleport in the browser.

On the proxy servers themselves I tried running curl 127.0.0.1:3080, and got an empty response from the proxy service:

curl: (52) Empty reply from server

This message was logged each time I tried ran curl to localhost:3080:

/usr/local/bin/teleport[2081943]: DEBU [MX:PROXY:] Detected an HTTP request. If this is for a health check, use an HTTPS request instead. multiplexer/multiplexer.go:248

Following the admin manual I start teleport on the proxies with the --insecure-no-tls flag. Has this changed in the 6.x release? Do the proxy servers need to have their own certificates even when SSL is terminated at a load balancer, or could it possibly be some other problem with my configuration?

I'm happy to provide any more detail if it helps with diagnosis. Since this is only a prototype infrastructure I in the process of deploying the cluster directly to 6.0 instead of upgrading from 5.2 as I did previously just to rule out any issues I might have introduced in the upgrade process.

[webvictim]

There haven't been any significant changes to that behaviour between 5.2.1 and 6.0.1 as far as I know.

What's the reason for using --insecure-no-tls? It shouldn't be required in most situations, and it makes it more difficult to get Teleport working correctly. Our AMIs haven't used it as part of the teleport start command with ACM for a while - see https://github.com/gravitational/teleport/blob/master/assets/aws/files/system/teleport-proxy-acm.service

See answer here: #5899 (reply in thread)

[aduston-rapid]

Thanks for looking into this. I'll try out 6.0.2 as soon as it's available.

To answer your question, the reason for using the --insecure-no-tls feature is that I'm setting up an HA cluster with multiple proxy servers. I'm using an ALB with an ACM cert for web and an NLB for the proxy/tunnel ports. In the Teleport Proxy HA section of the admin guide there is a note that says you have to set --insecure-no-tls on the proxies if you're terminating SSL at a load balancer.

If there's a recommended configuration to use that doesn't require the --insecure-no-tls in this case I'm happy to try it out. I did try a few different configurations for the ALB including setting the protocol for the target group to HTTPS, but couldn't make it work.

[webvictim]

That explanation makes sense. Interestingly, I think that AWS ALBs don't care too much about the certificates presented by upstream servers; they just accept them all blindly. It looks like I made this change around a year ago - if it helps, I can confirm that we have a number of Terraform clusters deployed in AWS with ACM using that same ha-autoscale-cluster Terraform and they all work fine without --insecure-no-tls.

[webvictim]

6.0.2 has been delayed slightly and should now be coming out this week instead.

[aduston-rapid]

Cool, I'm just coming back around to picking up this project again and I want to try this out. I suppose it makes minimal actual difference, but even without proper certs I suppose encrypted traffic is better than unencrypted in general.

[aduston-rapid]

Just for the sake of completeness, just wanted to update that this did work fine. I removed the --insecure-no-tls flag from the systemd unit, and tweaked the target group config in my Pulumi project to change the protocol to HTTPS. Doesn't seem to behave any differently than before. Target group config looks like this:

export const teleportWebTargets = new TargetGroup('teleport-web-targets', {
  name: 'teleport-web',
  port: 3080,
  protocol: 'HTTPS',
  vpcId: vpc.id,
  healthCheck: {
    enabled: true,
    protocol: 'HTTP',
    path: '/readyz',
    port: '3000',
  },

  tags: {
    Name: 'teleport-web',
    ...baseTags,
  },
});