HTTPS connection to scrapyd instances not possible

[franga2000]

Describe the bug
Running scrapyd instances across multiple servers at the moment requires either a complicated tunneling setup (like Wireguard) or directly exposing the scrapyd HTTP interface to the public Web. In the latter case, information (including basic auth credentials) is transmitted in clear text, which is obviously a problem.

Since scrapyd uses completely standard HTTP, one easy way to secure it is to use a HTTPS reverse proxy to encrypt the connection. However, the use of plain HTTP is hard-coded in various locations in scrapydweb. This means that the best we can do is set up a HTTP→HTTPS redirect, but that still means the connection goes through in plaintext first, exposing credentials.

To Reproduce
Steps to reproduce the behavior:

1. Set up HTTPS reverse proxy for scrapyd
2. Enter it into scrapydweb
3. The connection doesn't work, because it's still plain HTTP

Expected behavior
An HTTPS connection is established from the start.

[andre-response]

Were you able to solve this?