PIP 60: Support Proxy server with SNI routing
- Status: Discussion
- Author: Rajan Dhabalia
A proxy server is a go‑between or intermediary server that forwards requests from multiple clients to different servers across the Internet. The proxy server can act as a “traffic cop,” in both forward and reverse proxy scenarios, and adds various benefits in your system such as load balancing, performance, security, auto-scaling, etc.. There are already many proxy servers already available in the market which are fast, scalable and more importantly covers various essential security aspects that are needed by the large organization to securely share their confidential data over the network. Pulsar already provides proxy implementation PIP-1 which acts as a reverse proxy and creates a gateway in front of brokers. However, pulsar doesn’t provide support to use other proxies such as Apache traffic server (ATS), HAProxy, Nginx, Envoy those are more scalable and secured. Most of these proxy-servers support SNI ROUTING which can route traffic to a destination without having to terminate the SSL connection. Routing at layer 4 gives greater transparency because the outbound connection is determined by examining the destination address in the client TCP packets.
TLS Extension Definition adds support of Server Name Indication SNI and it has been available since January 2011. In SNI routing, the proxy server examines TLS connection on an initial handshake and extracts the "SNI" value. Proxy-server creates a connection with a destination host defined into SNI and forms a TLS tunnel between client and destination host. Figure 1: shows the layer-4 routing network activity diagram between client and application server via proxy-server (eg: ATS). In the figure, the client initiates TLS connection with the proxy-server by passing hostname of application-server in the SNI header. The proxy server examines SNI header, parses hostname of the target application server and creates an outbound connection with that application server host. Once, proxy server successfully completes TLS handshake with the application-server, the proxy creates a TLS tunnel between the client and the application server host to begin further data transactions. Many known proxy server solutions such as ATS, Fabio, Envoy, Nginx, HaProxy support SNI routing to create TLS tunnel between client and server host.
[Figure 1: Layer 4 routing]
Pulsar client creates TCP connection with the broker and exchanges binary data over TCP channel. Pulsar also supports TLS to create a secure connection between a client and the broker. As we have discussed in figure-1, the proxy server performs SNI routing on TLS connection and this mechanism can be perfectly fit into pulsar to introduce proxy between client and broker, and use proxy-server for both forward-proxy and reverse-proxy use cases.
In order to access a specific topic, the pulsar-client first finds out the target broker which owns the topic and then creates a connection channel with that target broker to publish/consume messages on that topic. This section explains how pulsar-client uses proxy-server to find a target broker for a specific topic and create a TCP channel with that target broker to access the topic.
Pulsar client connects to the broker in two phases: lookup and connect.
In lookup phase, a client does a lookup with the broker-service URL / VIP to connect with any broker in the cluster, send lookup request for a topic and receive the response with the name of the target broker that owns the topic. In the second phase, the client uses the lookup response which has the name of the target broker and connects with that target broker to publish/consume messages.
Figure 2 shows SNI routing for both the phases and how proxy-server creates a TCP tunnel between client and broker. In SNI Routing, pulsar-client always tries to create a connection with proxy-server and pass the target broker/service URL as part of the SNI header.
[Figure 2: Pulsar-client and broker connection using SNI Routing]
Lookup phase
Figure 2 shows that pulsar-client uses broker-service url (pulsar-vip) and proxy-server url (proxy-server:4443) to perform lookup using SNI routing. Pulsar-client uses proxy-server url to create a TLS connection with proxy-server and uses broker-service url to pass it in SNI header. At the time of TLS handshake, Proxy-server does SNI-routing by extracting broker-service name value from SNI header, finds appropriate mapped broker url from configured SNI mapping and connect with a broker to form a TCP tunnel between client and broker. Once, the TCP tunnel is established between pulsar client and broker, the client requests lookup for a specific topic and receives a lookup response with the name of target broker which owns the topic.
Connect phase
Once the client receives a lookup response with the target broker which owns the topic, the client requests proxy-server to create a TCP tunnel between client and broker. In order to create a TCP tunnel between client and target broker, the client always creates a connection with proxy-server and passes the target broker name into SNI header. Once the TCP tunnel is created, the client can communicate with the broker and publish/consume messages on that TCP connection.
In order to support SNI routing using Proxy-server, we have to mainly make changes into pulsar-client.
In order to add support of proxy-server into a pulsar, pulsar-client requires knowledge of proxy location and type of proxy routing (eg: SNI). So, we will add additional configuration in pulsar-client:
String proxyServiceUrl;
ProxyProtocol proxyProtocol; //eg: enum: ProxyProtocol.SNI
// allows adding more proxy options in configurations
We can also use proxy-server in geo-replication to create a proxy between two brokers. Therefore, we need similar configurations as pulsar-client into cluster metadata as well. If cluster metadata has a proxy configured then other clusters will connect to this pulsar cluster via proxy.