Why is HTTPS request hanging after 6 request in a row with WebKit based browsers and IIS?

[aggutierrez]

Expected Behavior

I've setup a couple of sample ASP.Net 6 APIs, one of them with Ocelot as the gateway, and tried to route requests to a given endpoint to a donstream API. Debugging with Rider everything works fine, and deployed to IIS works also fine if I configure HTTP bindings, but If I set up HTTPS bindings for both APIs and configure Ocelot routing to use HTTPS protocol, I would expect it to work OK, as through HTTP.

Actual Behavior

If I configure Ocelot to route through HTTPS, every 7th request from a WebKit based browser (I've tested Chrome and Edge in their latest versions) hangs for an undetermined time.

Some things I've tested so far

• Starting the gateway API with Kestrel, executing dontent MyProject.dll seems to work fine and I'm unable to reproduce this issue.
• Disabling HTTP/2 in both APIs in IIS seems to fix this issue too (although it's not a proper fix).

Steps to Reproduce the Problem

1. Create a blank ASP.Net 6 Web API project for the downstream API
2. Create a blank ASP.Net 6 Web API project for the gateway and install Ocelot (I've tested this with versions 18 and 20 with .Net 7)
3. Configure the gateway to route to the WeatherForecast endpoint of the donwstream API
4. Deploy both projects to IIS web sites and configure HTTPS bindings for the websites
5. Open a modern WebKit based browser and navigate to the gateway WeatherForecast endpoint so you get routed to the downstream API.
6. Browse 7 times to this same endpoint, till it hangs.

Specifications

• Version: Ocerlot v18 and v20
• Platform: ASP.Net 6 and ASP.Net 7
• Subsystem: Windows Server 2022 21H2 + IIS

[raman-m]

Is it a question or real issue, in your opinion, Alejandro?

[ggnaegi]

Hello @aggutierrez, are you experiencing the same issue as others with IIS?
#1630
#1591
If I have some time, I will try to reproduce the problem you discribed myself.

[aggutierrez]

Hi, @raman-m! It's mainly a question. I'm not sure it has 100% to do with Ocelot but I need to gather some info.

[raman-m]

Moved this thread to Q&A category

[aggutierrez]

Hi @ggnaegi! It behaves more or less like those issues, but in my case it's always at the 7th attempt. Also, I tried modifying the ocelot.json to redirect to the donwstream API using the IP address, instead of the domain name, as they suggest in one of the issues but that doesn't fix it and I keep reproducing this same behavior.

To be honest, to me it looks like it has to do with how WebKit browsers understand the end of a connection and how IIS implements HTTP/2. It looks like if the browsers never killed the previous threads, waiting for the server to finish the connection, and at some point they reach the maximum number of threads available, so the 7th request has to wait for a thread to be available. If I call the endpoint from Bombardier, for example, I've no issues and I can call any number of times.

[raman-m]

@aggutierrez wrote on Oct 13:

Some things I've tested so far

• Starting the gateway API with Kestrel, executing dontent MyProject.dll seems to work fine and I'm unable to reproduce this issue.
• Disabling HTTP/2 in both APIs in IIS seems to fix this issue too (although it's not a proper fix).

If you have a route to HTTP2 downstream service, you have to specify correct DownstreamHttpVersion property value.
Also, pay attention to #1673 please!

Another critical common problem with IIS hosting when Nth request being hanged is an IIS Gotcha. It could be #1657 or other issue...

Try to test #1657 user scenario and share a feedback plz...

---

As I said below, Ocelot allows to route HTTP2 traffic but Ocelot uses standard .NET HttpClient ecosystem. But you need to care about hosting environments: both Ocelot's and downstream ones.
It is not a responsibility of Ocelot to design & deploy client and downstream connection protocols and settings.
As a developer, you have to check direct connection from client to downstream first always, and after successful connection try to route traffic with Ocelot.

[aggutierrez]

That was of great help. I've been trying and just setting the hosting mode to OutOfProcess seems to fix this hickup 😄

[raman-m]

Good news! Enjoy Ocelot development.

[raman-m]

@aggutierrez I don't recommend to host Ocelot app by IIS. You must tune and provide correct settings for all IIS features, modules, handlers... you know this is a pain in the neck! IIS is a web server for classic web apps with ASP.NET front-ends. For ASP.NET Web API apps without front-ends it is better to use lite web servers like Kestrel.
Using self-hosted, Kestrel, Docker deployments are better.

I have no idea how to provide correct IIS settings for HTTP/2 protocol, but technically Ocelot should support HTTP2. You have to read tons of articles on how to setup IIS for HTTP/2.

[VigossX]

I have some ideas for this.
My website structure is like this:

One apigateway using ocelot, A and B application server behind gateway.
They are both depolyed on iis by three sites.

Apigateway use https, A and B server using http as they are as localhost server downstream.
When serveral requests coming in, apigateway log says they are https and using HTTP/2.
Then they become http and using HTTP/1.1 to downstream.
When requests' amount over 6, the 7th request will stalled for a long time maybe over one 1 minutes in the webkit browser. And finally reach backend.
But when we change to Firefox, no this issue.

Then I test, closing apigateway HTTP/2 function on iis. Then apigateway gets all requests by HTTP/1.1, it works, no stalled.
So I guess it has something with HTTP/2 socket.
Now I'm testing both using HTTP/2 on apigateway, and HTTP/2 on A and B server.
So I must using https://localhost for A and B server, I'm try out for the localhost certificate now.
Any one have some idea for this?

[ggnaegi]

Maybe this could help? #1673

[raman-m]

Hi @VigossX !
If you have a route to HTTP2 downstream service, you have to specify correct DownstreamHttpVersion property value.
Also, pay attention to #1673 plz

[VigossX]

@ggnaegi
I have some doubt for #1673 .
In 1673 you said, All components above should only talk to each other using HTTP/2 and no TLS (plain HTTP).

But in the ASP.NET CORE official document, it says in Kestrel when HTTP/2 is available, we need TLS 1.2 or later connection.
document
I think we need to use https when we want all components talk with HTTP/2.

[VigossX]

@raman-m
thanks, I just want have a test to confirm my suspicions. I have changed "DownstreamHttpVersion" to "2.0" and "DangerousAcceptAnyServerCertificateValidator" to true. finally, all become https and HTTP/2, the issue disappear.

Finally, I think I won't use this solution, I'll make gateway to use HTTP/1.1 so that I don't need to make local server become https.

[raman-m]

My dear friend from China,
Test everything you want right in your fork!
Why didn't you forked Ocelot repo? I don't see a fork in the list of repos of your account!
My recommendation is "Fork Ocelot repository first, and provide your researches".
Good luck!