feat: switch from Cowboy to Bandit, upgrade to Phoenix 1.7

[paulswartz]

Bandit is a new web server, designed around supporting the Plug interface that Phoenix uses.

It also logs errors much more aggressively, which Phoenix 1.7 handles much better than Phoenix 1.6.

Best to review this commit-by-commit: the reformatting commit adds a bit of noise.

Formatting has moved into the "Dev" tasks, as it now depends on the compiled dependencies. I'll remove it from the requirements when this merges.

[nlwstein]

Do you think it would be worth it to (have you already?) load test this module vs. Cowboy - I'm not opposed to just throwing it into production and reverting if necessary but it might be interesting to see how much extra stability / performance we can get out of it in a synthetic test too.

Specifically, # of concurrent connections per instance would be very interesting given the issue last week.

My only q w/ what's here is the http_1_options - I assume that was added for a reason so I just wanted to make sure we wouldn't benefit from the compression on HTTP2 as well.

[paulswartz]

@nlwstein I'm throwing it back up on dev-blue for further testing.

When I was testing it last night, I was able to get it up to around 1,000 /vehicle streams per server before I saw AWS scale up the number of instances.

[paulswartz]

@nlwstein doing some further testing:

• I can have 2,000 open connections without appreciably increasing the memory or CPU usage on dev-blue
• dev-green gets more traffic than dev (not sure why) but it's using 40% of the CPU and a bit less memory

I'd be okay with doing a temporary deploy to api-prod to see what it looks like with even more traffic, but I'll leave that up to you and @bklebe.

[bklebe]

I support trying a prod deploy. Maybe do it from this branch though so we can revert it easily?

[paulswartz]

So I went down a bit of a rabbit hole on this one looking into the various errors. It turns out that this was a known issue with how Bandit logs errors coming from Phoenix (see phoenixframework/phoenix#5445 for more), so Phoenix needed to be upgraded in order to address the errors. There were quite a few changes in Phoenix between 1.6 and 1.7, so the upgrade was a bit more involved than simply bumping the version in the mix.exs file.

All that being said: I'll run this on dev-green overnight to make sure there aren't any other unexpected issues, and then maybe we can try it on production again?

[github-actions]

Coverage of commit 683518e

Summary coverage rate:
  lines......: 89.2% (4098 of 4592 lines)
  functions..: 70.9% (2225 of 3139 functions)
  branches...: no data found

Files changed coverage rate:
                                                                        |Lines       |Functions  |Branches    
  Filename                                                              |Rate     Num|Rate    Num|Rate     Num
  ============================================================================================================
  apps/api_web/lib/api_web/event_stream.ex                              |83.0%     47|84.6%    13|    -      0
  apps/api_web/lib/api_web/event_stream/supervisor.ex                   |85.7%     14| 100%     7|    -      0
  apps/api_web/lib/api_web/router.ex                                    |92.3%     52|58.0%   157|    -      0
  apps/api_web/lib/api_web/swagger_helpers.ex                           |98.9%     95| 100%    35|    -      0
  apps/api_web/lib/api_web/view_helpers.ex                              |83.3%      6|50.0%     2|    -      0
  apps/api_web/lib/api_web/views/error_helpers.ex                       | 100%      7| 100%     3|    -      0
  apps/api_web/lib/api_web/views/route_view.ex                          |85.7%     21|37.0%    46|    -      0
  apps/api_web/lib/api_web/views/stop_view.ex                           | 100%     28|84.2%    57|    -      0
  apps/api_web/lib/api_web/web.ex                                       | 0.0%      1| 0.0%     6|    -      0

Download coverage report

[github-actions]

Coverage of commit f84d158

Summary coverage rate:
  lines......: 89.3% (4102 of 4595 lines)
  functions..: 70.9% (2225 of 3139 functions)
  branches...: no data found

Files changed coverage rate:
                                                                        |Lines       |Functions  |Branches    
  Filename                                                              |Rate     Num|Rate    Num|Rate     Num
  ============================================================================================================
  apps/api_web/lib/api_web/event_stream.ex                              |84.0%     50|84.6%    13|    -      0
  apps/api_web/lib/api_web/event_stream/supervisor.ex                   |85.7%     14| 100%     7|    -      0
  apps/api_web/lib/api_web/router.ex                                    |92.3%     52|58.0%   157|    -      0
  apps/api_web/lib/api_web/swagger_helpers.ex                           |98.9%     95| 100%    35|    -      0
  apps/api_web/lib/api_web/view_helpers.ex                              |83.3%      6|50.0%     2|    -      0
  apps/api_web/lib/api_web/views/error_helpers.ex                       | 100%      7| 100%     3|    -      0
  apps/api_web/lib/api_web/views/route_view.ex                          |85.7%     21|37.0%    46|    -      0
  apps/api_web/lib/api_web/views/stop_view.ex                           | 100%     28|84.2%    57|    -      0
  apps/api_web/lib/api_web/web.ex                                       | 0.0%      1| 0.0%     6|    -      0

Download coverage report