New incoming offline deals are blocked when boost receives 200+ online deals.

[cryptowhizzard]

Checklist

• This is not a question or a support request. If you have any boost related questions, please ask in the discussion forum.
• This is not a new feature request. If it is, please file a feature request instead.
• This is not an enhancement request. If it is, please file a improvement suggestion instead.
• I have searched on the issue tracker and the discussion forum, and there is no existing related issue or discussion.
• I am running the Latest release, or the most recent RC(release canadiate) for the upcoming release or the dev branch(master), or have an issue updating to any of these.
• I did not make any code changes to boost.

Boost component

• boost daemon - storage providers
• boost client
• boost UI
• boost data-transfer
• boost index-provider
• Other

Boost Version

Latest master october 2 2023

Describe the Bug

When boost receives a lot of online deals ( 200+ ) the channel for making new offline deals seems to get blocked.

Both client and SP receive error:
Error: send proposal rpc: failed to read response: stream reset

I verified the numbers for storage in the config. They are all above 1000 ( lotus-miner as well as boost ).

Logging Information

Error: send proposal rpc: failed to read response: stream reset

Repo Steps

1. Run '...'
2. Do '...'
3. See error '...'
   ...

[LexLuthr]

We need to move the deal acceptance flow to goroutines instead of handling them all in a single thread.

[cryptowhizzard]

This PR 1746 seems to improve a bit but the problem resides in provider_dealfilter.go.

Boost needs to output a lot of things about sectors and statuses to the external filter ( if configured ) and this is where the delay originates. Disabling everything except the deal parameters like client / provider / cid seems to resolve the problem in combination with this PR.

In my situation i am running a 30 PB+ machine and boost needs to collect a lot of info to send to that filter causing massive delays.

Please investigate per part what 's the exact cause of the delay in provider_dealfilter.go and let's see if we can find a way to work around that. I will migrate this now to my testing setup in next days so our customer's won't be affected anymore.

[LexLuthr]

@cryptowhizzard I would love for you to weigh in on #1747 (comment)
We can use the new issue to figure out what design would be most efficient.