[rpc send] retry when BlockhashNotFound happens

[ochaloup]

hi @butonium ,

this is not an urgent PR request. I think it can wait when you have a bit of time to check.
I'm trying to address failures happening in bot processing of marcrank.

I would like to ask you for a review or your perspective on this change.

This is a code I re-used from marinade-anchor, refactored it and now started to use it for marcrank and validator-manager bots.
It's code that manages to send transactions and to that "generic" stuff around RPC calls. Each of the bots then creates own code with business logic on top of this.

During the morning today there were multiple errors in marcrank script processing the transactions.
It's now all the time about

2023-11-09T10:15:41.604506456Z stderr F [2023-11-09T10:15:41.604312829Z ERROR marcrank::commands::stake_delta] TX ERR Failed to send transaction: ClientError { request: Some(SendTransaction), kind: RpcError(RpcResponseError { code: -32002, message: "Transaction simulation failed: Blockhash not found", data: SendTransactionPreflightFailure(RpcSimulateTransactionResult { err: Some(BlockhashNotFound), logs: Some([]), accounts: None, units_consumed: Some(0), return_data: None }) }) }

I reran the code and I cannot find much of the flaw in sending the transactions. The idea is to take a "Finalized" blockhash from the RPC call, to sign the transaction with that loaded blockhash and then send it to RPC to proceed.

It happens time to time and sometimes more often that the sending transaction ends with Blockhash not found. As the blockhash was searched as Finalized and it was loaded just before the call on sending happens I wonder this being some discrepancy of the particular RPC node to the network.
My idea is to retry few times (like 2-3 times) the call of the same transaction with a new blockhash until the transaction sending is considered a failure.

I would like to ask you about you feeling around this and/or maybe if you did some similar handling before and if can be done easier than this.

Thanks for your points in advance.

[ochaloup]

Closing as in rework for priority fee things.