Tendermint light client is failing due to context timeout when state syncing

[daniel1302]

The snapshot-testing has checks if the RPC server on remote host is healthy, but it is not restrictive enough. It checks only GET method on host given in the config. Sometimes entire state sync is failing with the following error:

2024-11-11T22:39:15.918+0100	ERROR	core	node/node.go:203	problem starting blockchain	{"error": "failed to start state sync: failed to set up light client state provider: post failed: Post \"http://<some-server>:26657\": context deadline exceeded"}

We should add checks against post method similar that state sync reactor in the tendermint is doing, and disable servers that are not healthy enough.

• Find timeout used in the statesync engine of the tendermint
• Check what post data is sent to the remote rpc server and use similar in the server health check.
• Remove all unhealthy servers from the statesync.rpc_servers in the tendermint config.