Circuit breaker implementation in scriptorium

[shubhi1092]

Description

This PR is to add circuit breaker functionality for scriptorium lambda. It is to handle the exceptions where service restart is not helpful and instead, we want to wait and retry again. For example, when mongo db is unavailable/down, and scriptorium is not able to write ops to the db, restarting the service doesnt help, instead we would wait and retry after some time. Circuit Breaker pattern helps in such cases by maintaining open/closed/halfOpen state.

So in scriptorium, all the calls to db are wrapped by the circuit breaker, and in case of such errors, the circuit will open and pause the lambda (i.e. pause the incoming messages). After some time, the circuit will go to halfOpen state and call a healthCheck function - if it succeeds, the circuit will close and resume the incoming messages, else it will stay open and paused.

We can configure various options, like error threshold, reset timeout, the errors for which we want to engage the circuit breaker, etc. Also if the circuit is not able to close or resume for some time (configurable), we will fallback to restarting the service to avoid being in an endless state of waiting.

This PR is for scriptorium, and once we validate and roll this out in production, we will add the same pattern for document lambdas too.

Summary of changes made in this PR:

• Circuit Breaker Implementation: Adds a circuit breaker pattern to scriptorium->db calls, with various configuration options for error thresholds, reset timeouts, and error filters.
• Pause and Resume Methods: Adds pause and resume methods for lambdas, context, documentContext, partition, partitionManager, kafkaRunner, rdKafkaConsumer, and lambda to manage message flow during circuit breaker states.
• Health Check for MongoDB: Adds a health check method to the MongoDB class and exposes a healthCheck property from the MongoManager class.

Testing

• Added unit tests for circuit breaker.
• Tested the scriptorium end to end functionality locally by forcing the db to be unavailable in the local setup.
• Testing in dev cluster (in progress)

We will roll this out slowly by testing in each ring.