MongoDB removal

[mturilli]

MongoDB has proven to introduce major limitations in both performance and usability of RP. Most notably:

• Users all complain about having to use a MDB server.
• We need to provide and maintain a MDB server for our users.
• Latency with remote MDB affects RP performance.

https://github.com/radical-cybertools/radical.pilot/tree/feature/nodb_2 prototypes RP without MDB. Currently, work on this is blocked by #2265 and #2312. Planned for September release.

Relates to #2191

[mturilli]

Relates to #2185

[andre-merzky]

Update: this is now being worked on in branch feature/nodb_2. The current implementation scheme is as follows:

• a proxy service implementation is provided which can create and manage proxy ZMQ bridges on demand
• a session contacts that service to create proxy bridges for
  • sending new tasks from client to agent (queue)
  • sending completed tasks from agent back to client (queue)
  • exchange of control messages between client and agent (pubsub)
    Note that the proxy lifetime is bound by heartbeat messages from both client and agent.
• if a service is not found, the session will create a temporary local proxy service instance (running in a thread) for the same functionality - the proxies' lifetime is then bound by the session lifetime

The current implementation works for the following use cases:

• agent can directly connect to the client host (the session-internal proxy service instance is sufficient)
• client and agent can connect to the same proxy host (that host needs to run a proxy service instance)

The implementation does not, as of yet, auto-start a proxy service instance on an intermediate host if that host does not yet run a proxy service. The implementation also does not yet tunnel communication via SSL. As such, it is actually only really usable for local execution (including running the client on a cluster login node) or for execution in a secure network environment.

[mtitov]

No MongoDB requirement for releases >=1.40