Services: RADICAL+SOMA Performance Monitoring

[mtitov]

SOMA repository: https://github.com/soma-monitoring-toolbox

Actionable items

• Extend support of service tasks (startup flag + termination)
  • Ensure having a comprehensive set of tests for service tasks
• Service/daemon to collect information about resource usage (per node)
• Service/daemon to collect information about task state (e.g., tracing events)

Collected information is pushed to the SOMA service using RPC calls.

Related tickets

Prioritized tickets

• Startup flag (service URL) for the SOMA service #3007
• Termination procedure for the SOMA service #3008

Other tickets

• Service startup flag: more flexible service startup flag #2953
  • As a temporary approach - consider using "readiness" file - when service is up it creates a corresponding file, and RP waits for this file before proceeding further
• Service termination: Possibility to terminate services #2967
• Expand service tasks with capability to run on every node #2899
• [FEATURE-REQUEST] Expose RP "services" as a special (multi-node) task #2543
• Track TAU use case and specific requirements #2711

p.s. for the reference, previous proposal for monitoring: #2308

[mtitov]

First step: implement component for task monitoring (original comment)

Next steps to build service/daemon to collect information about resources usage

• remove sub-agents
• place components directly on target nodes
• implement service as special components

[andre-merzky]

After discussion we plan to implement the following schema

• services are defined as normal RP tasks (special task type though)
• service tasks are tagged to a specific pilot instance by setting the pilot field in the TD
• they are submitted via the tmgr
• we introduce a new control message which signals the event that a service task is ready (which implies it has reached AGENT_EXECUTING state. That message type will have the fwd flag set by default
• the tmgr will collect those events and will issue ready-callbacks which are definable on service task types