improve observability for procedure

[waynexia]

What problem does the new feature solve?

Currently we are not able to check the state of each procedure, how long it takes, which resources it keeps, or which state it at. It's hard to debug or verify things related to one specific procedure.

This ticket proposes to improve the observability of procedures, by exposing the above things somewhere. Like an HTTP API or a table in information_schema.

Useful links

• procedure state
  

  greptimedb/src/common/procedure/src/procedure.rs

  Lines 301 to 315 in 1b58622

  	pub enum ProcedureState {
  	/// The procedure is running.
  	#[default]
  	Running,
  	/// The procedure is finished.
  	Done { output: Option<Output> },
  	/// The procedure is failed and can be retried.
  	Retrying { error: Arc<Error> },
  	/// The procedure is failed and commits state before rolling back the procedure.
  	PrepareRollback { error: Arc<Error> },
  	/// The procedure is failed and can be rollback.
  	RollingBack { error: Arc<Error> },
  	/// The procedure is failed and cannot proceed anymore.
  	Failed { error: Arc<Error> },
  	}

• acquiring locks
  

  greptimedb/src/common/procedure/src/local/runner.rs

  Lines 115 to 125 in 5303537

  	for key in self.meta.lock_key.keys_to_lock() {
  	// Acquire lock for each key.
  	let key_guard = match key {
  	StringKey::Share(key) => self.manager_ctx.key_lock.read(key.clone()).await.into(),
  	StringKey::Exclusive(key) => {
  	self.manager_ctx.key_lock.write(key.clone()).await.into()
  	}
  	};
  	guard.key_guards.push(key_guard);
  	}

• implement cluster_info table in information_schema feat: adds information_schema cluster_info table #3832

What does the feature do?

• Improve ProcedureState::Running to contain a String. This string shows a procedure's current stage.
• Count the procedure's execution time and store it.
• Expose those information to information_schema

Implementation challenges

No response

[Kelvinyu1117]

I would like to work on it.
How can I verify my change locally for this improvement?

[evenyag]

Improve ProcedureState::Running to contain a String. This string shows a procedure's current stage.

For this feature, one approach is adding a debug or trace log in the runner to print the current state of the procedure.

greptimedb/src/common/procedure/src/local/runner.rs

Lines 195 to 197 in 1b58622

	let state = self.meta.state();
	match state {
	ProcedureState::Running => {}

@WenyXu Do you have any suggestions?

[Kelvinyu1117]

For the execution time, it should be measuring how long does execute_procedure_in_loop() take right?
What time precision (milliseconds, microseconds, nanoseconds) do you want for the execution_time?

[evenyag]

Milliseconds should be enough.