Supporting Metadata Change Management

[mithmatt]

Context

We implemented some of the fundamentals required for supporting changes to metadata of entities:

• JSON PATCH for applying changes for PATCH API operations.
• RBAC support for MetadataOperations

Not all users of OpenMetadata can make changes to metadata of entities. Only entity owners and DataStewards currently have access to make metadata changes to entities.
We introduced metadata operation permissions such as SuggestDescription and SuggestTags in the backend, which can be associated roles.

Idea

Support a suggestion, review and approval process for metadata changes for users who cannot directly update metadata.

Change Proposal

• Users are able to edit different parts (description, tags, etc) of the entity metadata from the UI.
• If users do not have Update but have Suggest permission, the PATCH API Resource will store the JSON PATCH information and reference to the entity in addition to user who requested the change as a task in a change_management table
• All users can see the requested changes on the OpenMetadata UI (a new tab called tasks?)

Change Review

• By default, the owner of the entity is added in as a reviewer, and gets notification (change event + optional webhook) for the task
• Entity owner can go to the OpenMetadata UI tasks tab for the entity and view the task/change presented review. The UI and backend coordinate in showing the representation of the change (against latest version of metadata) in a similar way how versioning shows the diff between versions. Caveat: Some changes may not be mergeable since metadata evolved after the change was requested (similar to git merge conflict)
• Owner gets to approve/reject the changes or delegate review to some one else

Change Approval

• If change is rejected, the task is closed as rejected.
• if change is approved, the backend applies the change in the exact same way how the PATCH API works today. This generates metadata change event (as usual).
• Create a change event on task closure, and mark it as closed in the database.

[harshach]

We have state in tags already which takes in "Suggested" and "Confimred" options, lets make sure we build on top of it

OpenMetadata/catalog-rest-service/src/main/resources/json/schema/type/tagLabel.json

Line 28 in 67a1c23

"state": {

[mithmatt]

Yes, we should use these as part of this process.

When user/bot makes a change proposal and suggests a tag, we add the tag and mark it in Suggested state. In addition, we open a task for the owner or DataSteward to review (need to figure out details here).

Once the review is accepted, the tag state changes to Confirmed.

[harshach]

@mithmatt
overall this looks great. Only question I've is the if the "Task" becomes a separate entity. If so we need to make sure it's able to capture the Task from user induced events such as what you described vs a test failure vs org level tasks that everyone should do. In this case example would be Admin, Data Steward says we all need to tier our entities and add ownership. This should become tasks at the team level that the team owners can assign the ownership of tasks to others.
From there on we can build reports on how these tasks are being resolved etc..
Secondly, the tasks may need to integrate with existing solutions such as JIRA etc.. doesn't need to be in first cut but I am sure users will ask for it.

[mithmatt]

we need to make sure it's able to capture the Task from user induced events such as what you described vs a test failure vs org level tasks that everyone should do

Thinking through this further

Looks like there are different types of tasks here:

1. The change is known - eg: user suggests description change from "dim.address is X" to "dim.address is Y" . Review/approval is needed from someone who has the context. The change here is like a PR on github.
2. The change is unknown - eg: there is a bulk action to set the tier of all untiered datasets. This results in an open item to the owner of the dataset to set the tier of the dataset to any tier.
3. Production issue remediation request - a data quality test failure requires investigation by the team

Started this discussion with the limited scope of 1 above.
Good to see that we may be able to solve some of these with the same feature with a few tweaks.
Possibly support tasks with different types:

• CHANGE_REQUEST (can be shown like a diff on the UI for review to approve/reject) - covers 1
• OPEN_TASK (can be shown as open issues with link to required action and auto-resolves when the requested work is complete) - covers 2 and 3

team owners can assign the ownership of tasks to others

Yes, tasks should have assignee as a field (with reassigning capability)

the tasks may need to integrate with existing solutions such as JIRA etc

Within OpenMetadata we should aim to keep it lightweight and integrate with external systems as needed.
A possible approach could be tracking task with an external link to an issue in JIRA, and closing based on JIRA webhook request to OpenMetadata API.

[mithmatt]

There is great work happening on #2366
The ability to make suggestion on a thread and have assignees resolve them seems like something we can reuse for change management altogether without the need for a separate task as such.

We should explore this idea further as the feature on threads gets merged.

cc: @vivekratnavel

[vivekratnavel]

Yes, we should continue to explore how to notify users of tasks that they are assigned and also think about auto resolving some of the tasks when the action has been taken. For example, a task for updating the description of a table should be automatically resolved when the owner updates the description.

Overall, this would be a great addition to Open Metadata. Thank you @mithmatt

[mithmatt]

Some other unstructured thoughts I came up with:

We will need to discuss, find similarities and differences between task and thread to avoid mixing both, introducing technical and product complexity. Should look at existing systems such as Github (Issues and Discussions) and JIRA and draw parallels with what OpenMetadata customers would want to have.

Example of an automated change proposal: If dataset does not have owner, automation can recommend owner of upstream pipeline that produced the dataset as the owner of the dataset, and assign the task to the owner to approve/reject (taking ownership of the dataset).

[gavin-denyse]

Having the ability to link a Jira ticket and Github PR/MR to an entity would help with change management and who is actively working on the entities in a more automated way.